What Do You Use?

A lot of people ask what I use to protect my systems. I remember back a few years ago when I could say “nothing, I just don’t infect my machines by clicking ‘yes’ to everything on the web”. Unfortunately those days are long gone. Java, Flash, PDF, Operating System, Cross Site Cookies, and the like are all able to infect your system with potential “baddies” without any user input.

So, the question remains: what do I use?

It depends on what I’m going. But, from the internet on, let me lay it out for you.

I start with firewall rules. I allow, but log, everything outgoing, so long as it comes from the firewall box. If the router receives any requests from an internal IP other than the firewall box, it will log it and then deny the packets. The firewall box is a simple machine running only one item: untangle (7.4 at time of this writing). Untangle is an IPS/IDS solution that’s from open source software with a nice GUI interface.

After that, I have a wireless network and a wired network. They’re both on the same VLAN on the switch, so they can access almost anything. I had been running my wireless network on a separate VLAN, but transferring files between systems was becoming rather difficult. My wireless settings utilize a WPA2-Personal AES encryption algorithm. I also have implemented SSID hiding and MAC address filtering.

All DNS requests first hit my internal DNS server, and, if not found, are forwarded to the router. The router will only utilize OpenDNS DNS servers – any other requests on TCP or UDP 53 are dropped. So even IF you have your own DNS servers setup, you won’t be bypassing mine. OpenDNS is set to not allow for any malicious sites. Untangle is also set to IPS any malicious requests in quarantine and email me the results. If the Untangle box ever is compromised, I am also running a virtual machine Snort variant (snorby) with another system running syslog requests from all network devices. That way I can at least check what happened for later.

If any user wants to download something utilizing bittorrent (I use TED to download TV episodes I may have missed), I use a Virtual Machine running on top of Linux (or just Linux itself) to test out the products. It’s much easier to “fix” a virtual machine than it is to fix a real system (Delete!).

Each linux system is running ClamAV of some kind. Each Windows system is running Nod32.

Each windows client is also running the following software:
Threatfire
Secunia PSI
Malwarebytes

The one open system that anyone who visits can use is running Windows XP SP3 with SteadyState set to wipe everything after a reboot.

Paranoid? Sometimes. But there’s always a way to get into any system. I’m just trying to make it more difficult.

So what do you use? Comment and let me know.

Testing RAID and HD Speeds under *nix

I was given the task of purchasing a NAS/SAN solution for a secondary (dev) oracle database (11g). Pricing it out on the normal branded websites yielded roughly $12,000 to $30,000 depending on the options. A quick look at newegg gave me a lot more for less coin.

H55 Intel board, i3 2.93GHz dual core, 2GB DDR3 1333 dual channel, 6x 2TB 7200RPM SATA HDs in hot swapable cages, 4U case, Redundant 500Watt PSU, 8x RAID controller (SAS/SATA), Intel 10/100/1000. $2250. We could technically buy 5 of these for the lowest price of a branded system. But would it be fast enough?

Linux has this command for finding disk speeds:
/sbin/hdparm -t /dev/yourHDhere
And on my test system, here are the results:

Timing buffered disk reads: 202 MB in 3.01 seconds = 67.16 MB/sec

You can find out the path of the HD you want by:
df -h

That’s on a pretty decent VMWare box. On a real box, with much lower specs, the read times were abysmal:

Timing buffered disk reads: 10 MB in 3.32 seconds = 3.01 MB/sec

I guess IDE HD’s on P4 1.8GHz systems are not the best…
To find out your processor information:
cat /proc/cpuinfo

So how about on my NAS/SAN solution? Unfortunately the command /sbin/hdparm will not work as the OS on this machine is FreeBSD. However, there is a similar command for BSD:
diskinfo -t /dev/yourHDhere

So you can see that the average for software RAID5 is 190834KB/sec (186.36MB/sec). That’s pretty stellar.

Next I’ll be trying RAID5 on the hardware card, followed by RAID10 on that same card. I’ll post the results as soon as I can.

***EDIT***

RAID5 on an Areca ARC-1222
466.74MB/sec using the exact same hardware setup. Next will be trying RAID1+0.

RAID10 is actually slower. It’s showing 319.70MB/sec. I’m going to try with the following commands from /mnt/raidcontainer:
dd if=/dev/zero of=testfile bs=1M count=5000
and
dd if=/dev/zero of=testfile bs=8K count=5000
to see what is up.

254MB/sec on 1MB writes over 5000 attempts. I’m trying to find out how to not cache disk copies on the smaller file – it gave me over 1000MB/sec… 🙂

***EDIT again***
OK, so FreeBSD doesn’t allow for the oflag on dd, so we have to use conv=sparse instead (and I changed to 50000):
dd if=/dev/zero of=testfile bs=8K count=50000 conv=sparse
Which gives me 265MB/sec using 8K files. Not a bad idea of the range. I think that I must go back to HW RAID5 for a complete list of read and writes.

***EDIT 4***
I built another FreeNAS system (0.7.2.5543 x64) with the following specs:
H55 Intel board, i3 3.06GHz dual core, 4GB DDR3 1333 dual channel, 8x 2TB 7200RPM SATA HDs in hot swapable cages, 4U case, Redundant 658Watt PSU, 8x RAID controller (areca arc-1222) with battery backup cache, Intel 10/100/1000, dummy gig switch with jumbo frame support. $2800.
dd if=/dev/zero of=testfile bs=8K count=500000 conv=sparse

340MB/sec! Although the older rig averaged 44MB/sec per drive, this one is at 42.5MB/sec per drive. After this I think the primary issue will be the raid controller as a bottleneck.

VSFTP SSL and Filezilla

FTP Server needed to be secure (at least SSL 128) and was running on Ubuntu 10.04.1 x32. FTP client was required to be platform independent but needed to be tested on Windows 7 and Windows XP. I decided to use Filezilla for various reasons.

The easiest step was setting up the FTP server on the Ubuntu box. I followed along a bit on “Mike’s” blog (http://beginlinux.com/blog/2009/10/ubuntu-9-10-secure-ftp-with-ssl/). I was logged in as root to avoid all those sudo’s.

Update the system and install VSFTPD:
apt-get update
apt-get upgrade
apt-get install vsftpd

Edit the VSDTPD configuration:
nano /etc/vsftpd.conf

# Example config file /etc/vsftpd.conf
## Base Configuration
listen=YES
#listen_ipv6=YES
#anonymous_enable=YES
local_enable=YES
write_enable=YES
#local_umask=022
#anon_upload_enable=YES
#anon_mkdir_write_enable=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES

## Permissions
#chown_uploads=YES
#chown_username=whoever
chroot_local_user=YES
chroot_list_enable=NO
secure_chroot_dir=/var/run/vsftpd/empty
pam_service_name=vsftpd

## SSL Certificate Configuration
#implicit_ssl=YES
ssl_enable=YES
rsa_cert_file=/etc/ssl/certs/vsftpd.pem
rsa_private_key_file=/etc/ssl/certs/vsftpd.pem
allow_anon_ssl=NO
ssl_tlsv1=YES
#ssl_sslv2=YES
ssl_ciphers=HIGH
ssl_sslv3=YES
require_ssl_reuse=NO

## Force encrypted login/passwords
force_local_data_ssl=YES
force_local_logins_ssl=YES
listen_port=990
force_dot_files=NO
tcp_wrappers=NO
#listen_address=
#hide_file=
#anon_max_rate=
#local_max_rate=
pasv_min_port=6000
pasv_max_port=6500
#pasv_address=IPOFSERVER

Save and exit.

Create the self-signed SSL certificate. I’m assuming you’re like me and don’t have a budget for miscellaneous SSL certificates.
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout /etc/ssl/certs/vsftpd.pem -out /etc/ssl/certs/vsftpd.pem
Enter your information when prompted.

Restart the VSFTPD server:
/etc/init.d/vsftpd restart

Open ports on your firewall/router/gateway. Obviously change these to whatever you require and have in your conf file.
TCP 20-21 (data and login)
TCP 6000-6500 (PASV ports)

Connect using FileZilla:
Using the SiteManager (quick connect won’t work in this case), enter in the following:
Host - your IP address of the server (you can use the local IP for testing functionality of the server, but use the public IP for testing the firewall rules)
Port - 990
Server type - FTPES (FTP over explicit TLS/SSL)
Logon Type - Normal
User - username on the system
Password - password for that user
Click connect! Everything should work.

PS, this was done on a fresh install of ubuntu with SSH and LAMP installed.

***EDIT***
I was looking through the logs (/var/log/vsftpd.log) and watching connections:
watch cat /var/log/vsftpd.log
When I noticed the following:

No SSL session reuse on data channel

I added “require_ssl_reuse=NO” to the vsconfig

nano /etc/vsftp.conf
require_ssl_reuse=NO
Save and exit
/etc/init.d/vsftpd restart

Now I’m getting:

Connection terminated without SSL shutdown – buggy client?

If you’re receiving complaints that some cannot perform a directory listing, or you dislike having the following errors in your logs:
Server sent passive reply with unroutable address. Using server address instead.
GnuTLS error -53: Error in the push function.
Add the following to your NAT’d device (ie firewall/router):
nano /etc/vsftpd.conf
pasv_address=IPADDRESSOFYOUREXTERNALNAT
Save and restart the vsftp server
/etc/init.d/vsftpd restart

Edited to include pasv issue, remove sslv2, change the cert to 2048bit and valid for 2 years.

Exchange Errors

It was Friday around 3PM and most people were already gone for the day. One of the few employees left came by my desk and asked if something was up with the Exchange server. I checked Outlook – it said it was connected. Shift + F9 to force a reconnection and it failed out. Error 0x80004005. Last email received was 30 minutes ago. Great.

So error 0x80004005 says this:

The client operation failed. Microsoft Exchange Information Store.

Log onto the Exchange server and verify the services are all running. They all were running.
Check the event viewer log files – way too many errors to go through all of them while a production server was down, but this one stood out:

Event viewer showed this error:

A transient failure has occurred. The problem may resolve itself in a while. The service will retry in 56 seconds. Diagnostic information:
Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant.
Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant. —> Microsoft.Mapi.MapiExceptionLogonFailed: MapiExceptionLogonFailed: Unable to open message store. (hr=0x80040111)

Restart the Information Store service.
Now I get this error in the event viewer:

MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1276)

Crap. I verify that we’re running Exchange enterprise and not standard (database store limits). I check the management console and see that the database is offline and refuses to mount. Since the database was already offline, I decided to run a repair on the DB.

Open a command prompt:
eseutil /p "C:\program files\microsoft\exchange server\mailbox\first storage group\mailbox database.edb"
This took roughly 30 minutes on a 44GB database.

Try to mount the database again. It fails but gives me a different error this time:

Exchange is unable to mount the database that you specified. Specified database: Exchange07\Storage Group 1\Mailstore 1; Error code: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-515)

This means that I’m missing a required transaction log and that the DB won’t start without it. Great.

So I navigate to the first storage group log files (C:\program files\microsoft\exchange server\mailbox\first storage group)
I moved all of the files (excluding Mailbox Databse.edb and CatalogData-* folder) to another directory. Then I attempted to mount the DB again. Success!

Total downtime: 75 minutes
Active working time: 45 minutes

So what happened?
McAfee Antivirus grabbed a file from the transaction log before it could be written to the database. Since the transaction log was altered while it was trying to write to the database, the mailstore became slightly corrupt and required a repair. And because the transaction logs and the database were at different parts, even after the repair, the log files needed to be “destroyed” in order for the DB to be mounted again. Oh, the logs were recreated after successfully mounting the database.

LDAP Testing

Sometimes I forget how much we actually use LDAP in our daily lives. Active Directory is based on LDAP. Linux can use LDAP. Same for Mac. Cisco. Lots of stuff.

In the process of upgrading from a managed call center to an in-house Cisco call manager system, we find that we need to add a Unity user account and type in the LDAP path to this user (along with the path to the users we want to import).

Being as forgetful as I usually am, I had to open up adsiedit to find out the exact paths required.

OBJECT CLASS:
User, Group, Computer, or Container are all CN (Common Name)
Organizational Unit is OU
Domain is DC

So, in our case, I had to input “cn=Unity User,ou=Service Accounts,dc=domain,dc=com”

In order to test out if this works, as well as if a user actually has rights to see the LDAP information, I found a handy program called LDAP Administrator (or Viewer)
http://www.ldapadministrator.com/download.htm

Just feed that the information and it does the rest. It reminds me of a better ADSIedit program.

Decode eval gzinflate base64_decode

I had to install a wordpress theme for a customer. I’m not a web developer by any means, so lucky for me wordpress is pretty straight forward. The theme they wanted to use was “free”, but had a footer with links to the theme’s developer site. Unfortunately the developer’s site was no longer functioning. A brief search for their company yielded similar results – all landing website pages or 404 error sites.

The terms and conditions for the theme stated that you could remove the footer only if you paid $29.95 to the company. As there was no company to pay, a “gray area” was presented. Should I just remove the footer and be done with my day? The customer requested that I adhere to the terms and conditions but make sure that the links would not go to unknown websites for fear that a malicious link could be used. They also requested that an RSS feed link be removed from the footer.

I go to edit the footer and notice the following PHP code:
gzinflate(base64_decode('bVJRa9swEH4P5D9cTTrZEKdjG3tYbIeydexlfUhgMMYQtnWxRWTJk+RkWcl/n2S7bUirB+t8p/u+u+9ulSWM76EUuTFp8FUpizrIphNw52Uk5lI+x89PkkOtcZsGyaqtWyiE+qrjcqpBoY97RTgsSLWGVBY94zh3bvIp5qWQAlluBabDe$
Well how do I know this obfuscated PHP script isn’t going to send customers to malicious sites? I need to see the decoded PHP!

Scripting to the rescue!

Easiest way is to have a linux server laying around. I have a fully patched 10.04LTS Ubuntu server just for this purpose (VMWare Server).

Make the following PHP file:

< ? php /* Taken from http://www.php.net/manual/de/function.eval.php#59862 Directions: 1. Save this snippet as decrypt.php 2. Save encoded PHP code in coded.txt 3. Create a blank file called decoded.txt (from shell do CHMOD 0666 decoded.txt) 4. Execute this script (visit decrypt.php in a web browser or do php decrypt.php in the shell) 5. Open decoded.txt, the PHP should be decrypted */ echo "\nDECODE nested eval(gzinflate()) by DEBO Jurgen \n\n";
echo "1. Reading coded.txt\n";
$fp1 = fopen ("coded.txt", "r");
$contents = fread ($fp1, filesize ("coded.txt"));
fclose($fp1);
echo "2. Decoding\n";
while (preg_match("/eval\(gzinflate/",$contents)) {
$contents=preg_replace("/< \?|\?>/", "", $contents); eval(preg_replace("/eval/", "\$contents=", $contents)); } echo "3. Writing decoded.txt\n"; $fp2 = fopen("decoded.txt","w"); fwrite($fp2, trim($contents)); fclose($fp2);
?>

Then follow the directions in the script. Or, if you’re like me, just create the following:
coded.txt (this file will have the ENTIRE PHP portion – everything from “eval(gzinflate” to the last “);”)
decoded.txt (make sure it’s writeable – either 0666 or 0777)
Run the script
Open the decoded.txt file

Ramblings Of An IT Person