Monthly Archives: September 2010

Linux

Testing RAID and HD Speeds under *nix

Leave a comment

I was given the task of purchasing a NAS/SAN solution for a secondary (dev) oracle database (11g). Pricing it out on the normal branded websites yielded roughly $12,000 to $30,000 depending on the options. A quick look at newegg gave me a lot more for less coin.

H55 Intel board, i3 2.93GHz dual core, 2GB DDR3 1333 dual channel, 6x 2TB 7200RPM SATA HDs in hot swapable cages, 4U case, Redundant 500Watt PSU, 8x RAID controller (SAS/SATA), Intel 10/100/1000. $2250. We could technically buy 5 of these for the lowest price of a branded system. But would it be fast enough?

Linux has this command for finding disk speeds:
/sbin/hdparm -t /dev/yourHDhere
And on my test system, here are the results:

Timing buffered disk reads: 202 MB in 3.01 seconds = 67.16 MB/sec

You can find out the path of the HD you want by:
df -h

That’s on a pretty decent VMWare box. On a real box, with much lower specs, the read times were abysmal:

Timing buffered disk reads: 10 MB in 3.32 seconds = 3.01 MB/sec

I guess IDE HD’s on P4 1.8GHz systems are not the best…
To find out your processor information:
cat /proc/cpuinfo

So how about on my NAS/SAN solution? Unfortunately the command /sbin/hdparm will not work as the OS on this machine is FreeBSD. However, there is a similar command for BSD:
diskinfo -t /dev/yourHDhere

So you can see that the average for software RAID5 is 190834KB/sec (186.36MB/sec). That’s pretty stellar.

Next I’ll be trying RAID5 on the hardware card, followed by RAID10 on that same card. I’ll post the results as soon as I can.

***EDIT***

RAID5 on an Areca ARC-1222
466.74MB/sec using the exact same hardware setup. Next will be trying RAID1+0.

RAID10 is actually slower. It’s showing 319.70MB/sec. I’m going to try with the following commands from /mnt/raidcontainer:
dd if=/dev/zero of=testfile bs=1M count=5000
and
dd if=/dev/zero of=testfile bs=8K count=5000
to see what is up.

254MB/sec on 1MB writes over 5000 attempts. I’m trying to find out how to not cache disk copies on the smaller file – it gave me over 1000MB/sec… 🙂

***EDIT again***
OK, so FreeBSD doesn’t allow for the oflag on dd, so we have to use conv=sparse instead (and I changed to 50000):
dd if=/dev/zero of=testfile bs=8K count=50000 conv=sparse
Which gives me 265MB/sec using 8K files. Not a bad idea of the range. I think that I must go back to HW RAID5 for a complete list of read and writes.

***EDIT 4***
I built another FreeNAS system (0.7.2.5543 x64) with the following specs:
H55 Intel board, i3 3.06GHz dual core, 4GB DDR3 1333 dual channel, 8x 2TB 7200RPM SATA HDs in hot swapable cages, 4U case, Redundant 658Watt PSU, 8x RAID controller (areca arc-1222) with battery backup cache, Intel 10/100/1000, dummy gig switch with jumbo frame support. $2800.
dd if=/dev/zero of=testfile bs=8K count=500000 conv=sparse

340MB/sec! Although the older rig averaged 44MB/sec per drive, this one is at 42.5MB/sec per drive. After this I think the primary issue will be the raid controller as a bottleneck.

Linux, Microsoft

VSFTP SSL and Filezilla

1 Comment

FTP Server needed to be secure (at least SSL 128) and was running on Ubuntu 10.04.1 x32. FTP client was required to be platform independent but needed to be tested on Windows 7 and Windows XP. I decided to use Filezilla for various reasons.

The easiest step was setting up the FTP server on the Ubuntu box. I followed along a bit on “Mike’s” blog (http://beginlinux.com/blog/2009/10/ubuntu-9-10-secure-ftp-with-ssl/). I was logged in as root to avoid all those sudo’s.

Update the system and install VSFTPD:
apt-get update
apt-get upgrade
apt-get install vsftpd

Edit the VSDTPD configuration:
nano /etc/vsftpd.conf

# Example config file /etc/vsftpd.conf
## Base Configuration
listen=YES
#listen_ipv6=YES
#anonymous_enable=YES
local_enable=YES
write_enable=YES
#local_umask=022
#anon_upload_enable=YES
#anon_mkdir_write_enable=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES

## Permissions
#chown_uploads=YES
#chown_username=whoever
chroot_local_user=YES
chroot_list_enable=NO
secure_chroot_dir=/var/run/vsftpd/empty
pam_service_name=vsftpd

## SSL Certificate Configuration
#implicit_ssl=YES
ssl_enable=YES
rsa_cert_file=/etc/ssl/certs/vsftpd.pem
rsa_private_key_file=/etc/ssl/certs/vsftpd.pem
allow_anon_ssl=NO
ssl_tlsv1=YES
#ssl_sslv2=YES
ssl_ciphers=HIGH
ssl_sslv3=YES
require_ssl_reuse=NO

## Force encrypted login/passwords
force_local_data_ssl=YES
force_local_logins_ssl=YES
listen_port=990
force_dot_files=NO
tcp_wrappers=NO
#listen_address=
#hide_file=
#anon_max_rate=
#local_max_rate=
pasv_min_port=6000
pasv_max_port=6500
#pasv_address=IPOFSERVER

Save and exit.

Create the self-signed SSL certificate. I’m assuming you’re like me and don’t have a budget for miscellaneous SSL certificates.
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout /etc/ssl/certs/vsftpd.pem -out /etc/ssl/certs/vsftpd.pem
Enter your information when prompted.

Restart the VSFTPD server:
/etc/init.d/vsftpd restart

Open ports on your firewall/router/gateway. Obviously change these to whatever you require and have in your conf file.
TCP 20-21 (data and login)
TCP 6000-6500 (PASV ports)

Connect using FileZilla:
Using the SiteManager (quick connect won’t work in this case), enter in the following:
Host - your IP address of the server (you can use the local IP for testing functionality of the server, but use the public IP for testing the firewall rules)
Port - 990
Server type - FTPES (FTP over explicit TLS/SSL)
Logon Type - Normal
User - username on the system
Password - password for that user
Click connect! Everything should work.

PS, this was done on a fresh install of ubuntu with SSH and LAMP installed.

***EDIT***
I was looking through the logs (/var/log/vsftpd.log) and watching connections:
watch cat /var/log/vsftpd.log
When I noticed the following:

No SSL session reuse on data channel

I added “require_ssl_reuse=NO” to the vsconfig

nano /etc/vsftp.conf
require_ssl_reuse=NO
Save and exit
/etc/init.d/vsftpd restart

Now I’m getting:

Connection terminated without SSL shutdown – buggy client?

If you’re receiving complaints that some cannot perform a directory listing, or you dislike having the following errors in your logs:
Server sent passive reply with unroutable address. Using server address instead.
GnuTLS error -53: Error in the push function.
Add the following to your NAT’d device (ie firewall/router):
nano /etc/vsftpd.conf
pasv_address=IPADDRESSOFYOUREXTERNALNAT
Save and restart the vsftp server
/etc/init.d/vsftpd restart

Edited to include pasv issue, remove sslv2, change the cert to 2048bit and valid for 2 years.

Microsoft

Exchange Errors

2 Comments

It was Friday around 3PM and most people were already gone for the day. One of the few employees left came by my desk and asked if something was up with the Exchange server. I checked Outlook – it said it was connected. Shift + F9 to force a reconnection and it failed out. Error 0x80004005. Last email received was 30 minutes ago. Great.

So error 0x80004005 says this:

The client operation failed. Microsoft Exchange Information Store.

Log onto the Exchange server and verify the services are all running. They all were running.
Check the event viewer log files – way too many errors to go through all of them while a production server was down, but this one stood out:

Event viewer showed this error:

A transient failure has occurred. The problem may resolve itself in a while. The service will retry in 56 seconds. Diagnostic information:
Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant.
Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant. —> Microsoft.Mapi.MapiExceptionLogonFailed: MapiExceptionLogonFailed: Unable to open message store. (hr=0x80040111)

Restart the Information Store service.
Now I get this error in the event viewer:

MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1276)

Crap. I verify that we’re running Exchange enterprise and not standard (database store limits). I check the management console and see that the database is offline and refuses to mount. Since the database was already offline, I decided to run a repair on the DB.

Open a command prompt:
eseutil /p "C:\program files\microsoft\exchange server\mailbox\first storage group\mailbox database.edb"
This took roughly 30 minutes on a 44GB database.

Try to mount the database again. It fails but gives me a different error this time:

Exchange is unable to mount the database that you specified. Specified database: Exchange07\Storage Group 1\Mailstore 1; Error code: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-515)

This means that I’m missing a required transaction log and that the DB won’t start without it. Great.

So I navigate to the first storage group log files (C:\program files\microsoft\exchange server\mailbox\first storage group)
I moved all of the files (excluding Mailbox Databse.edb and CatalogData-* folder) to another directory. Then I attempted to mount the DB again. Success!

Total downtime: 75 minutes
Active working time: 45 minutes

So what happened?
McAfee Antivirus grabbed a file from the transaction log before it could be written to the database. Since the transaction log was altered while it was trying to write to the database, the mailstore became slightly corrupt and required a repair. And because the transaction logs and the database were at different parts, even after the repair, the log files needed to be “destroyed” in order for the DB to be mounted again. Oh, the logs were recreated after successfully mounting the database.