HTTP/1.1 507 Insufficient space to store resources

It doesn’t happen too often that I come across an error that I haven’t seen before.

Customer is running Exchange 2003 on a native 2003 server infrastructure. Users would log into the webmail area but when they’d click on any emails to read the nasty 507 error would show up. Another issue was that webmail was up-to-date but the local outlook client was not. Local client would say it was connected to Exchange, but would not actively update folders. Forced updates yielded no results.

A quick check of the eventlog showed this:

Source: MSExchangeIS
Category: Performance
Event ID: 9582
Description: The virtual memory necessary to run your Exchange server is fragmented in such a way that normal operation may begin to fail. It is highly recommended that you restart all Exchange services to correct this issue.

Well that’s an easy fix. I shut down the services:
IMAP4
Management
MTA Stacks
POP3
Routing Engine
Information Store
System Attendant

And then started them back up in the reverse order. Everything checked out OK. Now to get them to upgrade to Exchange 07/10…

Internet Connection Speeds

I figured I’d put this out so someone can read it all in one place.

Analog copper moved onto digital copper moved onto fiber optics. The best way to describe bandwidth and latency is to think of it this way:
Bandwidth is how much water can go through the pipe, latency is how much pressure. Or, bandwidth is how many lanes the road has and latency is how fast the cars are moving. Or (another car one) bandwidth is how fast the cars can drive and latency is how fast they can accelerate.
Narrowband references anything less than 56Kbps. Broadband references anything greater than 56Kbps.
FTTN – Fiber to the node – Connects the ISP to a street cabinet in your neighborhood via Fiber, and then copper to your home.
FTTC – Fiber to the Capinet/Curb – Connects the ISP to a cabinet within 300m of your home, and then copper to your home.
FTTB/FTTH – Fiber to the building/basement Fiber to the Home – Fiber comes all the way to a hand-off just outside of the living area of the home.

9600 baud modem. A whopping 1.2KB/sec. Still in use for null serial connections to many network devices.
14.4 modem. Analog, standard RJ11 phone line. 1.8KB/sec
19.2 modem. Analog, standard RJ11 phone line. 2.4KB/sec
33.6 modem. Analog, standard RJ11 phone line. 4.2KB/sec
56k flex modem. Analog, standard RJ11 phone line limited to 53kbps on the lines. 7KB/sec

64Kbps ISDN. Digital, standard RJ11 phone line. 64Kb in both directions. 8KB/sec
128Kbps ISND. Digital, standard RJ11 phone line. 128Kb in both directions. 16KB/sec

xDSL. Digital/Analog, usually running over standard RJ11 phone lines. Ranges from 128Kbps to 24Mbps. 16KB/sec to 3072KB/sec (theoretical max of 50Mbit)
Cable. Digital, usually running over RG59 coaxial cable. Ranges from 256Kbps to 50Mbps. 32KB/sec to 6400KB/sec (theoretical max of 400Mbit)

DS0 – Copper – 64Kbps. 8KB/sec
DS1 (T1) – Copper – 1.544Mbps.
DS2 (T2) – Copper – 6.312Mbps.
DS3 (T3) – Copper – 44.736Mbps.
DS4 (T4) – Fiber – 274.176Mbps.
DS5 (T5) – Fiber – 400.352Mbps.

OC1 – Fiber – 51.84Mbps. 6635.52KB/sec 6.48MB/sec 0.006GB/sec
OC3 – Fiber – 155.52Mbps. 19906.56KB/sec 19.44MB/sec 0.019GB/sec
OC12 – Fiber – 622.08Mbps. 79626.24KB/sec 77.76MB/sec 0.076GB/sec
OC24 – Fiber – 1244.16Mbps. 159252.48KB/sec 155.52MB/sec 0.152GB/sec
OC48 – Fiber – 2488.32Mbps. 318504.96KB/sec 311.04MB/sec 0.304GB/sec
OC96 – Fiber – 4976.64Mbps. 637009.92KB/sec 622.08MB/sec 0.608GB/sec
OC192 – Fiber – 9953.28Mbps. 1274019.84KB/sec 1244.16MB/sec 1.215GB/sec
OC768 – Fiber – 39813.12Mbps. 5096079.36KB/sec 4976.64MB/sec 4.86GB/Sec

Issue Copying Startup Files To USB Flash FreeNAS

I wanted to try out FreeNAS (based on FreeBSD but with quite a few awesome ‘NAS-like’ features) on a system at a jobsite:
System has 5x 2TB drives in RAID5 (6.7TB usable for storage) hotswappable, 2.5GHz Dual Core Pentium, 4GB DDR2 1066 RAM, 3x 10/100/100 NIC, FreeNAS .7 stable.

I got everything up and running without a problem (it is quite easy), running it off a 1GB USB Flash drive I had lying around. But I wanted to continue using my flash drive for other things and found an unused 128MB flash drive that no one would ever use again. I’d rather waste 128MB than 1024MB…

Formated the 128MB into FAT
Plugged the 128MB flash into the FreeNAS server while it was off
Booted the FreeNAS server using the FreeNAS .7 CD
Pressed 9 to install embedded onto the flash drive

Mount CDROM.
Installing system image on device da0.
gzip: stdout: broken pipe
Error: Failed to dd image on ‘/dev/da0’!
Unmount CDROM.
There was a problem while copying files.
Press ENTER to continue.

Argh. So I checked the flash drive on the main PC again – FreeNAS had written a config file and folder to the device!

It looks like if you boot up the system with a USB flash drive installed, the config file will always be written and you can’t format the device. So just plug the drive in AFTER everything is booted up.

Wiped 128MB again and formatted FAT
Plugged 128MB into FreeNAS - Auto mounted the device as da0
Try 9, 1 again

Mount CDROM.
Installing system image on device da0.
GEOM_LABEL: Label for provider da0a is ufsid/************.
Unmount CDROM.
FreeNAS has been installed on da0.
You can now remove the CDROM and reboot the PC.

So I did just that.

Block DNS DD-WRT

Here’s the project:
Secure a wireless access point for vendors/non-work-computers to use. Disallow access to anything that eats up bandwidth – it’s supposed to be a tool and not a play thing.

Background:
A primary internet connection has 62 public IP addresses. One is pointed to a WRT54-G (v1) router. I installed dd-wrt (v24 preSP2 build 13064) and configured accordingly. I signed up for OpenDNS service to block all the “bad stuff”.

Problem:
Merely putting in the DNS information into the DHCP server is not enough. Any savvy tech user can add their own DNS information into the equation and go from there. I needed a way to block DNS (port 53) on the router side of things so that no outside influences could bypass the security. DD-WRT was configured to block port 53, but that didn’t really work out too well. So I ended up adding my own iptable chain to the router not to block other DNS, but to force it to go through the router’s DNS. Makes things easier in the long run.

Solution:

Log into the router
Click on the Administration Tab
Click on the Commands Tab
In the commands box, paste the following:
iptables -t nat -A PREROUTING -p udp -i br0 --dport 53 -j DNAT --to $(nvram get lan_ipaddr)
iptables -t nat -A PREROUTING -p tcp -i br0 --dport 53 -j DNAT --to $(nvram get lan_ipaddr)
Click Save Firewall

After it reboots, test it out. Use 4.2.2.2 as a test DNS server to verify. Awesome, I know.

Security:
WRT54-G running DD-WRT with a non-standard password
SSID is not being broadcast
WPA2 Personal with AES Encryption
WRT54-G is connected to a portion of the main internet pipe on a public IP but is VLAN’d off from the rest of the network
DD-WRT admin access is not allowed over the WLAN
OpenDNS is blocking almost all access
DD-WRT does NOT allow connections before 7AM, after 7PM, or on weekends
I’m wearing pants

OK, had to lighten the mood a little bit. But that’s the gist of it.

Install Nagios on Ubuntu 9.x, Client on Windows

This is how I got nagios to install on my Ubuntu 9.10 x86 server (LAMP):
***EDIT*** I’ve upgraded the 9.10 x86 server to 10.04LTS x86 and Nagios is still running like a champ. FYI.

This assumes you’ve already got LAMP up and running on the box and have at least a little common sense when it comes to linux commands. I still use putty to connect via SSH to my box. I also cheated and made it so I don’t do sudo (sudo passwd root, then su into root). If you don’t want to do that, just add ‘sudo’ in front of all the commands.

Update your box:
apt-get update
apt-get upgrade
apt-get dist-upgrade

Install the GCC compiler development libraries:
apt-get install build-essential

Install GD development libraries:
apt-get install libgd2-xpm-dev

Create a new user “nagios”:
useradd -m -s /bin/bash nagios

Make a password for the new user:
passwd nagios

Create a new group “nagcmd”
groupadd nagcmd
usermod -a -G nagcmd nagios
usermod -a -G nagcmd www-data

Download the nagios files – I use /home/username as my download point:
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.0.tar.gz
wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.14.tar.gz

Extract and compile the files:
tar xzf nagios-3.2.0.tar.gz
cd nagios-3.2.0
./configure --with-command-group=nagcmd
make all
make install
make install-init
make install-config
make install-commandmode
make install-webconf

Create a nagiosadmin account for the web interface:
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

Restart Apache
apache2ctl restart

That should be it for the installation phase of the main application. You can verify that everything works by viewing the site: http://localhost/nagios (or http://ipofserver/nagios) and logging in using nagiosadmin/thepasswordyoupicked. But now we should probably put the plugins in place before actually going through any other setup.

Unpack the plugins:
tar xzf nagios-plugins-1.4.14.tar.gz
cd nagios-plugins-1.4.14

Configure and install the plugins:
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install

Verify that the sample nagios configuration files are fine:
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If there are no errors you can start nagios:
/etc/init.d/nagios start

http://localhost/nagios
Click on the Tactical Overview navbar link to see all that’s monitored right now. This may take a few moments to load as to not “kill” the machine with requests.

http://www.monitoringexchange.org/

INSTALL CLIENT ON WINDOWS MACHINE(S):
I’ve installed on WindowsXP 32, Windows Server 2003 32/64, and Windows 7 64bit without any issues.

Setting up for Windows Machine monitoring is a little more in depth:

Download the NSClient++:
http://nsclient.org/nscp/downloads
Extract to the C:\ directory
Open a CMD prompt and navigate to C:\NSClient++-0.3.7-Win32

Install the Service and System Tray:
The system tray gave me some issues on a Windows 7 x64 box. FYI.
nsclient++ /install
nsclient++ SysTray -install

Configure the NSClientpp Service:
Open up services (services.msc) and find NSCLientpp (Nagios)
Right-Click and select properties
Click on the Log On tab and make sure the Local System account can interact with the desktop

Edit the INI file:
Navigate to C:\NSClient++*\
Edit NSC.INI

Uncomment the following:

FileLogger.dll
CheckSystem.dll
CheckDisk.dll
NSClientListener.dll
NRPEListener.dll
SysTray.dll
CheckEventLog.dll
CheckHelpers.dll

Uncomment the hosts allowed file – either add the IP address of the Nagios server or just leave it blank and it will accept from all IPs.

Save the file, Start the service – I’ve had it work without interact with desktop, but it was recommended by Nagios to enable that feature.

Back on the Nagios server, edit the /usr/local/nagios/etc/objects/windows.cfg file to the liking of your windows machine.
You may have to add the host name/ip to your /etc/hosts file to rid yourself of errors

Verify your setup can run:
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Then restart nagios:
/etc/init.d/nagios restart

Now you should be done. Login to your http://serverip/nagios site with your nagiosadmin/passwordyouchoseearlier account information.

As a side note, I also edited /usr/local/nagios/etc/nagios.cfg for the admin_email= to send email to my account. I already have postfix installed from another project, so it *should* send me alert emails. AND I found that you have to edit /usr/local/nagios/etc/objects/contacts.cfg as well. After editing make sure you restart the nagios service.

***EDIT***

I was receiving the emails just fine, but they were coming from the “wrong domain”. Ubuntu/debian puts the domain name by default in the /etc/mailname file. Edit that file and then reload postfix.
nano /etc/mailname
postfix reload

If the file does not exist (you’re creating a new file with nano), type in your domain name (ie thelibrarie.com), and then save the file. Then edit the postfix configuration and uncomment myorgin.
nano /etc/mailname
nano /etc/postfix/main.c

myorigin = /etc/mailname

And then try it again. You can test it out by sending yourself some mail:
mail -s test username@domain.tld
body
Control + D
Control + D

I also realized that I put how to install the client on Windows machines, so I should probably mention that in the subject, eh?

***EDIT***
OK, I had to add another user to be able to view the status of the systems.
Add the User:
SSH to your nagios system
htpasswd /usr/local/nagios/etc/htpasswd.users USERNAMEHERE
apache2ctl restart

Add the permissions:
nano /usr/local/nagios/etc/cgi.cfg
Search under GLOBAL HOST/SERVICE VIEW ACCESS
Change:
authorized_for_all_services=nagiosadmin
authorized_for_all_hosts=nagiosadmin
to:
authorized_for_all_services=*
authorized_for_all_hosts=*

Or you can comma-delimit the users. Either way, I win.

Restart nagios.
/etc/init.d/nagios restart

***EDIT 2***
Need more plugins? http://nagios.manubulon.com/

***EDIT 3***
There will be a new post regarding editing Nagios to add images to the map/new users/parents of configurations etc.

PHPSysInfo and LM-Sensors

I wanted to get phpsysinfo working with the hardware sensors suite known as lm-sensors, but I really didn’t feel like working through all the problems myself. But, despite my attempts to be lazy, I actually had to work a little bit at this.

First of all, this is running on an Ubuntu Server 9.10 x86 – 2.6.31-14-generic-pae to be exact – using an old Dell Precision 360 box to be even more exact. I installed phpsysinfo version 2.5.4 and edited the config.php file to enable lm-sensors hardware monitoring.

Unfortunately I get this error when I navigate to http://myserverip/phpsysinfo:

No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
Return value: 0

Wonderful. So I have to install lm-sensors. Now another site told me to do the following:
sudo apt-get install lm-sensors i2c-tools read-edid sensord hddtemp sensors-applet
After the lengthy download (153MB I believe), hddtemp needed some input. The defaults are fine – I selected to start the hddtemp service every reboot and run it on the default port 7634 as this is an internal only server. If it’s an externally facing server you may want to exercise some caution when opening ports.

You can verify that this is working by navigating to http://yourserverip:7634
In my case I see:

|/dev/sg0|ST380011A|38|C||/dev/sda|ST380011A|38|C|

Which I can see the Seagate HD that’s 80GB IDE with two mount points running at 38C (100.4F).

Now to setup for other sensors. Go back to your terminal session and type:
sudo sensors-detect
Hit YES for everything. When the script finishes you should probably reboot:
reboot

Reload your http://yourserverip/phpsysinfo page – you should now see a bunch of other stuff added at the bottom:
lm-sensors

Remove Annoying Logon Background Image

I had to RDP to a server several times before this actually started to bother me:
Dell has their own image set as the default background during the login prompt. RDP over the internet enough and the slow loading of the background starts to eat away at you.

It’s a simple registry hack to fix it:

Start – Run
regedit
HKEY_USERS\.DEFAULT\Control Panel\Desktop
Find a key named Wallpaper
Change the key’s data to (None) instead of C:\windows\dellcrap\dellcrapimage.bmp
Close out of the registry editor
Wait a few minutes and it should be gone when you try to login.

This worked on Windows XP and Windows Server 2003.

Ramblings Of An IT Person