Rancid Ubuntu 14.04 LTS

Recently upgraded the Rancid server from 12.04.3 to 14.04 (and it started as 10.04 from my rancid on ubuntu 10 install page).

Not everything went flawlessly: the upgraded killed the configuration file that said drancid is to be used for my dell switches. Not a big deal, just calling it out. I just following my “drancid” post somewhere else on this blog.

Then I noticed that some of the switches were EOF prematurely; turned out it was due to the fact that the exec and login banners on these cisco devices had the character “#” in them and the new version of having an issue parsing those out. Another easy fix.

Final problem was an error 500 on the web interface. In my brief research this was due to the fact that perl was upgraded to 5.18.x from 5.14.x
It should be noted that I learned about the rmadison packagenamehere command (installed with devscripts) which was fairly helpful. It should also be noted that there was not a lot of information on this error which leads me to believe that not many are using rancid on a 14.x or above system.

Troubleshooting this issue I ran (change depending on the user you’re running rancid under)
sudo login -f root
source /etc/rancid/rancid.conf
NOPIPE=yes;export NOPIPE
rancid -d switchname
nano *.new
nano *.raw
The full logs are the *.new and the *.raw files in the directory you ran the commands in.

So someone (a netbsd op) pointed out that there is a fix for this issue. In my case I’m running debian/ubuntu, so I actually found this article by Mark Kamichoff.

nano /usr/lib/cgi-bin/cvsweb
I searched for “legend” to get to the following:

 <legend>General options</legend>
 <input type="hidden" name="copt" value="1" />
 EOF
-    for my $v qw(hidecvsroot hidenonreadable) {
+    for my $v (qw(hidecvsroot hidenonreadable)) {
       printf(qq{<input type="hidden" name="%s" value="%s" />\n},

Searched again for “mytz”:

   print '<i>';
-  if (defined @mytz) {
+  if (@mytz) {
     my ($est) = $mytz[(localtime($date{$_}))[8]];
     print scalar localtime($date{$_}), " $est</i> (";
   } else {

Nas4Free Smartctl

Running 9.2.0.1 – Shigawire (revision 972) on some supermicro server with an Intel Xeon.

Noticed that I wasn’t receiving my weekly status emails so I finally got around to checking it out (system is running as a backup of a backup for my personal files, so not the end of the world if it doesn’t work right).

Logged in via the web interface and didn’t notice anything really going on except that out of the 4 available cores on the processor 2 were maxing out at 100%. Process listing showed smartctl was the culprit. Went into the ZFS settings (forgot no smart settings in there), and then tried to load up the Disk > Management area. No go – spinning and spinning.

Logged in using SSH to restart the web services
/etc/rc.d/lighttpd restart

Back in business, but then the Disk > Management area was once again crashing it out.

Issued a kill -s HUP thePIDhere. They came right back.

Rebooted the server. Smartctl was still coming up eating all of my available CPU time for reports.

I ended up renaming the smartctl bin file and then killing the processes.
It’s located /usr/local/sbin, then mv smartctl to smartctl.old or something

From there I could load the Disk > Management page again and disable smart monitoring.

And yes, the disks have all checked OK.

MS SQL 2008 Space

So we were getting alerts that our SQL server was running out of space. It was chewing through ~150GB a month.

Looking into it, we found out that our CRM (MS CRM 4.0) database was the primary culprit.

crm1
* Highlighted portion from another technician

I finally noticed something a little bit off; the Last Database Log Backup was listed as 9/24/2014.
In the SQL Server Management Studio, under Databases, I right-click and choose properties of the CRM database
I then selected the Files page (left side)
The PRIMARY rows data mdf was 56GB
The Log ldf was ~185GB

That might explain it!

So we went back and looked through the notes regarding this server. We had changed the Recovery Model to Full (same location under the Options page) in order to export a copy for our development environment. And this was never changed back.

Change this setting back to Simple and hit OK.
Then Right-click on the same database and choose Tasks, Shrink, Files
File type change to Log

In my case it showed available freespace of 99%. I clicked OK and the data usage went down to 1.25GB. Pretty big savings if you ask me.

Exchange 2010 Remove Mobile Device Error

Exchange 2010 environment fully patched. One user hit the 10 device threshold on Exchange activesync devices and IT was tasked with removing the association from that device.

Right-click on the user and manage mobile device. Attempt to remove the partnership (not remote wipe) and had an error:
The ActiveSyncDevice Cannot be Found.

Awesome.

Powershell to the rescue:
List all devices for an account
get-activesyncdevice -mailbox USERNAME | select DeviceID, Identity

List all details for those devices
get-activesyncdevicestatistics -mailbox USERNAME

Remove that activesync
Remove-ActiveSyncDevice -Identity $(Get-ActiveSyncDevice -Mailbox USERNAME | where {$_.DeviceId -like "DEVICEID"} | select Identity).identity

Apparently what happens is that when we decided to move all OU groups around (and memberships) Exchange doesn’t like that very much and forgets to update itself.
http://exchangeblog.pl/en/2013/01/exchange-2010-remove-mobile-device-error-the-activesyncdevice-cannot-be-found/

Add User Pictures to Active Directory, Exchange

Running Active Directory in 2003 mode (with 2x 2003 and 2x 2008 servers) and Exchange 2010 in DAG.

http://www.codetwo.com/blog/codetwo-active-directory-photos-1-1-2-new-fields-in-filters/

Ran through and installed the above executable, ran with my domain admin credentials, but I could not get the pictures to show up in Outlook no matter what I did.

Finally figured out it is an issue with the offline address book – rebuilt on the Exchange side and no dice. But deleted the cached folders on my local machine and it worked fine. There’s also a registry key to help out:

Close Outlook
Delete everything in C:\users\yourusername\appdata\microsoft\outlook\ofline address books\
Open Outlook
Profit

Registry key:
[HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Cached Mode]
“DownloadOAB”=dword:00000000
Either create the key “Cached Mode” and then the 32bit dword “DownloadOAB” as 0 or save the above as a .reg file and then run that.

Linux Fun

I’ll probably break these out, but for now this is the title and location.

We’re finally hiring a linux administrator (13 years experience) to replace me and my “hobby”. Good thing is he’s way better than I can hope to be. Bad thing is he doesn’t start for another couple weeks and I have a lot of Linux-y stuff going on.

So here are some helpful commands:
What Linux Distribution Are You Using?
cat /etc/*-release
OR
lsb_release -a

What Kernel Version Am I Running?
uname -a

What Processors Am I Using?
cat /proc/cpuinfo

What Hardware Specs Do I Have? (Motherboard Model, BIOS revision, etc)
dmidecode
OR, if installed
hwinfo

How Do I Setup No-Password-Needed SSH?
ssh-keygen
enter
enter
ssh-copy-id username@systemname
userpassword

ZFS Replace A Drive
zpool offline poolname /dev/daX
zpool replace poolname /dev/daX /dev/daY
zpool status poolname
After Rebuild
zpool detach poolname /dev/daX

LVM – Create Physical, Volume, and Logical
pvcreate /dev/sdb1
vgcreate vgpool /dev/sdb1
lvcreate -L 3G -n lvstuff vgpool

LVM – Display Current Status
pvdisplay

LVM – Add A New Disk
fdisk /dev/daX
n, p, 1, t, 8e, w
pvcreate /dev/daX1

LVM – Extend LVM Pool To New Disk
vgextend vgpool /dev/daX1

LVM – Resize File System (required for shrinking/growing)
lvextend -L+8G /dev/vgpool/lvstuff
lvextend -L50GB /dev/vgpool/lvstuff (extend to total of 50gb)
resize2fs /dev/vgpool/lvstuff

ZFS Zpool Replace Failed Drive

So I am using ZFS on my Nas4Free installation (v5000 zfs) and I had a failed drive. Background: 20x 3TB SATA mirror and then striped in the same pool, 2x 3TB hotspares (not sure if these work – maybe more information later), and a raid card level ZIL of 120GB SSDs.

One of the drives in the mirror failed out. Had a technician replace the drive, but forgot to offline it before the drive was replaced. Since this was my first time with a pure ZFS environment (usually I had the raid controller do the heavy lifting and ZFS was just sitting there), I detached the drive. Caused all sorts of issues.

so then I had:

#zpool status -v
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da14 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
da15 ONLINE 0 0 0
da16 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
logs
da1p1 ONLINE 0 0 0
spares
da21 AVAIL
da22 AVAIL

We can see mirror-5 was just a single stripe disk. Awesome.

After adding in the drive and formatting via the GUI for ZFS, I then ran:
zpool attach zfs /dev/da14 /dev/da13
zpool attach YOURPOOLNAME /dev/YOURDEVICETHATSWORKING /dev/YOURNEWDEVICE

# zpool status
pool: zfs
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Aug 19 13:51:48 2014
3.23G scanned out of 1.41T at 276M/s, 1h29m to go
322M resilvered, 0.22% done

It actually finished in 40 minutes with no errors. Resilvering is a type of scrub, so a scrub cannot be running at the same time.

Then, just to be sure, I ran the scrub to verify data integrity:
zpool scrub zfs
zpool scrub YOURPOOLNAME

To Stop a scrub
zpool scrub -s YOURPOOLNAME

Verify status with (this checks all known data to be valid):
zpool status -v

# zpool status
pool: zfs
state: ONLINE
scan: scrub in progress since Tue Aug 19 17:06:45 2014
22.8G scanned out of 1.41T at 614M/s, 0h39m to go
0 repaired, 1.57% done

Ramblings Of An IT Person