Linux

Linux Fun

Leave a comment

I’ll probably break these out, but for now this is the title and location.

We’re finally hiring a linux administrator (13 years experience) to replace me and my “hobby”. Good thing is he’s way better than I can hope to be. Bad thing is he doesn’t start for another couple weeks and I have a lot of Linux-y stuff going on.

So here are some helpful commands:
What Linux Distribution Are You Using?
cat /etc/*-release
OR
lsb_release -a

What Kernel Version Am I Running?
uname -a

What Processors Am I Using?
cat /proc/cpuinfo

What Hardware Specs Do I Have? (Motherboard Model, BIOS revision, etc)
dmidecode
OR, if installed
hwinfo

How Do I Setup No-Password-Needed SSH?
ssh-keygen
enter
enter
ssh-copy-id username@systemname
userpassword

ZFS Replace A Drive
zpool offline poolname /dev/daX
zpool replace poolname /dev/daX /dev/daY
zpool status poolname
After Rebuild
zpool detach poolname /dev/daX

LVM – Create Physical, Volume, and Logical
pvcreate /dev/sdb1
vgcreate vgpool /dev/sdb1
lvcreate -L 3G -n lvstuff vgpool

LVM – Display Current Status
pvdisplay

LVM – Add A New Disk
fdisk /dev/daX
n, p, 1, t, 8e, w
pvcreate /dev/daX1

LVM – Extend LVM Pool To New Disk
vgextend vgpool /dev/daX1

LVM – Resize File System (required for shrinking/growing)
lvextend -L+8G /dev/vgpool/lvstuff
lvextend -L50GB /dev/vgpool/lvstuff (extend to total of 50gb)
resize2fs /dev/vgpool/lvstuff

Linux

ZFS Zpool Replace Failed Drive

Leave a comment

So I am using ZFS on my Nas4Free installation (v5000 zfs) and I had a failed drive. Background: 20x 3TB SATA mirror and then striped in the same pool, 2x 3TB hotspares (not sure if these work – maybe more information later), and a raid card level ZIL of 120GB SSDs.

One of the drives in the mirror failed out. Had a technician replace the drive, but forgot to offline it before the drive was replaced. Since this was my first time with a pure ZFS environment (usually I had the raid controller do the heavy lifting and ZFS was just sitting there), I detached the drive. Caused all sorts of issues.

so then I had:

#zpool status -v
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da14 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
da15 ONLINE 0 0 0
da16 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
logs
da1p1 ONLINE 0 0 0
spares
da21 AVAIL
da22 AVAIL

We can see mirror-5 was just a single stripe disk. Awesome.

After adding in the drive and formatting via the GUI for ZFS, I then ran:
zpool attach zfs /dev/da14 /dev/da13
zpool attach YOURPOOLNAME /dev/YOURDEVICETHATSWORKING /dev/YOURNEWDEVICE

# zpool status
pool: zfs
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Aug 19 13:51:48 2014
3.23G scanned out of 1.41T at 276M/s, 1h29m to go
322M resilvered, 0.22% done

It actually finished in 40 minutes with no errors. Resilvering is a type of scrub, so a scrub cannot be running at the same time.

Then, just to be sure, I ran the scrub to verify data integrity:
zpool scrub zfs
zpool scrub YOURPOOLNAME

To Stop a scrub
zpool scrub -s YOURPOOLNAME

Verify status with (this checks all known data to be valid):
zpool status -v

# zpool status
pool: zfs
state: ONLINE
scan: scrub in progress since Tue Aug 19 17:06:45 2014
22.8G scanned out of 1.41T at 614M/s, 0h39m to go
0 repaired, 1.57% done

Linux

Apache Forbid Direct Access

Leave a comment

Our all company meetings are “televised” via a gotowebinar/gotomeeting type setup and are recorded for future reference. The recordings are all in wmv format. The management team wants to allow end users to view these recordings, but they have strict policies in place as to how to secure said videos.

1) the video can only be accessed while on the internal network (either local or via VPN)
2) the video cannot be downloaded, copied, or shared with others

Number 1 is very easy; just host it on a system that’s not available externally. I ended up putting it on our intranet site. Check.
Number 2, however, was a little trickier since if I just put the .wmv up on the site it would be easily downloaded with a right-click save-as.

So I converted this to a flash file. FLVs are so much nicer since it can load up in the browsers of all the end users (not ipad owners.. )
Unfortunately one could still look at the source code and see the flv, right click from there and download. I initially looked at obfuscation of the code and found some good examples, but that doesn’t work entirely well since the browser will still take and show you the file locations.

So then I looked at .htaccess and how it could help.

cd /your/web/directory/and/files
nano .htaccess

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC]
RewriteRule \.(gif|jpg|flv)$ – [F]

apache2ctl restart

I changed the “localhost” portion to “intranet” in my case, but you can really utilize anything above.

Now you get a 403 forbidden error if you attempt to download those types of files.

Miscellaneous

Cisco Callmanager CUCM LDAP Logs

Leave a comment

So we had a working LDAP lookup system for our callmanager 8.x system up until very recently. What changed? We moved colocations and decided to decommission a few of the older domain controllers. No big deal, just point anything that had LDAP lookups to new DCs. Right?

So we noticed the callmanager was not populating any new employees. Hard to assign phones etc to them if that’s the case.

Connect
SSH to your CUCM box (our cucmadmin account was necessary)
I used putty.

List all logs, Take note of dates
file list activelog cm/trace/dirsync/log4j/ det date
The newest file should be on the bottom, looking like “dirsync0007.log” or similar

Open the log and see the errors
file tail activelog cm/trace/dirsync/log4j/dirsync00007.log
This will view the bottom part of the log (newest) live, so run a LDAP resync from the web interface of the CUCM and see results.
Unfortunately I had to wait the 10 minute timeout period to notice that it was looking at the old LDAP server despite the changes.

If I find out why it was still pointing to the old system I will update this post accordingly.

Computer Hardware

Supermicro X9SRW-F CPU Temperature Issues

Leave a comment

So I was not having a good time with this supermicro box – it’s a 2U with 8 SATA drives (3T each), areca raid controller, 4 core intel, 8gb ram. Not much. It ran wonderfully as a FreeNAS/NAS4Free box for the last 36 months and was just no being repurposed as an Oracle Playground (ie development/POC).

Installed CentOS 6.4 on it and gave it to the developer. Full building power outage later and it wasn’t booting backup. Turns out that my 8 drive RAID10 had some issues; 1 of the drives in a mirror died (click of death) and the other drive had corrupt data blocks. Awesome. Finding some working drives later, I rebuilt it again.

Now it was Fan full throttle, fan not full throttle, random fan issues. The logs stated something like CPU Temperature assertion, CPU Temperature desertion. Overheating? The room temperature is a balmy 72F. So I put the fans on Heavy I/O fan mode. This seemed to work for a few hours but then we were back to throttle up/down again.

I upgraded the IPMI firmware from 1.2.x to 3.0.x. I rebooted the machine. I even cold booted. I then updated the BIOS (download from supermicro, use Rufus to create a new bootable USB device, copy files, run ami.bat BIOSNAME.FILE). Still no dice.

Then I read what user “SpeedKills” posted on a forum board; simply RESET the IPMI to factory defaults.

Did that and no more fan issues.

Microsoft

Clear MSSQL Log Files

2 Comments

So first of all I’m not a SQL expert. Secondly I don’t recommend performing this on a production system. Third, this actually won’t happen on a production system as long as you’re performing backups – and your backup solution supports MSSQL – and I hope if it *is* a production system that you are actually backing it up.

Open the SQL Management Studio
Connect using the windows authentication piece, OR your SA password

Run the following in the query window
DBCC sqlperf(logspace)

This will show you the name of the database and associated size of log files.

Assuming that the INSERTNAMEHERE database is the one eating through the space
USE INSERTNAMEHERE;
GO
ALTER DATABASE INSERTNAMEHERE
SET RECOVERY SIMPLE;
GO
DBCC SHRINKFILE (INSERTNAMEHERE, 1);
GO
ALTER DATABASE INSERTNAMEHERE
SET RECOVERY FULL;
GO

You can then run the logspace again and verify
DBCC sqlperf(logspace)

This all worked nicely on my development system that doesn’t require/have backups in place.

Microsoft

Delete Exchange Users Old Mail

2 Comments

I was tasked with deleting a service account users old mail. Our CRM Mailbox user had 29.5GB of old mail floating around that needed to be purged. Adding access to the account was a no-no (and would probably crash out Outlook for a while) due to company policies. Powershell to the rescue?

I should probably mention that this is on Exchange 2010 in a DAG (2 servers) with SP3 hotfix 4 applied. Yes, I also know hotfix 5 is available.

I attempted to run the following script:
Search-Mailbox -Identity SOMEUSERNAME -SearchQuery '(Received:01/01/2012..12/31/2012)' -Deletecontent -Force

And I was told that the “search-mailbox” command was not found. Great. Verified that I was using the Exchange Management Powershell instead of just Windows PS. True story.

After a lot of searching I realized that our former Exchange Admin wasn’t very good at PowerShell. The Admin Tasks Management Powershell add-in was not available.

Check Available PowerShell Snap-ins
Get-PSSnapin
Notice Exchange Server Admin Tasks is not available.

Verify that the plugin is available
Get-PSSnapin -registered

Add the snap-in
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010

I ended up closing the powershell window and then re-opening it. My search mailbox and delete content commands now work!

Other commands I did:
Added my user to the Discovery Management role in the ECP (https://yourmailserver.tld/ecp)
Created two new groups to allow the import and export of mailboxes, and then the exchange support and diagnostics:
New-RoleGroup -Name "Exchange Mailbox Import Export" -Roles "Mailbox Import Export" -Members "domain\username" -DisplayName "Exchange Mailbox Import Export" -Description "This group handles permissions to the import and export commands in powershell."
New-RoleGroup -Name "Exchange Support and Diagnostics" -Roles "Support Diagnostics" -Members "domain\username" -DisplayName "Exchange Support Diagnostics" -Description "This group handles permissions to the support diagnostics cmdlets in exchange."

Ramblings Of An IT Person