I’ll probably break these out, but for now this is the title and location.

We’re finally hiring a linux administrator (13 years experience) to replace me and my “hobby”. Good thing is he’s way better than I can hope to be. Bad thing is he doesn’t start for another couple weeks and I have a lot of Linux-y stuff going on.

So here are some helpful commands:
What Linux Distribution Are You Using?
cat /etc/*-release
OR
lsb_release -a

What Kernel Version Am I Running?
uname -a

What Processors Am I Using?
cat /proc/cpuinfo

What Hardware Specs Do I Have? (Motherboard Model, BIOS revision, etc)
dmidecode
OR, if installed
hwinfo

How Do I Setup No-Password-Needed SSH?
ssh-keygen
enter
enter
ssh-copy-id username@systemname
userpassword

ZFS Replace A Drive
zpool offline poolname /dev/daX
zpool replace poolname /dev/daX /dev/daY
zpool status poolname
After Rebuild
zpool detach poolname /dev/daX

LVM – Create Physical, Volume, and Logical
pvcreate /dev/sdb1
vgcreate vgpool /dev/sdb1
lvcreate -L 3G -n lvstuff vgpool

LVM – Display Current Status
pvdisplay

LVM – Add A New Disk
fdisk /dev/daX
n, p, 1, t, 8e, w
pvcreate /dev/daX1

LVM – Extend LVM Pool To New Disk
vgextend vgpool /dev/daX1

LVM – Resize File System (required for shrinking/growing)
lvextend -L+8G /dev/vgpool/lvstuff
lvextend -L50GB /dev/vgpool/lvstuff (extend to total of 50gb)
resize2fs /dev/vgpool/lvstuff

So I am using ZFS on my Nas4Free installation (v5000 zfs) and I had a failed drive. Background: 20x 3TB SATA mirror and then striped in the same pool, 2x 3TB hotspares (not sure if these work – maybe more information later), and a raid card level ZIL of 120GB SSDs.

One of the drives in the mirror failed out. Had a technician replace the drive, but forgot to offline it before the drive was replaced. Since this was my first time with a pure ZFS environment (usually I had the raid controller do the heavy lifting and ZFS was just sitting there), I detached the drive. Caused all sorts of issues.

so then I had:

#zpool status -v
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da14 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
da15 ONLINE 0 0 0
da16 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
logs
da1p1 ONLINE 0 0 0
spares
da21 AVAIL
da22 AVAIL

We can see mirror-5 was just a single stripe disk. Awesome.

After adding in the drive and formatting via the GUI for ZFS, I then ran:
zpool attach zfs /dev/da14 /dev/da13
zpool attach YOURPOOLNAME /dev/YOURDEVICETHATSWORKING /dev/YOURNEWDEVICE

# zpool status
pool: zfs
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Aug 19 13:51:48 2014
3.23G scanned out of 1.41T at 276M/s, 1h29m to go
322M resilvered, 0.22% done

It actually finished in 40 minutes with no errors. Resilvering is a type of scrub, so a scrub cannot be running at the same time.

Then, just to be sure, I ran the scrub to verify data integrity:
zpool scrub zfs
zpool scrub YOURPOOLNAME

To Stop a scrub
zpool scrub -s YOURPOOLNAME

Verify status with (this checks all known data to be valid):
zpool status -v

# zpool status
pool: zfs
state: ONLINE
scan: scrub in progress since Tue Aug 19 17:06:45 2014
22.8G scanned out of 1.41T at 614M/s, 0h39m to go
0 repaired, 1.57% done

Our all company meetings are “televised” via a gotowebinar/gotomeeting type setup and are recorded for future reference. The recordings are all in wmv format. The management team wants to allow end users to view these recordings, but they have strict policies in place as to how to secure said videos.

1) the video can only be accessed while on the internal network (either local or via VPN)
2) the video cannot be downloaded, copied, or shared with others

Number 1 is very easy; just host it on a system that’s not available externally. I ended up putting it on our intranet site. Check.
Number 2, however, was a little trickier since if I just put the .wmv up on the site it would be easily downloaded with a right-click save-as.

So I converted this to a flash file. FLVs are so much nicer since it can load up in the browsers of all the end users (not ipad owners.. )
Unfortunately one could still look at the source code and see the flv, right click from there and download. I initially looked at obfuscation of the code and found some good examples, but that doesn’t work entirely well since the browser will still take and show you the file locations.

So then I looked at .htaccess and how it could help.

cd /your/web/directory/and/files
nano .htaccess

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC]
RewriteRule \.(gif|jpg|flv)$ – [F]

apache2ctl restart

I changed the “localhost” portion to “intranet” in my case, but you can really utilize anything above.

Now you get a 403 forbidden error if you attempt to download those types of files.

Perforce, a file repository and versioning system, decided that it wants to check to see if it can email/ldap authenticate every minute. And if it cannot connect to either for any reason, it will create an email with the report:

Unable to connect to SMTP host “mail.yourdomain.tld”!
Will try again in 60 seconds.

Awesome. But after an outage of 8 hours due to power issues, we had a queue of 480 emails ready to send. The SMTP relay didn’t like it and was throttling – which gave me time to remove the queue.

CentOS

postsuper -d ALL

Also, if you’d like to edit the AD_auth.pl file, it’s located here:
/p4/common/bin/triggers/AD_auth.pl
At least on our default install.

I’ve always used speedtest.net or speedtest.comcast.net etc to test the speed of my internet connection. Unfortunately this requires a browser that supports java/flash/html5/images etc. What about my linux based servers without a GUI?

Author Sivel made speedtest.cli in python. https://github.com/sivel/speedtest-cli

wget -O speedtest-cli https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
chmod +x speedtest-cli
./speedtest-cli

So we moved a RHEL6.x server from one colocation to another. Unfortunately since RHEL 6, Redhat decided to include network manager by default. This makes for some fun times when trying to manually change IP information.

Another unfortunate was that the uninstaller of the system failed to document where each ethernet cable was connected. We needed 2x LAN LACP, 1x backup, 1x IPMI. The IPMI was easy, but this server had 4 onboard gig ports and 2 pcie riser gig ports to contend with. And both sets were BCM drivers – Broadcom.

Network manager was less than helpful in telling me which ports to plug into. NM was showing eth4 (configured), eth5, eth6, eth7 (configured), eth8, eth9. WTF.

I needed to find out which ethernet devices were active now that I plugged them in.

ip link show
Showed me all of the interfaces, physical addresses, and if it was up or down (connect)

From there I learned that eth9 was now LAN and eth7 was backup – so I could edit the network files to change those.

NM, however, had different plans. Since the eth4 was still configured with the old IP range, routing wasn’t happy when I tried to access said range over the MPLS.
ifdown eth4

Error: Device ‘eth4’ (/org/freedesktop/NetworkManager/Devices/4) disconnecting failed: This device is not active.

F YOU!

ip link set dev eth4 down
or
ifconfig eth4 down
Worked much better.

Running rancid for all of our cisco gear is pretty nice, but we’re acquiring more and more non-cisco (dell/F10) switches for our iSCSI 10Gb and 40Gb needs. Price is a major contributor to that shift.

Anyway, we needed to add Dell PowerConnect 8132 and 8132F switches to the monitor.

This is geared towards the Debian/Ubuntu install I did in an earlier post. Files are from GIT (https://github.com/dotwaffle/rancid-git/tree/master/bin)

Download the required bin files
cd /var/lib/rancid/bin
wget http://it.thelibrarie.com/utilities/dlogin
wget http://it.thelibrarie.com/utilities/drancid
chmod +x dlogin
chmod +x drancid

Edit the Rancid manufacturer file
nano rancid-fe
Under %vendortable add the Dell brand
'dell' => 'drancid',
Save and Exit

Open your router.db file
nano /var/lib/rancid/IMO/router.db

Set your switch in the database
your.dell.switch:dell:up
Save and Exit

Either run rancid or wait until your cron job kicks off
sudo -u rancid -H /usr/bin/rancid-run

****EDIT 4/2/14****
So I ran the dlogin/drancid against one of my Dell switches and was receiving the following error in my /var/log/rancid/newestfilehere:

exec failed router manufacturer dell: No such file or directory

Dang, that’s no good.

cd /var/lib/rancid/bin
./dlogin IPADDRESS_OR_NAME_OF_DEVICE

-bash: ./dlogin: @EXPECT_PATH@: bad interpreter: No such file or directory

Well that solves that issue.

nano /var/lib/rancid/bin/dlogin
Add this to the very top:
#!/usr/bin/env expect
Save and quit
./dlogin IPADDRESS_OR_NAME_OF_DEVICE

root@rancid:/var/lib/rancid/bin# ./dlogin fp-dell8132sw1
fp-dell8132sw1
spawn ssh -c 3des -x -l dellsw fp-dell8132sw1
The authenticity of host ‘fp-dell8132sw1 (10.20.209.000)’ can’t be established.
RSA key fingerprint is 0b:05:08:c1:00:e9:00:30:0d:00:00:00:00:00:00:00.
Are you sure you want to continue connecting (yes/no)?
Host fp-dell8132sw1 added to the list of known hosts.
yes
Warning: Permanently added ‘fp-dell8132sw1,10.20.209.000’ (RSA) to the list of known hosts.
dellsw@fp-dell8132sw1’s password:

FP-DELL8132SW1>enable
Password:********

Success!

And make sure you do the same to @PERLV_PATH@: bad interpreter: No such file or directory
nano /var/lib/rancid/bin/drancid
Add the following to the top of the file

#!/usr/bin/perl -w

Save and close
./drancid to verify

EDIT!!!
apparently my hosting company doesn’t like me having perl, python, or other scripts laying around in a directory with no index page. So here’s the tarball of the files:
Tarball of 3 files