Running 9.2.0.1 – Shigawire (revision 972) on some supermicro server with an Intel Xeon.

Noticed that I wasn’t receiving my weekly status emails so I finally got around to checking it out (system is running as a backup of a backup for my personal files, so not the end of the world if it doesn’t work right).

Logged in via the web interface and didn’t notice anything really going on except that out of the 4 available cores on the processor 2 were maxing out at 100%. Process listing showed smartctl was the culprit. Went into the ZFS settings (forgot no smart settings in there), and then tried to load up the Disk > Management area. No go – spinning and spinning.

Logged in using SSH to restart the web services
/etc/rc.d/lighttpd restart

Back in business, but then the Disk > Management area was once again crashing it out.

Issued a kill -s HUP thePIDhere. They came right back.

Rebooted the server. Smartctl was still coming up eating all of my available CPU time for reports.

I ended up renaming the smartctl bin file and then killing the processes.
It’s located /usr/local/sbin, then mv smartctl to smartctl.old or something

From there I could load the Disk > Management page again and disable smart monitoring.

And yes, the disks have all checked OK.

I’ll probably break these out, but for now this is the title and location.

We’re finally hiring a linux administrator (13 years experience) to replace me and my “hobby”. Good thing is he’s way better than I can hope to be. Bad thing is he doesn’t start for another couple weeks and I have a lot of Linux-y stuff going on.

So here are some helpful commands:
What Linux Distribution Are You Using?
cat /etc/*-release
OR
lsb_release -a

What Kernel Version Am I Running?
uname -a

What Processors Am I Using?
cat /proc/cpuinfo

What Hardware Specs Do I Have? (Motherboard Model, BIOS revision, etc)
dmidecode
OR, if installed
hwinfo

How Do I Setup No-Password-Needed SSH?
ssh-keygen
enter
enter
ssh-copy-id username@systemname
userpassword

ZFS Replace A Drive
zpool offline poolname /dev/daX
zpool replace poolname /dev/daX /dev/daY
zpool status poolname
After Rebuild
zpool detach poolname /dev/daX

LVM – Create Physical, Volume, and Logical
pvcreate /dev/sdb1
vgcreate vgpool /dev/sdb1
lvcreate -L 3G -n lvstuff vgpool

LVM – Display Current Status
pvdisplay

LVM – Add A New Disk
fdisk /dev/daX
n, p, 1, t, 8e, w
pvcreate /dev/daX1

LVM – Extend LVM Pool To New Disk
vgextend vgpool /dev/daX1

LVM – Resize File System (required for shrinking/growing)
lvextend -L+8G /dev/vgpool/lvstuff
lvextend -L50GB /dev/vgpool/lvstuff (extend to total of 50gb)
resize2fs /dev/vgpool/lvstuff

So I am using ZFS on my Nas4Free installation (v5000 zfs) and I had a failed drive. Background: 20x 3TB SATA mirror and then striped in the same pool, 2x 3TB hotspares (not sure if these work – maybe more information later), and a raid card level ZIL of 120GB SSDs.

One of the drives in the mirror failed out. Had a technician replace the drive, but forgot to offline it before the drive was replaced. Since this was my first time with a pure ZFS environment (usually I had the raid controller do the heavy lifting and ZFS was just sitting there), I detached the drive. Caused all sorts of issues.

so then I had:

#zpool status -v
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da14 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
da15 ONLINE 0 0 0
da16 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
logs
da1p1 ONLINE 0 0 0
spares
da21 AVAIL
da22 AVAIL

We can see mirror-5 was just a single stripe disk. Awesome.

After adding in the drive and formatting via the GUI for ZFS, I then ran:
zpool attach zfs /dev/da14 /dev/da13
zpool attach YOURPOOLNAME /dev/YOURDEVICETHATSWORKING /dev/YOURNEWDEVICE

# zpool status
pool: zfs
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Aug 19 13:51:48 2014
3.23G scanned out of 1.41T at 276M/s, 1h29m to go
322M resilvered, 0.22% done

It actually finished in 40 minutes with no errors. Resilvering is a type of scrub, so a scrub cannot be running at the same time.

Then, just to be sure, I ran the scrub to verify data integrity:
zpool scrub zfs
zpool scrub YOURPOOLNAME

To Stop a scrub
zpool scrub -s YOURPOOLNAME

Verify status with (this checks all known data to be valid):
zpool status -v

# zpool status
pool: zfs
state: ONLINE
scan: scrub in progress since Tue Aug 19 17:06:45 2014
22.8G scanned out of 1.41T at 614M/s, 0h39m to go
0 repaired, 1.57% done

Our all company meetings are “televised” via a gotowebinar/gotomeeting type setup and are recorded for future reference. The recordings are all in wmv format. The management team wants to allow end users to view these recordings, but they have strict policies in place as to how to secure said videos.

1) the video can only be accessed while on the internal network (either local or via VPN)
2) the video cannot be downloaded, copied, or shared with others

Number 1 is very easy; just host it on a system that’s not available externally. I ended up putting it on our intranet site. Check.
Number 2, however, was a little trickier since if I just put the .wmv up on the site it would be easily downloaded with a right-click save-as.

So I converted this to a flash file. FLVs are so much nicer since it can load up in the browsers of all the end users (not ipad owners.. )
Unfortunately one could still look at the source code and see the flv, right click from there and download. I initially looked at obfuscation of the code and found some good examples, but that doesn’t work entirely well since the browser will still take and show you the file locations.

So then I looked at .htaccess and how it could help.

cd /your/web/directory/and/files
nano .htaccess

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC]
RewriteRule \.(gif|jpg|flv)$ – [F]

apache2ctl restart

I changed the “localhost” portion to “intranet” in my case, but you can really utilize anything above.

Now you get a 403 forbidden error if you attempt to download those types of files.

Perforce, a file repository and versioning system, decided that it wants to check to see if it can email/ldap authenticate every minute. And if it cannot connect to either for any reason, it will create an email with the report:

Unable to connect to SMTP host “mail.yourdomain.tld”!
Will try again in 60 seconds.

Awesome. But after an outage of 8 hours due to power issues, we had a queue of 480 emails ready to send. The SMTP relay didn’t like it and was throttling – which gave me time to remove the queue.

CentOS

postsuper -d ALL

Also, if you’d like to edit the AD_auth.pl file, it’s located here:
/p4/common/bin/triggers/AD_auth.pl
At least on our default install.

I’ve always used speedtest.net or speedtest.comcast.net etc to test the speed of my internet connection. Unfortunately this requires a browser that supports java/flash/html5/images etc. What about my linux based servers without a GUI?

Author Sivel made speedtest.cli in python. https://github.com/sivel/speedtest-cli

wget -O speedtest-cli https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
chmod +x speedtest-cli
./speedtest-cli

So we moved a RHEL6.x server from one colocation to another. Unfortunately since RHEL 6, Redhat decided to include network manager by default. This makes for some fun times when trying to manually change IP information.

Another unfortunate was that the uninstaller of the system failed to document where each ethernet cable was connected. We needed 2x LAN LACP, 1x backup, 1x IPMI. The IPMI was easy, but this server had 4 onboard gig ports and 2 pcie riser gig ports to contend with. And both sets were BCM drivers – Broadcom.

Network manager was less than helpful in telling me which ports to plug into. NM was showing eth4 (configured), eth5, eth6, eth7 (configured), eth8, eth9. WTF.

I needed to find out which ethernet devices were active now that I plugged them in.

ip link show
Showed me all of the interfaces, physical addresses, and if it was up or down (connect)

From there I learned that eth9 was now LAN and eth7 was backup – so I could edit the network files to change those.

NM, however, had different plans. Since the eth4 was still configured with the old IP range, routing wasn’t happy when I tried to access said range over the MPLS.
ifdown eth4

Error: Device ‘eth4’ (/org/freedesktop/NetworkManager/Devices/4) disconnecting failed: This device is not active.

F YOU!

ip link set dev eth4 down
or
ifconfig eth4 down
Worked much better.