We decided to test pulling out a HD and sticking it back in to see how the FreeNAS would react.

Specs:
FreeNAS 0.7.2 (5543)
Areca ARC-1222 w/ Battery
Hotswap ICYDock drives
Hitachi 2TB SATA HD’s (QTY 8 )
RAID 50 (so a stripe of 2x RAID 5’s, 12TB unformatted)

Pulled out the HD, put the HD back in. Drive lights on 7 of the drives turned to orange (means data was being accessed/changed) and the drive we pulled out and put back was green (only power).

Logged into the Areca Web Panel and added the HD as a hot spare. The RAID controller instantly tried to rebuild the array. After 15 minutes the rebuild process was still at 0.0% so something was definitely wrong.

Checked the FreeNAS and HD space was rapidly being used up (about 100MB/sec). Checked “top” to see what the problem was – fsck_ufs was running full tilt.

FreeBSD was checking the HD for errors after a forced reboot earlier in the day. So there was little free time for the RAID controller to rebuild.

Unfortunately the kill PID command was not working. Kill -9 PID also failed. I would have to reboot.

Edit this file to make sure this doesn’t happen again:

vi /etc/rc.conf
i
background_fsck="NO"
Escape
:wq!

Reboot. Now, due to the HD’s being run full tilt, we had to manually reboot the server. It might have worked out OK if we just left it after issuing the reboot command, but I’m not a patient man.