After playing around with ZFS for a while (it’s pretty nice, especially snapshots), I wanted to try non-solaris-type-deduplication. Opendedup was the answer.

I installed on a 2.8GHz Xeon Dell Poweredge 850 with 2GB RAM, and a mirrored array 500GB SATA drives.

Installed 10.04.3 LTS x64

Upgrade the Server
apt-get dist-upgrade
reboot

Install and Download
apt-get install openjdk-6-jre
apt-get install attr
wget http://opendedup.googlecode.com/files/sdfs-1.1.0_amd64.deb
wget http://opendedup.googlecode.com/files/sdfs_fuse_2.8.4.tar.gz
tar zxvf sdfs_fuse_2.8.4.tar.gz
cd sdfs_fuse_2.8.4
dpkg --install *.deb
apt-get install -f
dpkg -i sdfs-1.1.0_amd64.deb
apt-get install -f

Edit
nano /etc/security/limits.conf
Add the following above # End of file:

soft nofile 65535
hard nofile 65535

Create
/sbin/mkfs.sdfs --volume-name=VOLUMENAME --volume-capacity=CAPACITY
So in my case I wanted a 10GB volume for testing:
/sbin/mkfs.sdfs --volume-name=volume1 --volume-capacity=10GB

Mount
mkdir /mnt/volume1
/sbin/mount.sdfs -v VOLUMENAME -m MOUNTPOINT
/sbin/mount.sdfs -v volume1 -m /mnt/volume1

Verify
df -h

sdfs:/etc/sdfs/volume1-volume-cfg.xml:6442
10G 0 10G 0% /mnt/volume1

I then copied about 2GB worth of data. Looks like it parses through every 10 minutes or so – but since it’s the end of the day I’ll be checking in the morning.

There are a few common errors with simple fixes.
1. OutOfMemoryError – This is caused by the size of the DedupStorageEngine memory requirements being
larger than the heap size allocated for the JVM. To fix this edit the mount.sdfs script and increase the
-Xmx2g to something larger (e.g. -Xmx4g).

The files are actually stored in /opt/sdfs/volumes/VOLUMENAME
du -hc
Compare to df -h
df -h

In my example:
df -h
10G 5.4G 4.7G 54% /mnt/volume1
du -hc
3.1G total

So I’m using 5.4GB of space, but it’s only recorded as 3.1GB. Not a bad savings right there.

Next I’ll fill up the 10GB drive I created and see how much it really saves.

So I wanted a second volume shared for vmware using NFS. I ended up installing webmin and doing some of the configuration there for samba/cifs and nfs, but here’s my notes for commands on the opendedup server:

apt-get install nfs-kernel-server
nano /etc/exports
Add /export/volume2 *(insecure,no_subtree_check,rw,nohide,fsid=0)
/etc/init.d/nfs-kernel-server restart

Delete a Volume
If you ever need to delete a volume, make sure that you unmount it first:
umount /mnt/volume1
Then you can remove the data files:
rm -rf /opt/sdfs/volumes/volume1
The log files:
rm /var/log/sdfs/volume1*
And finally the configuration files:
rm /etc/sdfs/volume1*

Show Volume Info
/sbin/sdfscli --volume-info