Open Source Storage

After pricing storage units to provide the space we needed to backup our systems, offer file shares, and iSCSI targets we found them to all be extremely pricey. We decided to build our own storage.

pod2Pod1pod3

We were looking for an Open source solution for storage that would fall within our budget but be resilient and reliable. We found the backblaze storage pods and saw that the hardware design was open sourced and easily obtained from 45drives.com. We purchased the storage pod, 20 3TB hard-drives and 4 SSDs of caching. We decided to go with FreeBSD 10.0 for the system and used ZFS as the file system. Using ZFS we setup a number of mirror vdevs and striped across them leaving two drive for hot spares. We used the two SSD drives for the ZIL log and two SSD drives for caching.

The following is how to setup your own storage unit after getting the hardware.

STEP 1 (Install FREEbsd on your Storage Pod):

I went with FREEbsd 10.0 on the unit, because it was the newest stable release and allowed for easy setup of ZFS on the root volume.

FreeBSD-3

STEP 2 (Enable SSH for root user *optional can be a security risk):

Open /etc/ssh/sshd_config
# vi /etc/ssh/sshd_config
change the following lines:
From:
#PermitRootLogin no
PasswordAuthentication no

To:
PermitRootLogin yes
PasswordAuthentication yes

Restart ssh:
#service sshd restart

STEP 3 (Create ZFS storage pool):

Check which hard-drives are attached using the camcontrol command:

#camcontrol devlist
checkHDDs

Once you figure out which drives you wish to add create the ZFS pool with run a command similar to the following to create the pool:

# zpool create tank mirror ada0 ada2 mirror ada2 ada3
create a pair for each 2 devices you wish to mirror in the pool.
createpool

Next we can add a pair of SSD drives mirrored to act as the ZIL (this can increase write speeds):
# zpool add tank log mirror ada10 ada11
addLog

Next we can add a few SSDs for read caching to increase speed a bit:
# zpool add tank cache  ada12
# zpool add tank cache ada13
addCache

Next we can add a few drives as hot spares (note: FREEbsd cannot yet autoswap in a hot spare this needs to be done manually or with a script):
# zpool add tank spare ada9
# zpool add tank spare ada23
addSpare

Finally we can check on the newly create ZFS pool:
# zpool status tank
freebsd

STEP 4 (Setup iSCSI):

First we need to complile and build the iSCSI target daemon.
# cd /usr/ports/net/istgt
# make install clean

Next we need to create the configuration file:
# cd /usr/local/etc/istgt/
# vi istgt.conf

The following is a sample config file:

[Global]
Comment “Global section”
NodeBase “iqn.2011-06.net.example.tests” #Set to desired iqn
DiscoveryAuthMethod None
DiscoveryAuthGroup AuthGroup9999

#The following values were not editied
PidFile /var/run/istgt.pid
AuthFile /usr/local/etc/istgt/auth.conf
MediaDirectory /var/istgt
LogFacility “local7”

Timeout 30
NopInInterval 20

MaxSessions 16
MaxConnections 4
MaxR2T 32
MaxOutstandingR2T 16
DefaultTime2Wait 2
DefaultTime2Retain 60
FirstBurstLength 262144
MaxBurstLength 1048576
MaxRecvDataSegmentLength 262144
InitialR2T Yes
ImmediateData Yes
DataPDUInOrder Yes
DataSequenceInOrder Yes
ErrorRecoveryLevel 0

#Unit controller
[UnitControl]
Comment “Internal Logical Unit Controller”
AuthMethod None
AuthGroup AuthGroup10000
Portal UC1 127.0.0.1:3261
Netmask 127.0.0.1

#Portal of the Test Disk
[PortalGroup1]
Comment “SINGLE PORT TEST”
Portal DA1 172.23.28.11:3260  #Set to servers IP Address

#Initiators for the Test Disk
[InitiatorGroup1]
Comment “Initiator Group1”
InitiatorName “ALL” # change to IQN of connecting machine to lock down targets
Netmask 172.23.0.0/16 #Set to servers Netmask

#Test Disk
[LogicalUnit1]
Comment “Test Disk”
TargetName testdisk
TargetAlias “ISCSI Test Disk”

Mapping PortalGroup1 InitiatorGroup1
AuthMethod none
AuthGroup AuthGroup1
UseDigest Auto
UnitType Disk
LUN0 Storage /tank/test 20GB #location of iscsi target and size

[LogicalUnit110]
TargetName testdisk2
Mapping PortalGroup1 InitiatorGroup1
AuthMethod None
AuthGroup AuthGroup1
UnitType Disk
QueueDepth 32
LUN0 Storage /tank/test2 20GB # location of second iscsi target and size

Now edit /etc/rc.conf :
# vi /etc/rc.conf
Add the following line:
istgt_enable=”YES”

Now Create the target files declared in the istgt.conf
# touch /tank/test
# touch /tank/test2

Now start the istgt service:
# /usr/local/etc/rc.d/istgt start

You should now be able to connect to the created iSCSI targets. Security is not yet in place on the target so I would recommend reading up on istgt configuration to lock down your targets.

If you make a change to any of the istgt files you can restart the service for them to take affect with the following command:
# /usr/local/etc/rc.d/istgt restart

 

Leave a Reply

Your email address will not be published. Required fields are marked *