Home‎ > ‎CIS 192‎ > ‎

Storage Area Networks

Class presentation slides are here.

SAN is an alternative way to use Linux to get access to disks. Instead of accessing shared disks at the file level SAN accesses disks at the block level. This has some advantages and disadvantages over traditional networked file systems. There are very good articles that will help you understand SAN:

Setup LVM
To get the fullest benefit from SAN you want to combine it with LVM. LVM lets you create virtual disks that can be resized at will. That way your SAN can grow and shrink any time you need it to. I've added three disks to my VM. In order to do this procedure you will need to add at least one. Each of my disks are:
  • 16 GB
  • Thin Provisioned
IMPORTANT: Please Thin Provision your disks that way they will only occupy space on demand. 

My disks are /dev/sdb /dev/sdc and /dev/sdd.

1. Create a Physical Volume on each disk

root@filer:~# apt-get install lvm2
root@filer:~# pvcreate /dev/sdb /dev/sdc /dev/sdd
  Physical volume "/dev/sdb" successfully created
  Physical volume "/dev/sdc" successfully created
  Physical volume "/dev/sdd" successfully created

2. Create a new Volume Group and add your new Physical Volumes

root@filer:~# vgcreate SANGroup /dev/sdb /dev/sdc /dev/sdd 
  Volume group "SANGroup" successfully created
3. Create a new Logical Volume that takes all the storage in your newly created Volume Group

root@filer:~# lvcreate -n SANVolume -l 100%VG SANGroup 
  Logical volume "SANVolume" created

If you have done the above procedure correctly you should now see a new disk called /dev/SANGroup/SANVolume. Question: How big is that disk?

Setup AoE Server 
First you must have the tools installed.

root@filer:~# apt-get install vblade

Next you must take the block device you created with LVM and export it using AoE. 

root@filer:~# vblade 1 1 eth0 /dev/SANGroup/SANVolume &
pid 1938: e1.1, 100638720 sectors O_RDWR

Notice that I asked the vblade daemon to run in the background. If you don't do this the shell will not return to the prompt. 

Mount AoE on the Client
Now you are ready to have the client mount the share. Remember ATA over Ethernet is just that: Ethernet (layer 2). It is a non-routable protocol so your client MUST be on the same Ethernet network.

root@client:~# apt-get install aoetools

Now that the tools are installed you must initialize the AoE kernel modules and discover the disks that are available on your network:

root@client:~# modprobe aoe
root@client:~# aoe-interfaces eth0 
root@client:~# aoe-discover
root@client:~# aoe-stat
      e1.1        51.527GB   eth0 up            

If everything worked you now have a new device called /dev/etherd/e1.1. You can use that device just like you would use any other block device. A very important difference between SAN and other forms of file sharing is that the device must be formatted before it can be mounted. It's just a blob of data. 

root@client:~# mke2fs /dev/etherd/e1.1 
mke2fs 1.42 (29-Nov-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3145728 inodes, 12579840 blocks
628992 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
384 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done   

root@client:~# mount /dev/etherd/e1.1 /mnt

Taking your SAN down is the opposite of setup. 

On the client:
root@client:~# umount /dev/etherd/e1.1
root@client:~# rmmod aoe

On the server:
root@filer:~# killall vblade
root@filer:~# lvremove /dev/SANGroup/SANVolume 
Do you really want to remove active logical volume SANVolume? [y/n]: y
  Logical volume "SANVolume" successfully removed
root@filer:~# vgremove SANGroup 
  Volume group "SANGroup" successfully removed
root@filer:~# pvremove /dev/sdb /dev/sdc /dev/sdd
  Labels on physical volume "/dev/sdb" successfully wiped
  Labels on physical volume "/dev/sdc" successfully wiped
  Labels on physical volume "/dev/sdd" successfully wiped

This how-to showed you how to use LVM to bond disks together on the server. This could also be don on the client. The general procedure for that would look like:
  1. Use the vblade command to setup EACH disk with a different drive number
  2. On the client create a Physical Volume for each drive (e.g. /dev/etherd/e1.1)
  3. Add the PVs to a VG and create a LV
  4. Format and mount the LV
Make it work!