CouchSofa.org - Ob Stadt ob Dorf, was du brauchst ist couchsofa.org

If you have a lot of data to store it gets inconvenient to make backups and you might switch to a RAID.

My server had 4 1TB drives in a software RAID 5, so it could store 3TB of data. But some day came the time when even 3TB were full and I wanted to expand the raid.

Now some problems appeared:

1. the system partition was a RAID 1 on the same drives
2. the RAID partition was encrypted
3. so where do you store 3TB?

 

So I could try to grow the RAID but that would still leave me with an encrypted partition to grow. Well it turns out that it is very hard to find documentation on growing an encrypted partition, if it is even possible.
This problem lead me to an interesting setup.
You could encrypt every drive and than add it to the RAID, since growing a software RAID is rather easy and since the RAID sits upon the encryption layer the file system shouldn't 'know' of the encryption.

1. create backup of the data on sda1, sdb1 and sdc1 on sdd1, sde1 and some external drives
2. encrypt sda1, sdb1 and sdc1
   

   cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sda1
   cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sdb1
   cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sdc1

3. open crypto devices and create the RAID
   

   cryptsetup luksOpen /dev/sda1 crypt-1
   cryptsetup luksOpen /dev/sdb1 crypt-2
   cryptsetup luksOpen /dev/sdc1 crypt-3
   
   mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/mapper/crypt-1 /dev/mapper/crypt-2 /dev/mapper/crypt-3
   
   mkfs.ext3 /dev/md0
   
   mount -t ext3 /dev/md0 /mnt/raid

4. now copy the backups from sdd1 and sde1 on the RAID
5. encrypt sdd1 and sde1
   

   cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sdd1
   cryptsetup -c aes-cbc-essiv:sha256 luksFormat /dev/sde1

6. open the devices
   

   cryptsetup luksOpen /dev/sdd1 crypt-4
   cryptsetup luksOpen /dev/sde1 crypt-5

7. grow the RAID
   

   mdadm --add /dev/md0 /dev/mapper/crypt-4
   mdadm --add /dev/md0 /dev/mapper/crypt-5
   
   mdadm --grow /dev/md0 --raid-devices=5

8. grow the file system
   

   e2fsck -f /dev/md0
   resize2fs /dev/md0

Now the RAID could grow anytime a new drive is added (and encrypted).
However Ubuntu seems to have some problems with this setup. I think it tries to close the crypto devices and then stop the RAID. Now the RAID will think that it lost all devices and crash (data on the crypto disks may be harmed).
It is the same but vise versa on boot.

This problem is easy to solve when running a server (or a system you do not reboot frequently) by manually stopping the RAID and closing the crypto disks.