Debian Buster DRBD ZFS Storage Cluster


After using systems like Nexenta 3 and 4 and Compuverde (both not linked on purpose) I have learned that the only way to do Software Defined Storage for sure is to go for an ALL OPEN SOURCE solution. And what better platform for that to use than a Long Term Support Debian release. It just so happens that Debian Buster aka Debian 10 was recently released so I am going to use that.

High level overview

The cluster I am going to build has to have the following features:

  • High Availability (possibility to get 99,999% uptime)
  • Shared Nothing Resource (no shared SAS or anything like that, only a shared network connection)
  • Reasonably performant (40K IOPS, 800Mbit throughput)
  • No silent dataloss
  • Reasonably priced
  • Fully open source
  • Easy to maintain
  • Possibility to make snapshots
  • Possibility to do a-synchronus (long haul) replication
  • Easy to troubleshoot
  • Easy to upgrade

For this I want to use (for a small scale setup) a 2 node setup consiting of:

  • Asrock A300
  • AMD Ryzen 5 2600G
  • 32GB of RAM each (2666Mhz or 3200Mhz)
  • 256GB NVMe for OS (Intel 760p)
  • 2TB NVMe for DRBD+ZFS (Intel 660p)
  • Single 1Gbit/s network connection (optional additional USB3.0 Gbit network connection for performance/optimization)

Since Debian Buster has some newest (newer) versions of Pacemaker and DRBD not everything seems to be working right out of the box so I am documenting here what I am doing starting out with a 2 VM setup consisting of:

  • 2 vCPU
  • 8GB of RAM each
  • 16GB vDisk for OS
  • 100GB vDISK for DRBD

Let's build it..

So 2 vm's:

Install (in a nutshell):

  • debian10 amd64 netinstall
  • non-graphical
  • region Netherlands
  • rootdisk LVM all in one partition
  • no desktop, just SSH server and standard system utilities
  • reboot

After reboot:

  • Add 'contrib non-free' to all lines in /etc/apt/sources.list
  • apt update
  • apt install sudo postfix pcs drbd-utils zfs-dkms
  • do it twice since spl has to be installed before zfs can be built
  • reboot in between
  • run 'modprode zfs' in between so that zfsutils-linux and zfs-zed can also be installed

Now we create the logical volume on which we place the drbd:

  • pvcreate /dev/sdb
  • vgcreate vgroup1 /dev/sdb
  • lvcreate --extents 100%FREE -n lovo1 vgroup1

Then we move /etc/drbd.d/global_common.conf to /etc/drbd.d/global_common.conf.bak and put set protocol to C in /etc/drbd.d/global_common.conf

And we create the drbd res file.

Then we bring up the drbd-device on both nodes:

  • drbdadm create-md drbd1
  • drbdadm up drbd1

And on debstor1 we do:

  • drbdadm primary --force drbd1

This will start the syncing process between the two nodes.