Deploying LINSTOR storage for Proxmox

5 min readSep 12, 2018

Few time ago LINBIT released their new solution LINSTOR which is providing orchestration tool for manage multiple DRBD-arrays.

For example you can have few nodes, each one will have own LVM or ZFS pool, LINSTOR will automatically create new volumes there and replicate or distribute them using DRBD protocol.

LINSTOR supports thin-provisioning, snapshots and many other interesting things.

This solution is good suitable for virtual machines and containers.

LINSTOR Satellites

Presume you already have Proxmox cluster, I have three nodes pve1, pve2 and pve3.

So first step is install drbd-utils and build kernel module on every node.

Add linstor repo and install kernel-headers all needed packages from it:

wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - 
echo "deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \
  > /etc/apt/sources.list.d/linbit.listapt-get update
apt-get -y install pve-headers
apt-get -y install drbd-dkms drbdtop

After installing, let’s check your kernel module version:

modprobe drbd
cat /proc/drbd

If you see 8 version, then something has going wrong and you’ve loaded in-tree kernel module, check dkms status for more detail about dkms module building. For linstor you should use 9 version of drbd.

On every node install linstor-proxmox and linstor-satellite packages:

apt-get -y install linstor-proxmox linstor-satellite
systemctl start linstor-satellite.service
systemctl enable linstor-satellite.service

LINSTOR Controller

We will deploy controller using LXC-container.

Download debian template:

wget http://download.proxmox.com/images/system/debian-9.0-standard_9.3-1_amd64.tar.gz -P /var/lib/vz/template/cache/

Create controller container:

pct create 100 local:vztmpl/debian-9.0-standard_9.3-1_amd64.tar.gz \
  --hostname=linstor-controller \
  --net0=name=eth0,bridge=vmbr0,gw=10.1.0.1,ip=10.1.0.123/16

Run container, and exec into it:

pct start 100
pct exec 100 bash

Install updates

apt-get update 
apt-get -y upgrade

Add linstor repo and install linstor-controller and linstor-client:

wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - 
echo "deb http://packages.linbit.com/proxmox/ proxmox-5 drbd-9.0" \
  > /etc/apt/sources.list.d/linbit.list
apt-get update && apt-get install -y linstor-controller linstor-client
systemctl start linstor-controller.service
systemctl enable linstor-controller.service

Linstor requires configured locale. Configure locale:

sed -i '/en_US.UTF-8 UTF-8/ s/^# //' /etc/locale.gen
locale-gen

let’s configure timezone also:

dpkg-reconfigure tzdata

Preparing storage configuration

Create nodes

linstor node create pve1 10.1.0.11
linstor node create pve2 10.1.0.12
linstor node create pve3 10.1.0.13

For every node we will describe second interface which is dedicated for drbd replication:

linstor node interface create pve1 data 10.2.0.11
linstor node interface create pve2 data 10.2.0.12
linstor node interface create pve3 data 10.2.0.13

Example output of linstor node list command:

╭──────────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType  ┊ Addresses                        ┊ State  ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ pve1 ┊ SATELLITE ┊ 10.0.1.11,10.0.2.11:3366 (PLAIN) ┊ Online ┊
┊ pve2 ┊ SATELLITE ┊ 10.0.1.12,10.0.2.12:3366 (PLAIN) ┊ Online ┊
┊ pve3 ┊ SATELLITE ┊ 10.0.1.13,10.0.2.13:3366 (PLAIN) ┊ Online ┊
╰──────────────────────────────────────────────────────────────╯

Create pools:

On every node you should prepare lvm thinpool for drbd:

lvcreate -L 800G --thinpool drbdpool pve

Then you can add them into linstor:

linstor storage-pool create lvmthin pve1 drbdpool pve/drbdpool
linstor storage-pool create lvmthin pve2 drbdpool pve/drbdpool
linstor storage-pool create lvmthin pve3 drbdpool pve/drbdpool

Configure pools to use data interface:

linstor storage-pool set-property pve1 drbdpool PrefNic data
linstor storage-pool set-property pve2 drbdpool PrefNic data
linstor storage-pool set-property pve3 drbdpool PrefNic data

Example output of linstor storage-pool list command:

╭─────────────────────────────────────────────────────────────────
┊ StoragePool ┊ Node ┊ Driver        ┊ PoolName     ┊ FreeCapacity
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
┊ drbdpool    ┊ pve1 ┊ LvmThinDriver ┊ pve/drbdpool ┊ 800 GiB 
┊ drbdpool    ┊ pve2 ┊ LvmThinDriver ┊ pve/drbdpool ┊ 800 GiB 
┊ drbdpool    ┊ pve3 ┊ LvmThinDriver ┊ pve/drbdpool ┊ 800 GiB 
╰─────────────────────────────────────────────────────────────────

Now we can add our new storage into proxmox’s storage config:

cat >> /etc/pve/storage.cfg <<EOFdrbd: drbdpool
        content rootdir,images
        controller 10.1.0.123
        controllervm 100
        redundancy 3
EOF

Configuring HA for the controller

Now we will move our linstor container to the linstor storage.

First we will prepare virtual drive for it:

pct exec 100 bash
linstor resource-definition create vm-100-disk-0
linstor volume-definition create vm-100-disk-0 4G
linstor resource create vm-100-disk-0 --auto-place 3 -s drbdpool

Example linstor resource list output:

╭──────────────────────────────────────────╮
┊ ResourceName    ┊ Node ┊ Port ┊    State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-0   ┊ pve1 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-0   ┊ pve2 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-0   ┊ pve3 ┊ 7000 ┊ UpToDate ┊
╰──────────────────────────────────────────╯

By default linstor-satellite will remove configuration for all resources during service restart, then it will wait for new configuration from linstor-controller, but if we have linstor-controller on the same place than data, we should add it’s resource to preserve with --keep-res=vm-100 option, on all linstor-satellites:

echo -e "[Service]\nExecStart=\nExecStart=/usr/share/linstor-server/bin/Satellite --logs=/var/log/linstor-satellite --config-directory=/etc/linstor --keep-res=vm-100" | SYSTEMD_EDITOR=tee systemctl edit linstor-satellite.service

This action should prevent removing vm-100-disk-0 resource during service restart.

Now we will shutdown linstor-controller container and move all data from local drive to drbd drive:

pct shutdown 100
dd if=/var/lib/vz/images/100/vm-100-disk-0.raw of=/dev/drbd/by-res/vm-100-disk-0/0 bs=8M status=progress
e2fsck -f /dev/drbd/by-res/vm-100-disk-0/0 
resize2fs /dev/drbd/by-res/vm-100-disk-0/0

Update container config:

sed -i '/^rootfs:/d' /etc/pve/lxc/100.conf
echo 'rootfs: drbdpool:vm-100-disk-0,size=4G' >> /etc/pve/lxc/100.conf

Start container and if everything ok, remove old local drive:

pct start 100
rm -f /var/lib/vz/images/100/vm-100-disk-0.raw

Now we only need to add our container to the proxmox ha-manager:

ha-manager add ct:100 --max_relocate=3 --max_restart=3

PROFIT

Perfomance and tuning

Currently you have to disable user security (as there is currently no way to authenticate) for global controller options, see:

https://github.com/LINBIT/linbit-documentation/blob/master/UG9/en/administration-linstor.adoc#disable-user-security

For automatic split-bran resolution add theese options to the controller:

linstor controller drbd-options \
  --after-sb-0pri=discard-zero-changes \
  --after-sb-1pri=discard-secondary \
  --after-sb-2pri=disconnect

For my 10G network I found the next values is most optimal for fast sync:

linstor controller drbd-options \
  --max-buffers=36864 \
  --rcvbuf-size=2097152 \
  --sndbuf-size=1048576linstor controller drbd-options \
  --c-fill-target=10240 \
  --c-max-rate=737280 \
  --c-min-rate=20480 \
  --c-plan-ahead=10