Run kubernetes inside LXC container

I can tell you how you can run kubernetes master in LXC container, I use Proxmox and it is working really fine, this manual can serve as an alternative way for the classical several masters deployment.

In this case you can have only one master, and still have all the features of multimater.

Why proxmox?

Because proxmox itself is ready solution which provides a lot of things out of box.

With proxmox you have full features, like high availability, migrations, automatic backups, acl, and all of them with simple GUI.

Besides you can deploy simplest configuration of kubernetes with single master, because high availability will be managed by proxmox.

It is easy way for testing and small deployments.

For larger ones you can think about moving etcd to separate containers without shared storage in bottom, like: use only fast local drives and create etcd-cluster.
Nevertheless kubernetes-master can have dedicated ha-container on top of shared storage.

In addition, if you read this article, I presume that in most cases you already have some infrastructure on the proxmox and you want to have a same single management interface for them.

Why LXC?

There is no problem to run kubernetes inside some virtual-machine. But LXC-container provides flexibility which is not available for classic virutal machine.

Basically LXC-containers not provides full isolation from host, and all processes inside containers runs like usual host’s processes, they just using separate namespaces.

This method gives good performance but imposes some limitations in our case.

About these imposes and how to overcome them I will tell in this article.

Configuration

Since by default containers can’t load kernel modules, you need to configure all needed modules to load on hypervisors directly.

We will use overlay driver for docker, that’s only what we need:

echo overlay >> /etc/modules

Then we need to add more privileges to our container for make it possible to run containers inside, add those lines to your container config file:

lxc.apparmor.profile: unconfined
lxc.cap.drop: 
lxc.cgroup.devices.allow: a
lxc.mount.auto: proc:rw sys:rw

From the version v11.0 kubelet requires to have shared mode for the host mounts.

There is dirty hack for achieve that, inside LXC-container run:

echo '#!/bin/sh -e
mount --make-rshared /' > /etc/rc.local

It will run mount --make-rshared / command each new boot.

Then if you plan to use HA-feature, you should know that for now proxmox have one unpleasant bug#1842, that’s can kill your processes forcefully during container migration, which in turn can generate zombie processes and hold your storage.

That’s not good, there is simple solution for that:

sed -i 's/forceStop => 1/forceStop => 0/' /usr/share/perl5/PVE/HA/Resources/PVECT.pm

In addition you can obviously add those options to the your docker:

--storage-driver overlay2
--iptables=false
--ip-masq=false

Copy docker.service from /libto /etc for override:

cp /{lib,etc}/systemd/system/docker.service

Then add them to the ExecStart section.

On this all, after those steps standard kubeadm installation should work without any problems.