Containers are small clouds, right?

Cyclid is deliberately agnostic when it comes to where & how it creates build hosts; currently it supports Google Cloud, DigitalOcean and something called Mist.

Mist creates LXC containers for build hosts, and was created very early on as an important component of Cyclid because support for containers is important. The only problem is that LXC is rough around the edges; although it seemed easy enough, Mist is not very good and it only ever really worked for Ubuntu Trusty containers. Clearly not ideal.

As it happens, the LXC developers also realised that LXC wasn't ideal, so they created LXD. LXD solves a lot of the issues that LXC has and adds lots of nice features such as proper image management and a REST API. There's even a decent Rubygem on top of the REST API.

So all I have to do is write an LXD Builder plugin for Cyclid and that's the container problem solved, right? That's what I thought...

The Cloud Conundrum

First of all, let's be clear: LXD is an enormous improvement on LXC and I'd use LXD over LXC any day of the week. When it comes to my requirements for Cyclid though, it's still just as tricky to use as LXC.

The main problem is image management. In theory, Cyclid is also agnostic about the operating system, distribution & version that your builds run on. So it can't know in advance what any given job might request; all it can do is pass the request on to whatever is creating the actual instance and see if it can fulfil the request. In practice this is pretty reliable with most cloud providers; they all tend to work in a similar fashion and all tend to offer a broadly similar set of instances to choose from.

With LXD, there is no simple answer. In theory we could have LXD download a template from the Linuxcontainers.org public image server, but there is a major issue with that approach: none of the images there have cloud-init installed. So we can create a container we can't configure.

But wait! There are some templates which do have cloud-init installed...but those are only Ubuntu images. They're also served from a different public image server. They are more reliable than the old LXC templates, so at least we can now create containers other than Ubuntu Trusty, but if we want a container for anything other than Ubuntu we're still out of luck.

Okay, so if Ubuntu provide cloud images, surely other distributions do, too? The page at https://images.linuxcontainers.org/ even tells us that these are not official and that we should use the official images provided by the distribution.

It turns out that most distributions that aren't Ubuntu do provide cloud images; but not images that you can use with LXD. LXD has its own image format and most distributions only provide images in QCOW2 or raw format, for use with virtual machines like Qemu or VirtualBox.

So we can't rely on the public LXD images providing the functionality we need, and we can't use the cloud images that do provide what we need.

Plan C...

The remaining options are unattractive for various reasons:

1. Bootstrap instances without cloud-init. We can run commands directly in the container via. the API, so we could create the users & configure SSH that way. Except the Builder plugin would need to have knowledge of every possible distribution (package names, configuration files etc.) we'd support, which is impractical and inflexible.
2. Provide our own cloud images for popular distributions. Again, theoretically possible, but not something I want to do. I'd have to spend time updating image templates, and I'd have to host them and pay the bandwidth costs.
3. Give up entirely and make it the users problem: if you want to use a distribution with Cyclid on LXD, you have to provide the image(s) with cloud-init enabled. Welcome to the party!

...to Plan Z

Which brings us back to almost where we were at the start: LXD is better than LXC, but we can only use it to reliably create Ubuntu containers. That's probably what's going to happen, at least in the short term, because it's better than nothing (and still better than Mist). But far from ideal.

It really feels like LXD is missing a trick. It's not quite a cloud in a box, but with a little more effort and some attention to the operational aspects, it could be. Right now it seems to be aimed at traditional hands-on systems administration, where every container is lovingly hand crafted from the command line. That may be fine; heck that may really be what the vast majority of users care about, but it's a shame. Because I really, really, do not want to run OpenStack just to be able to reliably create a container...