Bug #15349

Migrate away from vmdebootstrap (and possibly from Vagrant)

Added by intrigeri 2018-02-27 13:59:18 . Updated 2020-05-15 08:56:59 .

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
2018-02-27
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

We use vmdebootstrap to build the VM image used for building Tails ISO images with Vagrant (source:vagrant/definitions/tails-builder/generate-tails-builder-box.sh). vmdebootstrap barely made it in Buster and might not be in Bullseye.

Sources of inspiration:

And wrt. replacing Vagrant altogether, see:


Files


Subtasks


Related issues

Related to Tails - Feature #7530: Evaluate switching to Docker for an easy, self-contained ISO build system Rejected 2014-07-09
Related to Tails - Feature #7580: Decide mid-term strategy regarding easy, self-contained build environment Resolved 2014-07-12 2014-08-31
Related to Tails - Feature #15864: Make onboarding of new developers easier In Progress 2018-08-30
Related to Tails - Feature #5691: Consider upgrading to current live-build Confirmed
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2018-02-27 14:00:03

  • Assignee set to intrigeri

I’ll do the initial triaging of this task by asking the Debian maintainers when they plan to remove the package from sid.

#2 Updated by intrigeri 2018-02-27 14:08:54

intrigeri wrote:
> […] by asking the Debian maintainers when they plan to remove the package from sid.

Done.

#3 Updated by intrigeri 2018-02-27 14:09:43

  • Description updated

#4 Updated by intrigeri 2018-02-27 14:10:33

#5 Updated by intrigeri 2018-02-27 14:11:23

  • Target version changed from Tails_4.0 to Tails_3.9

Let’s assume the worst case for now.

#6 Updated by intrigeri 2018-02-27 16:06:43

  • Description updated
  • Assignee deleted (intrigeri)
  • Target version changed from Tails_3.9 to Tails_4.0

#7 Updated by intrigeri 2018-07-30 02:56:29

intrigeri wrote:
> Lars Wirzenius tentatively agreed to keep the package in sid a bit longer, e.g. the end of 2019-02 and best/worst case the Buster release

… but later changed his mind: vmdebootstrap going away in September, switch now. I’ve checked with Lars privately and he confirmed that he really does not want to see vmdebootstrap in sid after September.

Let’s check the consequences:

  • time scope: between the time when vmdebootstrap is removed from testing/sid (September) and the time when our build system has switched to something else
  • who? people who satisfy all these criteria:
    • They want to build Tails ISO images.
    • They run testing/sid on their Tails development machine.
    • They have not installed vmdebootstrap before it gets removed from testing/sid.
  • impact: can’t build a Tails ISO

I doubt we want to migrate in a bugfix release and it’s too late to do this in time for 3.9. Most likely, the following major release will be 3.12, scheduled for 2019-01-29, with a freeze date around mid-January, so the deadline to have this work well tested and ready for QA would be sometime at the end of 2018. The impact described above would affect people for ~4 months, during which we can document workarounds (e.g. installing vmdebootstrap from Stretch). This was basically what I had in mind when I’ve negotiated a delay with Lars earlier this year and it still seems acceptable to me.

If we miss the end of 2018 deadline, then most likely the earliest we can do the switch is Tails 4.0 (~mid-2019) and then the problem will have affected people for ~9 months. I think that’s too long, so let’s try to avoid that if we can.

#8 Updated by intrigeri 2018-07-31 11:54:08

  • Description updated

#9 Updated by lamby 2018-08-17 15:15:17

#10 Updated by lamby 2018-08-24 20:36:31

  • Assignee set to lamby

Taking ticket to ensure IRL discussion is definitely captured. (Feel free to take back.)

This work is being targetted for Oct/Nov 2018.

#11 Updated by intrigeri 2018-08-26 06:09:54

  • Target version changed from Tails_4.0 to Tails_3.11

#12 Updated by intrigeri 2018-08-26 06:10:06

  • blocked by deleted (Feature #15334: Core work 2018Q3: Foundations Team)

#13 Updated by intrigeri 2018-08-26 06:10:16

#15 Updated by intrigeri 2018-09-12 07:04:35

  • Estimated time set to 24 h

#16 Updated by intrigeri 2018-12-03 15:22:31

  • Target version changed from Tails_3.11 to Tails_3.12

#17 Updated by intrigeri 2019-01-04 15:04:45

#18 Updated by intrigeri 2019-01-04 15:04:47

  • blocked by deleted (Feature #15506: Core work 2018Q4: Foundations Team)

#20 Updated by lamby 2019-01-07 19:09:29

  • Status changed from Confirmed to In Progress
  • Feature Branch set to feature/15349-migrate-away-from-vmdebootstrap

Spent some time at the IRL sprint on this. feature/15349-migrate-away-from-vmdebootstrap branch. Marking as In Progress.

#21 Updated by intrigeri 2019-01-08 07:27:47

lamby wrote:
> feature/15349-migrate-away-from-vmdebootstrap branch.

In which remote?

#22 Updated by anonym 2019-01-30 11:59:28

  • Target version changed from Tails_3.12 to Tails_3.13

#23 Updated by lamby 2019-03-12 15:30:57

  • Target version changed from Tails_3.13 to Tails_3.14

> lamby: how about we aim at having a PoC at the end of the sprint?

Let’s do this!

#24 Updated by intrigeri 2019-03-20 14:46:35

#25 Updated by intrigeri 2019-03-20 14:46:47

  • blocked by deleted (Feature #15507: Core work 2019Q1: Foundations Team)

#26 Updated by lamby 2019-04-04 17:20:33

I am writing this from the FT sprint. The consensus here that we should pause and change direction or otherwise widen the scope on this ticket for now. To elaborate, this is for three main reasons:

Firstly, the quasi-urgency to replace vmdebootstrap to create our base images was because vmdebootstrap had a Release Critical bug filed against it (in https://bugs.debian.org/910201) during September 2018 by the original maintainer, Lars Wirzenius. Entitled “vmdebootstrap should not be in buster”, here he opines that whilst it “has a bad architecture that makes the software inflexible, difficult to modify, hard to test, and it is not suitable for large number of use cases” he did admit that “a number of people have managed to make use of it”. Thus, I was tasked with finding a replacement as buster or sid lacking vmdebootstrap would make it technically inconvenient and awkward as well as “politically” strange given that we depend so much on Deban.

However, Niels Thykier and Jonathan Carter followed-up to the bug in March 2019 reporting that the removal of vmdebootstrap would affect builds of live images (see https://bugs.debian.org/922826) and, due to this, Niels Thykier added a “buster-ignore” tag in early April 2019, meaning that as the bug is no longer Release Critical with respect to the release of buster. It will now presumbly be part of the upcoming Debian release.

Secondly, it also appears that vmdebootstrap has gained a new — albeit likely temporary — maintainer which has actually resulted in a new upload to Debian unstable (version 1.11-2, see https://tracker.debian.org/news/1036916/accepted-vmdebootstrap-111-2-source-into-unstable/). There is therefore even less exigency to find a replacement project and there is even a potential route to report and fix bugs via this new “upstream” and, possibly, even to propose enhancements.

The last reason is that, given the now-lack of haste required we can spend time later looking at larger solutions to building Tails that do not involve Vagrant. This is because the weak-ish consensus is that nobody is truly happy with using Vagrant especially as it’s the cause of most build problems, and it is at least in my personal experience. A number of (entirely un-evaluated) options were mooted IRL including using lxc or Docker (which was actually proof-of-concepted some time ago, but it may not be in buster due to $reasons). intrigeri elaborated that Vagrant was chosen because at the time no other virtualisation options supported building in a reasonably secure manner; the other solutions at the time required highly-privileged access to the point where the “build system could even power down the host.”

As part of the work on this ticket, I evaluated a number of replacement options, possibly still extant on this ticket’s corresponding feature branch, but I have not re-checked at this time.

Firstly, I spent a few hours looked at debos (https://github.com/go-debos/debos). However, it did not support one of the filesystem image options that we needed so I hit a bit of dead-end in getting an actually-working image. Whilst it seemed like it would be essentially trivial to add (my impression was that the Go code is quite straightforward to hack on and it appears to continue to receive good/regular upstream attention), I concluded that requiring changes that are not in the version in buster itself would essentially leave us in the same place as vmdebootstrap. In other words, requiring that we specify using package [versions] from unstable (or, worse, upstream git!) to build.

I then spent even longer looking at vmdb2 assuming that it would be more-or-less a drop-in replacement to vmdebootstrap. However, I ironically hit almost entirely similar issues regarding a single, missing filesystem configuration option, which was curious given that its aforementioned intention to be a drop-in replacement.

So, after hitting similar issues in both potential replacement projects, I then evaluated not using any helper or build tool whatsoever! To do this, I quickly hacked my local copy of vmdebootstrap to log which shell commands it was actually running and then duplicating them in the build script. I did this on the hypothesis that we aren’t actually doing anything that strange or weird when building; essentially “just” calling debootstrap and copying that into a large enough disk image that then gets consumed by Vagrant. This appeared to work to some degree (although I was missing something at the very end…) and was indeed fairly easy to demonstrate to some degree. However, after looking at the resulting diff etc. I remained unclear that this is long-term sustainable as it essentially replicates one of the above tools without real error checking nor cleanup. For example, the loopback devices created by losetup(8) need some management that is difficult to accomplish in a shell script. Thus we would likely just be adding to the flaky nature of the build system (see also, Vagrant!), rather than making it any simpler or at least at the same level of flakiness.

I did not get around to evaluating FAI (https://fai-project.org/) as I had the preconception and very slight experience in that it is a little more complex so I intended to try it last. Ironically this apparent “complexity” may indicate that it would support all of our requirements, although with a little more of a learning curve.

I did try and research some other things but they are probably not relevant to the current status of this issue. Anyway, I hope this gives an overview and update of this ticket and, unless there are any objections and given the now lack of priority, I will remove myself as an assignee of this ticket. It may be worth renaming this ticket (or perhaps creating a separate one with some kind of blocking relation?) regarding questions about replacing Vagrant itself. After all, replacing this virtualisation solution might imply or even require a specific build tool anyway.

#27 Updated by lamby 2019-04-05 16:37:20

  • Assignee deleted (lamby)

Unassigning as-per new Target version policy (re. https://tails.boum.org/contribute/working_together/roles/foundations_team/#tasks-management)

#28 Updated by intrigeri 2019-04-05 16:38:09

@lamby, happy to mangle the ticket metadata and move on once you’ve clarified where the feature/15349-migrate-away-from-vmdebootstrap can be found (and if it can’t, fine, we’ll do without it).

#29 Updated by lamby 2019-04-05 16:43:16

Sure, @intrigeri. So I can’t actually see somewhere that I’ve pushed it to (no matching remote…) but this is a fairly new installation and it might have been blown away when I removed my GitHub mirror. Oh, no, wait.. I can find a random WIP commit that appears to be using vmdb2…? Attaching now without looking at it whatsoever just so it doesn’t get lost.

#30 Updated by intrigeri 2019-04-08 14:42:59

  • Priority changed from Elevated to Normal
  • Target version changed from Tails_3.14 to Tails_5.0
  • Feature Branch deleted (feature/15349-migrate-away-from-vmdebootstrap)

@lamby, if you can remember or have kept notes: it would be sweet to know which filesystem configuration option are missing in debos and vmdb2, so that if for 5.0 we end up sticking to Vagrant with the need to migrate away from vmdebootstrap again, we don’t need to do the same work from scratch. If you haven’t kept track of this info, forget it.

Setting target version = 5.0 to ensure this is on our radar for Bullseye. Lowering priority for now; we can raise it again when it’s clearer what’s going to happen 1. with vmdebootstrap vs. Debian Live; 2. with our own build system vs. Vagrant.

#31 Updated by intrigeri 2019-04-08 14:43:42

  • related to Feature #7530: Evaluate switching to Docker for an easy, self-contained ISO build system added

#32 Updated by intrigeri 2019-04-08 14:44:04

  • related to Feature #7580: Decide mid-term strategy regarding easy, self-contained build environment added

#33 Updated by intrigeri 2019-04-08 14:46:19

  • Subject changed from Migrate away from vmdebootstrap to Migrate away from vmdebootstrap (and possibly from Vagrant)
  • Description updated

#34 Updated by lamby 2019-04-09 08:41:00

> it would be sweet to know which filesystem configuration option are missing in debos and vmdb2,

Unfortunately not. :( However, on the positive-side I do remember hitting them “fairly” quickly, so it is not something that would require in-depth. I think it was about having lots of free space in the image, so it might not be something that’s insurmountable with a hybrid $TOOL-but-resize-afterwards approach… but don’t quote me too strongly on that.

#35 Updated by intrigeri 2020-02-14 08:55:31

  • Description updated

(Document that vagrant-libvirt is currently unmaintained.)

#36 Updated by intrigeri 2020-05-15 08:25:19

  • related to Feature #15864: Make onboarding of new developers easier added

#37 Updated by intrigeri 2020-05-15 08:36:05

  • related to Feature #5691: Consider upgrading to current live-build added

#38 Updated by intrigeri 2020-05-15 08:56:59

> Related to Feature Feature #5691: Consider upgrading to current live-build added

I marked these issues as related because fundamentally, it feels a bit weird to me that our code base uses 2 different tools to solve essentially the same problem, i.e. building system disk images: 1 tool to create the build environment (currently vmdebootstrap), 1 tool to build Tails images from inside that build system (currently live-boot).

Now, if, on the build environment side, we end up dropping Vagrant in favor of something that does not need a disk image (e.g. Docker), that has its own powerful build tools which brings other advantages (e.g. layering), then the question would be different, and it probably would not make much sense to try to unify on a single tool to solve what would become 2 different problems. In that case I would see value in treating these 2 questions as separate ones.