Tails

#2 Updated by intrigeri 2018-02-27 14:08:54

intrigeri wrote:
> […] by asking the Debian maintainers when they plan to remove the package from sid.

Done.

#7 Updated by intrigeri 2018-07-30 02:56:29

intrigeri wrote:
> Lars Wirzenius tentatively agreed to keep the package in sid a bit longer, e.g. the end of 2019-02 and best/worst case the Buster release

… but later changed his mind: vmdebootstrap going away in September, switch now. I’ve checked with Lars privately and he confirmed that he really does not want to see vmdebootstrap in sid after September.

Let’s check the consequences:

• time scope: between the time when vmdebootstrap is removed from testing/sid (September) and the time when our build system has switched to something else
• who? people who satisfy all these criteria:
  • They want to build Tails ISO images.
  • They run testing/sid on their Tails development machine.
  • They have not installed vmdebootstrap before it gets removed from testing/sid.
• impact: can’t build a Tails ISO

I doubt we want to migrate in a bugfix release and it’s too late to do this in time for 3.9. Most likely, the following major release will be 3.12, scheduled for 2019-01-29, with a freeze date around mid-January, so the deadline to have this work well tested and ready for QA would be sometime at the end of 2018. The impact described above would affect people for ~4 months, during which we can document workarounds (e.g. installing vmdebootstrap from Stretch). This was basically what I had in mind when I’ve negotiated a delay with Lars earlier this year and it still seems acceptable to me.

If we miss the end of 2018 deadline, then most likely the earliest we can do the switch is Tails 4.0 (~mid-2019) and then the problem will have affected people for ~9 months. I think that’s too long, so let’s try to avoid that if we can.

#9 Updated by lamby 2018-08-17 15:15:17

• Updated/stablish link for README in upstream sources: https://sources.debian.org/src/vmdebootstrap/latest/debian/NEWS/

• Do we have any idea which of the replacements cited on https://lists.debian.org/debian-devel-announce/2018/07/msg00002.html we would want to move to?

• (In addition to suggest installing from stretch, we also have https://snapshot.debian.org/

#14 Updated by lamby 2018-09-01 19:04:25

RM bug filed https://bugs.debian.org/907751

#19 Updated by lamby 2019-01-05 18:34:07

https://wiki.debian.org/vmdebootstrap?action=diff&rev1=8&rev2=9

#21 Updated by intrigeri 2019-01-08 07:27:47

lamby wrote:
> feature/15349-migrate-away-from-vmdebootstrap branch.

In which remote?

#26 Updated by lamby 2019-04-04 17:20:33

I am writing this from the FT sprint. The consensus here that we should pause and change direction or otherwise widen the scope on this ticket for now. To elaborate, this is for three main reasons:

Firstly, the quasi-urgency to replace vmdebootstrap to create our base images was because vmdebootstrap had a Release Critical bug filed against it (in https://bugs.debian.org/910201) during September 2018 by the original maintainer, Lars Wirzenius. Entitled “vmdebootstrap should not be in buster”, here he opines that whilst it “has a bad architecture that makes the software inflexible, difficult to modify, hard to test, and it is not suitable for large number of use cases” he did admit that “a number of people have managed to make use of it”. Thus, I was tasked with finding a replacement as buster or sid lacking vmdebootstrap would make it technically inconvenient and awkward as well as “politically” strange given that we depend so much on Deban.

However, Niels Thykier and Jonathan Carter followed-up to the bug in March 2019 reporting that the removal of vmdebootstrap would affect builds of live images (see https://bugs.debian.org/922826) and, due to this, Niels Thykier added a “buster-ignore” tag in early April 2019, meaning that as the bug is no longer Release Critical with respect to the release of buster. It will now presumbly be part of the upcoming Debian release.

Secondly, it also appears that vmdebootstrap has gained a new — albeit likely temporary — maintainer which has actually resulted in a new upload to Debian unstable (version 1.11-2, see https://tracker.debian.org/news/1036916/accepted-vmdebootstrap-111-2-source-into-unstable/). There is therefore even less exigency to find a replacement project and there is even a potential route to report and fix bugs via this new “upstream” and, possibly, even to propose enhancements.

The last reason is that, given the now-lack of haste required we can spend time later looking at larger solutions to building Tails that do not involve Vagrant. This is because the weak-ish consensus is that nobody is truly happy with using Vagrant especially as it’s the cause of most build problems, and it is at least in my personal experience. A number of (entirely un-evaluated) options were mooted IRL including using lxc or Docker (which was actually proof-of-concepted some time ago, but it may not be in buster due to $reasons). intrigeri elaborated that Vagrant was chosen because at the time no other virtualisation options supported building in a reasonably secure manner; the other solutions at the time required highly-privileged access to the point where the “build system could even power down the host.”

As part of the work on this ticket, I evaluated a number of replacement options, possibly still extant on this ticket’s corresponding feature branch, but I have not re-checked at this time.

Firstly, I spent a few hours looked at debos (https://github.com/go-debos/debos). However, it did not support one of the filesystem image options that we needed so I hit a bit of dead-end in getting an actually-working image. Whilst it seemed like it would be essentially trivial to add (my impression was that the Go code is quite straightforward to hack on and it appears to continue to receive good/regular upstream attention), I concluded that requiring changes that are not in the version in buster itself would essentially leave us in the same place as vmdebootstrap. In other words, requiring that we specify using package [versions] from unstable (or, worse, upstream git!) to build.

I then spent even longer looking at vmdb2 assuming that it would be more-or-less a drop-in replacement to vmdebootstrap. However, I ironically hit almost entirely similar issues regarding a single, missing filesystem configuration option, which was curious given that its aforementioned intention to be a drop-in replacement.

So, after hitting similar issues in both potential replacement projects, I then evaluated not using any helper or build tool whatsoever! To do this, I quickly hacked my local copy of vmdebootstrap to log which shell commands it was actually running and then duplicating them in the build script. I did this on the hypothesis that we aren’t actually doing anything that strange or weird when building; essentially “just” calling debootstrap and copying that into a large enough disk image that then gets consumed by Vagrant. This appeared to work to some degree (although I was missing something at the very end…) and was indeed fairly easy to demonstrate to some degree. However, after looking at the resulting diff etc. I remained unclear that this is long-term sustainable as it essentially replicates one of the above tools without real error checking nor cleanup. For example, the loopback devices created by losetup(8) need some management that is difficult to accomplish in a shell script. Thus we would likely just be adding to the flaky nature of the build system (see also, Vagrant!), rather than making it any simpler or at least at the same level of flakiness.

I did not get around to evaluating FAI (https://fai-project.org/) as I had the preconception and very slight experience in that it is a little more complex so I intended to try it last. Ironically this apparent “complexity” may indicate that it would support all of our requirements, although with a little more of a learning curve.

I did try and research some other things but they are probably not relevant to the current status of this issue. Anyway, I hope this gives an overview and update of this ticket and, unless there are any objections and given the now lack of priority, I will remove myself as an assignee of this ticket. It may be worth renaming this ticket (or perhaps creating a separate one with some kind of blocking relation?) regarding questions about replacing Vagrant itself. After all, replacing this virtualisation solution might imply or even require a specific build tool anyway.

#28 Updated by intrigeri 2019-04-05 16:38:09

@lamby, happy to mangle the ticket metadata and move on once you’ve clarified where the feature/15349-migrate-away-from-vmdebootstrap can be found (and if it can’t, fine, we’ll do without it).

#34 Updated by lamby 2019-04-09 08:41:00

> it would be sweet to know which filesystem configuration option are missing in debos and vmdb2,

Unfortunately not. :( However, on the positive-side I do remember hitting them “fairly” quickly, so it is not something that would require in-depth. I think it was about having lots of free space in the image, so it might not be something that’s insurmountable with a hybrid $TOOL-but-resize-afterwards approach… but don’t quote me too strongly on that.

#38 Updated by intrigeri 2020-05-15 08:56:59

> Related to Feature Feature #5691: Consider upgrading to current live-build added

I marked these issues as related because fundamentally, it feels a bit weird to me that our code base uses 2 different tools to solve essentially the same problem, i.e. building system disk images: 1 tool to create the build environment (currently vmdebootstrap), 1 tool to build Tails images from inside that build system (currently live-boot).

Now, if, on the build environment side, we end up dropping Vagrant in favor of something that does not need a disk image (e.g. Docker), that has its own powerful build tools which brings other advantages (e.g. layering), then the question would be different, and it probably would not make much sense to try to unify on a single tool to solve what would become 2 different problems. In that case I would see value in treating these 2 questions as separate ones.