Feature #7530

Evaluate switching to Docker for an easy, self-contained ISO build system

Added by intrigeri 2014-07-09 12:58:42 . Updated 2016-08-18 12:43:35 .

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
2014-07-09
Due date:
% Done:

100%

Feature Branch:
feature/7530-docker
Type of work:
Research
Starter:
0
Affected tool:
Deliverable for:

Description

See the parent ticket and blueprint for the rationale. We should evaluate various aspects of Docker.


Files

docker.diff (2208 B) intrigeri, 2014-12-05 10:51:52

Subtasks

Feature #7531: Evaluate stability of Docker's API and configuration format Rejected

10

Feature #7532: Evaluate Docker's availability on target platforms Resolved intrigeri

100

Feature #7533: Look for a Docker "basebox" creation & maintenance helper Rejected

100

Feature #8781: Manage and use an apt-cacher-ng container with Docker Rejected

10


Related issues

Related to Tails - Bug #15349: Migrate away from vmdebootstrap (and possibly from Vagrant) In Progress 2018-02-27

History

#1 Updated by intrigeri 2014-07-10 21:49:44

  • Description updated

#2 Updated by intrigeri 2014-07-12 09:50:15

  • blocks Feature #7580: Decide mid-term strategy regarding easy, self-contained build environment added

#3 Updated by intrigeri 2014-07-12 12:34:06

  • Description updated
  • Blueprint set to https://tails.boum.org/blueprint/evaluate_Docker/

#4 Updated by intrigeri 2014-11-10 16:17:04

Any news on this front?

#5 Updated by intrigeri 2014-11-30 12:58:41

I’ve (finally) done the research I was supposed to do: see sibbling tickets, parent ticket and the corresponding blueprints.

#6 Updated by BitingBird 2014-11-30 15:49:26

  • Status changed from Confirmed to In Progress

#7 Updated by intrigeri 2014-12-05 10:52:10

Initial PoC attached, thank to jvoisin’s crew

#8 Updated by anonym 2015-01-20 13:44:35

  • Feature Branch set to feature/7530-docker

Some more testing/work has been done (based on jvoisin’s patch — thanks!) in the feature/7530-docker branch. Due to no apt proxy being configured, curl fails to fetch the TBB tarballs unless the following is applied:

--- a/config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt
+++ b/config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt
@@ -1 +1 @@
-http://dist.torproject.org/torbrowser/
+https://dist.torproject.org/torbrowser/


Indeed, with that applied inside the feature branch I managed to build (in-disk, without any apt caching) a working Tails iso image inside an up-to-date Jessie VM:

sudo apt-get install --no-install-recommends docker.io aufs-tools cgroupfs-mount
sudo adduser $USER docker
su $USER
make


Progress!

#9 Updated by anonym 2015-01-21 16:11:49

  • Assignee changed from jvoisin to intrigeri
  • QA Check set to Info Needed

intrigeri, I think we’ve duplicated some work. See my branch feature/7530-docker_anonym, commit 7bd957b vs your commit 62903ef (which also contains a random fix I committed separately as commit a9d29d6 in my branch). Since we are going to introduce another Dockerfile for the apt-cacher-ng container, isn’t it smarter to move the tails_builder Dockerfile into its own subdirectory like in my branch? If so, what about resetting feature/7530-docker to my branch’s state?

Also, notice that my branch introduces RAM builds, which should be good news for the SSD thrashing you experienced. :)

#10 Updated by intrigeri 2015-01-21 16:51:06

> intrigeri, I think we’ve duplicated some work.

I was hoping that fetching and pushing as often as I could, plus being on IRC, would avoid that. Let’s try to coordinate better the next steps.

> Since we are going to introduce another Dockerfile for the apt-cacher-ng container, isn’t it smarter to move the tails_builder Dockerfile into its own subdirectory like in my branch?

Sure.

> If so, what about resetting feature/7530-docker to my branch’s state?

Agreed, except:

  • The comment in .gitignore: “Needed by Docker” makes little sense to me in that context. We’re not ignoring it because it’s needed, but rather because it’s auto-generated and we don’t need to track it in Git.
  • The name of the copy_tails_apt_key target in Makefile, that I find less idiomatic than my version.

> Also, notice that my branch introduces RAM builds, which should be good news for the SSD thrashing you experienced. :)

Woohooo \o/

#11 Updated by intrigeri 2015-01-21 16:56:03

  • Assignee changed from intrigeri to anonym
  • QA Check deleted (Info Needed)

#12 Updated by anonym 2015-01-22 13:08:54

  • Assignee changed from anonym to jvoisin

intrigeri wrote:
> > intrigeri, I think we’ve duplicated some work.
> I was hoping that fetching and pushing as often as I could, plus being on IRC, would avoid that. Let’s try to coordinate better the next steps.

Right. Any way, I’m not sure I’ll have too much time working on this until after the 1.3 release. I did do some quick investigation of the apt-cacher-ng container, but I mostly got trash. I’ll push the useful part, which is just enough to build the container (but not linking it to the builder), to the feature/7530-docker-with-apt-cacher-ng branch in case you (or someone else) want to procrastinate on this. :)

For the record, I directed the container to use an existing apt-cacher-ng instance and use gzip compression for mksquashfs with this ugly hack (we probably want to use docker run -e instead of setting ENV, but sadly there’s nothing like it for docker build for which we also would like APT caching):

--- a/docker/tails_builder/Dockerfile
+++ b/docker/tails_builder/Dockerfile
@@ -1,6 +1,9 @@
 FROM debian:wheezy
 MAINTAINER tails@boum.org

+ENV http_proxy http://10.0.2.15:3142
+ENV MKSQUASHFS_OPTIONS -comp gzip
+
 COPY provision/assets/apt/deb.tails.boum.org.key   /tmp/deb.tails.boum.org.key
 COPY provision/assets/apt/sources.list             /etc/apt/sources.list
 COPY provision/assets/apt/preferences              /etc/apt/preferences.d/tails


and with a prepped cache it all took 17m 45s, of which building the container took 45s. 17m is what a VirtualBox-based Vagrant build takes for me, and since I’m running docker inside a VirtualBox VM with enough RAM for a RAM Build this expected results but good to have verified.

> > If so, what about resetting feature/7530-docker to my branch’s state?
>
> Agreed, except:
>
> * The comment in .gitignore: “Needed by Docker” makes little sense to me in that context. We’re not ignoring it because it’s needed, but rather because it’s auto-generated and we don’t need to track it in Git.
> * The name of the copy_tails_apt_key target in Makefile, that I find less idiomatic than my version.

I’ve force pushed my branch with fixes for these concerns. Thanks for the pointers!

#13 Updated by intrigeri 2015-01-23 11:38:11

> I did do some quick investigation of the apt-cacher-ng container,

Cool. Moved the discussion about this part to the Feature #8781 subtask.

#14 Updated by sajolida 2015-09-07 10:46:29

  • Target version changed from Hole in the Roof to 2016

#15 Updated by intrigeri 2015-09-23 03:10:52

  • blocked by deleted (Feature #7580: Decide mid-term strategy regarding easy, self-contained build environment)

#16 Updated by intrigeri 2015-09-23 03:12:31

  • Target version deleted (2016)

At the summit we decided to “put aside the Docker thing for a while” and do Bug #10232 instead.

#17 Updated by intrigeri 2016-06-08 07:41:17

We now have vagrant-libvirt working fine, and for our CI we will probably use it as well (worst case we’ll use Bug #10232 rather than Docker I think). So IMO, we should just close all Docker-related tickets as “rejected”. jvoisin, anonym?

#18 Updated by BitingBird 2016-06-29 07:25:03

  • QA Check set to Info Needed

#19 Updated by intrigeri 2016-08-18 12:43:35

  • Status changed from In Progress to Rejected
  • Assignee deleted (jvoisin)

Vagrant!

#20 Updated by intrigeri 2019-04-08 14:43:42

  • related to Bug #15349: Migrate away from vmdebootstrap (and possibly from Vagrant) added