Feature #6220

Automated Debian package build infrastructure

Added by intrigeri 2013-08-07 13:10:22 . Updated 2019-09-19 06:24:55 .

Status:
Confirmed
Priority:
Normal
Assignee:
bertagaz
Category:
Continuous Integration
Target version:
Start date:
2013-08-07
Due date:
% Done:

0%

Starter:
0
Affected tool:
Deliverable for:

Description

It’s been done before, no need to reinvent the wheel. See the blueprint for potentially useful tools.

Team: bertagaz


Subtasks


Related issues

Related to Tails - Feature #6090: Automated builds Resolved 2013-07-26 2015-02-28
Related to Tails - Feature #14455: Reproducible Builds Stage 2 Confirmed 2017-08-26
Related to Tails - Feature #6219: Document our autobuild setup Rejected 2013-08-07
Blocked by Tails - Feature #7036: Move custom software to our main Git repository In Progress

History

#1 Updated by Dr_Whax 2014-04-04 00:03:55

The Debile link is not working, this one is: http://anonscm.debian.org/gitweb/?p=pkg-debile/debile.git

#2 Updated by intrigeri 2014-04-04 09:35:18

  • Description updated

#3 Updated by intrigeri 2014-04-04 09:38:04

  • Description updated
  • Blueprint set to https://tails.boum.org/blueprint/automated_builds_and_tests/#resources-debian-pkg

#4 Updated by intrigeri 2014-08-05 12:31:17

  • Blueprint changed from https://tails.boum.org/blueprint/automated_builds_and_tests/#resources-debian-pkg to https://tails.boum.org/blueprint/automated_builds_and_tests/Debian_packages/

#5 Updated by intrigeri 2014-08-14 05:15:42

  • related to Feature #7036: Move custom software to our main Git repository added

#6 Updated by intrigeri 2014-08-14 05:16:11

#7 Updated by sajolida 2014-11-03 21:37:02

  • Target version set to Sustainability_M1

#8 Updated by sajolida 2015-08-14 11:48:26

  • Description updated

#9 Updated by sajolida 2015-09-07 10:49:52

  • Target version changed from Sustainability_M1 to 2016

#10 Updated by Dr_Whax 2016-08-20 12:22:35

  • Description updated
  • Assignee set to bertagaz
  • Target version changed from 2016 to 2018

#11 Updated by intrigeri 2017-08-11 16:32:09

At this point I’m happy we did not set this up server-side, in a centralized manner, because I’ve entirely changed my mind wrt. how we should approach the problem at hand.

IMO we should build our custom packages as part of building an ISO (using good ol’ sbuild/pbuilder or something that better isolates package builds from one another, such as https://gitlab.com/uhoreg/whalebuilder or one of the other options, and track their source code as Git submodules i.e. Feature #7036 (for non-native packages, likely we want to track the branch that contains the Debian packaging).

This advantages I see with this approach, compared to what we currently do, and to extending our central infrastructure to build packages automatically:

  1. We get more value out of reproducible ISO builds, by removing binary input and the required trust in the content of these packages, in the developer who built+uploaded them, and in our custom APT repo infrastructure.
  2. Developers who don’t have Git commit access can work on our custom software and build an ISO with their modified version in a straightforward way (no need to have a pbuilder setup nor to copy .deb’s to config/chroot_local-packages, and the result of the work can be handled as any other Git pull request).
  3. No need for all developers to set up and maintain a local pbuilder setup with N chroots.
  4. We can get rid of our custom APT repository and, perhaps more importantly, of the complex & error-prone workflow that comes with it (e.g. for merging/resetting APT suites). Everything related to config/APT_overlays.d/ can go away, which simplifies our codebase and infrastructure.
  5. More parameters that affect the resulting ISO are encoded in Git, which makes them easier to audit, log, and diff. The only remaining thing that’s not encoded in Git is the content of the APT snapshots we use (we already encode in Git which set of APT snapshots we use).
  6. We simplify what it means to have Git commit access: developers who have Git commit access don’t also need upload rights to our custom APT repo. This simplifies sysadmin (no need to keep OpenPGP keys up-to-date) and makes it easier for developers to understand how the whole thing works.
  7. The apt-cacher-ng cache can be shared between ISO build process and .deb packages build process.
  8. We get finer-grained control over what exact state of the pbuilder/sbuild/whalebuilder chroot we use for building packages. E.g. we might want to use exactly the same APT snapshots that are used for building the rest of the ISO, which can matter a lot if we switch to Debian testing (Feature #12615): building against current Debian testing can create packages that are not installable on the (older) snapshot of Debian testing we use in the SquashFS.
  9. More of us have the skills and access needed to work, within our Git tree, on such a Debian packages build system, than to work on a centralized package build system hosted our infrastructure. This implies that we have more options wrt. who implements, reviews & maintains the new thing, instead of increasing our reliance on a tiny sysadmin team that would use a totally different stack (Puppet, Jenkins, jenkins-debian-glue, etc.).
  10. Compared to what we do today, we have the option to enforce some quality checks on the packaging and upstream code (e.g. Lintian or autopkgtests).
  11. No need to design & implement yet another mapping between Git branches, Jenkins jobs and APT suites.
  12. No race conditions / async design skills required: we’re guaranteed that the expected version of every custom package we want has been built when we need it.

#12 Updated by intrigeri 2017-09-28 12:14:20

  • Target version deleted (2018)

(as per updated roadmap)

#14 Updated by intrigeri 2017-10-03 06:28:14

#15 Updated by Anonymous 2018-08-19 08:46:49

Inspiration: https://auto.debian.net/

#16 Updated by Anonymous 2018-08-19 08:48:25

#17 Updated by intrigeri 2019-09-19 06:24:16

  • related to deleted (Feature #7036: Move custom software to our main Git repository)

#18 Updated by intrigeri 2019-09-19 06:24:19

  • blocked by Feature #7036: Move custom software to our main Git repository added

#19 Updated by intrigeri 2019-09-19 06:24:55

It might be that once we have Feature #7036, there’s very few reasons left (if any) to bother doing what this very ticket is about.