Feature #10748

Generate a manifest of packages used at build time

Added by intrigeri 2015-12-13 12:46:35 . Updated 2016-06-08 01:25:04 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Build system
Target version:
Tails_2.4
Start date:
2015-12-13
Due date:
% Done:

100%

Feature Branch:
feature/10748-gen-packages-manifest
Type of work:
Code
Blueprint:

https://tails.boum.org/blueprint/freezable_APT_repository/

Starter:
Affected tool:
Deliverable for:
269

Description

At ISO build time, generate a list of used packages and version, including packages used at build time but not shipped in the ISO.

Output: a machine-readable file that sums up all the information we’ll later need to create a tagged, partial snapshot of the APT repositories we use, that contains only packages that were used at ISO build time.

Subtasks

Related issues

Related to Tails - Feature #5548: Research ways to distribute source Resolved
Has duplicate Tails - Feature #6297: Save list of packages used at ISO build time Duplicate
Blocks Tails - Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests Resolved 2016-05-11

History

#1 Updated by intrigeri 2015-12-13 12:46:59

  • Blueprint set to https://tails.boum.org/blueprint/freezable_APT_repository/

#2 Updated by intrigeri 2015-12-13 12:54:39

  • blocked by Feature #6297: Save list of packages used at ISO build time added

#3 Updated by intrigeri 2015-12-13 14:56:04

  • % Done changed from 10 to 20
  • Feature Branch changed from kibi:feature/5926-freezable-APT-repository to feature/5926-freezable-APT-repository

Merged kibi’s work, tested basic functionality, integrated into our build system. I’ll track what’s left to check and test to the blueprint.

#4 Updated by intrigeri 2015-12-14 02:43:04

OK, so everything left to check depends on me looking into tails-prepare-tagged-apt-snapshot-import first.

#5 Updated by intrigeri 2015-12-14 02:50:56

#6 Updated by intrigeri 2015-12-14 02:53:06

  • blocks deleted (Feature #6297: Save list of packages used at ISO build time)

#7 Updated by intrigeri 2015-12-14 02:53:50

#8 Updated by intrigeri 2015-12-14 02:54:22

#9 Updated by intrigeri 2015-12-14 02:54:44

  • blocked by Feature #6297: Save list of packages used at ISO build time added

#10 Updated by intrigeri 2015-12-14 02:55:55

  • blocks Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots added

#11 Updated by intrigeri 2015-12-14 06:20:10

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Ready for QA to Info Needed

The main potential issues I’ve discovered are being discussed over email:

#12 Updated by intrigeri 2016-02-05 20:48:04

  • Assignee changed from CyrilBrulebois to intrigeri
  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.0 to Tails_2.2
  • QA Check deleted (Info Needed)

Re-assigning to me so I have it in mind, and will schedule time with Cyril to address the identified problems in the delivered code.

#13 Updated by intrigeri 2016-03-08 13:45:48

  • Target version changed from Tails_2.2 to Tails_2.3

#14 Updated by intrigeri 2016-03-10 13:45:01

intrigeri wrote:
> The main potential issues I’ve discovered are being discussed over email:
>
> * “TailsFeature #10748: build manifest vs. multiple origins”

It would be good to check if this change in APT will impact that problem:

apt (1.1~exp9) experimental; urgency                    =medium

  A new algorithm for pinning has been implemented, it now assigns a
  pin priority to a version instead of assigning a pin to a package.

  This might break existing corner cases of pinning, if they use multiple
  pins involving the same package name or patterns matching the same
  package name, but should overall lead to pinning that actually works
  as intended and documented.

 -- Julian Andres Klode <jak@debian.org>  Mon, 17 Aug 2015 14:45:17 +0200

#15 Updated by intrigeri 2016-03-10 14:18:58

intrigeri wrote:
> The main potential issues I’ve discovered are being discussed over email:
>
> * “TailsFeature #10748: build manifest vs. multiple origins”

We won’t handle that when generating the manifest, but on Feature #10749.

#16 Updated by intrigeri 2016-03-10 15:58:19

  • Feature Branch changed from feature/5926-freezable-APT-repository to feature/10748-gen-packages-manifest

#17 Updated by intrigeri 2016-03-25 11:40:15

  • blocked by deleted (Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots)

#18 Updated by intrigeri 2016-03-25 22:56:48

  • Subject changed from Generate a manifest of packages used at build time per-origin to Generate a manifest of packages used at build time

#19 Updated by intrigeri 2016-03-25 23:00:03

  • blocks deleted (Feature #6297: Save list of packages used at ISO build time)

#20 Updated by intrigeri 2016-03-25 23:02:26

  • has duplicate Feature #6297: Save list of packages used at ISO build time added

#21 Updated by intrigeri 2016-03-25 23:03:00

  • related to Feature #5548: Research ways to distribute source added

#22 Updated by intrigeri 2016-03-25 23:04:20

  • Description updated

#23 Updated by intrigeri 2016-03-26 00:12:51

  • Feature Branch changed from feature/10748-gen-packages-manifest to feature/10748-gen-packages-manifest, puppet-tails:feature/10748-gen-packages-manifest

#24 Updated by intrigeri 2016-04-25 13:01:08

  • Target version changed from Tails_2.3 to Tails_2.4

#25 Updated by intrigeri 2016-05-11 08:28:25

  • Feature Branch changed from feature/10748-gen-packages-manifest, puppet-tails:feature/10748-gen-packages-manifest to feature/10748-gen-packages-manifest, puppet-tails

#26 Updated by intrigeri 2016-05-11 08:29:24

  • Feature Branch changed from feature/10748-gen-packages-manifest, puppet-tails to feature/10748-gen-packages-manifest

#27 Updated by intrigeri 2016-05-11 08:29:42

  • blocks Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests added

#28 Updated by intrigeri 2016-05-11 08:30:21

  • blocked by deleted (Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests)

#29 Updated by intrigeri 2016-05-11 08:30:55

  • blocks Feature #11412: Drop support for packages-from-acng-log in our Puppet manifests added

#30 Updated by intrigeri 2016-05-11 08:31:43

  • blocks Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots added

#31 Updated by intrigeri 2016-05-11 08:32:28

  • Assignee changed from intrigeri to anonym
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

Please review and merge :)

#32 Updated by anonym 2016-05-12 10:45:36

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Info Needed

Code looks good, and makes sense. Great job! I admittedly didn’t look at Cyril’s generate-build-manifest, but I assume you have, and I know you are in a much better position to do so. :)

However, I thought I should at least make a low-effort sanity check of the generated .build-manifest compared to the packages downloaded according to the .buildlog. Try running the below, but set ISO to some of your own builds.

ISO=tails-i386-devel-2.4-20160512T1527Z-7137ac2.iso
BUILD_MANIFEST="${ISO}.build-manifest"
BUILDLOG="${ISO}.buildlog"
sed -nE 's/^  ( |-) package: //p' "${BUILD_MANIFEST}" | sort -u > /tmp/pkgs-from-manifest
sed -nE \
  -e 's/^I: Retrieving (\S+) .*$/\1/p' \
  -e 's/^Get:[0-9]+\s+\S+\s+\S+\s+(\S+)\s+.*$/\1/p' \
  "${BUILDLOG}" | sort -u > /tmp/pkgs-from-buildlog
diff -Naur /tmp/pkgs-from-manifest /tmp/pkgs-from-buildlog


The diff contains some crap you’ll have to filter out manually, but shows two packages that’s omitted but definitely should not be:

  • lockfile-progs: this one is even shipped in Tails, so I do not see why it should not be in the manifest.
  • squashfs-tools: this one is not in Tails, but used during build to squash the filesystem so it should be in the manifest.

So, either explain why I am confused and reassign the ticket back to me, or, if you confirm that this is wrong, keep the ticket and change QA Check to Dev Needed.

#33 Updated by intrigeri 2016-05-12 12:09:15

  • QA Check changed from Info Needed to Dev Needed

#34 Updated by intrigeri 2016-05-12 12:31:27

> However, I thought I should at least make a low-effort sanity check of the generated .build-manifest compared to the packages downloaded according to the .buildlog.

Thanks for trying that! I would have noticed at least one of these two issues later on, when trying to build from a (partial) tagged snapshot that has only the packages listed in the build manifest, but still it’s good to detect problems early :)

> * lockfile-progs: this one is even shipped in Tails, so I do not see why it should not be in the manifest.

Fixed in commit:4b4e5b8. tl;dr: yes, apt-get purge can install packages.

> * squashfs-tools: this one is not in Tails, but used during build to squash the filesystem so it should be in the manifest.

OK, so this one is a weird one.

It is pulled by the build system outside of the chroot that we customize (that includes using our apt-get wrapper), compress and ship, and it affects the ISO just as much as anything else in the build VM, so arguably it’s part of the environment that we do not try to capture. OTOH, in practice it’ll be pulled from http://time-based.snapshots.deb.tails.boum.org/debian/ most of the time, so when we build from a tagged snapshot it’ll be pulled from http://tagged.snapshots.deb.tails.boum.org/debian/, and then the build will fail because that package will be missing. So, even if I could have a good reason to dismiss this one, I have to deal with it… somehow. I’ll see if I can handle it nicely (not too hopeful), and worst case we’ll have a list of “extra” packages that should always be added to the build manifest, regardless of whether we have detected that they have been pulled during the build process.

#35 Updated by intrigeri 2016-05-12 14:11:43

  • Assignee changed from intrigeri to anonym
  • % Done changed from 50 to 60
  • QA Check changed from Dev Needed to Ready for QA

Both fixed.

#36 Updated by anonym 2016-05-13 08:28:45

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 60 to 100
  • QA Check changed from Ready for QA to Pass

Yup, now the two problematic packages are present in the build-manifest. While reviewing your recent addition to generate-build-manifest I had a look at the full script, not the Perl particularities, but the general approach, and it looks straightforward and good.

No more complaints => merged!

#37 Updated by intrigeri 2016-05-23 13:06:25

  • blocked by deleted (Feature #10749: Create partial APT snapshot from a build manifest and a set of time-based snapshots)

#38 Updated by anonym 2016-06-08 01:25:04

  • Status changed from Fix committed to Resolved