Feature #15524

Iteration 1: Write release process documentation for custom packages

Added by segfault 2018-04-11 16:11:49 . Updated 2018-10-15 21:10:06 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2018-04-11
Due date:
% Done:

100%

Feature Branch:
feature/15524-veracrypt-release-documentation
Type of work:
Contributors documentation
Blueprint:

Starter:
Affected tool:
Deliverable for:
299

Description

In the releases between when we ship our VeraCrypt work in Tails (3.9) and when Tails is based on Buster, we need to maintain the custom packages we create (see Feature #15521).

For this we should include a step in the release process to check whether any of the packages we base ours on have an update, and in that case merge the update into our packages.


Subtasks


Related issues

Related to Tails - Feature #14728: Track security updates during the Tails code freeze Confirmed 2017-09-26

History

#1 Updated by bertagaz 2018-05-10 11:09:22

  • Target version changed from Tails_3.7 to Tails_3.8

#2 Updated by intrigeri 2018-06-03 11:51:55

When working on this you can assume the RM knows the basics of Debian packaging so I think this can boil down to:

  • document the list of custom packages and their inter-build-dependencies as you did on Feature #14481
  • something like “download the updated Debian source package, the old Debian source package that our custom package was forked off, and our custom source package, debdiff the 2 last ones, apply that debdiff on top of the 1st one, dch and boom”

So I don’t think we need one dedicated page per package under wiki/src/contribute/release_process/, one single page referenced from contribute/release_process should be enough.

#3 Updated by intrigeri 2018-06-03 13:07:37

  • Target version changed from Tails_3.8 to Tails_3.9

This can wait: as long as it’s done (i.e. implemented, reviewed and merged) in time for 3.9 we’re good.

#4 Updated by segfault 2018-07-30 15:23:09

  • Assignee changed from segfault to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:
> * something like “download the updated Debian source package, the old Debian source package that our custom package was forked off, and our custom source package, debdiff the 2 last ones, apply that debdiff on top of the 1st one, dch and boom”

How is the RM supposed to download the old Debian source package? It doesn’t seem possible to download these via apt source (I tried to download the previous version of udisks2 on stable via apt source udisks2=2.1.7-3 and it doesn’t work). Should we upload them somewhere?

#5 Updated by intrigeri 2018-08-01 03:10:44

  • Assignee changed from intrigeri to segfault
  • QA Check deleted (Info Needed)

segfault wrote:
> intrigeri wrote:
> > * something like “download the updated Debian source package, the old Debian source package that our custom package was forked off, and our custom source package, debdiff the 2 last ones, apply that debdiff on top of the 1st one, dch and boom”
>
> How is the RM supposed to download the old Debian source package?

With apt-get source or dget.

> It doesn’t seem possible to download these via apt source (I tried to download the previous version of udisks2 on stable via apt source udisks2=2.1.7-3 and it doesn’t work). Should we upload them somewhere?

Debian publishes the source of all packages so we don’t have to upload them ourselves. Stretch has 2.1.8-1 and I see no Debian release with 2.1.7-3 which I suspect is why what you’ve tried did not work.

#6 Updated by segfault 2018-08-01 18:21:03

  • Assignee changed from segfault to intrigeri
  • QA Check set to Info Needed

> Debian publishes the source of all packages so we don’t have to upload them ourselves. Stretch has 2.1.8-1 and I see no Debian release with 2.1.7-3 which I suspect is why what you’ve tried did not work.

Yes, but 2.1.7-3 was once released, and we are talking about old Debian packages, i.e. ones that are not the current version. So what I’m looking for is a way to download those, which does not seem possible with apt source or dget.

#7 Updated by intrigeri 2018-08-02 00:42:01

  • Assignee changed from intrigeri to segfault
  • QA Check changed from Info Needed to Dev Needed

> we are talking about old Debian packages, i.e. ones that are not the current version.

Right, I missed that part of the context, sorry. Note that I think we only need that for packages that were not maintained in Git when Stretch was released (e.g. most GNOME stuff). For packages that had a Vcs-Git back then already, better fork that Vcs-Git and then updating the package is essentially a Git merge + gbp dch.

> So what I’m looking for is a way to download those, which does not seem possible with apt source or dget.

https://snapshot.debian.org/

#8 Updated by intrigeri 2018-09-05 16:26:57

  • Target version changed from Tails_3.9 to Tails_3.10.1

#9 Updated by segfault 2018-09-10 21:22:41

What should the distribution field be set to in new versions of our custom packages?

#10 Updated by segfault 2018-09-10 21:34:22

  • Assignee changed from segfault to intrigeri
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch set to segfault:feature/15524-veracrypt-release-documentation

I wrote a draft. I didn’t document the package uploading because I don’t know how that is done.

#11 Updated by intrigeri 2018-09-11 07:22:49

> What should the distribution field be set to in new versions of our custom packages?

It should be based on the name of the branch (bin/add-APT-overlay does the translation for you) that will bring the update, which itself should be based on the corresponding ticket number.

#12 Updated by intrigeri 2018-09-11 08:07:02

  • Assignee changed from intrigeri to segfault
  • QA Check changed from Ready for QA to Dev Needed

> I wrote a draft.

Great!

  • I would not hard-code the “version in Tails 3.9” info. I don’t think it’s terribly useful and we would need additional steps to update it.
  • s/a new version in Debian stable/a new version in Debian stable or stretch-security/
  • Let’s make the instructions independent from 3.9 (otherwise we would need additional steps to update them): instead of pulling our forked source package from the 3.9 suite, let’s pull it from our stable, testing or devel suite, depending on which release the updated packages should go in. And then drop check-valid-until=no for these APT sources.
  • Why do we need deb (binary) APT sources? I’d rather not encourage developers to add those to their main system unless it’s really needed.
  • “and the original version” feels ambiguous to me; right now I know what it means but in 6 months, well… Maybe “the version of the official Debian package it was forked off” or similar?
  • Please specify which kind of chroot one must build the package in. E.g. https://tails.boum.org/contribute/release_process/tails-greeter/ reads “(use a Stretch/amd64 chroot)”.
  • It seems to me that the “Download the new source package” section assumes that one has no APT source configured with a newer udisks2 than the one in Debian stable and stretch-security. That’s not always the case, o.g. on my system this would download the version from sid, which is not what we want.
  • s/>> tails.diff/> tails.diff/ otherwise things will break as soon as one repeats the process, be it to update the same package or another one.
  • I would “automate” the process a bit by first storing all the needed info in variables (source package name, version in current Tails stable/testing/devel, version we forked off last time, new version we want to rebase on top of, etc.) so that once it’s done, the rest of the process is a matter of copy’n’paste without having to adjust every second command by hand. Not a blocker, YMMV :)
  • s/hit Debian stable/have hit Debian stable/
  • Ideally this would not be part of the release process: it feels a bit too late to act, there won’t be any 3rd-party QA, and I don’t think this should be on the RM’s plate anyway. To me it looks much more like FT work. But we have no good way to track the need for such updates yet (Feature #14728 will help) so unless you have a better idea, let’s keep things as-is and add a note about it on Feature #14728.

I won’t bother testing the doc right now. Let’s do it the first time we need to follow it :)

> I didn’t document the package uploading because I don’t know how that is done.

You can copy the “Add the Debian package to Tails” section from https://tails.boum.org/contribute/release_process/tails-greeter/ :)

(Slightly off-topic: BTW, if you’re curious, https://tails.boum.org/contribute/APT_repository/custom/ has more detailed info.)

#13 Updated by intrigeri 2018-09-11 08:09:03

  • related to Feature #14728: Track security updates during the Tails code freeze added

#14 Updated by segfault 2018-09-26 20:07:23

  • Assignee changed from segfault to intrigeri
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:
> > I wrote a draft.
>
> Great!
>
> * I would not hard-code the “version in Tails 3.9” info. I don’t think it’s terribly useful and we would need additional steps to update it.

I think it is useful. How would the workflow to check for a new version look like if the version is not listed in the instructions?

> * s/a new version in Debian stable/a new version in Debian stable or stretch-security/

Done.

> * Let’s make the instructions independent from 3.9 (otherwise we would need additional steps to update them): instead of pulling our forked source package from the 3.9 suite, let’s pull it from our stable, testing or devel suite, depending on which release the updated packages should go in.

Done.

> * Why do we need deb (binary) APT sources? I’d rather not encourage developers to add those to their main system unless it’s really needed.

I somehow thought that those were also required to download the sources. I removed them.

> * “and the original version” feels ambiguous to me; right now I know what it means but in 6 months, well… Maybe “the version of the official Debian package it was forked off” or similar?

Done.

> * Please specify which kind of chroot one must build the package in. E.g. https://tails.boum.org/contribute/release_process/tails-greeter/ reads “(use a Stretch/amd64 chroot)”.

Done.

> * It seems to me that the “Download the new source package” section assumes that one has no APT source configured with a newer udisks2 than the one in Debian stable and stretch-security. That’s not always the case, o.g. on my system this would download the version from sid, which is not what we want.

Fixed.

> * s/>> tails.diff/> tails.diff/ otherwise things will break as soon as one repeats the process, be it to update the same package or another one.

Fixed.

> * I would “automate” the process a bit by first storing all the needed info in variables (source package name, version in current Tails stable/testing/devel, version we forked off last time, new version we want to rebase on top of, etc.) so that once it’s done, the rest of the process is a matter of copy’n’paste without having to adjust every second command by hand. Not a blocker, YMMV :)

> * s/hit Debian stable/have hit Debian stable/

Done.

> * Ideally this would not be part of the release process: it feels a bit too late to act, there won’t be any 3rd-party QA, and I don’t think this should be on the RM’s plate anyway. To me it looks much more like FT work. But we have no good way to track the need for such updates yet (Feature #14728 will help) so unless you have a better idea, let’s keep things as-is and add a note about it on Feature #14728.

OK, I don’t have a better idea.

> I won’t bother testing the doc right now. Let’s do it the first time we need to follow it :)

ACK.

> > I didn’t document the package uploading because I don’t know how that is done.
>
> You can copy the “Add the Debian package to Tails” section from https://tails.boum.org/contribute/release_process/tails-greeter/ :)

Done.

> (Slightly off-topic: BTW, if you’re curious, https://tails.boum.org/contribute/APT_repository/custom/ has more detailed info.)

OK, thanks.

#15 Updated by intrigeri 2018-10-10 14:40:40

  • Assignee changed from intrigeri to segfault
  • QA Check changed from Info Needed to Dev Needed

Hi!

All your fixes look good, woohoo! The remaining fixes I’m requested below should be easy so I’m very confident I’ll merge your branch after the next iteration of the feedback loop :)

>> * I would not hard-code the “version in Tails 3.9” info. I don’t think it’s terribly useful and we would need additional steps to update it.

> I think it is useful. How would the workflow to check for a new version look like if the version is not listed in the instructions?

Either look into the .packages file on our master branch, or use dpkg in a Tails system, or use APT to query the tagged APT snapshot for the latest Tails release.
Now, if you prefer to keep this duplicated info on this document, fine, but please add steps to update it.

>> * Ideally this would not be part of the release process: it feels a bit too late to act, there won’t be any 3rd-party QA, and I don’t think this should be on the RM’s plate anyway. To me it looks much more like FT work. But we have no good way to track the need for such updates yet (Feature #14728 will help) so unless you have a better idea, let’s keep things as-is and add a note about it on Feature #14728.

> OK, I don’t have a better idea.

Fine, please update Feature #14728 then :)

Also, I’ve noticed some regressions since the last version I’ve reviewed:

  • I believe apt source -t ${NEW_VERSION} ${PACKAGE_NAME} won’t work. Maybe you meant apt source ${PACKAGE_NAME}=${NEW_VERSION}?
  • without "bugfix/" or "feature/" feels wrong.
  • All these unquoted variables hurt my eyes. Yeah, shell scripting is hard :/ Our preferred/standard way to write variables in such documents is ${VARIABLE:?} which is basically equivalent to set -u ; "$VARIABLE" i.e. an error will be raised if whoever pastes these command lines has not set $VARIABLE (mistakes do happen :)

#16 Updated by segfault 2018-10-14 11:33:20

  • Assignee changed from segfault to intrigeri
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch changed from segfault:feature/15524-veracrypt-release-documentation to feature/15524-veracrypt-release-documentation

intrigeri wrote:
> Hi!
>
> All your fixes look good, woohoo! The remaining fixes I’m requested below should be easy so I’m very confident I’ll merge your branch after the next iteration of the feedback loop :)
>
> >> * I would not hard-code the “version in Tails 3.9” info. I don’t think it’s terribly useful and we would need additional steps to update it.
>
> > I think it is useful. How would the workflow to check for a new version look like if the version is not listed in the instructions?
>
> Either look into the .packages file on our master branch, or use dpkg in a Tails system, or use APT to query the tagged APT snapshot for the latest Tails release.

All of these methods need manual effort. I think the process will be a lot faster if we provide the version to compare with in the document. And since we don’t expect many updates, I expect the overhead of keeping this list up to date to be less than the time we save each time we check for updates.

> Now, if you prefer to keep this duplicated info on this document, fine, but please add steps to update it.

Done.

> >> * Ideally this would not be part of the release process: it feels a bit too late to act, there won’t be any 3rd-party QA, and I don’t think this should be on the RM’s plate anyway. To me it looks much more like FT work. But we have no good way to track the need for such updates yet (Feature #14728 will help) so unless you have a better idea, let’s keep things as-is and add a note about it on Feature #14728.
>
> > OK, I don’t have a better idea.
>
> Fine, please update Feature #14728 then :)

Done.

> Also, I’ve noticed some regressions since the last version I’ve reviewed:
>
> * I believe apt source -t ${NEW_VERSION} ${PACKAGE_NAME} won’t work. Maybe you meant apt source ${PACKAGE_NAME}=${NEW_VERSION}?
> * without "bugfix/" or "feature/" feels wrong.
> * All these unquoted variables hurt my eyes. Yeah, shell scripting is hard :/ Our preferred/standard way to write variables in such documents is ${VARIABLE:?} which is basically equivalent to set -u ; "$VARIABLE" i.e. an error will be raised if whoever pastes these command lines has not set $VARIABLE (mistakes do happen :)

Fixed.

#17 Updated by segfault 2018-10-14 15:01:22

  • Status changed from Confirmed to In Progress

Applied in changeset commit:79a140cae3f2fe02127d3472854223abf7cf5cce.

#18 Updated by intrigeri 2018-10-15 11:40:04

  • Assignee changed from intrigeri to segfault
  • % Done changed from 0 to 80

Great! I’ve fixed with commit:8e03989e38f4b194290152a15f28df7815ecb91e a regression that had been introduced by the commit about quoting and merged into master. Please review my fix.

#19 Updated by segfault 2018-10-15 19:10:29

  • Assignee changed from segfault to intrigeri
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:
> Great! I’ve fixed with commit:8e03989e38f4b194290152a15f28df7815ecb91e a regression that had been introduced by the commit about quoting and merged into master. Please review my fix.

Done, thanks for fixing this

#20 Updated by intrigeri 2018-10-15 19:31:34

  • Status changed from In Progress to Fix committed
  • Assignee deleted (intrigeri)
  • % Done changed from 80 to 100

We’re done here => doing the paperwork.

#21 Updated by intrigeri 2018-10-15 21:10:06

  • Status changed from Fix committed to In Progress

Applied in changeset commit:8e03989e38f4b194290152a15f28df7815ecb91e.

#22 Updated by intrigeri 2018-10-15 21:10:07

  • Status changed from In Progress to Resolved

Applied in changeset commit:3c6224f5cb747c1a561e26856e15dbae964aeeba.