Bug #14459

Some branches fail to build an ISO because we merge their base branch too late

Added by intrigeri 2017-08-28 07:55:37 . Updated 2017-09-13 06:14:32 .

Status:
Resolved
Priority:
Normal
Assignee:
intrigeri
Category:
Continuous Integration
Target version:
Tails_3.2
Start date:
2017-08-28
Due date:
% Done:

100%

Feature Branch:
bugfix/14459-merge-base-branch-earlier
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Our ISO build system errors out when building from a branch based on master:

> + apt-snapshots-serials prepare-build
> + apt-mirror debian
> Not building from a tag, but last version in changelog was released
> + DEBIAN_MIRROR=
> + apt-mirror debian-security
> Not building from a tag, but last version in changelog was released
> + DEBIAN_SECURITY_MIRROR=
> + apt-mirror torproject
> Not building from a tag, but last version in changelog was released
> + TORPROJECT_MIRROR=
> + [ -n  ]
> + exit 1

I believe this is caused by auto/config running apt-snapshots-serials prepare-build before auto/build had a chance to merge the base branch (generally: stable) into the topic branch that’s being built: a branch based on master generally satisfies the “Not building from a tag, but last version in changelog was released” error condition until their base branch is merged into it.

The simplest, ad-hoc fix would probably be to move a chunk of code from auto/config to auto/build. I’ll handle this.

But thinking about the root cause, a more fundamental issue becomes obvious: merging the base branch happens too late, and by running a fair amount of (build) code before we do that, we create a frankenstein ISO whose build system partly comes from the current status of the topic branch, and partly comes from the current status of it base branch. IMO we should do the base branch merge first thing in the build process, in a manner that’s isolated from other build steps, so that all build code (including Rakefile) except the tiny script that performs the merge will be. anonym, bertagaz: if you agree we have a fundamental problem here, please file a ticket about it.

Subtasks

Related issues

Related to Tails - Bug #12557: The mergebasebranch option does the base branch merge too late Confirmed 2017-05-17
Related to Tails - Bug #14857: Some doc branches fail to build an ISO because the base branch of master is "testing" after a major release Resolved 2017-10-17
Blocks Tails - Feature #11355: Re-enable Jenkins notifications on ISO build/test failure In Progress 2017-08-28
Blocks Tails - Feature #13234: Core work 2017Q3: Foundations Team Resolved 2017-06-29
Blocks Tails - Bug #12641: Comment changes in POT files make ISO builds non-reproducible Resolved 2017-06-05

History

#1 Updated by intrigeri 2017-08-28 07:55:57

  • blocks Feature #11355: Re-enable Jenkins notifications on ISO build/test failure added

#2 Updated by intrigeri 2017-09-02 09:30:26

  • Subject changed from Some documentation branches fail to build on Jenkins to Some documentation branches fail to build an ISO

#3 Updated by intrigeri 2017-09-02 09:56:55

BTW, the same root cause is shared by another recurring painful problem: topic branches stop building once the APT snapshots they encode disappears, even though their base branch encodes newer, valid APT snapshots.

#4 Updated by intrigeri 2017-09-02 10:05:05

  • Description updated

#5 Updated by intrigeri 2017-09-02 10:09:08

  • Subject changed from Some documentation branches fail to build an ISO to Some branches fail to build an ISO because we merge their base branch too late

#6 Updated by intrigeri 2017-09-02 10:25:48

#7 Updated by intrigeri 2017-09-02 10:48:52

  • related to Bug #12557: The mergebasebranch option does the base branch merge too late added

#8 Updated by intrigeri 2017-09-02 11:39:02

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA
  • Feature Branch set to bugfix/14459-merge-base-branch-earlier

I’ve tested my fix this way a branch forked off current master, whose base branch is set to stable, fails to build as expected (it has commits on top of the last entry in debian/changelog); cherry-picking my commit on top of that branch fixes the problem.

I wanted to also test a topic branch whose APT snapshots have expired, but this will still fail even with my fix unless the corresponding Vagrant box already exists: we’re still building it before we merge the base branch. So something more involved (Bug #12557) is needed to address this problem.

FWIW I’ve also verified that (locally) after merging my branch into stable and devel doesn’t break the build; I didn’t wait for the build to finish, I’ve only waited for lb build to have been run by auto/build.

The topic branch is based on master on purpose, as it should be merged into master (to fix the original problem this ticket was about) in addition to the usual stable & devel. (Once this is done we may have to also merge its base branch into every active topic branch; we’ll see.)

#9 Updated by intrigeri 2017-09-04 20:13:19

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Hold on! This branch breaks reproducibility on Jenkins (not for actual releases though, I think) because it encodes BUILD_BASENAME, that contains a timestamp, in /usr/share/amnesia/build/variables.

#10 Updated by intrigeri 2017-09-05 06:59:50

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Dev Needed to Ready for QA

#11 Updated by intrigeri 2017-09-11 12:00:31

  • blocks Bug #12641: Comment changes in POT files make ISO builds non-reproducible added

#12 Updated by anonym 2017-09-12 17:48:23

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

Applied in changeset commit:f5ed6d9ae1e50e6d3cb65db89de24a4a21e637f9.

#13 Updated by anonym 2017-09-12 17:53:12

  • Assignee deleted (anonym)
  • QA Check changed from Ready for QA to Pass

I agree this isn’t ideal and we at some point will need Bug #12557, but the feature branch still clearly improves the situation significantly, so => merged!

#14 Updated by anonym 2017-09-12 18:54:12

  • Assignee set to intrigeri
  • % Done changed from 100 to 90
  • QA Check changed from Pass to Ready for QA

So while I verified that your branch could successfully build an ISO image, I didn’t look closely at their file names. I’ve already pushed a fixup with commit:1bef9097d16d94752e6a0657a2230fc09c544691. Please have a look!

#15 Updated by anonym 2017-09-12 21:55:47

  • Status changed from Fix committed to In Progress

Applied in changeset commit:1bef9097d16d94752e6a0657a2230fc09c544691.

#16 Updated by intrigeri 2017-09-13 06:14:32

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100
  • QA Check changed from Ready for QA to Pass

Looks OK for now as we hardcode amd64 in a few other places anyway. Thanks!

#17 Updated by intrigeri 2017-10-17 16:28:20

  • related to Bug #14857: Some doc branches fail to build an ISO because the base branch of master is "testing" after a major release added