Bug #16226

Most branches FTBFS on Jenkins since the 3.11 release

Added by intrigeri 2018-12-15 17:27:18 . Updated 2019-01-27 10:55:28 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Build system
Target version:
Start date:
2018-12-15
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

It seems that the 3.11 release process was left in a state that breaks the build of most branches: the only branch that currently builds on Jenkins is the devel one. If you need help to get things back into working order, let me know :)


Subtasks


Related issues

Related to Tails - Bug #16060: Improve ASP code: configuration window Rejected 2018-10-16
Blocks Tails - Bug #16110: Button to remove package in ASP GUI has a wrong label Resolved 2018-11-08

History

#1 Updated by CyrilBrulebois 2018-12-16 15:41:51

If I didn’t screw up too much, apt repositories and git branches should be fine by now:

  • devel was already fine, as noted.
  • stable no longer fails immediately (but the build is still running).
  • with this updated stable branch, I suppose other branches getting merged will get better over time.

Notes about the Prepare for the next development cycle section:

  • given Bug #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.
  • for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

Keeping this ticket assigned to me until Jenkins has had a chance to catch up, and until there’s agreement we should delete 3.10.1 bits entirely.

Another thing that needs doing is merging devel into feature/buster, but that was filed already (Bug #16212).

#2 Updated by CyrilBrulebois 2018-12-16 15:41:58

  • Status changed from Confirmed to In Progress

#3 Updated by CyrilBrulebois 2018-12-16 15:49:37

Apparently I failed at understanding how to revert freeze exceptions (release_process.mdwn and friends have some instructions, but the actual config/chroot_apt/preferences had different instructions for the kernel)…

+ apt-get install --yes build-essential libelf-dev linux-headers-4.18.0-3-amd64                                      
E: Unable to correct problems, you have held broken packages.                                                        
Reading package lists...                                                                                             
Building dependency tree...                                                                                          
Reading state information...                                                                                         
Some packages could not be installed. This may mean that you have                                                    
requested an impossible situation or if you are using the unstable                                                   
distribution that some required packages have not yet been created                                                   
or been moved out of Incoming.                                                                                       
The following information may help to resolve the situation:                                                         

The following packages have unmet dependencies:                                                                      
 linux-headers-4.18.0-3-amd64 : Depends: linux-kbuild-4.18 (>= 4.18.20-2) but 4.18.10-2 is to be installed           
E: Unable to correct problems, you have held broken packages.                                                        
E: config/chroot_local-hooks/12-kernel-modules-build-environment failed (exit non-zero). You should check for errors.

#4 Updated by alant 2018-12-16 17:09:02

  • blocks Bug #16034: ASP: Fix race condition when writing to packages file added

#5 Updated by alant 2018-12-16 17:10:46

  • blocks Bug #16110: Button to remove package in ASP GUI has a wrong label added

#6 Updated by alant 2018-12-16 17:44:06

  • related to Bug #16060: Improve ASP code: configuration window added

#7 Updated by intrigeri 2018-12-17 11:46:30

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check set to Info Needed

I kinda remember this part of our doc is, let’s say, “suboptimal” :/ I’ll check that doc and will either fix it or provide here the info you’re missing.

#8 Updated by intrigeri 2018-12-17 11:50:26

> * given Bug #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.

Now that a workaround was documented, please go ahead.

> * for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

Great, that’s actually the right thing to do. I’ll check our release process doc and will clarify/fix the relevant bits. That’s a PITA and I’m sorry about it, but I’m confident we’re currently improving that doc much faster than we’re adding unclear/buggy bits to it, thanks to your input!

#9 Updated by intrigeri 2018-12-17 13:59:34

> I kinda remember this part of our doc is, let’s say, “suboptimal” :/ I’ll check that doc and will either fix it or provide here the info you’re missing.

I’ve tried to fix the doc about freeze exceptions; feedback welcome! So I think we should now:

  1. revert commit:049701430a24bc3ebe430837acc0b00f9f2ed8ee on stable: as I’ve tried to clarify in commit:a48e24875bc609255b2c784ba3b5d5606074d7e9, we shall not lift freeze exceptions on branches that are still frozen
  2. merge stable into master
  3. merge stable into devel
  4. on devel, revert the revert of 049701430a24bc3ebe430837acc0b00f9f2ed8ee

#10 Updated by intrigeri 2018-12-17 14:02:32

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Info Needed to Dev Needed

>> * for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

> Great, that’s actually the right thing to do. I’ll check our release process doc and will clarify/fix the relevant bits.

May I ask what lead you to think it was customary to add a 3.13 changelog entry on stable?

I see we have:

* increment the version number in stable's `debian/changelog` to match
the next bugfix release, so that
next builds from the `stable` branch do not use the APT suite meant
for the last release:

cd "${RELEASE_CHECKOUT}" && \
git checkout stable && \
dch --newversion "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
"Dummy entry for next release." && \
git commit debian/changelog \
-m "Add dummy changelog entry for ${NEXT_PLANNED_BUGFIX_VERSION:?}."

And

* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
 *bugfix* Tails release; if the next release is a bugfix release, use
that one; otherwise, use `${VERSION}.1`

So given the next release (3.12) is a major one, NEXT_PLANNED_BUGFIX_VERSION should be set to 3.11.1, no?

Happy to fix the doc once I understand where the confusion comes from :)

#11 Updated by CyrilBrulebois 2018-12-17 22:12:15

Failure to read the documentation, I suppose. :/

Train of thoughts was probably: Planned = look at calendar, skip 3.12 as that’s major, and encode 3.13. Wasn’t caught by peer review.

#12 Updated by intrigeri 2018-12-18 08:17:18

> Train of thoughts was probably: Planned = look at calendar, skip 3.12 as that’s major, and encode 3.13. Wasn’t caught by peer review.

OK, thanks! Well, this happens and the impact is basically non-existent (you’ve self-corrected it where it mattered later) so let’s say the doc is good enough.

#13 Updated by CyrilBrulebois 2018-12-18 13:26:56

intrigeri wrote:
> > * given Bug #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.
>
> Now that a workaround was documented, please go ahead.

Apparently we’re getting reports it’s not working, so I won’t be deleting it right away?

#14 Updated by CyrilBrulebois 2018-12-18 13:29:45

intrigeri wrote:
> I’ve tried to fix the doc about freeze exceptions; feedback welcome! So I think we should now:
>
> # revert commit:049701430a24bc3ebe430837acc0b00f9f2ed8ee on stable: as I’ve tried to clarify in commit:a48e24875bc609255b2c784ba3b5d5606074d7e9, we shall not lift freeze exceptions on branches that are still frozen
> # merge stable into master
> # merge stable into devel
> # on devel, revert the revert of 049701430a24bc3ebe430837acc0b00f9f2ed8ee

Thanks for the commit, I probably wouldn’t have made that mistake with this devel vs. stable clarification.

Should I go ahead with the steps you proposed, build things locally, and push branches if everything seems to build fine?

(As a side note, and as you probably noticed already, linux 4.19.9-1 landed in unstable yesterday.)

#15 Updated by intrigeri 2018-12-19 08:00:17

>> > * given Bug #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.
>>
>> Now that a workaround was documented, please go ahead.

> Apparently we’re getting reports it’s not working, so I won’t be deleting it right away?

It’s a tough one. In general we never leave obsolete images around because we don’t want people to use a Tor Browser with known security vulns, and finding + verifying the old image is just too hard for the vast majority of our users anyway. Most people would be safer using an up-to-date Tor Browser outside of Tails than using Tails 3.10.1, but there are obviously downsides. I would let our help desk decide: if they are actively suggesting users to downgrade their Tails to 3.10.1 (I would be surprised), then keep it around; otherwise, delete.

#16 Updated by intrigeri 2018-12-19 08:00:51

> Should I go ahead with the steps you proposed, build things locally, and push branches if everything seems to build fine?

Just go ahead and push, no need to build locally first.

> (As a side note, and as you probably noticed already, linux 4.19.9-1 landed in unstable yesterday.)

Yep.

#17 Updated by CyrilBrulebois 2018-12-19 09:10:48

Pushed all three branches; I had triggered builds past night anyway, and from checking build manifests, the stable one looks good (only a bump in serial for debian-security), as well as the devel one (a bunch of binaries from a single source package got upgraded compared to the last devel build in jenkins, because of an update in backports).

#18 Updated by intrigeri 2018-12-19 10:05:03

> Pushed all three branches; I had triggered builds past night anyway, and from checking build manifests, the stable one looks good (only a bump in serial for debian-security), as well as the devel one (a bunch of binaries from a single source package got upgraded compared to the last devel build in jenkins, because of an update in backports).

woohoo! \o/

#19 Updated by CyrilBrulebois 2018-12-19 13:17:23

Jenkins looks way greener (or blue-er) now, sorry it took so long; just didn’t want to risk having to revert the revert of the revert… etc. by rushing anything. The commit history is messy enough already.

#20 Updated by alant 2018-12-21 15:39:47

CyrilBrulebois wrote:
> Jenkins looks way greener (or blue-er) now,

Thanks a lot!

> sorry it took so long; just didn’t want to risk having to revert the revert of the revert… etc. by rushing anything. The commit history is messy enough already.

Don’t worry.

#21 Updated by alant 2018-12-21 15:48:05

  • blocked by deleted (Bug #16034: ASP: Fix race condition when writing to packages file)

#22 Updated by intrigeri 2019-01-27 10:55:28

  • Status changed from In Progress to Resolved
  • Assignee deleted (CyrilBrulebois)
  • QA Check deleted (Dev Needed)

intrigeri wrote:
> >> > * given Bug #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.
> >>
> >> Now that a workaround was documented, please go ahead.
>
> > Apparently we’re getting reports it’s not working, so I won’t be deleting it right away?
>
> It’s a tough one. In general we never leave obsolete images around because we don’t want people to use a Tor Browser with known security vulns, and finding + verifying the old image is just too hard for the vast majority of our users anyway. Most people would be safer using an up-to-date Tor Browser outside of Tails than using Tails 3.10.1, but there are obviously downsides. I would let our help desk decide: if they are actively suggesting users to downgrade their Tails to 3.10.1 (I would be surprised), then keep it around; otherwise, delete.

Help Desk has not suggested that but anyway, it’s too late to delete that ISO: it’ll go away on Tuesday at the same time as 3.11.

The main problem this ticket was about is resolved.