Feature #11966

Reproducible website build

Added by intrigeri 2016-11-19 15:57:13 . Updated 2017-03-14 11:02:25 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
2016-11-19
Due date:
% Done:

100%

Feature Branch:
web/11966-deterministic-website
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:
289

Description

We include a local copy of the website in the ISO image. It must build reproducibly, somehow. ikiwiki has a deterministic option that should help.


Subtasks


Related issues

Related to Tails - Bug #11987: ikiwiki includes "Posted" timestamps in some generated web pages Resolved 2016-11-22
Related to Tails - Feature #12051: Use ikiwiki 3.20161219+ instead of our patched one Resolved 2016-12-21
Related to Tails - Bug #13531: Use ikiwiki 3.20161219+, again Duplicate 2017-07-29
Blocks Tails - Bug #12032: The SquashFS creation is not deterministic Resolved 2016-12-14

History

#1 Updated by intrigeri 2016-11-19 15:59:48

  • Feature Branch set to web/11966-deterministic-website
  • Deliverable for set to 289

#2 Updated by intrigeri 2016-11-19 16:20:00

  • Assignee set to intrigeri

#3 Updated by intrigeri 2016-11-20 07:17:21

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Progress!

  • ikiwiki’s “deterministic” option was enabled; not sure if it buys us much, but it should only improve things;
  • the pagestats output issue was reported with a patch: https://ikiwiki.info/bugs/pagestats_output_is_not_deterministic/ — next step is probably to upload a patched ikiwiki to our builder-jessie overlay APT suite;
  • the email address mangling issue was fixed on the topic branch (by using a new “discount” package), that I’ll merge into feature/5630-deterministic-builds once anonym has fixed his Git history there; same for the “Created” timestamp issue.

#4 Updated by intrigeri 2016-11-20 08:10:29

Once my branch will be merged and a patched ikiwiki uploaded, it looks like the “Last edited” timestamp may be the only remaining problem.

#5 Updated by intrigeri 2016-11-20 08:49:41

  • % Done changed from 10 to 20

intrigeri wrote:
> * the pagestats output issue was reported with a patch: https://ikiwiki.info/bugs/pagestats_output_is_not_deterministic/ — next step is probably to upload a patched ikiwiki to our builder-jessie overlay APT suite;

Uploaded, and web/11966-deterministic-website ensures it is installed in the Vagrant build box.

> it looks like the “Last edited” timestamp may be the only remaining problem.

Fixed on the topic branch.

#6 Updated by intrigeri 2016-11-20 11:03:57

  • Assignee changed from intrigeri to anonym
  • Target version set to Tails_2.9.1
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

I did two ISO builds with cleanall from feature/5630-deterministic-builds, that has web/11966-deterministic-website merged in, and the resulting SquashFS’es contain exactly the same website (empty diff output). So please merge the topic branch into master.

#7 Updated by anonym 2016-11-21 15:13:10

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Looks great!

#8 Updated by intrigeri 2016-11-22 08:07:49

  • Status changed from Fix committed to Resolved

This was merged into master.

#9 Updated by intrigeri 2016-11-22 08:07:58

  • related to Bug #11987: ikiwiki includes "Posted" timestamps in some generated web pages added

#10 Updated by sajolida 2016-11-22 12:45:16

I’m quite happy we got rid of the “Last edited” timestamp!

#11 Updated by intrigeri 2016-12-21 08:52:23

  • related to Feature #12051: Use ikiwiki 3.20161219+ instead of our patched one added

#12 Updated by intrigeri 2017-03-14 06:37:23

  • Status changed from Resolved to In Progress
  • Assignee set to intrigeri
  • Target version deleted (Tails_2.9.1)
  • % Done changed from 100 to 50
  • QA Check deleted (Pass)

Actually it’s still not 100% reproducible: some of the pages that list other pages (with inline) produce different results, apparently depending on the mtime of source files. I’m on it.

#13 Updated by intrigeri 2017-03-14 06:41:03

  • blocks Bug #12032: The SquashFS creation is not deterministic added

#14 Updated by intrigeri 2017-03-14 11:02:25

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

I’ve fixed all issues I’ve seen happen so far. Will reopen if more problems show up.

#15 Updated by intrigeri 2017-07-29 07:04:43

  • related to Bug #13531: Use ikiwiki 3.20161219+, again added