Bug #14933

stable branch is not reproducible: differences in some .fa.html website files

Added by intrigeri 2017-11-08 10:38:08 . Updated 2018-01-09 20:52:04 .

Status:
Resolved
Priority:
High
Assignee:
Category:
Target version:
Start date:
2017-11-08
Due date:
% Done:

100%

Feature Branch:
bugfix/14933-non-deterministic-inline-posted-timestamp
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:
301


Files


Subtasks


Related issues

Related to Tails - Bug #12726: There should be a date on the notes in the News section of the website Resolved 2017-06-17

History

#1 Updated by intrigeri 2017-11-08 10:39:51

I’ll have a very quick first look in the hope I can easily find out where the root cause is. If the fix is not trivial I’ll coordinate with anonym to see who handles this.

#2 Updated by intrigeri 2017-11-08 11:23:33

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 10
  • Type of work changed from Research to Code

I think we didn’t notice this earlier because the fix for Bug #12726 had been merged into master but not into stable. Yesterday I’ve merged master into stable (precisely thinking about such potential problems) which is why we now see it. I’m glad we noticed this before we’re closer to the 3.3 release.

So, in the source of affected pages we have [[!inline pages="doc/encryption_and_privacy/gpgapplet.warning" raw="yes" sort="age"]]. The mtime of wiki/src/doc/encryption_and_privacy/gpgapplet.warning.fa.po is used as the timestamp I see in the diffoscope output matches:

08:08:29 + TAILS_GIT_DIR=/home/vagrant/amnesia
08:08:29 + [ ! -d /home/vagrant/amnesia ]
08:08:29 + git clone /amnesia.git/.git /home/vagrant/amnesia
08:08:29 Cloning into '/home/vagrant/amnesia'...
08:08:34 done.

I think the mtime is used (instead of the time of last change in Git) because our ikiwiki.setup disables the rcs parameter. I suspect we should normalize such timestamps of files in wiki/src before building the website in auto/build, just like we already do for config/binary_local-includes and config/chroot_local-includes. anonym, wanna give it a try or should I?

Now, honestly I’m a bit confused: I don’t understand why this problem only appears for this specific inlined page and only in Farsi. The way I understand the problem currently, all inlined pages without an explicit meta date should trigger the exact same problem.

Might it be the problem I was refering to in “non-deterministic inlined posts’ date: any inline can be affected as long as it uses a template that displays the date; this is the problem that made us do 104d7f89ffd0a83962eb6eb891e1c2d390cf62b0 initially; your validation code addresses that… as long as it covers all affected inline directives; I think it does” (Bug #12726)? Apparently I was wrong and we don’t check all affected inline directives. But if merely normalizing mtimes works, let’s not bother.

#3 Updated by intrigeri 2017-11-08 11:23:43

  • related to Bug #12726: There should be a date on the notes in the News section of the website added

#4 Updated by intrigeri 2017-11-10 12:18:06

  • Assignee changed from anonym to intrigeri

#5 Updated by intrigeri 2017-11-10 12:41:11

  • QA Check set to Ready for QA
  • Feature Branch set to bugfix/14933-non-deterministic-inline-posted-timestamp

(I have to make this RfQA otherwise Jenkins won’t test reproducibility.)

#6 Updated by intrigeri 2017-11-10 15:39:22

  • Assignee changed from intrigeri to anonym
  • % Done changed from 10 to 50

Woohoo, my fix works. The ISO is still not reproducible in our CI environment:
https://jenkins.tails.boum.org/view/Tails_ISO/job/reproducibly_build_Tails_ISO_bugfix-14933-non-deterministic-inline-posted-timestamp/1/artifact/build-artifacts/diffoscope.html but for a different reason, that I’ve reported on Bug #14946. I think this problem won’t affect release builds but I’m not 100% sure and anyway I’ve found a rather trivial fix. In any case that should not block fixing this problem.

#7 Updated by intrigeri 2017-11-10 15:42:48

  • blocks Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option added

#8 Updated by anonym 2017-11-11 14:53:44

intrigeri wrote:
> Woohoo, my fix works.

Yay!

> The ISO is still not reproducible in our CI environment:
> https://jenkins.tails.boum.org/view/Tails_ISO/job/reproducibly_build_Tails_ISO_bugfix-14933-non-deterministic-inline-posted-timestamp/1/artifact/build-artifacts/diffoscope.html but for a different reason, that I’ve reported on Bug #14946. I think this problem won’t affect release builds but I’m not 100% sure

I’m 99% sure this isn’t a problem for releases: they are built from a tag which makes mergebasebranch into a noop.

#9 Updated by intrigeri 2017-11-11 16:58:47

  • Assignee changed from anonym to bertagaz

#10 Updated by bertagaz 2017-11-13 11:09:47

  • Status changed from In Progress to Fix committed
  • Assignee deleted (bertagaz)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Apparently that seems to do the trick, so it’s been merged together with Bug #14946.

#11 Updated by bertagaz 2017-11-15 10:38:10

  • Status changed from Fix committed to Confirmed
  • Target version changed from Tails_3.3 to Tails_3.5
  • % Done changed from 100 to 70
  • QA Check changed from Pass to Dev Needed

bertagaz wrote:
> Apparently that seems to do the trick, so it’s been merged together with Bug #14946.

During the 3.3 release, we noticed this problem was not really gone: when $TAILS_MERGE_BASE_BRANCH is not set, the wiki is built in vagrant/provision/assets/build-tails, where the timestamps have not been yet cleaned. When it gets refreshed later in auto/build, the problem with the openpgp_applet fa.html files is not fixed.

#12 Updated by intrigeri 2017-11-15 10:59:35

  • Assignee set to anonym
  • Priority changed from Elevated to High

#13 Updated by anonym 2017-11-15 11:37:20

  • blocked by deleted (Bug #14946: Topic branches that lag behind their base branch don't build reproducibly with mergebasebranch build option)

#14 Updated by intrigeri 2017-12-07 11:49:21

anonym will investigate this by December 11 and then might delegate it to me.

#15 Updated by anonym 2017-12-12 07:51:43

intrigeri wrote:
> anonym will investigate this by December 11 and then might delegate it to me.

I’ll look at it today (off-by-one error, whoops! :)).

#16 Updated by anonym 2017-12-12 11:42:42

bertagaz wrote:
> bertagaz wrote:
> > Apparently that seems to do the trick, so it’s been merged together with Bug #14946.
>
> During the 3.3 release, we noticed this problem was not really gone: when $TAILS_MERGE_BASE_BRANCH is not set, the wiki is built in vagrant/provision/assets/build-tails, where the timestamps have not been yet cleaned. When it gets refreshed later in auto/build, the problem with the openpgp_applet fa.html files is not fixed.

I can confirm that building with mergebasebranch produces the same ISO each time on my system:

md5:     25adfb21ffed211ac154aa6669d87163  tails-amd64-3.3.iso
sha1:    7bf7f11f206803d4690dfa1ce7d4da35735ea262  tails-amd64-3.3.iso
sha256:  82b1286442be5f72d68d36166a352de7b12472d0e94692ca64fcc6e6a113e343  tails-amd64-3.3.iso
sha512:  0ef9de2d3965c06738a4556d8f6d4fa5aa9fe1196069c0a624de23558bfb141b566b6473078e50a3f89c62fb0da792609415dcf73429c69181ee32adf97fe005  tails-amd64-3.3.iso

So, most likely Jenkins built that exact image, but the artifacts from all jobs built from that commit are absent. I’ve instead asked jvoison to rebuild Tails 3.3 with mergebasebranch set. If he gets the same SHAAAA as me, we’re good, imho.

As for the actual fix, see commit:3a622350a5 which removes the early wiki build — I’m probably the only one using it (together with keeprunning), which isn’t very often, and if I ever really need it I can just drop ./build-wiki into build-tails temporarily.

#17 Updated by anonym 2017-12-12 12:07:14

  • Status changed from Confirmed to In Progress

Applied in changeset commit:3a622350a5525d5e5e17efe8f40bcf145b8d4549.

#18 Updated by cypherpunks 2017-12-12 13:00:22

$ TAILS_BUILD_OPTIONS="mergebasebranch" rake build 
[…]
$ md5sum tails-amd64-3.3.iso
25adfb21ffed211ac154aa6669d87163  tails-amd64-3.3.iso
$ sha1sum tails-amd64-3.3.iso
7bf7f11f206803d4690dfa1ce7d4da35735ea262  tails-amd64-3.3.iso
$ sha256sum tails-amd64-3.3.iso
82b1286442be5f72d68d36166a352de7b12472d0e94692ca64fcc6e6a113e343  tails-amd64-3.3.iso
$ sha512sum tails-amd64-3.3.iso
0ef9de2d3965c06738a4556d8f6d4fa5aa9fe1196069c0a624de23558bfb141b566b6473078e50a3f89c62fb0da792609415dcf73429c69181ee32adf97fe005  tails-amd64-3.3.iso
$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:    17.04
Codename:   zesty
$ uname -a
Linux jv 4.10.0-42-generic #46-Ubuntu SMP Mon Dec 4 14:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$                                            

#19 Updated by anonym 2017-12-13 16:48:41

  • Assignee changed from anonym to intrigeri
  • % Done changed from 70 to 80
  • QA Check changed from Dev Needed to Ready for QA

Success! Please review’n’merge into stable!

#20 Updated by intrigeri 2017-12-13 18:54:38

Code review passes. I’ve updated the branch so it has a chance to build on Jenkins.

#21 Updated by intrigeri 2017-12-14 07:22:25

  • Status changed from In Progress to Fix committed
  • % Done changed from 80 to 100

Applied in changeset commit:19befbd9ca6955b52610a8a720e67f191b1cc566.

#22 Updated by intrigeri 2017-12-14 07:23:18

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

Merged! Let’s see how it goes next time we release :)

#23 Updated by intrigeri 2018-01-04 18:26:34

  • Target version changed from Tails_3.5 to Tails_3.4

#24 Updated by anonym 2018-01-09 20:52:04

  • Status changed from Fix committed to Resolved