Tails

#1 Updated by intrigeri 2017-05-19 11:18:34

• related to Feature #11971: Consider migrating some of /lib/live/config/* to systemd unit files added

#2 Updated by intrigeri 2017-05-19 11:19:26

• Description updated

#3 Updated by intrigeri 2017-05-19 11:23:33

• File 12567.tar.bz2 added
• Assignee changed from intrigeri to lamby

Here are the 2 sets of /var/cache/fontconfig files + the diffoscope output.

Chris:

• would you have time to work on this by June 5, and ideally earlier (in my dreamworld this would be solved by the end of May);
• how are you doing wrt. the time budget we agreed on in March? if there’s nothing left, how much time do you think we should allocate for this issue?

Sorry for the timing, I’ve just learnt about this issue today :(

#8 Updated by lamby 2017-05-26 17:47:10

• File diff.txt added

The root cause is that fontconfig embeds the mtime of each font directory in a “checksum” member of a “_FcCache” struct. This is so that it can identify which cache files remain valid and/or require regeneration.

Unfortunately, we can’t just replace the checksum value with SOURCE_DATE_EPOCH as it will mean the cache files will not be valid from fontconfig’s PoV at runtime, defeating the entire point of generating them.

Took me a little while to track that down and then work through a number of solutions. I’ve attached something approximating the cleanest one with the smallest diff. Enjoy :)

FYI this also resulted in the following bug being filed: https://bugs.debian.org/863427

#9 Updated by intrigeri 2017-05-27 07:13:42

• Assignee changed from lamby to anonym
• QA Check set to Ready for QA

Thanks!

anonym, I expect you’ll evaluate this with diffoscope locally, and then once it’s merged into testing https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_testing/ should help (although these jobs still have issues — Bug #12579 — I’ve seen the one for the testing branch work fine once yesterday).

#11 Updated by anonym 2017-05-31 19:47:03

• Status changed from Confirmed to In Progress
• Assignee changed from anonym to lamby
• % Done changed from 0 to 20
• QA Check changed from Ready for QA to Info Needed

I fail to build a package with your patch applied. I tried:

apt source fontconfig=2.11.0-6.7.0tails1
cd fontconfig-2.11.0
quilt import /tmp/10-fc-cache-list-dirs.patch
dch
pdebuild

but then I get:

[...]
patching file debian/fontconfig.postinst
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored
patching file fc-cache/fc-cache.1
patching file fc-cache/fc-cache.c
patching file fc-cache/fc-cache.sgml
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch '10-fc-cache-list-dirs.patch' is correctly applied by quilt, use 'quilt refresh' to update it
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -E -b -B .pc/10-fc-cache-list-dirs.patch/ --reject-file=- < fontconfig-2.11.0.orig.ydiTR1/debian/patches/10-fc-cache-list-dirs.patch gave error exit status 1

This is not a reversed patch AFAICT, so I really don’t get what is going on. Any ideas?

#12 Updated by anonym 2017-05-31 19:53:23

• Assignee changed from lamby to intrigeri

IMHO this (i.e. me being a Debian packaging n00b) is a crappy thing to be blocked by lamby on, so if you can think of what I should look closer at, perhaps this can be resolved quickly without disturbing lamby. :)

#13 Updated by anonym 2017-05-31 19:53:37

• Assignee changed from intrigeri to lamby

#14 Updated by lamby 2017-05-31 20:01:32

• Assignee changed from lamby to anonym

Perhaps quilt import does not work when we are patching files under debian/? Try with regular “patch -p1 < diff.txt”?

#15 Updated by anonym 2017-05-31 21:08:08

• QA Check deleted (Info Needed)

lamby wrote:
> Perhaps quilt import does not work when we are patching files under debian/? Try with regular “patch -p1 < diff.txt”?

Ah, now I get it, thanks a lot! So I extracted the debian/ parts of that patch and applied it directly. I’ve got a package built now that I will test.

#16 Updated by anonym 2017-05-31 22:41:10

• File 12567-2.tar.gz added
• Assignee changed from anonym to lamby
• QA Check set to Dev Needed

Sadly it seems your patch is not enough, but it greatly reduces the number of differences: now I get 5; I used to get 75. :)

See attached tarball for the offending files + diffoscope report.

#18 Updated by lamby 2017-06-01 06:38:21

• Assignee changed from lamby to anonym

Thanks fo the update! Two things:

a) Can you let me know how you performed the test, exactly?

b) I can probably look at this this evening, otherwise tomorrow.

#19 Updated by intrigeri 2017-06-01 06:47:53

• Feature Branch set to bugfix/12567-fontconfig-fixup

I’m adding a CI reproducibility test for the topic branch. anonym, please point lamby to the artifacts (on nightly.t.b.o) once it has run once :)

#20 Updated by intrigeri 2017-06-01 09:40:29

• Assignee changed from anonym to lamby

lamby wrote:
> a) Can you let me know how you performed the test, exactly?

https://nightly.tails.boum.org/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/builds/2017-06-01_06-48-13/archive/build-artifacts/ has the build log and diffoscope output. Don’t hesitate (re-)asking anonym if that’s not enough info :)

#21 Updated by lamby 2017-06-02 17:48:29

• File diff.txt added
• Assignee changed from lamby to anonym

Updated diff attached.

FYI I did some diffoscope work to help diagnose this:

https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?h=experimental&id=df8360b1230c0c019cc8f0faa0f50cb5b1f7c265

#22 Updated by intrigeri 2017-06-02 18:30:48

• QA Check changed from Dev Needed to Ready for QA

Excellent, thanks :)

#23 Updated by anonym 2017-06-02 23:15:01

• File 12567-3.tar.bz2 added
• Assignee changed from anonym to lamby
• QA Check changed from Ready for QA to Dev Needed

I still see the “same” 5 differences after building new packages with that one-line fixup to debian/fontconfig.postinst applied. :/

See the attached tarball for the full fontconfig cache + diffoscope report.

An ETA would be appreciated again!

For the record, I started a job for this on Jenkins which should be done within two hours and hopefully reproduces (hah!) the same differences: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/3/

#24 Updated by lamby 2017-06-03 07:59:21

• File diff.txt added
• Assignee changed from lamby to anonym

anonym wrote:
> I still see the “same” 5 differences after building new packages with that one-line fixup to debian/fontconfig.postinst applied. :/

Hah, so I woke up in the night thinking that I missed something about deferencing the symlinks…

Alas, no time to test this this morning (but can probably find time later) but I thiiiink the attached should fix it.

#27 Updated by anonym 2017-06-03 15:28:33

• Status changed from In Progress to Fix committed
• % Done changed from 20 to 100

Applied in changeset commit:9ea44ce4b22feb1b1341cc2dbb39eb5f982d9c15.

#28 Updated by anonym 2017-06-03 15:35:51

• Assignee changed from anonym to lamby
• QA Check changed from Dev Needed to Info Needed

Jenkins managed to reproduce a pair of ISOs! https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_bugfix-12567-fontconfig-fixup/4/

I’ve also done two reproductions locally, so I think we’re good! Merged!

Lamby, are you willing to, and will your time budget allow you to do the upstreaming part (reassign back to me when answering)? If so, all that’s needed is your latest patch (so forget about the find -L idea).

#29 Updated by lamby 2017-06-03 22:01:23

• Assignee changed from lamby to anonym

> I’ve also done two reproductions locally, so I think we’re good! Merged!

Wooo! Gotta love the power of stepping away from the computer to come up with the solution, even if it’s a “oh crap!”.

> are you willing to, and will your time budget allow you to do the upstreaming part

Indeed I am. Will be a pleasure…

> the rush is my fault for not picking up this thread in the reproducibility
> effort sooner.

No problem. Is there anything I could have done better mind you? I mean, apart from do a full test Tails build (where the offending poppler-data (!) package would have been installed and flagged..).

[Assigning back as requested]

#31 Updated by anonym 2017-06-08 14:20:07

• Assignee changed from anonym to lamby

For the record, we’re currently having a Tails reprodicible builds (remote-)party (Feature #12608#note-10) and it is looking really good! We’ve reproducibly built the same Tails on 5 different (real, hardware-wise) systems! And for extra robustness we’ve faked the system time so it appears we built it a month in the future, and forced different QEMU CPU models and machine types. It Just Works™! Congrats to us! :)

lamby wrote:
> Forwarded upstream here: https://lists.freedesktop.org/archives/fontconfig/2017-June/thread.html (when it regenerates anyway…)

I still don’t see it. Did it bounce?

> Debian bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082

Thanks!

#34 Updated by anonym 2017-06-09 08:53:08

• Assignee deleted (lamby)
• QA Check changed from Info Needed to Pass

Excellent! Then there are no more loose ends here!

#35 Updated by intrigeri 2017-06-12 16:12:16

• Status changed from Fix committed to Resolved

#36 Updated by intrigeri 2018-01-18 10:14:24

• related to Bug #15187: fontconfig cache is not reproducible in Buster added