Bug #15187

fontconfig cache is not reproducible in Buster

Added by intrigeri 2018-01-17 17:22:51 . Updated 2019-04-06 16:33:42 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Tails_4.0
Start date:
2018-01-17
Due date:
2018-09-30
% Done:

100%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

See attached diffoscope output.

I don’t remember if we expected Buster to have everything we need, or if there’s still some upstream or Debian bug report open about that. I guess that’s https://bugs.debian.org/864082, which is not marked as forwarded upstream (strange, probably a mere oversight). We could of course build a patched fontconfig package as we do for Stretch, but it would be nice to seize this opportunity to try harder to upstream this.

Files

feature-buster_diffoscope.html (453016 B) intrigeri, 2018-01-17 17:19:17
0001-Ensure-cache-checksums-are-determinstic.patch (3878 B) lamby, 2018-05-03 03:46:57

Subtasks

Related issues

Related to Tails - Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 Resolved 2017-05-19
Related to Tails - Feature #16285: feature/buster branch is not reproducible Resolved 2019-01-05
Blocks Tails - Feature #15944: Port Tails to Buster Resolved 2018-09-12
Blocked by Tails - Bug #15857: Make feature/buster build Resolved 2018-08-29
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2018-01-18 10:14:23

  • related to Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 added

#2 Updated by intrigeri 2018-01-18 10:36:28

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Type of work changed from Communicate to Code

So there were two issues:

  • https://bugs.debian.org/857892, fixed in the version of fontconfig that’s in Buster (we apply a similar patch to upstream’s on Stretch) => case closed
  • https://bugs.debian.org/864082 (i.e. the solution to Bug #12567), that has two parts:
    • a patch against the upstream source, that I’ve just marked as forwarded upstream; upstream has replied with a different design proposal that totally makes sense to me; no follow-up from our part so far
    • a Debian packaging change that depends on the new feature proposed upstream

Next step: implement what upstream suggested for the second problem. It looks easy: I suspect that even with my non-existing C skills I could get something worth sending upstream in a few hours (including integration with the upstream build system + Debian packaging and testing in an ISO build). I would start from this code (the Python option is more tempting but I don’t think it’s acceptable to add this dependency to fontconfig in Debian).

#3 Updated by lamby 2018-01-18 22:28:29

Thanks for marking it as forwarded upstream. Usually when this happens (and “upstream” is a mailing list) it’s because posts get marked for moderation and I can only really keep polling the HTML archives for so so many days until I get distracted by something else :)

Let me know if I can help here - very welcome to jump in and resolve this, just let me know and assign over etc. etc.

#4 Updated by intrigeri 2018-04-13 14:33:16

  • Assignee set to lamby
  • Estimated time set to 3 h

#5 Updated by intrigeri 2018-04-13 15:57:24

  • Due date set to 2018-09-30

#6 Updated by intrigeri 2018-04-13 15:57:40

#9 Updated by lamby 2018-05-03 03:48:30

I’ve reworked the patch on upstream’s advice. I’ve:

- Sent it here to https://bugs.debian.org/864082
- Forwarded it to the upstream mailing list (although I think it is stuck in their spam queue, alas, but it should eventually be available https://lists.freedesktop.org/archives/fontconfig/2018-May/thread.html)
- Attached it here

#11 Updated by intrigeri 2018-05-07 11:38:28

  • Assignee changed from intrigeri to lamby

> I’ve reworked the patch on upstream’s advice. I’ve:

> - Sent it here to https://bugs.debian.org/864082
> - Forwarded it to the upstream mailing list
> […]

Great! Can you handle the next steps i.e. ensure this is reviewed and merged upstream (or worst case, in Debian), pinging the relevant folks as needed and following-up on whatever comment they may have?

#12 Updated by lamby 2018-05-08 19:29:15

Sure. Already on it :)

#18 Updated by lamby 2018-05-18 07:46:50

https://bugs.debian.org/864082#49 - Pinged Debian maintainer to cut a new release and updated forwarded URI

#19 Updated by lamby 2018-05-24 07:50:26

Pinged Debian bug https://bugs.debian.org/864082#56

#20 Updated by intrigeri 2018-05-24 09:29:12

lamby wrote:
> https://lists.freedesktop.org/archives/fontconfig/2018-May/006289.html merged upstream

Congrats!

#21 Updated by intrigeri 2018-05-24 09:30:03

Will we need changes in the Debian packaging, aside of importing the changes from upstream?

#22 Updated by lamby 2018-05-24 09:33:39

intrigeri wrote:
> Will we need changes in the Debian packaging, aside of importing the changes from upstream?

No.

#23 Updated by lamby 2018-08-04 12:40:57

Pinged bug and Keith IRL…

#24 Updated by lamby 2018-08-24 20:33:15

Pinged upstream bug

#25 Updated by intrigeri 2018-09-12 06:45:37

  • Target version changed from Tails_4.0 to Tails_3.11

#26 Updated by intrigeri 2018-09-12 06:45:44

#27 Updated by lamby 2018-09-26 18:36:40

Finally merged in debian as 2.13.1-1

#28 Updated by intrigeri 2018-10-08 13:59:37

#29 Updated by intrigeri 2018-10-08 13:59:41

  • blocked by deleted (Feature #15334: Core work 2018Q3: Foundations Team)

#30 Updated by lamby 2018-10-08 14:10:19

  • Assignee changed from lamby to CyrilBrulebois
  • QA Check set to Ready for QA

#31 Updated by lamby 2018-10-08 14:10:54

  • blocked by Bug #15857: Make feature/buster build added

#32 Updated by lamby 2018-10-21 13:53:26

  • Assignee changed from CyrilBrulebois to intrigeri

Hm, according to https://bugs.debian.org/864082#101:

> Unfortunately, fontconfig still installs unreproducibly. Try […]

Hmpf!

I think I would need some hour(s) to even confirm/investigate this, hence assigning over to you, intri.

#33 Updated by intrigeri 2018-10-24 17:13:50

  • Assignee changed from intrigeri to lamby
  • Estimated time changed from 3 h to 7 h
  • QA Check changed from Ready for QA to Dev Needed

Sure, adding 4 hours that should hopefully be enough to debug this, get a fix upstream and in Buster :)

#34 Updated by lamby 2018-10-29 15:08:51

Can reproduce:


$ diffoscope --markdown=- debian-fontconfig*/var/cache/fontconfig

# Comparing debian-fontconfig1/var/cache/fontconfig & debian-fontconfig2/var/cache/fontconfig

## file list

    @@ -1,5 +1,5 @@
    +7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
    +95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
     CACHEDIR.TAG
    -a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
    -afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
    -c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
    -ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
    +cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
    +dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

## stat {}

    @@ -1,8 +1,8 @@

       Size: 4096       Blocks: 8          IO Block: 4096   directory
     Links: 2
     Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

    -Modify: 2018-10-29 15:00:44.915312284 +0000
    +Modify: 2018-10-29 15:02:00.475850377 +0000

      Birth: -

## Comparing debian-fontconfig1/var/cache/fontconfig/CACHEDIR.TAG & debian-fontconfig2/var/cache/fontconfig/CACHEDIR.TAG

### stat {}

    @@ -1,8 +1,8 @@

       Size: 200        Blocks: 8          IO Block: 4096   regular file
     Links: 1
     Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

    -Modify: 2018-10-29 15:00:44.915312284 +0000
    +Modify: 2018-10-29 15:02:00.475850377 +0000

      Birth: -

#36 Updated by lamby 2018-11-12 13:23:21

Upstream were interested in my patch but it sparked a dicussion around removing this mechanism entirely (yay?) but that has some deeper issues with Flatpak/bind-mounting, etc. I’ve sent a gentle ping of sorts offering my assistance: https://lists.freedesktop.org/archives/fontconfig/2018-November/006405.html

#37 Updated by lamby 2018-11-22 14:12:56

I’ve pinged upstream here. I’m not really sure what else I can do at the moment. https://lists.freedesktop.org/archives/fontconfig/2018-November/006416.html

#38 Updated by intrigeri 2018-11-30 14:05:41

  • Estimated time changed from 7 h to 4 h

(Removing what’s already been accounted for in Q2.)

#39 Updated by intrigeri 2018-12-03 15:22:30

  • Target version changed from Tails_3.11 to Tails_3.12

#40 Updated by lamby 2019-01-04 15:01:02 

From: Chris Lamb <chris@chris-lamb.co.uk>
To: fontconfig@lists.freedesktop.org
Cc: 864082@bugs.debian.org
Subject: Next steps for a reproducible Fontconfig?
Date: Friday, 4 January 2019 1:29 PM

[Adding 864082@bugs.debian.org to CC]

Dear fontconfig maintainers,

I've just spent a coffee-or-two unpicking this to get the latest status
and to load the history back into my brain.

As a bit of background, I'm working on the Reproducible Builds
effort and fontconfig — in its usual usage, or at least in Debian
at the time — generated unreproducible cache files.

This was due to it using the timestamps of each directory in the
`checksum` member of the `_FcCache` struct. This is so that it can
identify which cache files remain valid and/or require regeneration
(or similar logic).

So therefore in June 2017 I sent an initial patch:

  https://lists.freedesktop.org/archives/fontconfig/2017-June/005948.html

… which, after some (private?) discussion regarding the implementation, resulted in:

  https://lists.freedesktop.org/archives/fontconfig/2018-May/006285.html

… and that was merged after some further round-trips in f098adac54:

   https://lists.freedesktop.org/archives/fontconfig/2018-May/006289.html

… which was released as part of:

   $ git tag --contains f098adac54 | head -n1
   2.13.1

So far, so good. However, Johannes Schauer then reported that
fontconfig "still" installs unreproduciby:

  https://bugs.debian.org/864082#101

… so I prepared a new patch:

  https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html

… and that was "soft NACK'd" in the sense that Keith mentions:

I've dug into this a bit more and I think an architectural change in the
cache files made last year is probably not what we want.

       — https://lists.freedesktop.org/archives/fontconfig/2018-October/006376.html

(I am now inferring that it was this "architectural change"
resulted in the regression Johannes reported, rather than the bug
being incomplete from the beginning.)

Anyway, the upshot from my proposal was that some larger/different
changes are/were "requested" instead.

Behdad Esfahbod also chimed in with:

I don't like the new mechanism either, but I think it was added to resolve
bind-mounted font dirs

       — https://lists.freedesktop.org/archives/fontconfig/2018-October/006381.html

… in the context of Flatpak apps. Keith then addressed all this
with a branch which he published here:

  https://gitlab.freedesktop.org/keithp/fontconfig

… the most salient commit being (I think?):

  https://gitlab.freedesktop.org/keithp/fontconfig/commit/a04751b2e624d034becec7588159ef2f9a8dfc1b

Since then, I don't believe there has been any review of this
branch both in the sense of the code itself but also in terms of
the architectural changes that it implies. I might be able to help
on the former front but without knowing the "lore" of Fontconfig I
simply cannot comment on the latter parts.

Anyway, I'd love to get this resolved once and for all ideally get
it into Debian buster which is about to start "freezing" very
soon.

What would be the best way for me to help here? Can I entreat Keith
to merge his branch? I can put some cycles onto this issue if that is
of some assistance.


Best wishes,

-- 
Chris Lamb
chris-lamb.co.uk / @lolamby

https://lists.freedesktop.org/archives/fontconfig/2019-January/006420.html / https://bugs.debian.org/864082#120

#41 Updated by intrigeri 2019-01-04 15:03:29

#42 Updated by intrigeri 2019-01-04 15:04:12

  • blocked by deleted (Feature #15506: Core work 2018Q4: Foundations Team)

#43 Updated by lamby 2019-01-05 10:03:26

  • Subject changed from fontconfig cache is not reproducible on Buster to fontconfig cache is not reproducible in Buster

#44 Updated by lamby 2019-01-05 16:26:06

  • blocks Feature #16285: feature/buster branch is not reproducible added

#45 Updated by lamby 2019-01-06 16:21:03

  • Assignee changed from lamby to intrigeri

Please check the master branch at https://salsa.debian.org/lamby/pkg-tails-fontconfig.

ps. Should the Target Version be 4.0 here?

#46 Updated by lamby 2019-01-06 16:21:34

  • Assignee changed from intrigeri to lamby

Taking ticket back ; upstream’s test suite fails at the last second!!

#47 Updated by lamby 2019-01-06 16:54:57

  • Assignee changed from lamby to intrigeri
  • Target version changed from Tails_3.12 to Tails_4.0

Apologies for that - it had got 90% of the way through the testsuite and I sent the message prematurely instead of waiting before it had definitely passed before hitting Submit… :D

Try 8abd008bc6fcecbdf45a45f2fd7a504054ddfe19 on master at gitsalsa.debian.org:lamby/pkg-tails-fontconfig.git@

#48 Updated by intrigeri 2019-01-07 08:49:31

  • Assignee changed from intrigeri to lamby
  • % Done changed from 10 to 20
  • QA Check changed from Dev Needed to Ready for QA

Thanks! Built and uploaded to feature-buster.

The URL for the 1st CI job that will benefit from this will probably be: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-buster/71/. Please keep an eye on it and check whether this fixes the fontconfig cache reproducibility issues :)

#49 Updated by lamby 2019-01-07 19:07:41

  • Status changed from In Progress to Fix committed
  • % Done changed from 20 to 90
  • QA Check changed from Ready for QA to Pass

This is fixed in Tails, but keeping this open for the time being as a) the build is not entirely reproducible yet and it b) it would be nice to get a fix upstream and/or in Debian.

Marking as “passed QA”, “fix committed” & “90% done” however.

#50 Updated by lamby 2019-01-08 13:12:59

  • blocked by deleted (Feature #16285: feature/buster branch is not reproducible)

#51 Updated by lamby 2019-01-08 13:13:29

  • related to Feature #16285: feature/buster branch is not reproducible added

#52 Updated by intrigeri 2019-01-19 13:36:12

  • Estimated time changed from 4 h to 0 h

Removing what’s been accounted for in 2018Q4. Shout if you need more :)

#54 Updated by intrigeri 2019-01-25 08:21:53

> Some upstream movement https://lists.freedesktop.org/archives/fontconfig/2019-January/006464.html

Yeah, I’m super happy the Flatpak folks and fontconfig crew have been collaborating towards a solution that satisfies Flatpak’s needs, without breaking reproducibility! :)

#55 Updated by intrigeri 2019-03-20 14:47:50

#56 Updated by intrigeri 2019-03-20 14:47:53

  • blocked by deleted (Feature #15507: Core work 2019Q1: Foundations Team)

#57 Updated by intrigeri 2019-04-02 15:22:28

  • Status changed from Fix committed to In Progress

Let’s mark this fix committed once we have a reproducible feature/buster build.

#58 Updated by intrigeri 2019-04-06 16:33:42

  • Status changed from In Progress to Resolved
  • Assignee deleted (lamby)
  • % Done changed from 90 to 100

feature/buster is now reproducible: https://jenkins.tails.boum.org/view/RM/job/reproducibly_build_Tails_ISO_feature-buster/114/