Bug #12740

Various .cache files not reproducible in some environments

Added by anonym 2017-06-19 10:21:32 . Updated 2017-09-28 18:41:51 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
2017-06-23
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
Affected tool:
Deliverable for:
289

Description

See Feature #12608#note-18 for arnaud’s diffoscope report. There are also .torrent:s for the good ISO, and for arnaud’s.

The files in question are:

/usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders.cache
/usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache
/usr/lib/x86_64-linux-gnu/gtk-2.0/2.10.0/immodules.cache
/usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules.cache
/usr/share/applications/mimeinfo.cache


Here’s an excerpt from the diff:

├── /usr/share/applications/mimeinfo.cache
│ │ @@ -9,15 +9,15 @@
│ │  application/mathml+xml=libreoffice-math.desktop;
│ │  application/msexcel=libreoffice-calc.desktop;
│ │  application/mspowerpoint=libreoffice-impress.desktop;
│ │  application/msword=libreoffice-writer.desktop;
│ │  application/mxf=org.gnome.Totem.desktop;
│ │  application/ogg=audacity.desktop;org.gnome.Totem.desktop;
│ │  application/oxps=evince.desktop;
│ │ -application/pdf=libreoffice-draw.desktop;evince.desktop;gimp.desktop;inkscape.desktop;
│ │ +application/pdf=libreoffice-draw.desktop;gimp.desktop;inkscape.desktop;evince.desktop;
│ │  application/pgp-encrypted=seahorse-pgp-encrypted.desktop;
│ │  application/pgp-keys=seahorse-pgp-keys.desktop;
│ │  application/pgp-signature=seahorse-pgp-signature.desktop;
│ │  application/pkcs10=gcr-viewer.desktop;
│ │  application/pkcs10+pem=gcr-viewer.desktop;
│ │  application/pkcs12=gcr-viewer.desktop;
│ │  application/pkcs12+pem=gcr-viewer.desktop;
[...]

Files


Subtasks

Bug #12909: /var/cache/cracklib/src-dicts not reproducible Resolved

100

Bug #13439: mimeinfo.cache not reproducible Resolved

100

Bug #13440: GTK immodules.cache not reproducible Resolved

100

Bug #13441: giomodule.cache not reproducible Resolved

100

Bug #13442: gdk-pixbuf's loaders.cache not reproducible Resolved

100


History

#1 Updated by anonym 2017-06-19 10:24:30

  • Assignee set to lamby
  • QA Check set to Info Needed

Could you have a look?

#2 Updated by intrigeri 2017-06-22 13:49:31

#3 Updated by lamby 2017-06-23 09:14:46

  • Assignee changed from lamby to anonym

This looks like 4 separate issues related to filesystem ordering. Happy to write patches to fix upstream, alternatively I can look into whether they are safe to just delete. Let me know either way.

#4 Updated by anonym 2017-07-07 15:21:27

  • Assignee changed from anonym to lamby

lamby wrote:
> This looks like 4 separate issues related to filesystem ordering.

Would it help to track these separately, or will that just add overhead?

> Happy to write patches to fix upstream, alternatively I can look into whether they are safe to just delete. Let me know either way.

Our experience with deleting the fontconfig cache wasn’t great; I fear dropping these caches from the ISO will result in some of them being generated at boot time, making it slower, and using more RAM (on the tmpfs).

Here upstream fixes would be great! Do you think you can handle all four of them in a single work day? If so, feel free to unset the “Info Needed” status and proceed!

#5 Updated by lamby 2017-07-07 15:56:04

  • Assignee changed from lamby to anonym

> Do you think you can handle all four of them in a single work day?

Alas, I seriously doubt it given a) there are four lights^Wissues, and b) they are part of GTK-ish components so the REPL iteration time is going to be a little slow :)

> Would it help to track these separately

I think so yeah, even if we don’t push ahead with the changes. I find tickets with rather generic names like “various” are usually a little unwieldy and often lead to wasted effort. :)

#6 Updated by anonym 2017-08-07 13:54:23

  • Status changed from Confirmed to In Progress
  • QA Check deleted (Info Needed)

#7 Updated by anonym 2017-08-07 14:03:17

  • Target version changed from Tails_3.1 to Tails_3.2

#8 Updated by anonym 2017-08-17 12:16:28

  • Assignee changed from anonym to lamby

lamby, a user managed to reproduce Bug #13439 to Bug #13442 (the original children of this ticket, i.e. not Bug #12909). Note that the issues Bug #13439 to Bug #13442 has so far always occurred all together, and Bug #12909 seems to be independent. I wonder if these four issues is not separate, but share the same cause, as I originally assumed when creating this ticket.

Any way, I see no progress on these four child tickets since their creation a month ago, and I’m wondering if there is some way I can help to get things rolling, or are you just busy on other fronts?

#9 Updated by anonym 2017-08-17 12:16:38

  • QA Check set to Info Needed

#10 Updated by lamby 2017-08-17 14:44:35

anonym wrote:
> Any way, I see no progress on these four child tickets since their creation a month ago, and I’m wondering if there is some way I can help to get things rolling, or are you just busy on other fronts?

Oh, what a shame. I was not aware I should (and/or was “authed”) to work on it. Indeed, you were quite careful recently to check admin blah with intri beforehand, and I saw no reason why that wouldn’t remain. In other words, would love to get onto this, just waiting for the nod.

#11 Updated by anonym 2017-08-18 10:17:55

lamby wrote:
> anonym wrote:
> > Any way, I see no progress on these four child tickets since their creation a month ago, and I’m wondering if there is some way I can help to get things rolling, or are you just busy on other fronts?
>
> Oh, what a shame. I was not aware I should (and/or was “authed”) to work on it. Indeed, you were quite careful recently to check admin blah with intri beforehand, and I saw no reason why that wouldn’t remain. In other words, would love to get onto this, just waiting for the nod.

Ok, this is definitely my fault. I have no definite memory, but it feels like a comment on this ticket was lost (I do have a tendency to update tickets and forget to press “Submit” which could explain it…).

Any way, I see no apparent way for tackling this. I say, spend a day on investigating whether these .cache files have something in common that could give a single explanation for all of them. If it turns out that these are separate issues I suppose you’ll still have some clues about what’s problematic with most of them. If you at that point think you’d need just another day to fix them all, then please go ahead, otherwise let us know and we take a closer look on how to proceed. Ok? All clear?

I’ve now attached an excerpt of the full diffoscope output that show all differences for the .cache files.

#12 Updated by lamby 2017-08-18 17:10:15

anonym wrote:
> Ok, this is definitely my fault. I have no definite memory, but it feels like a comment on this ticket was lost (I do have a tendency to update tickets and forget to press “Submit” which could explain it…).

Seriously, no worries. The only reason I wrote so much was so that we can all improve our processes! And I’ve also done the whole “forgetting to press ‘Send’” many times too :)

> spend a day on investigating whether these .cache files have something in common that could give a single explanation for all of them.

Sounds good; am on it :)

#13 Updated by lamby 2017-08-29 18:46:10

  • Assignee changed from lamby to anonym
  • QA Check changed from Info Needed to Ready for QA

All of these now have patches and/or waiting for merge/QA etc. Assigning to thee, anonym…

Enjoy :)

#14 Updated by intrigeri 2017-09-10 17:37:43

  • QA Check changed from Ready for QA to Dev Needed

(Two of the subtasks are apparently not fully fixed yet.)

#15 Updated by anonym 2017-09-15 17:41:47

intrigeri wrote:
> (Two of the subtasks are apparently not fully fixed yet.)

Now they are most likely fixed, thanks to lamby’s new patches. Let’s see! I’ll keep this ticket open until 3.2 final, when we hopefully know whether they are enough.

#16 Updated by lamby 2017-09-15 18:07:38

anonym wrote:
> intrigeri wrote:
> > (Two of the subtasks are apparently not fully fixed yet.)
>
> Now they are most likely fixed, thanks to lamby’s new patches. Let’s see! I’ll keep this ticket open until 3.2 final, when we hopefully know whether they are enough.

\o/ \o/ \o/

So glad to help out :)

#17 Updated by intrigeri 2017-09-21 10:18:31

Every subtask is either “resolved” or “fix committed”. Anything else to do here?

#18 Updated by intrigeri 2017-09-21 10:19:31

  • Target version changed from Tails_3.2 to Tails_3.3

Ooops, I missed “I’ll keep this ticket open until 3.2 final, when we hopefully know whether they are enough”. Then if we’re only going to know after the 3.2 release if that’s enough, then this ticket can’t be a blocker for 3.2 => postponing.

#19 Updated by anonym 2017-09-21 13:14:22

  • Target version changed from Tails_3.3 to Tails_3.2

intrigeri wrote:
> Ooops, I missed “I’ll keep this ticket open until 3.2 final, when we hopefully know whether they are enough”. Then if we’re only going to know after the 3.2 release if that’s enough, then this ticket can’t be a blocker for 3.2 => postponing.

No, I just asked all previous testers of Tails’ reproducibility to try to reproduce 3.2~rc1 — if we don’t see any of this ticket’s children among the reports, then I’ll close this ticket. Well, I won’t if we don’t get positive responses from those that historically were affected by this ticket’s children.

#20 Updated by anonym 2017-09-21 14:28:48

kurono was affected by Bug #13440 and Bug #13442 when he tried to reproduce 3.2~alpha2, but is not any more with 3.2~rc1 which was reproduced successfully! This looks promising as it is a data point supporting the hypothesis that lamby’s patches now are complete!

Woo! You rock, lamby!

#21 Updated by lamby 2017-09-21 14:35:39

anonym wrote:
> kurono was affected by Bug #13440 and Bug #13442 when he tried to reproduce 3.2~alpha2, but is not any more with 3.2~rc1 which was reproduced successfully! This looks promising as it is a data point supporting the hypothesis that lamby’s patches now are complete!
>
> Woo! You rock, lamby!

\o\ /o/

So, now, is that the official position of the Tails Project…? Can we get the press team onto publicising that? ;)

#22 Updated by anonym 2017-09-21 14:44:51

  • Assignee changed from anonym to lamby
  • QA Check changed from Dev Needed to Info Needed

lamby, do you have some ideas about what environmental discrepancies could have been the trigger for these sources of non-determinism? We have seen that:

  • these issues (possibly excluding Bug #12909) always seem to come together, so the “trigger” is likely the same for all of them
  • none of Tails’ “core” or infra was ever affected
  • most other people that tried to reproduce Tails were affected

I’m not intending much time to be spent on this, so I’m mostly interested in what you can come up with from the top of your head when looking back at your patches, which I then will casually investigate. I figure if we can identify the trigger and fix it now, then we might save ourselves some future headache — finding this problem now should be easier since we have up-to-date info (env vars etc.) about all the systems that attempted to reproduce Tails, and this whole issue is still fresh in our brains.

#23 Updated by anonym 2017-09-21 14:49:46

lamby wrote:
> So, now, is that the official position of the Tails Project…? Can we get the press team onto publicising that? ;)

Good point! I think so! :) There was some internal controversy about what the requirements for calling Tails reprodicible should be, but I hope we can agree that it’s achieved once all known issues are fixed. So unless some of the others that were affected by this ticket’s children report failures, I guess we are there, i.e. “Tails is reproducible” (!). :)

#24 Updated by Anonymous 2017-09-21 15:04:48

anonym wrote:
> lamby wrote:
> > So, now, is that the official position of the Tails Project…? Can we get the press team onto publicising that? ;)
>
> Good point! I think so! :) There was some internal controversy about what the requirements for calling Tails reprodicible should be, but I hope we can agree that it’s achieved once all known issues are fixed. So unless some of the others that were affected by this ticket’s children report failures, I guess we are there, i.e. “Tails is reproducible” (!). :)

Awesome!!!

I think we should not prepare such communication right now. But there are two tickets to communicate about this to the tech community and to our users (could be done in a monthly report for example).

#25 Updated by anonym 2017-09-21 15:10:11

u wrote:
> I think we should not prepare such communication right now. But there are two tickets to communicate about this to the tech community and to our users (could be done in a monthly report for example).

I know about Feature #12626, but not of any ticket about communicating this to our users.

#26 Updated by lamby 2017-09-21 15:57:05

> ideas about what environmental discrepancies could have been the trigger for these sources of non-determinism?

My immediate gut guess is that different filesytem. eg. ext-family vs. btfs vs. xfs.

> internal controversy about what the requirements for calling Tails reprodicible should be

Ah, sorry, you misunderstand; I was talking about it being the official Policy of the Tails Project that “lamby rocks”… g

#27 Updated by intrigeri 2017-09-21 16:28:52

> Ah, sorry, you misunderstand; I was talking about it being the official Policy of the Tails Project that “lamby rocks”… g

As per our non-existing Democratic Constitution that I embody (that much is pretty obvious), this ought to be discussed on the tails-project@boum.org mailing list instead of behind closed curtains within the (sadly limited) ranks of our Great Redmine Cabal.

Needless to say, I’ll wholeheartedly second this proposal. Now, here’s the politician in me speaking: I think this proposal will get more traction if it comes with at least a rough design wrt. how it will be implemented. I doubt integrating this into our Social Contract will work well. Perhaps a banner on our website? Or a simple, catchy statement on the desktop wallpaper in Tails itself? Of course, we shall take into account how it’ll work on the long term, striving towards minimization of maintenance costs: what’s the process to update this official policy? I suggest it should automatically expire after N years; and the closest we have to a GR (i.e. myself and/or sajolida convincing everyone that we’re right) can revoke it at any time earlier. What do you think?

[To all innocent bystanders: yes, I’m somewhat joking.]

#28 Updated by anonym 2017-09-22 00:01:10

lamby wrote:
> > ideas about what environmental discrepancies could have been the trigger for these sources of non-determinism?
>
> My immediate gut guess is that different filesytem. eg. ext-family vs. btfs vs. xfs.

Note that the same filesystems (ext4, tmpfs) are always used in the guest VM that actually builds Tails. Did you overlook this, or does the host OS’ filesystems in fact really matter?

> > internal controversy about what the requirements for calling Tails reprodicible should be
>
> Ah, sorry, you misunderstand; I was talking about it being the official Policy of the Tails Project that “lamby rocks”… g

Aaah! Indeed! Let’s continue this discussion on tails-project@. I anticipate zero resistance. :)

#29 Updated by lamby 2017-09-22 08:15:32

anonym wrote:

> Note that the same filesystems (ext4, tmpfs) are always used in the guest VM that actually builds Tails. Did you overlook this, or does the host OS’ filesystems in fact really matter?

Ah, interesting. Does the git checkout happen inside there or outsie? I’m just trying to think why, for example, files might be checked out in a different order… :)

#30 Updated by intrigeri 2017-09-22 08:40:01

My theory is still that depending on the initial state of the apt-cacher-ng cache, packages that trigger cache refreshes might be installed in a different order.

#31 Updated by lamby 2017-09-22 08:45:14

intrigeri wrote:
> My theory is still that depending on the initial state of the apt-cacher-ng cache, packages that trigger cache refreshes might be installed in a different order.

Hm, or something about the parallel APT HTTP downloads? Acquire::Queue-Mode “access”; etc. :)

#32 Updated by intrigeri 2017-09-22 09:34:14

> Hm, or something about the parallel APT HTTP downloads? Acquire::Queue-Mode “access”; etc. :)

Yep.

#33 Updated by anonym 2017-09-26 12:15:14

  • Status changed from In Progress to Fix committed
  • Assignee deleted (lamby)
  • QA Check changed from Info Needed to Pass

intrigeri’s theory sounds reasonable, and easy to figure out in the future, should we need it. I think we can just ignore it for now.

So, given that we have only received successes for the reproduction of Tails 3.2~rc1 (incl. several participants that previously always were hit by these issues) I think wee can call this one resolved! Good job, everyone!

#34 Updated by lamby 2017-09-26 13:35:09

anonym wrote:
> intrigeri’s theory sounds reasonable, and easy to figure out in the future, should we need it. I think we can just ignore it for now.

Agreed, although I would just add that even if we could prove it now it might still mean we fix the underlying problems in the fture instead of setting various (rare) APT settings or defining “one must use ext4” or whatever the issue or issues are. :)

Anyway, yay. :)

#35 Updated by anonym 2017-09-28 18:41:52

  • Status changed from Fix committed to Resolved