Tails

#4 Updated by intrigeri 2018-01-01 19:44:55

> Applied in changeset commit:aafdf8da72df5e9ffdada2eabc920325a6e13520.

Great to see someone picking this up! (Wanna assign it to yourself? Otherwise drop the target version: it feels weird to see a ticket for next release without any assignee.)

One comment: this seems to assume all “latest” serials are kept in sync, which is not true. You can very well have an ISO build where each “latest” pointer is resolved to different values. So I guess we need to store each such value, either serializing them all in one single variable with JSON or similar, or with one variable per repo (urgh).

#6 Updated by bertagaz 2018-01-03 23:01:54

intrigeri wrote:
> We’ve triaged this during the monthly meeting “Volunteers to handle important tickets flagged for next release, but without assignee” topic. If you’re taking responsibility for this, please assign this ticket to yourself and then set whatever target version you want, but the status of a ticket flagged for next release, but without assignee, is too unclear.

Ack. Too bad for Bug #14944. This ticket’s priority is ‘normal’ though.

#7 Updated by intrigeri 2018-01-04 05:47:43

> Too bad for Bug #14944.

I don’t understand what you mean: we certainly did not imply that you could not work on this if you fancy it or feel responsible for it.

#8 Updated by bertagaz 2018-01-04 15:48:01

intrigeri wrote:
> I don’t understand what you mean: we certainly did not imply that you could not work on this if you fancy it or feel responsible for it.

Ack, I was confused and surprised to see that change.

> One comment: this seems to assume all “latest” serials are kept in sync, which is not true. You can very well have an ISO build where each “latest” pointer is resolved to different values. So I guess we need to store each such value, either serializing them all in one single variable with JSON or similar, or with one variable per repo (urgh).

If by that you mean that in the build process when apt-snapshots-serials prepare-build is run, the latest snapshots are rotated at the same time and thus at least one of the serial returned during the resolving is different from the others, I agree. I did not think about that.

Now I think this scenario is possible but is rarely happening (and hits devel and branches based on it only). The development cost to implement a solution to that is quite high IMO, at least for me. OTOH the proposed patch is fixing the situaion for Bug #14944 for a very huge part of the builds. I have other things to do for 3.5, so I don’t think I’ll have the time to work on implementing more than what is proposed.

So I think we should merge the proposed patch, which would greatly help for Bug #14944, and see later if we need this much to spend our work time on fixing the problem you’re pointing.

Not sure how to proceed…

#9 Updated by intrigeri 2018-01-04 17:23:35

Hi!

>> One comment: this seems to assume all “latest” serials are kept in sync, which is not true. You can very well have an ISO build where each “latest” pointer is resolved to different values. So I guess we need to store each such value, either serializing them all in one single variable with JSON or similar, or with one variable per repo (urgh).

> If by that you mean that in the build process when apt-snapshots-serials prepare-build is run, the latest snapshots are rotated at the same time

No, that’s not what I meant (or I misunderstood what you think I meant). Each of the APT archives we maintain snapshots of is updated at different times: the load is spread over the entire day. So when we start an ISO build it’s entirely possible that the latest snapshot for the debian one is YYYYMMDD02 while the latest snapshot for the torproject archive is YYYYMMDD01. Whenever this happens, with the proposed patch the 2nd ISO build will fail because APT will try to fetch a snapshot of torproject indices that does not exist yet. And symetrically, if the first build used different “latest” serials for 2+ APT archives, with the proposed patch the second build will necessarily produce a different ISO, while right now there’s a good chance it would manage to reproduce the output of the first build.

Clearer?

> Now I think this scenario is possible but is rarely happening

Do you still think that with the above clarification? If yes, I’m curious why: according to git log -p config/APT_snapshots.d/, it happened 3 times in the 5 times we froze APT snapshots for a release in 2017. Granted, that’s a very small sample; if you have better data, please share.

> The development cost to implement a solution to that is quite high IMO, at least for me.

Understood. Let’s take this into account. If it may help, I think that I could get this part done painlessly (but I’d rather not handle the integration with Jenkins). Shall I give it a try (as a volunteer)?

> OTOH the proposed patch is fixing the situaion for Bug #14944 for a very huge part of the builds.

I have no idea if the proposed patch will fix more problems than it’ll introduce regressions. Our daily builds start at a deterministic time (based on a hash of the branch name) so:

• It will fix the problem for a subset of the branches that currently fail to build reproducibly due to APT snapshots being updated during the first build. This is good.
• It will break the second ISO build (due to trying to use non-existing snapshots) for another subset of the branches, that currently build reproducibly just fine. That’s more false positives.
• It will successfully build a different second ISO, i.e. fail to build reproducibly for another subset of the branches, that currently build reproducibly just fine: if the first build was using different “latest” serials for 2+ APT archives, the second build will necessarily be different. That’s more false positives.

Similarly, for some part of the builds triggered by Git changes it will fix the problem, for some others it will break the second build, and for some other it will make the second build fail to reproduce.

I won’t do the maths right now. So either you do the maths yourself, or we simply try and remain ready to quickly revert if the proposed change makes things worse.
Or I implement the serialization thing I’ve proposed, that shouldn’t take me more time than arguing, or doing the maths, or pretending I’m not doing them ;)

What do you think?

> I have other things to do for 3.5, so I don’t think I’ll have the time to work on implementing more than what is proposed.

Fair enough!

#11 Updated by Anonymous 2018-01-15 09:50:12

@bertagaz: Is this ticket still relevant? If yes, can you assign it to yourself or the relevant person? Thanks!

#16 Updated by intrigeri 2018-12-02 13:40:57

intrigeri wrote:
> I pushed a draft implementation. I refrained from using Perl and instead did Ruby so that you and anonym understand what’s going on and can take ownership of this tiny piece of code if needed in the future. The prepare-build part seems to work just fine. I didn’t know what your plan was wrt. storing the serials of the snapshots used for a given build, so for now I’ve merely added a cat-json action as an example of how it would look like; I’m happy to adjust it to suit whatever interface is required by the integration technique you have in mind; who knows, perhaps calling it with tmp/APT_snapshots.d as an argument would be enough?

Here’s an implementation plan. It might not be the nicest approach but I’ve picked it because it only requires changes in tails.git, so it should be easy to try out on a topic branch without impacting any other branch.

During the first build (build_Tails_ISO_*), after figuring out which APT snapshots shall be used (apt-snapshots-serials prepare-build), run echo "APT_SNAPSHOT_SERIALS='$(apt-snapshots-serials-cat-json tmp/APT_snapshots.d)'" >> tails-build-env.list. Later, once the build has succeeded, the collect_build_environment Jenkins builder will run puppet-tails:files/jenkins/slaves/isobuilders/collect_build_environment that will append other variables to this file.

Then the reproducibly_build_Tails_ISO job already loads tails-build-env.list so rake and auto/build should do the right thing, i.e. pass that inherited chunk of JSON to apt-snapshots-serials prepare-build.

#18 Updated by intrigeri 2018-12-02 15:51:49

To test this, we need to start builds at a time that will trigger the failure condition:

• baseline i.e. reproduce the problem: start a build of the devel branch a few minutes before 05:15, 11:15, 17:15, or 23:15 (UTC)
• validate the fix: start a build of this branch a few minutes before 05:15, 11:15, 17:15, or 23:15 (UTC)

The debian-security snapshot update starts at 05:15, 11:15, 17:15, 23:15 UTC. It generally lasts less than an ISO build. Starting the first builds a few minutes before one of these times should be enough to almost guarantee that the 2nd builds will use a different snapshot and fail to reproduce.

#26 Updated by intrigeri 2019-01-07 10:14:52

FTR when invalid JSON is passed, the build fails as expected:

+ [ -n {"torproject":"2019010603","debian":"2019010603","debian-security":"2019010603",} ]
+ echo Fixing 'latest' APT snapshots serials to: '{"torproject":"2019010603","debian":"2019010603","debian-security":"2019010603",}'.
Fixing 'latest' APT snapshots serials to: '{"torproject":"2019010603","debian":"2019010603","debian-security":"2019010603",}'.
+ apt-snapshots-serials prepare-build {"torproject":"2019010603","debian":"2019010603","debian-security":"2019010603",}
/usr/lib/ruby/2.3.0/json/common.rb:156:in `parse': 822: unexpected token at '{"torproject":"2019010603","debian":"2019010603","debian-security":"2019010603",}' (JSON::ParserError)
    from /usr/lib/ruby/2.3.0/json/common.rb:156:in `parse'
    from /usr/lib/ruby/2.3.0/json/common.rb:335:in `load'
    from /tmp/tails-build.K6g1xPRu/auto/scripts/apt-snapshots-serials-load-json:16:in `<main>'