Tails

#7 Updated by intrigeri 2019-11-28 17:58:47

segfault and I discussed this a bit a few days ago. Here’s what I recall from our conclusions. This is probably heavily biased by my own views on this topic so please assume it’s my own opinion only.

The caching problem is a well-known one. One of its hardest part is to be able to detect when a cache is stale and must be invalidated. That is, in order to cache the website in a way that produces the expected results, we need to know when we can invalidate the cache. This is equivalent to: we need to know when a fresh build of the website would yield a different output. It follows that we need to encode, along with the cached built website, the status of anything that affects such output. This includes:

• last commit on wiki/src and ikiwiki.setup (so before reusing the cached built website, we can check if anything changed there); we do something similar for the Vagrant build box.
• version of the tools used to build the cached version, such as ikiwiki and its most important dependencies (e.g. po4a and the library used to render Markdown to HTML)

IMO we should not aim for perfection here (e.g. I doubt it’s worth listing all ikiwiki dependencies recursively) but for some trade-off that gives us something that’s good enough (for some value of good enough that we can agree on).

(And while I’m on this topic, a meta point: I’m personally exposed to lots of input wrt. the pain points of our development process. I’m missing a way to collectively prioritize them. Right now I’m commenting here, and I’m not filing tickets for the various other ideas/problems that I’ve heard of, because I believe that the lack of this build feature is the most important pain point in our dev process; I venture this is confirmed by the fact some of us have local, custom hacks to avoid building the website when they build Tails images; if this is not resolved once we’re there, I’d like to propose this ticket as a candidate for FT work for our next roadmap.)

#14 Updated by segfault 2020-01-09 12:14:57

The branch fixes some issues which qualify as FT work and some which do not. I only tracked the time I spent reviewing the whole branch, not each commit / issue. Would it be ok if I just clocked half of the time as FT work?

#15 Updated by intrigeri 2020-01-09 12:19:12

Hi @segfault,

> I only tracked the time I spent reviewing the whole branch, not each commit / issue. Would it be ok if I just clocked half of the time as FT work?

Reviewing code submissions is FT work, regardless of whether the development work was paid or not ⇒ clock it all!

#16 Updated by intrigeri 2020-01-09 12:22:18

> LGTM, nice Python code again :)

/me blushes again!

> I pushed a commit fixing some minor style issues, please review and then feel free to merge the branch.

LGTM! I’ll happily merge once 4.2.1 is out and Jenkins has successfully built from the updated branch.

#18 Updated by intrigeri 2020-01-14 19:51:21

Dear segfault, now that 4.2.2 is behind us I’m going to merge this. Just to be sure I tried to build with cachewebsite enabled first, and… it failed. A part of commit:2f5785fde147dcc5eadba93f9f64f52ab4c638d8 broke the build, so I’ve reverted that part in commit:b1a68a6349cbd3ee47004fc54926e774a87483dc. If there’s a more clever way to fix that, or if I misunderstood something, please let me know :)

And while we’re at it, segfault and @anonym, I’d love your input on Bug #17439.

#20 Updated by segfault 2020-01-15 17:50:02

intrigeri wrote:
> Dear segfault, now that 4.2.2 is behind us I’m going to merge this. Just to be sure I tried to build with cachewebsite enabled first, and… it failed. A part of commit:2f5785fde147dcc5eadba93f9f64f52ab4c638d8 broke the build, so I’ve reverted that part in commit:b1a68a6349cbd3ee47004fc54926e774a87483dc.

Oh shit, sorry about that!

#21 Updated by segfault 2020-01-15 17:51:45

I also forgot something else I wanted to note after the review:

I would also like to have an option to not rebuild the website even if the wiki changed. Most of the time I build a Tails image, I don’t care about the website at all, I just want to test my feature / bugfix. In fact, I’m almost always using this script I wrote to build Tails, which simply patches out building the website, and the only issue I had with this is that the Unsafe Browser doesn’t start if /usr/share/doc/tails/website/misc/unsafe_browser_warning doesn’t exist. That would be fixed if any cached website build would be used.

#22 Updated by intrigeri 2020-02-04 12:09:06

Hi,

> I would also like to have an option to not rebuild the website even if the wiki changed. Most of the time I build a Tails image, I don’t care about the website at all, I just want to test my feature / bugfix. In fact, I’m almost always using this script I wrote to build Tails, which simply patches out building the website, and the only issue I had with this is that the Unsafe Browser doesn’t start if /usr/share/doc/tails/website/misc/unsafe_browser_warning doesn’t exist. That would be fixed if any cached website build would be used.

I understand this desire and I would be fine with having such a build option supported explicitly (it feels better than folks doing what you’re doing on their own).
I don’t plan to work on this myself but feel free to capture this idea in a ticket and to work on it :)
Regarding the interface, it could be worth coordinating with Bug #17439: for example, I could imagine replacing the current cachewebsite with cachewebsite={yes,disable,force}, where “force” would do what you want, except of course in the cases when we require a clean build (building from a tag). I’m happy to leave room for this while working on Bug #17439, to ease this follow-up work and make the interface nicer.

#23 Updated by intrigeri 2020-02-04 12:17:29

> cachewebsite={yes,disable,force}

Gah, I meant cachewebsite={yes,no,force}