Bug #12053

Website build takes 10-20 minutes longer since upgrade to 3.20160905~bpo8+1

Added by intrigeri 2016-12-21 09:22:24 . Updated 2017-02-10 15:56:33 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Tails_2.11
Start date:
2016-12-21
Due date:
% Done:

100%

Feature Branch:
web/12053-faster-build
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

On November 19 we upgraded ikiwiki on our isobuilders from 3.20141016.3 to 3.20160905~bpo8+1 (commit:a459578, for Feature #11966, but it was going to happen at some point anyway). Since then, ISO build times went crazy because the web site takes much more time to build.

Subtasks

History

#1 Updated by sajolida 2016-12-26 16:36:19

I get very similar build times on my Tails with different versions of ikiwiki:

|        3.20161219 | 34m5.039s |
| 3.20160905~bpo8+1 | 34m7.126s |

That’s also similar to the times I reported with earlier versions in Feature #11151.

#2 Updated by intrigeri 2016-12-26 17:01:35

> I get very similar build times on my Tails with different versions of ikiwiki:

>

> |        3.20161219 | 34m5.039s |
> | 3.20160905~bpo8+1 | 34m7.126s |
>

So, the regression introduced between 3.20141016.3 and 3.20160905 isn’t fixed in 3.20161219 (and probably still isn’t fixed).
That’s no big surprise to me, but still a useful data point :)

#3 Updated by intrigeri 2016-12-28 17:40:50

On my machine, at commit:8044f99f7632f0bff87cee7f4dad37efba1fec06:

  • 3.20161219, deterministic: 16:25
  • 3.20161219, non-deterministic: 17:21 => no big difference, so next results all have deterministic enabled
  • 3.20160121: 17:10
  • 3.20150614: 15:57
  • 3.20150610: 6:19
  • 3.20150329: 6:12
  • 3.20141016: 6:09

#4 Updated by intrigeri 2016-12-28 18:36:41

On my machine, commit:29d3d41b8d4ca7680e2074c0988c9710d44be20f, with ikiwiki 3.20161219, gets the build back down to 6:30.

#5 Updated by intrigeri 2016-12-28 18:41:26

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to sajolida
  • Target version changed from Tails_2.11 to Tails 2.10
  • % Done changed from 0 to 30
  • QA Check set to Ready for QA
  • Feature Branch set to web/12053-faster-build
  • Type of work changed from Research to Code

Please review’n’merge into master. I only gave a cursory look at the resulting built website, so it would be nice if you made sure that non-obvious parts (e.g. the IA) are still OK.

#6 Updated by sajolida 2017-01-20 11:33:36

  • Assignee changed from sajolida to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Woh, that’s sooo cool that you found that! My build time went down from ~34 minutes to ~14. You’re my savior! Now I wonder whether the huge difference I observed between build in Tails and in Debian still exists :)

Regarding the work itself, I couldn’t spot any problems by having ‘sort=“age”’ everywhere, including in the IA. Still, I have two concerns:

  • You added a missing @[[!meta date@ to one post (1bc6bf2) but doing grep -L '!meta date="' news/*.mdwn I still spot 29 reports without dates. Would this be problematic?
  • Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?
  • The merge with master now conflicts after my work on Feature #11297. I don’t mind fixing this myself if you think I should be the one responsible for that :)

#7 Updated by sajolida 2017-01-20 11:50:26

I’ve added some date with e2362df :)

#8 Updated by intrigeri 2017-01-20 12:21:21

> * You added a missing @[[!meta date@ to one post (1bc6bf2) but doing grep -L '!meta date="' news/*.mdwn I still spot 29 reports without dates. Would this be problematic?

Apparently you’ve fixed that already after reassigning to me. Thanks. I’ll skip my usual rant about limiting risks of duplicating work :p

> * Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?

I don’t know what you mean with “broken translation string” so I’ll build and try to figure it myself :) Stay tuned!

> * The merge with master now conflicts after my work on Feature #11297. I don’t mind fixing this myself if you think I should be the one responsible for that :)

I’ve just solved that problem, which took less time than arguing about who should resolve merge conflicts introduced by working on master before merging ready for QA branches that affect the same files.

Note that there was one more conflict due to commit:a3a1d83 (I had done this on the master branch already to avoid new reports creating more problems in case this branch wasn’t merged promptly).

#9 Updated by intrigeri 2017-01-20 12:43:47

  • Assignee changed from intrigeri to sajolida
  • % Done changed from 30 to 80
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:
> > * Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?
>
> I don’t know what you mean with “broken translation string” so I’ll build and try to figure it myself :) Stay tuned!

Fixed on the branch (I had in mind to do that immediately after the merge, but I didn’t want to do it earlier in case the review took time, otherwise we would have had to deal with potentially lots of painful merge conflicts).

So, I think we’re good to go: please review and merge :)

#10 Updated by anonym 2017-01-24 20:48:54

  • Target version changed from Tails 2.10 to Tails_2.11

#11 Updated by spriver 2017-02-01 19:47:26

  • Assignee changed from sajolida to spriver

#12 Updated by spriver 2017-02-08 20:32:40

  • Assignee changed from spriver to intrigeri

I reviewed the changes, I’m fine with them. But when I’m issuing a --rebuild of the wiki, I’ll have 168 .po files changed (generating fuzzy entries, at least with some I checked). Is this correct?

#13 Updated by intrigeri 2017-02-10 14:01:48

> I reviewed the changes, I’m fine with them. But when I’m issuing a --rebuild of the
> wiki, I’ll have 168 .po files changed (generating fuzzy entries, at least with some
> I checked). Is this correct?

Good catch!

I’ve manually unfuzzied them all and am now going to proceed with merging! :)

#14 Updated by intrigeri 2017-02-10 15:56:33

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass