Feature #11151

Speed up the build of the website

Added by sajolida 2016-02-21 17:54:14 . Updated 2016-05-13 08:17:59 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2015-12-23
Due date:
% Done:

100%

Feature Branch:
Type of work:
Website
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

As suggested in Bug #11135#note-5 we could measure the time to build the wiki in the past (without the Installation Assistant for example) and now, and try to set a realistic goal.

Then we should investigate on concrete changes to speed things up, like Bug #11135.


Subtasks

Feature #10792: Try to group inc/steps inlines Resolved

0

Bug #11135: Disable translations of /news/reports Resolved

100

Feature #11156: Disable translations about very old releases Resolved

100

Feature #11158: Disable backlinks on website Rejected

0

Bug #11227: Website: Broken links at https://tails.boum.org/security/ Resolved sajolida

100

Feature #11249: Do stats on which pages are the least translated Resolved

0


Related issues

Related to Tails - Feature #9323: Adapt current documentation with the Installation Assistant Resolved 2015-11-27

History

#1 Updated by sajolida 2016-02-21 17:54:27

  • Tracker changed from Bug to Feature
  • Subject changed from Speed the build of the wiki to Speed the build of the website

#2 Updated by sajolida 2016-02-21 19:52:38

Reference time, on a X200 with a P8700, building from RAM on Tails takes at commit edf97d3:

real    48m53.368s
user    48m4.740s
sys 0m15.344s

#3 Updated by intrigeri 2016-02-21 20:27:08

On intri’s laptop:

current

git reset —hard 89d07a0245d7f09707aefa1fa0ad8c82ab5e8ac7 && time ./build-website —rebuild

1036.49s user 6.94s system 100% cpu 17:17.74 total

old school (no Farsi, no IA)

git reset --hard caa53db1bf9f0157207ba92f56daf23f527ac6e3^ && time ./build-wiki --rebuild

334.42s user 3.51s system 99% cpu 5:38.40 total

current minus Farsi

git reset --hard 89d07a0245d7f09707aefa1fa0ad8c82ab5e8ac7
patch -p1<<EOF
--- a/ikiwiki.setup
+++ b/ikiwiki.setup
@@ -199,7 +199,6 @@ po_master_language: en|English
 # slave languages (translated via PO files) format: ll|Langname
 po_slave_languages:
   - de|Deutsch
-  - fa|Farsi
   - fr|Français
   - pt|Português
 # PageSpec controlling which pages are translatable
EOF
find wiki/src -name *.fa.po -delete
time ./build-website --rebuild

738.55s user 5.02s system 100% cpu 12:18.17 total

current minus IA

git reset --hard 89d07a0245d7f09707aefa1fa0ad8c82ab5e8ac7
rm -rf wiki/src/{install,upgrade}
time ./build-website --rebuild

604.88s user 5.70s system 99% cpu 10:10.96 total

Conclusions

no IA, no Farsi = about 5.5 minutes, and then IA alone adds 7 minutes, Farsi alone adds 5 minutes, and adding both together adds 12 minutes => almost 17.5 minutes

#4 Updated by sajolida 2016-02-21 20:29:17

From the same commit after doing rm -r install upgrade I get:

real    30m22.595s
user    29m6.884s
sys 0m11.628s

which is 39% less and matches more or less what intrigeri gets.

Let’s try to reduce the build time by 1/3 then.

#5 Updated by sajolida 2016-02-21 21:21:24

After merging Bug #11135 and Bug #9671 I get:

real    43m19.589s
user    41m47.072s
sys 0m15.312s

which is 11% less already.

#6 Updated by sajolida 2016-02-22 17:45:41

Remove the old installation instructions (/install and /upgrade) also speeds about 4%.

#7 Updated by sajolida 2016-02-22 17:46:06

  • related to Feature #9323: Adapt current documentation with the Installation Assistant added

#8 Updated by intrigeri 2016-02-22 20:35:23

At commit:9244ee23feb893aa1bf956b825bf0d0aa225e145 (after merging the branches for Bug #11135, Bug #9671 and Feature #11156) I see:

779.86s user 4.99s system 101% cpu 12:56.60 total

i.e. –25% :)

#9 Updated by intrigeri 2016-02-22 20:35:36

  • Subject changed from Speed the build of the website to Speed up the build of the website
  • Status changed from Confirmed to In Progress

#10 Updated by anonym 2016-02-23 12:13:41

On my laptop (Intel i7-3520M CPU with the core used by ikiwiki running at 3.2 GHz, and a pretty fast SSD):

current

real    20m23.051s
user    20m39.604s
sys     0m7.780s

old school (no Farsi, no IA)

real    6m18.782s
user    6m20.312s
sys     0m3.664s

current minus Farsi

real    13m2.951s
user    13m19.244s
sys     0m5.444s

current minus IA

real    10m6.676s
user    10m8.244s
sys     0m4.436s

So very comparable figures to what you’ve all seen.

So, in my case the new target (- 1/3) would sill be 20m23s * 2/3 ~= 14m which is still pretty long. :/

#11 Updated by sajolida 2016-03-24 14:00:41

  • related to Bug #11227: Website: Broken links at https://tails.boum.org/security/ added

#12 Updated by intrigeri 2016-03-25 19:47:41

  • related to deleted (Bug #11227: Website: Broken links at https://tails.boum.org/security/)

#13 Updated by emmapeel 2016-04-29 08:35:05

In my Thinkpad the current website takes

real    9m18.094s
user    8m58.084s
sys 0m9.280s

I have 7.6 GiB memory, and an Intel® Core™ i5 CPU M 520 @ 2.40GHz × 4 processor

#14 Updated by sajolida 2016-04-30 14:39:06

  • Status changed from In Progress to Resolved
  • Assignee deleted (sajolida)

When building Feature #9323 I get 11% less in comparison with master today, for a total of 25m41.124s which is 47% less than what I had initially. So I can close this ticket :) \o/

#15 Updated by sajolida 2016-05-13 06:57:14

On an X201 with an i3 M350:

real   10m55.939s
user   10m57.772s
sys    0m5.136s

#16 Updated by sajolida 2016-05-13 08:17:59

Today I’ve got 36m20.637s on my machine. That’s 26% less than my reference time.