Feature #14588

Self-host our website

Added by intrigeri 2017-09-04 09:47:54 . Updated 2018-11-15 18:00:06 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Infrastructure
Target version:
Tails_3.11
Start date:
2018-10-03
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

New design

ikiwiki and web hosting

ikiwiki would run on www.lizard which also serves the generated content and runs ikiwiki.cgi (until Feature #9174 is done):

  • ikiwiki on www.lizard pushes changes back to the master/central tails.git repo (PO files updates, changes done in the web interface aka. ikiwiki.cgi). That’s exactly what the current production setup does.
  • Ensure logging policy is OK:
    • nginx’ own logs: no IPs: OK
    • Journal, if relevant: OK, I’ve not found anything nginx-related in the Journal, even after hitting a 404
  • There’s currently a cronjob that extracts connection stats from access.log (Tails boots, downloads of the detached ISO signature) and emails them to tails@b.o. We need to import that too.
  • The current setup is heavily based on Apache features, while we usually run nginx on our infra:

language negotiation

I got language negotiation to work fine with this included in the http {} block:

map $http_accept_language $lang {
    default en;
    ~*^de de;
    ~*^fa fa;
    ~*^es es;
    ~*^fr fr;        
    ~*^it it;
    ~*^pt pt;
}

… and this included in the vhost:

location / {
    try_files $uri $uri/index.$lang.html $uri/index.en.html $uri/index.html =404;
}

Our initial options were:

Master/central tails.git repo

The master/central tails.git repo would move to git.puppet.t.b.o (sic) i.e. hosted on lizard’s gitolite:

  • we’ll be able to use gitolite to manage ACLs which is better than the current setup at b.o, e.g. we can manage users & keys ourselves, give access to some more people to a subset of branches (e.g. we could allow some developers to push to feature/* and bugfix/* so their stuff is built and tested in Jenkins, but they would not be allowed to push to protected branches such as master, stable, testing and devel)
  • maybe we can use a better CNAME that does not say “puppet”; but anyway, unless using the Tor onion service, a SSH config is needed because that Gitolite runs behind a non-standard port, so contributors with push access can as well use the onion service and we don’t bother about DNS names
  • Git validation hook (file size, obsolete rewritten history, etc.) copied from the one that’s set up at b.o
  • Git hook that triggers an ikiwiki update with the pingee plugin
  • post-update hook that pushes to all mirrors (copied from b.o)

Migration plan

migrate the master/central tails.git repo

  1. copy everything over to lizard and have gitolite push updates to all mirrors (see design above); mirrors, Git validation hook
  2. have b.o disable their Git hook that pushes updates to mirrors, except lizard
  3. migrate the other repos hosted at boum.org (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary) and push to boum.org and immerda on update
  4. announce downtime to Git committers
  5. forbid Git committers write access to the former master/central tails.git
  6. drop the ACL that allows the former repo to push to the new one
  7. have b.o disable their Git hook that pushes updates to lizard
  8. have the new master/central tails.git push to b.o on updates so that our website, still running at b.o at this point, is updated
  9. update contribute/git doc
  10. tell Git committers they can push again and ask them to update their config

bugfixes

  1. fix updating master Git repo upon PO files updates / edit on the live website

prepare the new web hosting setup

  1. set up the basic web server (including LE) and ikiwiki stuff on www.lizard under some temporary vhost name
  2. CGI: tweak fcgiwrap.socket so that only the www-data user has access to the socket
  3. have the master/central repo trigger an ikiwiki update (on Git push) on www.lizard as well, using the pingee plugin => we can test how it behaves
  4. deal with the other repos needed to build our website (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary): give them each a post-update hook that triggers a git pull in a non-bare clone on www.lizard and then calls the pingee plugin
  5. import relevant stuff from the Apache vhost at b.o (e.g. HSTS)
  6. try to set Content-Language header correctly (not needed since we specify this in our HTML)
  7. logging
  8. htaccess
  9. avoid HTTP cache issues such as Bug #16049
  10. see design above for more things to do
  11. ensure there’s enough space in www.lizard:/var/log/nginx to host the amount of logs we need to keep around (even without hosting our website, at some point that directory took too much space: Bug #12425)
  12. set up rss2email
  13. prevent ikiwiki.cgi from recording IP addresses in commit messages

migrate to the new web hosting setup

Once happy with the new hosting setup:

  1. allow ikiwiki on www.lizard to push to the canonical repo’s master branch
  2. rename/copy it to support tails.b.o, adjust ikiwiki.setup accordingly, rebuild
  3. copy X.509 cert+key from the old website to the new one
  4. point tails.b.o in /etc/hosts to lizard on a test machine and ensure the website hosted there works as expected
  5. point DNS to lizard
  6. update /etc/hosts:
    1. on www: commit e4b20b4
    2. on other systems: look for jenkins.tails.boum.org in manifests/classes.pp
  7. update Git hooks: s/new\.tails\.boum\.org/tails\.boum\.org/ (or maybe just use www.lizard)
  8. drop/update everything (e.g. hooks) about new.tails.b.o
  9. ensure Let’s Encrypt certificate renewal has a chance to work

And once we’re convinced the new hosting setup works well enough:

  1. cronjob that emails stats from logs
  2. have b.o disable their rss2email feed, then set the email recipient for ours to the production one
  3. replace content of .htaccess with a pointer to https://git-tails.immerda.ch/puppet-tails/tree/templates/website/nginx/rewrite_rules.conf.erb and notify the translation platform team (who runs a staging version of our website on Apache)
  4. drop the temporary vhost and its files (whose name include news.tails.boum.org)
  5. drop permissions from b.o to all Git repos on lizard and immerda
  6. ask b.o to delete our website, Git repo, cronjob, and the corresponding SSH keys
  7. If time allows, do Feature #9174 and/or Feature #12408 (time does not allow)

Subtasks

Related issues

Related to Tails - Feature #10034: Translation web platform Resolved 2015-08-14
Related to Tails - Bug #6907: ikiwiki po plugin does not play well with inline directives Resolved 2018-12-03 2019-03-12
Related to Tails - Feature #11815: Have Tails::Download::HTTPS require TLS 1.2+ Resolved 2016-09-20
Related to Tails - Feature #16028: /mirrors.json is not synced with gitolite@d53ykjpeekuikgoq.onion Resolved 2018-10-03
Related to Tails - Bug #16123: Test suite broken on Jenkins since we self-host our website Resolved 2018-11-13
Related to Tails - Bug #16124: URLs without explicit .html are not redirected anymore Resolved 2018-11-13
Related to Tails - Bug #16142: Consider giving sajolida access to the web logs of our website Rejected 2018-11-19
Related to Tails - Bug #16542: Refresh ikiwiki wrappers when ikiwiki.setup.erb is updated Resolved 2019-03-07
Related to Tails - Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins Resolved 2015-10-28
Blocks Tails - Feature #12408: Ensure our website is ready for temporary surge of new users Confirmed 2017-03-29
Blocks Tails - Feature #9174: Migrate our blueprints out of tails.git and tails.boum.org Confirmed 2015-04-07
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 2017-06-30
Blocks Tails - Bug #13450: Implement CSP HTTP header Rejected 2017-07-10
Blocks Tails - Feature #16091: Rethink our caching of static (CSS, JavaScript, more?) files Confirmed 2018-11-02
Blocks Tails - Bug #12113: Make sure tails.boum.org is available over IPv6 Confirmed 2017-01-05

History

#1 Updated by intrigeri 2017-09-11 12:30:20

  • blocks Feature #12408: Ensure our website is ready for temporary surge of new users added

#2 Updated by sajolida 2018-01-21 12:44:29

  • blocks Feature #9174: Migrate our blueprints out of tails.git and tails.boum.org added

#3 Updated by Anonymous 2018-03-01 10:26:33

#4 Updated by intrigeri 2018-04-08 13:02:00

  • blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#5 Updated by intrigeri 2018-04-08 13:02:37

  • Target version changed from 2019 to Tails_3.10.1

#6 Updated by intrigeri 2018-05-02 10:16:18

  • Description updated
  • Status changed from Confirmed to In Progress

#7 Updated by intrigeri 2018-05-02 10:17:44

  • Description updated

#8 Updated by intrigeri 2018-05-02 10:23:27

  • Description updated

#9 Updated by intrigeri 2018-05-02 12:43:49

  • Description updated

#10 Updated by intrigeri 2018-05-02 12:48:04

  • Description updated

#11 Updated by intrigeri 2018-05-02 12:54:34

  • Description updated

#12 Updated by intrigeri 2018-05-02 12:57:41

  • Description updated

#13 Updated by intrigeri 2018-05-02 13:00:38

  • Description updated

#14 Updated by intrigeri 2018-05-02 13:09:12

  • Description updated

#15 Updated by intrigeri 2018-05-02 13:48:52

  • Description updated

#16 Updated by intrigeri 2018-05-02 14:27:25

  • Description updated

#17 Updated by intrigeri 2018-05-02 14:29:56

  • Description updated

#18 Updated by intrigeri 2018-05-02 14:39:15

  • Description updated

#19 Updated by intrigeri 2018-05-02 14:40:19

  • Description updated

#20 Updated by intrigeri 2018-05-02 14:55:27

  • Description updated

#21 Updated by intrigeri 2018-05-02 16:12:10

  • Description updated

#22 Updated by intrigeri 2018-05-02 16:23:38

  • Description updated

#23 Updated by intrigeri 2018-05-02 16:42:41

  • Description updated

#24 Updated by intrigeri 2018-07-10 09:06:17

#25 Updated by intrigeri 2018-08-18 09:42:08

  • blocks Bug #13450: Implement CSP HTTP header added

#26 Updated by Anonymous 2018-09-03 14:27:30

  • related to Bug #12113: Make sure tails.boum.org is available over IPv6 added

#27 Updated by intrigeri 2018-09-25 08:55:51

#28 Updated by intrigeri 2018-09-27 13:08:11

  • Description updated

#29 Updated by intrigeri 2018-09-27 13:35:59

  • Description updated

#30 Updated by intrigeri 2018-09-27 13:47:06

  • Description updated

#31 Updated by intrigeri 2018-09-27 14:05:49

  • Description updated

#32 Updated by intrigeri 2018-09-27 14:06:33

  • Description updated

#33 Updated by intrigeri 2018-09-27 14:27:05

  • Description updated

#34 Updated by intrigeri 2018-09-27 14:27:37

  • Description updated

#35 Updated by intrigeri 2018-09-27 15:55:37

  • blocked by deleted (Feature #15202: Onboard new members to the mirror team)

#36 Updated by intrigeri 2018-09-27 15:57:28

  • Description updated

#37 Updated by intrigeri 2018-09-27 16:32:42

  • Description updated

#38 Updated by intrigeri 2018-09-27 16:37:27

  • Description updated

#39 Updated by intrigeri 2018-09-27 22:20:19

  • Description updated
  • % Done changed from 0 to 20

#40 Updated by intrigeri 2018-09-28 07:08:58

  • Description updated

#41 Updated by intrigeri 2018-09-28 09:50:14

  • Description updated

#42 Updated by intrigeri 2018-09-28 11:15:23

  • Description updated

#43 Updated by intrigeri 2018-09-28 11:17:26

  • Description updated

#44 Updated by bertagaz 2018-09-29 08:42:25

While fixing some unrelated stuff on isobuilder4, I noticed that apache2 was now installed and running on our isobuilders. My research pointed to commit puppet-tails:b24dd2d6ee58d500367f8d5ffc849a260b7f89b3 which I think is the root cause. The timing of apache2 installation shown by etckeeper seems to point to that commit. It’s probably due to the fact that the xapian-omega package recommends apache2 | httpd-cgi.

#45 Updated by intrigeri 2018-10-02 13:32:24

> While fixing some unrelated stuff on isobuilder4, I noticed that apache2 was now installed and running on our isobuilders. My research pointed to commit puppet-tails:b24dd2d6ee58d500367f8d5ffc849a260b7f89b3 which I think is the root cause. The timing of apache2 installation shown by etckeeper seems to point to that commit. It’s probably due to the fact that the xapian-omega package recommends apache2 | httpd-cgi.

Good catch, thanks! Fixed.

#46 Updated by intrigeri 2018-10-07 10:43:45

  • Description updated

#47 Updated by intrigeri 2018-10-10 15:41:23

  • Target version changed from Tails_3.10.1 to Tails_3.11

I’ve not scheduled my next work session on this yet but it’s clear it won’t be before 3.10.

#48 Updated by intrigeri 2018-10-12 12:02:04

  • blocked by deleted (Feature #15798: Jenkins access for new FT members)

#49 Updated by intrigeri 2018-10-12 20:38:22

  • related to Bug #6907: ikiwiki po plugin does not play well with inline directives added

#50 Updated by intrigeri 2018-10-16 08:30:54

  • Description updated

#51 Updated by intrigeri 2018-10-30 07:06:51

  • related to Feature #11815: Have Tails::Download::HTTPS require TLS 1.2+ added

#52 Updated by intrigeri 2018-11-02 15:37:28

  • blocks Feature #16091: Rethink our caching of static (CSS, JavaScript, more?) files added

#53 Updated by Anonymous 2018-11-02 19:52:56

  • related to deleted (Bug #12113: Make sure tails.boum.org is available over IPv6)

#54 Updated by Anonymous 2018-11-02 19:53:03

  • blocks Bug #12113: Make sure tails.boum.org is available over IPv6 added

#55 Updated by intrigeri 2018-11-09 07:35:57

  • related to Feature #16028: /mirrors.json is not synced with gitolite@d53ykjpeekuikgoq.onion added

#56 Updated by intrigeri 2018-11-09 07:36:16

  • Priority changed from Normal to Elevated
  • % Done changed from 100 to 10

#57 Updated by intrigeri 2018-11-09 10:25:18

  • Description updated

#58 Updated by intrigeri 2018-11-09 10:33:00

  • Description updated

#59 Updated by intrigeri 2018-11-09 10:55:31

  • Description updated

#60 Updated by intrigeri 2018-11-09 11:06:42

  • Description updated

#61 Updated by intrigeri 2018-11-09 11:20:03

  • Description updated

#62 Updated by intrigeri 2018-11-09 12:20:25

  • Description updated

#63 Updated by intrigeri 2018-11-09 13:21:03

  • Description updated

#64 Updated by intrigeri 2018-11-09 14:27:55

  • Description updated

#65 Updated by intrigeri 2018-11-09 14:54:13

  • Description updated

#66 Updated by intrigeri 2018-11-09 15:14:51

  • Description updated

#67 Updated by intrigeri 2018-11-09 15:32:00

  • Description updated

#68 Updated by intrigeri 2018-11-09 15:34:06

  • Description updated

#69 Updated by intrigeri 2018-11-10 09:09:03

  • Description updated
  • % Done changed from 10 to 20

#70 Updated by intrigeri 2018-11-10 09:48:49

  • Description updated

#71 Updated by intrigeri 2018-11-10 10:44:32

  • Description updated

#72 Updated by intrigeri 2018-11-10 11:04:56

  • Description updated

#73 Updated by intrigeri 2018-11-10 11:54:29

  • Description updated

#74 Updated by intrigeri 2018-11-10 13:37:23

  • Description updated

#75 Updated by intrigeri 2018-11-10 13:41:55

  • Description updated

#76 Updated by intrigeri 2018-11-10 14:13:26

  • Description updated

#77 Updated by intrigeri 2018-11-10 14:22:44

  • Description updated

#78 Updated by intrigeri 2018-11-10 16:24:37

  • Description updated
  • Assignee changed from intrigeri to groente
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

Seems to work fine. All relevant commits in puppet-tails.git should reference this ticket.

#80 Updated by intrigeri 2018-11-11 10:03:37

FTR I took a good look at Munin (both zooming in www.lizard -specific metrics and global lizard ones) and it’s hard to notice any impact of switching the production website to lizard. The only significant change I could spot is that our outgoing network traffic has increased by ~1-2 sustained MBit/s, which is expected and matches the decrease we can see on boum.org’s side. I’m now very tempted to stress-test the new setup and adjust if/as needed for Feature #12408 but I’ll take it easy: preliminary testing with ab(1) showed that with 10k connections and concurrency=500, the new setup is tremendously more efficient and reliable than the previous one (as in: the new setup can handle this load nicely while the old one would fall apart), which is a pretty good first step :)

#81 Updated by intrigeri 2018-11-13 09:39:16

  • related to Bug #16123: Test suite broken on Jenkins since we self-host our website added

#82 Updated by groente 2018-11-13 10:18:14

  • Assignee changed from groente to intrigeri
  • QA Check changed from Ready for QA to Pass

#83 Updated by intrigeri 2018-11-15 09:45:56

  • related to Bug #16124: URLs without explicit .html are not redirected anymore added

#84 Updated by intrigeri 2018-11-15 18:00:06

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

#85 Updated by intrigeri 2018-11-19 14:00:58

  • related to Bug #16142: Consider giving sajolida access to the web logs of our website added

#86 Updated by intrigeri 2019-03-07 08:30:18

  • related to Bug #16542: Refresh ikiwiki wrappers when ikiwiki.setup.erb is updated added

#87 Updated by intrigeri 2019-08-10 19:00:04

  • related to Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins added