Bug #14770

"Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver

Added by intrigeri 2017-10-04 08:21:30 . Updated 2019-11-18 13:45:33 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Tails_4.1
Start date:
2017-10-04
Due date:
% Done:

0%

Feature Branch:
bugfix/12689-more-reliable-OpenPGP-keyserver
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

The temporary solution implemented in Bug #12211 is not robust enough: while doing Feature #12291 I see lots of failures in August. Thankfully we’ve discussed (and agreed on) better options on Bug #12211.

Subtasks

Related issues

Related to Tails - Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver Resolved 2017-02-03
Related to Tails - Bug #10378: The "Tails OpenPGP keys" scenario is fragile Resolved 2015-10-15
Related to Tails - Feature #9519: Make the test suite more deterministic through network simulation In Progress 2015-06-02
Related to Tails - Bug #15415: Unreliable key server operations Resolved 2018-03-14
Related to Tails - Bug #17169: Seahorse can't sync keys with keyservers: Request Entity Too Large Confirmed
Blocked by Tails - Bug #12689: gpg --recv-key often hangs due to unreliable keyserver Resolved 2017-06-13
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by intrigeri 2017-10-04 08:21:46

  • blocked by Feature #12292: Deal with September 2017 false positive scenarios added

#2 Updated by intrigeri 2017-10-04 08:21:55

  • related to Bug #12211: Adapt GnuPG automated tests after switching to an Onion keyserver added

#3 Updated by intrigeri 2017-10-04 08:23:04

  • related to Bug #10378: The "Tails OpenPGP keys" scenario is fragile added

#4 Updated by anonym 2017-11-07 13:38:29

  • Target version changed from Tails_3.3 to Tails_3.5

#5 Updated by intrigeri 2017-12-03 21:06:20

  • related to Feature #9519: Make the test suite more deterministic through network simulation added

#6 Updated by intrigeri 2017-12-03 21:06:31

Wrt. the best long-term option we’ve selected on Bug #12211 (“Run a local mock keyserver onion. This doesn’t depend on the Internet => potentially 100% robust”), I’ve read that Schleuder 3 does mock the keyserver in its test suite. Likely that’s in Ruby so we could maybe reuse it easily :)

#7 Updated by anonym 2017-12-07 12:36:13

  • Target version deleted (Tails_3.5)

#8 Updated by intrigeri 2017-12-16 13:34:57

  • Type of work changed from Research to Code

intrigeri wrote:
> bertagaz said on Feature #12290#note-8 that these failures had disappeared in September, so I’m not flagging these scenarios as fragile: instead, while doing Feature #12292 anonym should check this and act accordingly, hence making this a Research ticket on anonym’s plate for this month.

I’ve just seen this happen again (https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/1085/) so I’ve tagged these scenarios as fragile.

#9 Updated by intrigeri 2017-12-16 13:35:18

  • Description updated

#10 Updated by intrigeri 2017-12-16 13:35:25

  • blocks deleted (Feature #12292: Deal with September 2017 false positive scenarios)

#11 Updated by intrigeri 2017-12-16 13:38:14

  • Status changed from Confirmed to In Progress

Applied in changeset commit:1db04699bd95dccbee96e0cc51b0fc233fce75a8.

#12 Updated by sajolida 2018-03-14 16:50:10

  • related to Bug #15415: Unreliable key server operations added

#13 Updated by intrigeri 2018-11-16 09:52:48

  • related to Bug #12689: gpg --recv-key often hangs due to unreliable keyserver added

#14 Updated by intrigeri 2018-11-16 09:56:43

  • related to deleted (Bug #12689: gpg --recv-key often hangs due to unreliable keyserver)

#15 Updated by intrigeri 2019-08-30 12:27:18

  • Assignee deleted (anonym)

Every such scenario has failed a few dozens of times on Jenkins in the last 2 months. But that’s an actual bug in Tails: Bug #12689. So this ticket is not actionable until that bug is fixed.

#16 Updated by intrigeri 2019-08-30 12:27:50

  • blocked by Bug #12689: gpg --recv-key often hangs due to unreliable keyserver added

#17 Updated by intrigeri 2019-09-09 06:13:13

  • Subject changed from "Fetching OpenPGP keys" scenarios are fragile to "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver

(Let’s disambiguate from other tickets that track other reasons why such scenarios are fragile.)

#18 Updated by intrigeri 2019-10-19 15:59:15

  • Assignee set to intrigeri
  • Feature Branch set to bugfix/12689-more-reliable-OpenPGP-keyserver

#19 Updated by intrigeri 2019-10-19 16:01:33

#20 Updated by intrigeri 2019-10-20 13:30:00

While working on the underlying Tails bug (Bug #12689), I discovered one more reason why what was set up on Bug #12211 is fragile: it implicitly relies on the assumption that every member of the actual target pool (pool.sks-keyservers.net) can be queried via hkp://$IP:11373/ with any HTTP Host header of our choosing (in this case, the .onion run by Chutney). As it happens, this assumption is invalid for at least one member of that pool: 193.224.163.43 has Apache VirtualHost:s explicitly configured for the hostnames it supports, but the fallback VirtualHost that we hit is not a keyserver.

Demonstration:

  • curl --resolve pool.sks-keyservers.net:11371:193.224.163.43 'http://pool.sks-keyservers.net:11371/pks/lookup?op=get&options=mr&search=0x7C84A74CFB12BC439E81BA78C92949B8A63BB098' works fine because it sends the correct Host header in the HTTP request
  • curl 'http://193.224.163.43:11371/pks/lookup?op=get&options=mr&search=0x7C84A74CFB12BC439E81BA78C92949B8A63BB098' fails with a 404 error, because http://193.224.163.43:11371/ merely serves the default Apache homepage and is not backed by a keyserver

I can’t think of a way to fix this, apart of having our test suite use as its upstream keyserver one that meets this now-explicit requirement, instead of any random member of the HKP pool.

#21 Updated by intrigeri 2019-10-20 15:16:18

  • related to Bug #17169: Seahorse can't sync keys with keyservers: Request Entity Too Large added

#22 Updated by intrigeri 2019-10-20 15:28:02

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Target version set to Tails_4.0

(Same as Bug #12689#note-27)

#23 Updated by intrigeri 2019-10-21 06:10:06

  • Target version changed from Tails_4.0 to Tails_4.1

#24 Updated by hefee 2019-11-12 15:45:35

  • Status changed from Needs Validation to Fix committed

applied in c73634f6f076542621c9126b3b3cdc434b4dee7e.

#25 Updated by intrigeri 2019-11-18 13:45:33

  • Status changed from Fix committed to Resolved