Bug #12689

gpg --recv-key often hangs due to unreliable keyserver

Added by dachary 2017-06-13 12:37:20 . Updated 2019-11-18 13:43:25 .

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Target version:
Tails_4.1
Start date:
2017-06-13
Due date:
% Done:

100%

Feature Branch:
bugfix/12689-more-reliable-OpenPGP-keyserver
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

description

    gpg --recv-key "2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77"

but it hangs and timesout.

Steps to Reproduce

  • Install https://tails-dl.urown.net/tails/stable/tails-amd64-3.0/tails-amd64-3.0.iso with persistence and boot with an admin password.
  • Connect to internet (via wifi, no firewall)
  • Verify connectivity with sudo apt-get update
  • Open a terminal
  • $ cat /home/amnesia/.gnupg/dirmngr.conf
    use-tor
    keyserver hkp://jirk5u4osbsr34t5.onion
  • gpg —debug-all —recv-key “2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77”

Expected Behavior

The key is imported.

Actual Behavior

gpg: reading options from ‘/home/amnesia/.gnupg/gpg.conf’
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /home/amnesia/.gnupg
gpg: DBG: chan_3 <- # Config: /home/amnesia/.gnupg/dirmngr.conf
gpg: DBG: chan_3 <- OK Dirmngr 2.1.18 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.1.18
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET — 0x22245C81E3BAEB4138B36061310F561200F4AD77
gpg: keyserver receive failed: Connection timed out

Subtasks

Related issues

Related to Tails - Feature #12210: Deal with automated tests of onion services vs Chutney Confirmed 2017-02-03
Related to Tails - Bug #17169: Seahorse can't sync keys with keyservers: Request Entity Too Large Confirmed
Related to Tails - Feature #17298: Document how to upgrade gnupg's config to the new key server Resolved
Has duplicate Tails - Feature #16575: Use a more reliable OpenPGP key server by default Duplicate 2019-03-19
Has duplicate Tails - Bug #17090: Use keys.openpgp.org as the default key server Duplicate
Blocks Tails - Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver Resolved 2017-10-04
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

History

#1 Updated by redshiftzero 2017-06-13 19:49:10

I also see this behavior, it looks like the issue is with the hkp server jirk5u4osbsr34t5.onion, e.g.:

gpg —keyserver hkp://pool.sks-keyservers.net —recv-key “2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77”

works.

#2 Updated by goupille 2017-06-14 09:26:01

  • Assignee set to goupille
  • Type of work changed from Debian to Research

I couldn’t reproduce this issue with tails 3.0, could you try yourself ?

#3 Updated by dachary 2017-06-14 12:22:16

Today it works for me as well. I guess the server was down for a day or so ?

#4 Updated by intrigeri 2017-06-14 14:42:11

  • Status changed from New to Rejected
  • Assignee deleted (goupille)

Our automated test suite also has identified some on & off issues with that keyserver pool. We can keep track of this problem this way, and if it comes back more often we should come back here.

#5 Updated by dachary 2017-06-14 16:10:11

Is there a workaround for when the default server is down ? I suppose it is included by default because it is trusted. Are there other trusted key servers to be used when this one is not available ?

#6 Updated by intrigeri 2017-06-14 16:45:58

https://sks-keyservers.net/overview-of-pools.php

(But no, there’s not really any “trusting” involved.)

#7 Updated by fowlslegs 2017-06-14 21:03:16

gpg --keyserver hkp://pool.sks-keyservers.net --recv-key "2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77"

fails on Tails 3 beta 4 with error message

> gpg:keyserver receive failed: No keyserver available.

So at least for me @redshiftzero’s workaround does not seem to work. That said the default keyserver is described as an “experimental Tor OnionBalance hidden service is running as hkp://jirk5u4osbsr34t5.onion consisting of the servers marked with Tor support in the status list as backend,” which IMO Tails should re-consider using by default precisely because of it’s experimental nature and the fact it doesn’t seem highly reliable.

As @intrigeri correctly points out and as discussed further in https://github.com/freedomofpress/securedrop/pull/1804, there is not much “trusting” of keyservers. Would it not be better to simply go back to using the HKPS SKS keyservers, which seem to have greater uptime/ reliability? I’ll leave that question up to the Tails team, but if I were part of it I think I would be arguing yes. As much as I like to use/ support use of onion services, use for keyservers/ keyserver pools combined with OnionBalance does in fact seem too experimental for inclusion in the default Tails GPG configuration.

#8 Updated by eloquence 2018-10-12 22:21:44

I still encounter the same hangs with the default configuration, even in Tails 3.9.1. This issue was rejected a year ago in favor of monitoring long term trends; would it be possible to get an update on what the automated test suite shows in terms of reliability of the default configuration? Apologies if this is already covered in another issue I missed in search.

#9 Updated by intrigeri 2018-11-16 09:52:48

  • related to Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver added

#10 Updated by intrigeri 2018-11-16 09:56:43

  • related to deleted (Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver)

#11 Updated by intrigeri 2018-11-16 09:59:03

  • Status changed from Rejected to Confirmed
  • Assignee set to emmapeel
  • QA Check set to Info Needed

> I still encounter the same hangs with the default configuration,
> even in Tails 3.9.1.

> This issue was rejected a year ago in favor of monitoring long term trends; would it be possible to get an update on what the automated test suite shows in terms of reliability of the default configuration?

Sadly, we’re not going to get the data I expected this way: we don’t use this keyserver in our test suite anymore since we started using Chutney (Bug #12211).
So to monitor long term trends, we’ll need to rely on feedback from our users, such as your report ⇒ dear emmapeel, how frequently do users report issues with fetching OpenPGP keys from keyservers?

#12 Updated by intrigeri 2019-03-08 14:44:42

  • Assignee deleted (emmapeel)
  • QA Check deleted (Info Needed)

As a user of this keyserver myself, I confirm it’s not up and working all the time.

#13 Updated by intrigeri 2019-03-23 19:05:45

  • has duplicate Feature #16575: Use a more reliable OpenPGP key server by default added

#14 Updated by intrigeri 2019-03-23 19:06:08

Feature #16575 has more info and ideas.

#15 Updated by intrigeri 2019-04-03 05:43:35

The “keyserver network dying” thread on the monkeysphere mailing list (started on 2019-04-02) suggests that not only the Onion keyservers pool has problems these days.

#16 Updated by intrigeri 2019-08-30 12:26:28

  • Subject changed from gpg --recv-key hangs to gpg --recv-key often hangs due to unreliable keyserver

This is one of the main cause of test suite failures these days (Bug #14770).

#17 Updated by intrigeri 2019-08-30 12:27:49

  • blocks Bug #14770: "Fetching OpenPGP keys" scenarios are fragile: communication failure with keyserver added

#18 Updated by intrigeri 2019-08-30 12:27:58

#19 Updated by intrigeri 2019-09-01 14:24:10

Buster 10.1 will have gnupg2 2.2.12-1+deb10u1. This includes one change that’s relevant here: “use keys.openpgp.org as the default keyserver”. Note, however, that this change won’t affect Tails as-is, because we configure our own keyserver (git grep -F jirk5u4osbsr34t5.onion; and IIRC Torbirdy does the same for Enigmail); these settings can have been copied to persistent ~/.gnupg/dirmngr.conf.

#20 Updated by sajolida 2019-09-24 15:38:05

  • related to Bug #17090: Use keys.openpgp.org as the default key server added

#21 Updated by intrigeri 2019-09-30 17:15:56

  • has duplicate Bug #17090: Use keys.openpgp.org as the default key server added

#22 Updated by intrigeri 2019-09-30 17:16:09

  • related to deleted (Bug #17090: Use keys.openpgp.org as the default key server)

#23 Updated by intrigeri 2019-10-19 15:35:57

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri

See commit:bdcd73bfc5508dbcde552ade36f1db216d32f958 that removes the code to migrate keyservers in persistent GnuPG configuration. It could be useful, if/when we migrate to another keyserver than hkp://jirk5u4osbsr34t5.onion, if we want to forcibly migrate users’ persistent config.

I’ll give a quick try to switching to keys.openpgp.org’s Onion service (which should be the cheapest way for us to use it, and preserves end-to-end encryption and authentication of the server in Seahorse, which doesn’t support hkps://), and our test suite will tell us what kind of extra work is needed.

#24 Updated by intrigeri 2019-10-19 15:59:08

  • Feature Branch set to bugfix/12689-more-reliable-OpenPGP-keyserver

#25 Updated by intrigeri 2019-10-20 09:47:59

  • related to Feature #12210: Deal with automated tests of onion services vs Chutney added

#26 Updated by intrigeri 2019-10-20 15:16:12

  • related to Bug #17169: Seahorse can't sync keys with keyservers: Request Entity Too Large added

#27 Updated by intrigeri 2019-10-20 15:27:28

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Target version set to Tails_4.0
  • Type of work changed from Research to Code

I’ve got something that I’m reasonably satisfied with:

  • Works when tested manually on the command line and in Seahorse.
  • Most affected test cases got more robust.

Caveats:

  • Enigmail still uses the old-school keyserver onion pool. See commit:b53b55c599be81ccb8b1bd7b836c1c558caf3ac9 for the rationale.
  • I don’t know if Bug #17169 happens more often than it used to. This feature is disabled by default (and a bad idea in most cases) so I don’t see it as a blocker.
  • I’m not updating persistent ~/.gnupg/dirmngr.conf automatically. Given keyservers are mostly used by technical users, IMO it’ll be good enough to add a note about this in the release notes.

I realize it’s very late in the 4.0 cycle to merge this so I’m happy to postpone to 4.1 if no reviewer has time to look at the branch, or if the reviewer prefers it this way :)

#28 Updated by intrigeri 2019-10-21 06:10:07

  • Target version changed from Tails_4.0 to Tails_4.1

#29 Updated by hefee 2019-11-12 14:50:42

  • Assignee set to hefee

#30 Updated by Anonymous 2019-11-12 15:30:28

  • Status changed from Needs Validation to Fix committed
  • % Done changed from 0 to 100

Applied in changeset commit:tails|c73634f6f076542621c9126b3b3cdc434b4dee7e.

#31 Updated by hefee 2019-11-12 15:46:13

  • Assignee deleted (hefee)

#32 Updated by intrigeri 2019-11-18 13:43:25

  • Status changed from Fix committed to Resolved

#33 Updated by intrigeri 2019-12-12 07:39:46

  • related to Feature #17298: Document how to upgrade gnupg's config to the new key server added