Bug #15548

Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

Added by goupille 2018-04-20 14:28:04 . Updated 2020-03-15 10:21:00 .

Status:
Confirmed
Priority:
Normal
Assignee:
goupille
Category:
Tor configuration
Target version:
Start date:
2018-05-09
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:

Starter:
Affected tool:
Deliverable for:

Description

Several users reported having issues with obfs4 since a few weeks. I tried with the following bridges


obfs4 76.74.178.195:9443 406A8B5869B72221036291407EC3688C69995F80 cert=FY2R16JOoE2VNCU2gVLWBj6Gg+YBP7mTLU5zl12Fz9iC5TQG6SqE71CFhD3zIuJcEFrcMQ iat-mode=0
obfs4 91.89.79.175:43099 681DD069768F8DF5C94284FFE8CEA600C6EAFB86 cert=Dux9bBro6xfxb7rUr8wdp0TephA7wMV0MjPMAcNqkU/r1Q+56EpHypOFZHRZFDukZNCHYg iat-mode=0
obfs4 34.203.58.40:9443 A3C8B95009DB61BE34B2628C9B9B4E66FCA32FA3 cert=etaJXxltJNQxJlKxVgGXWakop242cHNtzxG/GKroXYPTlzvmsYh818WrcIJ5cZ4DbZREJA iat-mode=0

and ended up with Tor launcher stuck on “Loading Network Consensus” for at least 15 minutes. I saw this behaviour two times on three trials.

I attached a whisperback report and the content of /var/log/tor/log, they are showing a weird jump back in the time :

Apr 20 14:17:56.000 [info] connection_edge_process_inbuf(): data from edge while in 'waiting for connect response' state. Sending it anyway. package_partial=1, buflen=214
Apr 20 14:17:56.000 [info] handle_control_authenticate(): Authenticated control connection (23)
Apr 20 11:30:00.000 [notice] Interrupt: exiting cleanly.
Apr 20 11:30:00.000 [info] or_state_save(): Saved state to "/var/lib/tor/state"

Files

bug report - obfs4 (567230 B) goupille, 2018-04-20 14:24:19
journal.txt (410680 B) mercedes508, 2018-06-15 12:22:00
torlog.txt (310522 B) mercedes508, 2018-06-15 12:22:11

Subtasks


Related issues

Related to Tails - Bug #15743: Too much log when using obfs4 in Tails Resolved 2018-07-19
Related to Tails - Bug #16972: tordate sometimes breaks obfs4 by messing with a correct clock Resolved
Related to Tails - Feature #15599: Improve known issues about clock going backwards Resolved 2018-05-09
Related to Tails - Bug #17525: obfs4 documentation does not mention the need for a hardware clock set to UTC Confirmed
Has duplicate Tails - Bug #9268: obfs4 bridges often don't work (maybe MTU?) Duplicate 2015-04-21
Blocked by Tails - Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC Resolved 2018-01-15
Blocked by Tails - Feature #5774: Robust time syncing In Progress 2015-05-17

History

#1 Updated by intrigeri 2018-04-30 10:46:07

  • Subject changed from Tails can't establish a connection with obfs4 bridges to Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC
  • Category set to Tor configuration
  • Assignee changed from intrigeri to goupille
  • Target version set to Tails_3.7
  • QA Check set to Info Needed
Apr 20 14:17:55 amnesia time[16223]: Waiting for a Tor consensus file to contain a valid time interval
Apr 20 14:17:55 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE cached-descriptors.new
Apr 20 14:17:56 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE unverified-microdesc-consensus.tmp
Apr 20 14:17:56 amnesia time[16259]: A Tor consensus file now contains a valid time interval.
Apr 20 14:17:56 amnesia time[16262]: We do not have a Tor verified consensus, let's use the unverified one.
Apr 20 14:17:56 amnesia time[16263]: Waiting for the chosen Tor consensus file to contain a valid time interval...
Apr 20 14:17:56 amnesia time[16265]: The chosen Tor consensus now contains a valid time interval, let's use it.
Apr 20 14:17:56 amnesia time[16269]: Tor: valid-after=2018-04-20 10:00:00 | valid-until=2018-04-20 13:00:00
Apr 20 14:17:56 amnesia time[16272]: Current time is 2018-04-20 14:17:56
Apr 20 14:17:56 amnesia time[16277]: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]
Apr 20 11:30:00 amnesia systemd[1]: Stopping Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Stopped Anonymizing overlay network for TCP.
-- Subject: Unit tor@default.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has finished shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Starting Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun starting up.

So there was a mismatch between the system clock (that we initialize to whatever the hardware clock says) and the Tor consensus we got.
To me this looks like “Problems when the system clock goes backwards” from https://tails.boum.org/support/known_issues/.

Could you please ensure the hardware clock is set to UTC (not local time) and retry?

#2 Updated by intrigeri 2018-04-30 10:46:37

  • related to Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#3 Updated by intrigeri 2018-04-30 10:51:46

Assuming goupille confirms this was caused by a RTC set to localtime >> UTC: if we get something done on Bug #15168 soon, great, this will fix this problem; otherwise, I’m starting to think we should notice when we would like to set the clock backwards and point the user to our doc that explains how they should set their RTC to UTC.

#4 Updated by goupille 2018-04-30 13:25:02

  • Assignee changed from goupille to intrigeri

those logs are coming from a VM, I rebooted the VM three times and could reproduce the error the first and the third times. the lgos are coming from the last session. libvirt says that the VM clock is set to UTC.

jumping to 11:30:00 exactly seems really weird, iirc, gnome’s time was set to paris time when the error occured, so UTC should have been 12:17:56, no ?

#5 Updated by intrigeri 2018-04-30 13:50:26

  • Assignee changed from intrigeri to goupille

goupille wrote:
> jumping to 11:30:00 exactly seems really weird,

That’s not random, it’s our time sync system in play: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]. This happens because “Current time is not in valid Tor range”, which should never happen if the RTC is in UTC and the host system’s clock is correct.

> iirc, gnome’s time was set to paris time when the error occured

My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync’ is done, as expected. So if you’re seeing localtime instead, then it strongly suggests that either your VM actually has a RTC set to Paris time (and not to UTC) — which could be a configuration error or a libvirt/QEMU bug — or the host system has a wrong clock or timezone settings. In any case, the result is the same: Tails is given Paris time and believes it’s UTC, which results in “Current time is not in valid Tor range”.

Please share:

  • the content of /etc/timezone on the host system
  • the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
  • the full clock section of your test VM’s definition (virsh dumpxml $VM)

#6 Updated by intrigeri 2018-04-30 13:57:35

Also, if you want to double check, start Tails, enable offline mode in the Greeter, and check the system time after login: it should be the correct UTC time; if it’s not, then the underlying virtualization stack or host system is either buggy or badly configured.

#7 Updated by goupille 2018-04-30 16:08:36

intrigeri wrote:

> My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync’ is done, as expected.

my mistake, my VMs are displaying UTC time even before the time sync…

> * the content of /etc/timezone on the host system
> * the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
> * the full clock section of your test VM’s definition (virsh dumpxml $VM)

I sent you an email with those informations (ticket number in the subject)

#8 Updated by intrigeri 2018-04-30 17:34:30

> my mistake, my VMs are displaying UTC time even before the time sync…

OK, then please retry and upon failure, send me:

  • the content of the Journal
  • the Tor log
  • the accurate time (with timezone) from outside of Tails

Rationale: I want to check if the Tor consensus is OK of not.

#9 Updated by Mackpetrova 2018-05-03 08:03:08

Set network proxy to manual and use those settings

#10 Updated by Anonymous 2018-05-09 09:29:43

this has been a problem for soooo long.
time syncing has been failing for months if not years, whenever a bridge (obfs4) is selected right at the startup (non persistent mode).
would be great if this would fixed for good.

it is hilariously stupid to need to select “use a bridge” at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info… (_)

#11 Updated by bertagaz 2018-05-10 11:09:33

  • Target version changed from Tails_3.7 to Tails_3.8

#12 Updated by mercedes508 2018-06-15 12:22:25

intrigeri wrote:
> > my mistake, my VMs are displaying UTC time even before the time sync…
>
> OK, then please retry and upon failure, send me:
>
> * the content of the Journal
> * the Tor log
> * the accurate time (with timezone) from outside of Tails
>
> Rationale: I want to check if the Tor consensus is OK of not.

I’m attaching these logs from a user experiencing this issue.
The timezone is UTC.

#13 Updated by intrigeri 2018-06-15 13:20:19

> I’m attaching these logs from a user experiencing this issue.

Could you reproduce it?

> The timezone is UTC.

You mean the hardware clock?

#14 Updated by intrigeri 2018-06-26 16:28:06

  • Target version changed from Tails_3.8 to Tails_3.9

#15 Updated by Anonymous 2018-08-16 11:13:19

Anonymous wrote:
> it is hilariously stupid to need to select “use a bridge” at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info… (_)

Please read our code of conduct: https://tails.boum.org/contribute/working_together/code_of_conduct/
We very much appreciate being excellent to each other, thank you.

#16 Updated by Anonymous 2018-08-16 11:13:52

hi goupille, is this still a thing?

#17 Updated by Anonymous 2018-08-18 12:04:12

  • related to Bug #9268: obfs4 bridges often don't work (maybe MTU?) added

#18 Updated by intrigeri 2018-09-05 16:27:01

  • Target version changed from Tails_3.9 to Tails_3.10.1

#19 Updated by intrigeri 2018-10-24 17:03:46

  • Target version changed from Tails_3.10.1 to Tails_3.11

#20 Updated by intrigeri 2018-11-17 20:57:09

Dear help desk, it would be nice to have conclusive, actionable input here: we’ve been trying to confirm a correlation between the hardware clock timezone and this issue (and Bug #9268) for more than a year, they regularly make it into the help desk’s hot topics, and it’s still not clear to me what I should do about it.

Let’s focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?

If the answer is “yes”, then we need to investigate further here.

Else, if the answer is “no” or “we don’t know” or calming silence, let’s mark this ticket as blocked by Bug #15168 (or even reject it) and prioritize that one higher: at least when we’ll have fixed Bug #15168 we’ll see whether users still have issues with obfs4 :)

Thanks in advance!

#21 Updated by intrigeri 2018-11-20 12:31:51

  • Subject changed from Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC to Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

> Let’s focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?
> If the answer is “yes”, then we need to investigate further here.
> Else, if the answer is “no” or “we don’t know” or calming silence, let’s mark this ticket as blocked by Bug #15168 (or even reject it) and prioritize that one higher: at least when we’ll have fixed Bug #15168 we’ll see whether users still have issues with obfs4 :)

Actually, let’s do this instead: if the answer is “yes”, file a new ticket. If the answer is “no”, then I won’t need more input and I’ll keep working on it here and on Bug #15168. Thanks!

While looking at Bug #15743 I worked a bit on this today. I got a fresh set of obfs4 bridges that work fine with a RTC in UTC.

If I set the RTC to UTC+1 (<clock offset='variable' adjustment='3600' basis='utc'/> in libvirt), tor never manages to bootstrap. I see:

  • either connection_or_note_state_when_broken(): Connection died in state 'handshaking (proxy) with SSL state (No SSL object)'
  • or clock_skew_warning(): Received directory with skewed time (DIRSERV:148.163.90.126:443): It seems that our clock is ahead by 59 minutes, 58 seconds, or that theirs is behind, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings

If I set the RTC to UTC-1, tor does bootstrap as I was lucky enough to still be within the consensus validity bounds, which is not guaranteed with a 1-hour incorrect system time, and I see: [info] clock_skew_warning(): Received NETINFO cell with skewed time (OR:148.163.90.126:443): It seems that our clock is behind by 1 hours, 0 minutes, or that theirs is ahead, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings.

So at least this confirms that we have a problem for obfs4 if the RTC is incorrect or set to a timezone too far away from UTC. What’s interesting is that in some cases, we don’t even manage to establish a SSL connection to the bridge, so we don’t get any time info from it at all, so that’s not a failure mode we can recover from even with our dirty “tordate” hacks in 20-time.sh => I think the only way to improve UX in this case is Bug #15168.

#22 Updated by intrigeri 2018-11-20 12:32:02

  • related to deleted (Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC)

#23 Updated by intrigeri 2018-11-20 12:32:08

  • blocked by Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#24 Updated by intrigeri 2018-11-20 12:32:30

  • Assignee deleted (goupille)
  • Target version deleted (Tails_3.11)
  • QA Check deleted (Info Needed)

#25 Updated by intrigeri 2018-11-20 12:32:43

  • Type of work changed from Research to Code

#26 Updated by intrigeri 2018-11-20 12:34:39

  • related to Bug #15743: Too much log when using obfs4 in Tails added

#27 Updated by mercedes508 2019-03-28 13:47:44

a89500911ff03e90098e12848a5d639e
f4df66ca-7cf4-2484-7ef9-69586df3d173 (message ID from non WB report)

#28 Updated by intrigeri 2019-08-11 15:59:13

#29 Updated by intrigeri 2019-08-11 16:07:29

  • related to deleted (Bug #9268: obfs4 bridges often don't work (maybe MTU?))

#30 Updated by intrigeri 2019-08-11 16:07:46

  • has duplicate Bug #9268: obfs4 bridges often don't work (maybe MTU?) added

#31 Updated by intrigeri 2019-08-11 16:50:05

  • related to Bug #16972: tordate sometimes breaks obfs4 by messing with a correct clock added

#32 Updated by intrigeri 2019-08-11 16:53:11

intrigeri wrote:
> > I’m attaching these logs from a user experiencing this issue.
>
> Could you reproduce it?
>
> > The timezone is UTC.
>
> You mean the hardware clock?

Dear @mercedes508, AFAICT you did not answer these questions. I guess it’s now too late to find out. Still, analyzing the logs lead me to discover a potential bug (Bug #16972) in this area. I’ll propose a branch that might help for such situations.

This being said, let’s now keep this ticket focused on the “hardware clock too far away from UTC”. If you folks get bug reports about obfs4 and a hardware clock that has the correct UTC time (better ask the user to check in the BIOS), please file new tickets :)

#33 Updated by intrigeri 2019-08-11 17:01:09

  • related to Feature #15599: Improve known issues about clock going backwards added

#34 Updated by intrigeri 2019-08-11 17:01:38

  • % Done changed from 100 to 0

#35 Updated by goupille 2019-09-11 10:17:54

  • Assignee set to goupille

Bug report: f24d7bd837478c87bd0afc8b2f8c8460

#36 Updated by intrigeri 2019-11-07 14:41:13

I’ve mentioned this to Philipp Winter (the lead of Tor’s anti-censorship team, that works on obfs4proxy these days) this week and he would like us to file a proper bug report to them, describing the problem (reproducer, impact).

#37 Updated by intrigeri 2019-11-09 08:17:39

goupille wrote:
> Bug report: f24d7bd837478c87bd0afc8b2f8c8460

Looks like a system clock that’s set to localtime wich is a few hours after UTC, which confirms what we already know.

#38 Updated by intrigeri 2019-11-09 08:51:06

intrigeri wrote:
> I’ve mentioned this to Philipp Winter (the lead of Tor’s anti-censorship team, that works on obfs4proxy these days) this week and he would like us to file a proper bug report to them, describing the problem (reproducer, impact).

Reported an upstream ticket (https://trac.torproject.org/projects/tor/ticket/32439) and pointed Philipp Winter to it.

#39 Updated by intrigeri 2020-03-15 10:21:00

Dear goupille,

I’m not sure why you assigned this to yourself 6 months ago. Does this still make sense? If not, feel free to unassign yourself :)

#40 Updated by intrigeri 2020-03-15 10:24:14

  • related to Bug #17525: obfs4 documentation does not mention the need for a hardware clock set to UTC added