Bug #8603

Unexpected redirection while sending Host header with wget

Added by sajolida 2015-01-07 20:45:10 . Updated 2015-03-20 18:29:42 .

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Target version:
Start date:
2015-01-07
Due date:
% Done:

100%

Feature Branch:
bugfix/quote-wrappers-arguments
Type of work:
Code
Blueprint:

Starter:
Affected tool:
check-mirrors
Deliverable for:

Description

While testing check-mirrors with torsocks in experimental (Feature #8074) I noticed that wget is having a weird behaviour that it didn’t had in the past when sending explicit Host headers. It adds an unexpected redirection that bypasses the Host header (and fetches from a different IP address in the context of check-mirrors).

Success without Host header:

amnesia@amnesia:~$ wget http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --max-redirect=0
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:27:25--  http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 836 [application/pgp-signature]
Remote file exists.

Failure with Host header:

amnesia@amnesia:~$ wget http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org"
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:28:03--  http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 836 [application/pgp-signature]
Remote file exists.

Spider mode enabled. Check if remote file exists.
--2015-01-07 20:28:04--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 50.254.151.65
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|50.254.151.65|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://cime.net/ [following]
0 redirections exceeded.

Second failure with Host header but no —max-redirect, note that the redirected IP address of the imaginary redirection changes:

amnesia@amnesia:~$ torsocks /usr/bin/wget http://176.9.53.178/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --header=Host: dl.amnesia.boum.org
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:36:14--  http://176.9.53.178/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 176.9.53.178:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
Remote file does not exist -- broken link!!!

Spider mode enabled. Check if remote file exists.
--2015-01-07 20:36:15--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 96.126.119.95
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|96.126.119.95|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Files

trace (31119 B) sajolida, 2015-01-08 17:34:21

Subtasks


Related issues

Related to Tails - Feature #8074: Test check-mirrors with torsocks Resolved 2014-10-12
Related to Tails - Feature #6623: Have wget directly use the Tor SOCKS proxy Resolved 2014-01-27
Blocked by Tails - Bug #8830: Buggy passed-through argument quoting in connect-socks, totem, wget and whois wrappers Resolved 2015-02-02

History

#1 Updated by sajolida 2015-01-07 20:46:04

  • Description updated

#2 Updated by sajolida 2015-01-07 20:47:20

  • Description updated

#3 Updated by sajolida 2015-01-07 20:47:43

  • related to Feature #8074: Test check-mirrors with torsocks added

#4 Updated by intrigeri 2015-01-08 10:19:02

  • Assignee changed from intrigeri to sajolida
  • QA Check set to Info Needed
  • Affected tool set to WhisperBack

Looks like a bug in wget: it shouldn’t be doing DNS resolution when connecting to a specific IP, even when a Host header is passed. FWIW, I can’t reproduce your 2nd test (that fails) on current sid, with a manually torsocks’ified wget. If you can easily test on Tails/experimental with current testing/sid’s wget, it would be helpful.

In your 3rd attempt, you explicitly use torsocks, but not in the first two ones. Was that intended? It feels weird, since in the experimental branch we’re already wrapping wget with torsocks.

#5 Updated by intrigeri 2015-01-08 10:32:58

  • Affected tool changed from WhisperBack to check-mirrors

#6 Updated by sajolida 2015-01-08 17:22:30

  • Assignee deleted (sajolida)

Note that the same commands in the context of 1.2.2 (same wget package) work fine, so that might as well be a bug in torsocks.

I tried to install wget/testing in experimental and I think that’s not possible because it requires an upgrade of libc6:

root@amnesia:~# apt-get install wget=1.16-1 libgnutls-deb0-28=3.3.8-5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 libgnutls-deb0-28 : Depends: libc6 (>= 2.17) but 2.13-38+deb7u6 is to be installed
                     Depends: libgmp10 (>= 2:6) but 2:5.0.5+dfsg-2 is to be installed
                     Depends: libhogweed2 (>= 2.7) but 2.4-3 is to be installed
                     Depends: libnettle4 (>= 2.7) but 2.4-3 is to be installed
                     Depends: libp11-kit0 (>= 0.20.7) but 0.12-3 is to be installed
 wget : Depends: libc6 (>= 2.17) but 2.13-38+deb7u6 is to be installed
        Depends: libpsl0 (>= 0.4.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

When I try to install `wget/install` I get a reproducible segfault in at least experiemental and 1.2.1:

root@amnesia:~# apt-get install wget/testing
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '1.16-1' (Debian:testing [i386]) for 'wget'
Selected version '3.3.8-5' (Debian:unstable [i386]) for 'libgnutls-deb0-28' because of 'wget'
Segmentation fault

But maybe that’s because `apt-get update` fails with:

Reading package lists... Done
W: Ignoring Provides line with DepCompareOp for package php-psr-log-implementation
W: You may want to run apt-get update to correct these problems

Which might be transient. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774815

On the other hands many other packages install fine.

#7 Updated by sajolida 2015-01-08 17:34:43

Attaching a full trace of the event.

#8 Updated by sajolida 2015-01-08 17:35:57

  • QA Check deleted (Info Needed)

#9 Updated by intrigeri 2015-01-21 11:36:28

  • Assignee set to sajolida
  • QA Check set to Info Needed

OK, thanks. Please retry in an ISO built from the feature/jessie branch, then: it’ll have wget 1.16. I’ve just merged the fix for Bug #8715 into that branch, hopefully it won’t break the build… if it does, get the latest autobuilt Jessie ISO.

#10 Updated by sajolida 2015-01-31 11:02:19

  • Assignee changed from sajolida to intrigeri
  • QA Check changed from Info Needed to Dev Needed

Ok, I finally did those tests and I think that there is a bug in the wget wrapper.

When I do wget without torsocks I get the same imaginary redirection. This time I tried with bash -x and I think that the arguments are not passed correctly by the wrapper:

amnesia@amnesia:~$ bash -x wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org"
+ unset http_proxy
+ unset HTTP_PROXY
+ unset https_proxy
+ unset HTTPS_PROXY
+ exec torsocks /usr/bin/wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header=Host: dl.amnesia.boum.org
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:47:44--  http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 801 [application/pgp-signature]
Remote file exists.
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:47:44--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 195.154.14.189
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|195.154.14.189|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
Remote file does not exist -- broken link!!!

If I give the correct arguments to torsocks I get the expected result from wget.

amnesia@amnesia:~$ torsocks /usr/bin/wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org"
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:57:45--  http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 801 [application/pgp-signature]
Remote file exists.

#11 Updated by sajolida 2015-01-31 11:02:41

  • related to Feature #6623: Have wget directly use the Tor SOCKS proxy added

#12 Updated by intrigeri 2015-02-01 08:44:27

> Ok, I finally did those tests and I think that there is a bug in the wget wrapper.

Indeed. It looks like it should be:

exec torsocks /usr/bin/wget "$@"

instead of

exec torsocks /usr/bin/wget $*

Can you please test that fix?

#13 Updated by Tails 2015-02-01 08:54:49

  • Status changed from Confirmed to In Progress

Applied in changeset commit:9ee271f5afa99de6adc4e3853907b855e3f5c1dc.

#14 Updated by intrigeri 2015-02-01 14:33:56

  • Feature Branch set to bugfix/quote-wrappers-arguments

#15 Updated by intrigeri 2015-02-02 11:02:05

  • blocked by Bug #8830: Buggy passed-through argument quoting in connect-socks, totem, wget and whois wrappers added

#16 Updated by Tails 2015-02-02 11:24:52

Applied in changeset commit:009a35821f512ae161b972d2344d7c811722edd1.

#17 Updated by intrigeri 2015-02-02 11:27:56

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 0 to 50
  • QA Check changed from Dev Needed to Ready for QA

#18 Updated by intrigeri 2015-02-02 11:28:22

  • Priority changed from Normal to Elevated

That’s a regression, bumping priority.

#19 Updated by sajolida 2015-02-02 21:16:12

It seems like your patch modifies --header="Host: dl.amnesia.boum.org" into '--header=Host: dl.amnesia.boum.org'. I’m afraid that won’t work either. Did you test it?

#20 Updated by sajolida 2015-02-02 21:25:17

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

#21 Updated by intrigeri 2015-02-02 22:40:42

> It seems like your patch modifies --header="Host: dl.amnesia.boum.org" into '--header=Host: dl.amnesia.boum.org'. I’m afraid that won’t work either. Did you test it?

Not by hand, but I wrote an automated test that supposedly tests it.
Did it fail for you in practice?

#22 Updated by intrigeri 2015-02-04 09:03:33

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Dev Needed to Ready for QA

Tested by hand, works fine.

#23 Updated by Tails 2015-02-05 15:53:39

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

Applied in changeset commit:c73d1df140c6840ea04c4e992c150b034351c2b5.

#24 Updated by bertagaz 2015-02-05 15:54:46

  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Pass

#25 Updated by BitingBird 2015-02-24 22:45:36

  • Status changed from Fix committed to Resolved

#26 Updated by sajolida 2015-03-20 18:29:42

  • Subject changed from Unexpected redirection while send Host header in wget to Unexpected redirection while sending Host header with wget