Bug #16132
Connectivity issue between isobuilders and Redmine breaks CI
0%
Description
I see lots of ISO builds failing on Jenkins because /usr/local/bin/output_ISO_builds_and_tests_notifications --apikey_file /etc/jenkins/redmine_apikey --rfqa_only --no_notify_builds --no_notify_tests fails to connect to Redmine e.g. https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-15428-rename-htp-pools/3/console. I’ve noticed a few such failures on isobuilder2, not checked the others.
The URL that’s queried in in the build failure log. Trying it via curl on isobuilder2 does not always work. I see no connection to that URL from the python-requests/2.12.4 user agent in Redmine’s access.log.
Pinging redmine.tails.boum.org from isobuilder2 sometimes goes to 198.252.153.98 and sometimes to 10.10.0.3, so I suspect one of them works but not the other (firewall or routing issue?).
If that’s something I broke when renaming labs, please give this back to me. But I haven’t noticed this in the last week (didn’t pay that much attention either).
Subtasks
Related issues
| Blocks Tails - Feature #13242: Core work: Sysadmin (Maintain our already existing services) | Confirmed | 2017-06-29 |
History
#1 Updated by intrigeri 2018-11-17 18:01:31
- blocks Feature #13242: Core work: Sysadmin (Maintain our already existing services) added
#2 Updated by intrigeri 2018-11-17 18:06:52
- Subject changed from Connectivity issue between isobuilder2 and Redmine to Connectivity issue between isobuilder2 and Redmine breaks CI
- Priority changed from Normal to Elevated
#3 Updated by intrigeri 2018-11-17 18:07:38
- Subject changed from Connectivity issue between isobuilder2 and Redmine breaks CI to Connectivity issue between isobuilders and Redmine breaks CI
Also happens on isobuilder1.
#4 Updated by groente 2018-11-17 19:37:41
- Assignee changed from groente to intrigeri
- QA Check set to Ready for QA
intrigeri wrote:
> Also happens on isobuilder1.
pretty sure this is caused by redmine.tails.boum.org being an alias for buse.tails.boum.org - which is in the local hosts file - whereas labs.riseup.net has an A record, so traffic there always went to the external ip.
it’s still a bit odd that it sometimes resolves to the internal and sometimes to the external address, i guess dnsmasq is to blame for that.
eitherway, i see two options: give redmine.tails.boum.org an A record, or open up firewalling for traffic to port 443 on 10.10.0.3.
my preference would be to change the DNS record from a CNAME to an A record, please confirm if you agree.
#5 Updated by intrigeri 2018-11-17 19:52:05
- Assignee changed from intrigeri to groente
- QA Check changed from Ready for QA to Dev Needed
> eitherway, i see two options: give redmine.tails.boum.org an A record, or open up firewalling for traffic to port 443 on 10.10.0.3.
> my preference would be to change the DNS record from a CNAME to an A record, please confirm if you agree.
Works for me: getting deterministic behaviour is higher priority than some vague desire I have of sending more of the connections between our machines through the VPN.
#6 Updated by groente 2018-11-17 20:28:07
- Status changed from Confirmed to Resolved
that seems to have done the trick