Tails

#4 Updated by intrigeri 2016-12-01 12:55:15

What’s your ETA on this one? I wonder how much effort we should put into Bug #12009, i.e. whether we can reasonably “wait” for Feature #11972 do be done.

#5 Updated by intrigeri 2016-12-01 13:59:56

Commit 5c41d23d15da324c18e0920a3c9c1c7f467b938e in puppet-tails does weird things with regexps. Even if we’re lucky enough that it does what you mean it to do, I find it hard to understand both the intent, and the effect of the new regexp.

#6 Updated by intrigeri 2016-12-01 14:12:48

• I don’t understand what’s the deal wrt. tails::builder vs. tails::iso_builder, but I expect that’s temporary and you’ll refactor this before asking for a proper review :)
• Please s/uses_vagrant/use_vagrant/ for better consistency.

Otherwise, current Puppet code looks OK.

#12 Updated by intrigeri 2016-12-04 15:42:39

More questions, after I thought I would look at jenkins-jobs.git too (next time, please point me to all the repos/branches where there’s stuff to review):

• why do we set defaultcomp manually?
• why do we set cpus=4 manually? I thought we had a mechanism to autodetect this.
• is GIt_COMMIT a typo? It feels strange.
• please quote strings that contain variables in macros/builders.yaml. This allows detecting mistakes such as git reset --hard $UNSET_VARIABLE.

Also:

• in sign_artifacts, please don’t recycle exit codes for unrelated errors.
• (OT) in compare_artifacts, is if ! cmp -s ${ARTIFACTS_DIR}/tails-*.iso really needed? I was hoping that diffoscope would do that check itself.

#13 Updated by intrigeri 2016-12-06 10:03:43

Another question: I’ve seen a number of build failures like https://jenkins.tails.boum.org/view/Tails_ISO/job/build_Tails_ISO_vagrant_wip-11972-use-vagrant-in-jenkins/86/console:

17:36:00 This is the first time that the Tails builder virtual machine is
17:36:00 started. The virtual machine template is about 300 MB to download,
17:36:00 so the process might take some time.
17:36:00 
17:36:00 Please remember to shut the virtual machine down once your work on
17:36:00 Tails is done:
17:36:00 
17:36:00     $ rake vm:halt
17:36:00 
17:36:01 Bringing machine 'default' up with 'libvirt' provider...
17:36:02 Name `tails-builder-amd64-jessie-20160226_default` of domain about to create is already taken. Please try to run
17:36:02 `vagrant up` command again.
17:36:02 rake aborted!

Was this fixed on the branch already?

#14 Updated by anonym 2016-12-08 15:12:04

Code review:

> 093a92f Mount whole Tails repo in vagrant builder VM and copy it.

As I’ve already expressed in Feature #11980#note-12, using rsync instead of git will copy crap (e.g. build artifacts) from the host into the builder. Let’s move back to git! In the commit message you say:

> […] there are no Git ref to the base branch being merged

Just pass --no-local to git clone and all refs will be copied. I guess this will take longer time than rsync now, but I have an idea Feature #11979 that will fix it later (spoiler: move what we now call WORKSPACE (i.e. the Git repo) to the extra disk).

-mv -f tails-* "$ARTIFACTS_DIR"
+if [ "$TAILS_RAM_BUILD" ]; then
+       mv -f tails-* "$WORKSPACE/"
+fi

This feels wrong — when we are not building in RAM, we still build inside BUILD_DIR, so the artifacts must still be copied back to WORKSPACE.

A related issue (not due to this branch!) is that we apparently only run remove_build_dirs before we set up the new BUILD_DIR if we are building from RAM, which feels wrong for the same reason. I guess we’ve been confused and assumed that disk builds are very different from RAM builds, when they in fact are not. I’d appreciate if you’d fix it while you’re at it, but won’t block on this.

> 2aed0ce Respect the ‘ARTIFACTS’ environment variable if set.

Minor nitpicking (not blocking): ENV stores strings, so the quoting and #{} just reduces readability here:

+        "#{user}@#{hostname}:#{artifact}", "#{ENV['ARTIFACTS']}"

Once you’ve fixed all these (except those I’ve explicitly said are not blocking) I’m happy with this branch.

#15 Updated by intrigeri 2016-12-08 15:17:32

anonym: I’m glad that you looked at this branch, and I would love if you could handle the rest of the review’n’merge process: you’re obviously much more knowledgeable than I am in the Vagrant bits :)

#16 Updated by anonym 2016-12-08 16:11:08

bertagaz, also please add /*.timestamp to .gitignore.

intrigeri wrote:
> anonym: I’m glad that you looked at this branch, and I would love if you could handle the rest of the review’n’merge process: you’re obviously much more knowledgeable than I am in the Vagrant bits :)

Sure, as long as you double-check my potentially crazy suggestions. :)

#19 Updated by intrigeri 2017-03-16 12:16:00

I had a quick look at the diff:

• vagrant/definitions/tails-builder/postinstall.sh uses a non-frozen version of our custom APT repo, which means that we will affect all the baseboxes that we already have generated in the past whenever we update a package in our builder-jessie APT suite. This feels a bit wrong. Any reason for doing it this way?
• AFAICT this branch will break the build on all isobuilders that haven’t been switched to Vagrant yet, so unless this migration is rushed, regardless of whether we merge it early or late, we’ll have to live with fewer isobuilders. If not too painful, I would prefer avoiding this, i.e. it would be nice if this branch added support for building in Vagrant without breaking what currently works. Now, if the migration can be completed within 2-3 days max, and at a sensible time wrt. our dev/release cycle, don’t bother and go for it.
• Passing -m 0 to mkfs.ext4 would avoid wasting disk space.
• Why do we create a partition on /dev/vdb? We could simply format the drive itself, which makes various operations easier.
• Please mount /var/cache/apt-cacher-ng with relatime.
• Is /var/log/apt-cacher-ng/main_*.html still correct while we configured a different LogDir?
• Currently /var/cache/apt-cacher-ng is mounted by hand, which means that no automatic fsck is performed, and developer experience will suffer whenever this FS is corrupted. Perhaps add it to fstab instead? (augtool can be handy)
• Using journal_checksum would make the ext4 filesystems more robust, which could improve the developer experience.

#20 Updated by intrigeri 2017-03-16 12:24:22

I’ve tried building from this branch with http_proxy=http://10.36.24.33:3142 TAILS_BUILD_OPTIONS="ram extproxy" rake build and it seems that the /amnesia.git working copy is used as if it were a bare repo, or a .git tree, and then the build fails:

==> default: Should be mounting folders
==> default:  /amnesia.git, opts: {:type=>:"9p", :readonly=>true, :guestpath=>"/amnesia.git", :hostpath=>"/home/intrigeri/tails/git", :disabled=>false, :__vagrantfile=>true, :target=>"/amnesia.git", :accessmode=>"passthrough", :mount=>true, :mount_tag=>"86502a772c54ee52df905a151ae3a9d"}
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Creating shared folders metadata...
==> default: mounting p9 share in guest
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
fatal: Not a git repository: '/amnesia.git'
Connection to 192.168.121.220 closed.
rake aborted!
VagrantCommandError: 'vagrant ["ssh", "-c", "MKSQUASHFS_OPTIONS='-comp gzip' TAILS_PROXY='http://10.36.24.33:3142' TAILS_PROXY_TYPE='extproxy' TAILS_RAM_BUILD='1' TAILS_GIT_COMMIT='50a8f71e94ac2ec8ee0d2d6ac972fd3a85dec817' TAILS_GIT_REF='wip/11972-use-vagrant-in-jenkins' build-tails"]' command failed: 128
/home/intrigeri/tails/git/Rakefile:71:in `run_vagrant'
/home/intrigeri/tails/git/Rakefile:353:in `block in <top (required)>'
Tasks: TOP => build

#21 Updated by intrigeri 2017-03-16 12:39:14

Also, the rsync -a should perhaps have the --delete option? Otherwise files we’ve deleted in Git may still be in the tree used for building.

#22 Updated by intrigeri 2017-03-16 12:40:24

intrigeri wrote:
> […] and then the build fails:

Note that git status is happy when run in /amnesia.git inside the VM, so I’m not quite sure why fatal: Not a git repository: '/amnesia.git'.

#23 Updated by intrigeri 2017-03-16 17:04:48

Here’s the fix to the fatal: ambiguous argument 'origin/stable': unknown revision or path not in the working tree. error seen e.g. on https://jenkins.tails.boum.org/job/build_Tails_ISO_vagrant_wip-11972-use-vagrant-in-jenkins/183/console, that prompted you to switch from git clone to rsync: after cloning, run git config remote.origin.fetch +refs/remotes/origin/*:refs/remotes/origin/* && git fetch. And then you’ll have access to the refs you want :)

#25 Updated by intrigeri 2017-04-06 15:24:25

bertagaz will split this ticket to disentangle the tails.git part (that’s mostly ready) from the Puppet bits.

#28 Updated by anonym 2017-04-25 16:14:10

bertagaz wrote:
> I’ve been through all the notes of this ticket to check every issue raised was addressed. I’ve pushed commit:0d11da9cc7fc50bb33db296443351e5a6bee4121 and commit:09ca60ea29a1491f675d5b1ae768980570ac45af which were the last remainings from intrigeri’s review in Feature #11972#note-19.

I pushed a fixup (commit:045d3294aaea891f81c0a47034300df5465aef8f) on commit:0d11da9. That commit also made me realize that we should try to avoid the need of fsck by unmounting the disk, so I pushed commit:0b3c939bf7e6fbb1503ab87f9d1cea34a2ee50cb to do just that.

> The only one I’m not sure of is:
>
> intrigeri wrote:
> > * vagrant/definitions/tails-builder/postinstall.sh uses a non-frozen version of our custom APT repo, which means that we will affect all the baseboxes that we already have generated in the past whenever we update a package in our builder-jessie APT suite. This feels a bit wrong.

I agree that this feels wrong.

> > Any reason for doing it this way?

From my side it’s just that I didn’t think about it. Thanks for noticing! :)

> No reason apart we did not use a frozen version of our custom APT repo in the build system. Makes sense to use a frozen version in the basebox though. Where can I find an exemple of the URI of a frozen version of our repo, I have troubles finding that?

AFAICT, we only do time-based snapshots of our upstream APT repos, i.e. Debian and Tor Project, not of our own custom APT repo. When we create tagged snapshots (for releases) we do include the packages from our custom APT repo that were used when building. Unfortunately “used” here refers to what’s installed when running live-build, so anything installed before that, including live-build itself, is not included. In other words, we currently have no snapshots that includes most of these packages (incl. live-build).

intrigeri, what about adding our custom APT repo’s builder-jessie suite to the set of repos we do time-based snapshots of? Otherwise I’m not sure how we can solve this.

#30 Updated by anonym 2017-04-25 17:26:26

intrigeri wrote:
> I’m sorry that the pointers in https://tails.boum.org/contribute/APT_repository/time-based_snapshots/ were not enough to highlight this (hints welcome wrt. how this could be improved, on a dedicated ticket please, because I checked and could not find what’s missing), but we do have snapshots of a few dists already: https://time-based.snapshots.deb.tails.boum.org/tails/.

Speaking for myself, I’m not very used to interacting with APT repos over HTTP, and generally when I have tried I just keep on getting 403 Forbidden (especially with our repos). I don’t think our docs can be improved without becoming too verbose, or by listing what we snapshot, which will get outdated.

#31 Updated by intrigeri 2017-04-26 06:36:33

> No reason apart we did not use a frozen version of our custom APT repo in the build system.

Right. FYI that’s because:

1. we use our custom APT repo for freeze exceptions, so it can’t be frozen;
2. we control what’s uploaded in there so we can enforce our freeze ourselves (as opposed to what happens with the Debian repos).

> Makes sense to use a frozen version in the basebox though. Where can I find an exemple of the URI of a frozen version of our repo, I have troubles finding that?

I’m interested in improving our doc so you can find such things yourself next time, so may I ask: where exactly did you look (and failed to find)?

#32 Updated by intrigeri 2017-04-26 06:50:15

> Speaking for myself, I’m not very used to interacting with APT repos over HTTP, and generally when I have tried I just keep on getting 403 Forbidden (especially with our repos).

Please report a bug next time you see that, as our APT repos should be publicly browseable (except some reprepro config and database internal bits that are not meant for consumption by APT). Thanks in advance!

#34 Updated by bertagaz 2017-04-27 13:17:12

bertagaz wrote:
> intrigeri wrote:
> > Or, give me a dedicated ticket if you can wait a few days and don’t feel comfortable applying the change yourself (it should be relatively straightforward given the existing examples, some understanding of reprepro, and… an up-to-date set of remote backups :)
>
> I’m updating my backups, but that should take some time.

Wait, now I realize that we actually don’t backup time-based snapshots, so this step is probably not that important, apart for the reprepro configuration I guess.

#35 Updated by bertagaz 2017-04-28 08:45:48

bertagaz wrote:
> Now I’ll had the builder-jessie time-based snapshots in our Puppet/reprepro. This indeed seems straightforward at the puppet level.

Done, commit:048bc8cf3ea704e8e2768a818b4148ce0ea406f4 in puppet-tails, as well as pushed commit:d7e220c39345c325a404522e33b3c6df29852881 in the main tails.git repo. I had to bump the serial of the basebox, as the previously used Debian snapshot has disappeared. I have not uploaded the related now basebox.

This seems to work, the builder-jessie APT repo snapshot is used in the vagrant VM.

But now the build is broken with “NoMethodError: undefined method `active?' for nil:NilClass” error, as seen in https://jenkins.tails.boum.org/job/build_Tails_ISO_vagrant_wip-11972-use-vagrant-in-jenkins/342/console. I’m not sure where that comes from. Will inverstigate.

#36 Updated by anonym 2017-04-28 09:39:52

bertagaz wrote:
> bertagaz wrote:
> > Now I’ll had the builder-jessie time-based snapshots in our Puppet/reprepro. This indeed seems straightforward at the puppet level.
>
> Done, commit:048bc8cf3ea704e8e2768a818b4148ce0ea406f4 in puppet-tails, as well as pushed commit:d7e220c39345c325a404522e33b3c6df29852881 in the main tails.git repo. I had to bump the serial of the basebox, as the previously used Debian snapshot has disappeared. I have not uploaded the related now basebox.
>
> This seems to work, the builder-jessie APT repo snapshot is used in the vagrant VM.

Awesome!

> But now the build is broken with “NoMethodError: undefined method `active?' for nil:NilClass” error, as seen in https://jenkins.tails.boum.org/job/build_Tails_ISO_vagrant_wip-11972-use-vagrant-in-jenkins/342/console. I’m not sure where that comes from. Will inverstigate.

That was my fault. I have pushed an (untested) fix.

#37 Updated by bertagaz 2017-04-28 10:59:44

anonym wrote:
> bertagaz wrote:
> > Done, commit:048bc8cf3ea704e8e2768a818b4148ce0ea406f4 in puppet-tails, as well as pushed commit:d7e220c39345c325a404522e33b3c6df29852881 in the main tails.git repo. I had to bump the serial of the basebox, as the previously used Debian snapshot has disappeared. I have not uploaded the related now basebox.
> >
> > This seems to work, the builder-jessie APT repo snapshot is used in the vagrant VM.

By testing I realized I made a mistake in the pining for live-build and syslinux-utils. I’ll fix that and it should be good to be reviewed.

> > But now the build is broken with “NoMethodError: undefined method `active?' for nil:NilClass” error, as seen in https://jenkins.tails.boum.org/job/build_Tails_ISO_vagrant_wip-11972-use-vagrant-in-jenkins/342/console. I’m not sure where that comes from. Will inverstigate.
>
> That was my fault. I have pushed an (untested) fix.

Seems to work!

#41 Updated by bertagaz 2017-05-03 13:13:22

Added Feature #12505 to track the puppet part of the deployement.

#43 Updated by bertagaz 2017-05-07 11:37:31

anonym wrote:
> Looks good! (I noticed a somewhat related issue, but I’ll deal with that on Feature #12409).

Great! As said on Bug #11006, I’m re-installing the isobuilders and will merge the branch once done.

#46 Updated by intrigeri 2017-05-10 11:38:51

Any reason this is “Fix committed” and not “Resolved”? It seems to me that this was deployed to production already.