Tails

#3 Updated by intrigeri 2017-09-02 17:05:31

intrigeri wrote:
> * MD RAID: unclear whether MD forwards discard requests, RHEL 6.5 doc says it does but that might be thanks to their custom kernel

We’ll see.

> * lvm.conf: augeas 1.3+ (in Stretch) supports it (/files/etc/lvm/lvm.conf/devices/dict/issue_discards/)

Enabled on all virtualization hosts.

> * fstab: trivially supported by Puppet (unless we decide to go for fstrim instead)

I think I’ll go with /usr/share/doc/util-linux/examples/fstrim.{service,timer} instead of fiddling with tons of fstab entries.

> * crypttab: augeas seems to support it

Enabled on all LUKS volumes on all our virtualization hosts.

We’ll see if that’s enough to allow manual fstrim once these boxes have rebooted. If it works, then I’ll enable the systemd service + timer.

#5 Updated by intrigeri 2017-10-11 07:07:06

Added discard='unmap' to all libvirt guest definitions.

#6 Updated by intrigeri 2017-10-11 08:32:55

Actually virtio-blk does not support discard yet (https://wiki.qemu.org/ToDo/Block#virtio-blk_discard_support_.5BPeter_Lieven.5D) but some patches have been discussed on LKML a few months ago.

#7 Updated by intrigeri 2019-03-07 15:19:19

intrigeri wrote:
> Actually virtio-blk does not support discard yet

After reading https://mpolednik.github.io/2017/01/23/virtio-blk-vs-virtio-scsi/, it seems that the best thing to do (not only wrt. discard) is to switch to virtio-scsi. Unfortunately this will rename all drives (e.g. vda → sda) in VMs so it’ll be a quite disruptive operation, that requires some carefully coordinated Puppet changes, initramfs & bootloader updates, and downtime. I say let’s skip that until we notice actual performance decrease caused by the lack of discard support.

Next step: look at long-term I/O performance trends in 6-12 months to see if SSD performance decreases.

#9 Updated by intrigeri 2019-04-24 07:22:38

intrigeri wrote:
> Actually virtio-blk does not support discard yet

It does in QEMU 4.0, that we won’t have in Buster yet, so that’s for whenever we upgrade our virtualization host to Bullseye, i.e. probably in the 2nd half of 2021.

#10 Updated by intrigeri 2019-12-27 10:37:34

On my sid system, I have a version of QEMU that supposedly supports TRIM for virtio-blk, but lsblk --discard in the guest disagrees.

I’ve switched a VM to virtio-scsi and TRIM support does work. One minor caveat is that iothreads can only be assigned per-SCSI-controller, and not per disk, so to give each disk its own iothread, one needs 1 virtio-scsi controller per disk, and to assign each disk to its own controller.

#11 Updated by intrigeri 2020-01-02 10:41:45

intrigeri wrote:
> > * fstab: trivially supported by Puppet (unless we decide to go for fstrim instead)
>
> I think I’ll go with /usr/share/doc/util-linux/examples/fstrim.{service,timer} instead of fiddling with tons of fstab entries.

The recommended way these days is indeed fstrim.{service,time} and not the discard mount option. I’ve enabled fstrim.timer on all our systems: https://git.tails.boum.org/puppet-tails/commit/?id=28220dd0c78ceb115098e928aad3ea00e98df1a3

So the only remaining blocker here is the libvirt/QEMU storage layer.

#12 Updated by intrigeri 2020-02-15 08:12:01

intrigeri wrote:
> I’ve switched a VM to virtio-scsi and TRIM support does work.

I did the same for all the VMs that are part of my local Jenkins setup; let’s see how it goes :)

Another caveat is that on Stretch, fstrim -a (that’s run by fstrim.service) has buggy de-duplication of fstab entries when bind-mounts are involved, which for example prevents it from doing its job on isobuilders. That’s been fixed in Buster: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864806, https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/commit/?id=155d48f590a50bb5dc265162ff2f9a971daed543.