Feature #11817: Optimize I/O settings on lizard

Feature #11817

Optimize I/O settings on lizard

Added by intrigeri 2016-09-20 09:57:13 . Updated 2017-02-28 14:22:55 .

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Infrastructure

Target version:

Tails_2.11

Start date:

2016-09-20

Due date:

% Done:

100%

Feature Branch:

Type of work:

Sysadmin

Blueprint:

Starter:

Affected tool:

Deliverable for:

280

Description

I/O scheduler = noop in KVM guests
I/O scheduler on the host: “The cfq scheduler is not recommended for hardware that does not suffer a large seek penalty, such as […] solid state disks.” ⇒ elevator=deadline for SSDs, or “5.3.5.1. Tuning cfq for fast storage” https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Performance_Tuning_Guide/
libvirt/QEMU I/O mode:
- It seems that we want <driver name='qemu' type='raw' io='native'/> according to https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liaat/liaatbpkvmasynchio.htm and http://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/libvirt-aio-mode.html.
- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html#sect-Virtualization_Tuning_Optimization_Guide-BlockIO-IO_mode

Subtasks

Related issues

Blocks Tails - ~~Feature #11179~~: Enable automatic NUMA balancing on lizard

Resolved

2016-02-29

History

#1 Updated by intrigeri 2016-09-20 09:57:56

Status changed from Confirmed to In Progress
% Done changed from 0 to 10

intrigeri wrote:
> * I/O scheduler = noop in KVM guests

Done.

#2 Updated by intrigeri 2016-09-20 09:58:11

Deliverable for set to 270

#3 Updated by intrigeri 2016-09-20 10:01:09

> * I/O scheduler on the host: “The cfq scheduler is not recommended for hardware that does not suffer a large seek penalty, such as […] solid state disks.” ⇒ elevator=deadline for SSDs, or “5.3.5.1. Tuning cfq for fast storage” https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Performance_Tuning_Guide/

This will require a Puppet module to manage sysfs (https://github.com/github/puppet-sysfs or https://github.com/luxflux/puppet-sysfs), or some tiny systemd unit file that runs a few echo commands.

#4 Updated by intrigeri 2016-09-20 10:32:25

% Done changed from 10 to 20

intrigeri wrote:
> > * I/O scheduler on the host: “The cfq scheduler is not recommended for hardware that does not suffer a large seek penalty, such as […] solid state disks.” ⇒ elevator=deadline for SSDs, or “5.3.5.1. Tuning cfq for fast storage” https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Performance_Tuning_Guide/
>
> This will require a Puppet module to manage sysfs (https://github.com/github/puppet-sysfs or https://github.com/luxflux/puppet-sysfs), or some tiny systemd unit file that runs a few echo commands.

Done as well.

I’ll come back to it in a few weeks to check the munin graphs.

#5 Updated by intrigeri 2016-09-20 10:34:16

Description updated

#6 Updated by intrigeri 2016-09-20 10:53:10

Description updated

#7 Updated by intrigeri 2016-09-20 10:54:08

Left to do (before benchmarking): set io=native for all VM disks.

#8 Updated by intrigeri 2016-09-20 15:13:08

% Done changed from 20 to 30

> Left to do (before benchmarking): set io=native for all VM disks.

Done as well. I’ll come back to it in a few weeks to check the munin graphs.

#9 Updated by intrigeri 2016-09-20 15:17:03

blocks ~~Feature #11179~~: Enable automatic NUMA balancing on lizard added

#10 Updated by intrigeri 2016-09-22 08:47:09

We might also want:

virtio-blk “data plane”, that is supported in libvirt 1.2.8+ (see new syntax in http://wiki.qemu.org/ChangeLog/2.1) via the iothread disk attribute and the iothreads domain attribute; see https://wiki.mikejung.biz/KVM_/_Xen#virtio-blk_iothreads_.28x-data-plane.29; this apparently implies using cache=directsync
~~guest clocksource: TSC~~ (found one single reference, not backed by any explanation)
~~disable delayed accounting (nodelayacct)~~ (found one single reference, and others that say it has very little impact)
~~modprobe virtio_blk use_bio=1 on SSD-backed guests~~ (found one single reference, and parameter not available anyway at least on Debian)

#11 Updated by intrigeri 2016-10-09 13:27:58

% Done changed from 30 to 40

Measurable differences I could spot since I’ve tweaked things 19 days ago:

no performance degradation I could spot
improved read latency for https://munin.riseup.net/static/dynazoom.html?cgiurl_graph=/munin-cgi/munin-cgi-graph&plugin_name=tails.boum.org/lizard.tails.boum.org/diskstats_latency/lizard_apt_snapshots&size_x=800&size_y=400&start_epoch=1473142337&stop_epoch=1475993537
improved read disk throughput for https://munin.riseup.net/tails.boum.org/lizard.tails.boum.org/diskstats_throughput/lizard_bitcoin_data.html and https://munin.riseup.net/tails.boum.org/lizard.tails.boum.org/diskstats_throughput/lizard_apt_snapshots.html
improved IO/s for https://munin.riseup.net/tails.boum.org/lizard.tails.boum.org/diskstats_iops/lizard_apt_snapshots.html and https://munin.riseup.net/tails.boum.org/lizard.tails.boum.org/diskstats_iops/lizard_bitcoin_data.html

So I call this initial experiment step a success and will proceed with the next step (virtio-blk data plane).

#12 Updated by intrigeri 2016-10-09 13:41:57

Once all this is done I’ll update our VM creation doc accordingly.

#13 Updated by intrigeri 2016-10-09 14:01:19

Tried to enable iothreads (lizard’s etckeeper ae7bbf3) on apt.lizard and bitcoin.lizard. The resulting QEMU command line looks good, but virsh qemu-monitor-command --hmp bitcoin 'info qtree' says that x-data-plane is off, which might or might not be expected with iothreads enabled (perhaps that’s just because the interface to these options changed in QEMU 2.1). I’ll check Munin graphs in a week or three and see if this impacts anything.

#14 Updated by intrigeri 2016-10-11 20:10:13

% Done changed from 40 to 50

intrigeri wrote:
> Tried to enable iothreads (lizard’s etckeeper ae7bbf3) on apt.lizard and bitcoin.lizard. The resulting QEMU command line looks good, but virsh qemu-monitor-command --hmp bitcoin 'info qtree' says that x-data-plane is off, which might or might not be expected with iothreads enabled (perhaps that’s just because the interface to these options changed in QEMU 2.1). I’ll check Munin graphs in a week or three and see if this impacts anything.