Tails

#9 Updated by intrigeri 2016-09-20 15:17:27

(I don’t want to mix two concurrent experiments so I’ll wait.)

#11 Updated by intrigeri 2016-11-06 10:04:46

Before enabling:

sudo numastat -c qemu-system-x86_64

Per-node process memory usage (in MBs)
PID              Node 0 Node 1  Total
---------------  ------ ------ ------
8121 (sudo)           3      1      4
23663 (qemu-syst   8855     34   8889
23685 (qemu-syst    110    534    644
23709 (qemu-syst     89    529    618
23727 (qemu-syst    101    528    629
23745 (qemu-syst     79  23575  23653
23775 (qemu-syst    641     12    653
23793 (qemu-syst    168   8732   8900
23840 (qemu-syst    101  23570  23672
23866 (qemu-syst    130    539    668
23885 (qemu-syst  23645     12  23657
23925 (qemu-syst   1120     14   1134
23943 (qemu-syst    151   2745   2895
23961 (qemu-syst     90    530    620
24031 (qemu-syst    145   8738   8883
24140 (qemu-syst    100   1563   1663
24159 (qemu-syst   1193     17   1209
24179 (qemu-syst    189   4147   4336
24198 (qemu-syst     92  23577  23669
24235 (qemu-syst    155   1552   1707
24295 (qemu-syst   3342  20335  23676
24446 (qemu-syst   8869     32   8901
24468 (qemu-syst  23644     21  23665
31929 (qemu-syst  23645     22  23667
31970 (qemu-syst  23648     16  23664
46498 (qemu-syst   1083     25   1108
---------------  ------ ------ ------
Total            121386 121397 242783

#13 Updated by intrigeri 2017-05-28 13:59:09

Ouch, things are still not ideally balanced:

$ sudo numastat
                           node0           node1
numa_hit               393402402       453822993
numa_miss                 284650       544264909
numa_foreign           544105750          443809
interleave_hit             59805           59815
local_node             393402402       453822993
other_node                     0               0

$ sudo numastat -c qemu-system-x86_64 

Per-node process memory usage (in MBs)
PID              Node 0 Node 1  Total
---------------  ------ ------ ------
12071 (qemu-syst  26505     37  26542
12119 (qemu-syst   1566     24   1590
12160 (qemu-syst    775      6    780
12203 (qemu-syst      6   2744   2750
12269 (qemu-syst   1256     11   1267
12451 (qemu-syst    890  25650  26539
12511 (sudo)          1      2      4
12790 (qemu-syst   3106   1048   4155
12837 (qemu-syst    462   1122   1584
12977 (qemu-syst     11  26530  26541
13040 (qemu-syst     35    532    567
13083 (qemu-syst      9  14546  14555
13131 (qemu-syst      5    874    879
13172 (qemu-syst      7    553    560
13213 (qemu-syst    556      4    560
13254 (qemu-syst     27    533    560
13296 (qemu-syst  10489  16054  26543
13350 (qemu-syst     10    553    563
13393 (qemu-syst     10    554    564
13438 (qemu-syst  14551      5  14556
13492 (qemu-syst  24073   2469  26541
13548 (qemu-syst      9  14545  14553
13608 (qemu-syst      8    555    563
---------------  ------ ------ ------
Total             84367 108950 193316

So I’ve shut down all isobuilders and isotesters, and tried:

• turning off automatic NUMA balancing, running numad, starting isotesters one after the other: several are badly aligned
• turning on automatic NUMA balancing, stopping numad, starting 5 isotesters sequentially: several are badly aligned
• turning off both automatic NUMA balancing and numad, starting 5 isotesters sequentially: all correctly aligned

So I’m not quite sure what’s going on. Possible explanations and ideas:

• One of the algorithms used by automatic NUMA balancing is Migrate-on-Fault, so it’s plausible that the situation gets better over time, and looking at stats immediately after starting the VMs might be mostly worthless.
• We use <vcpu placement='static'>, while <vcpu placement='auto'> would query numad; this is best combined with <numatune><memory mode='strict' placement='auto'/></numatune>, that queries numad too.

#14 Updated by intrigeri 2017-05-28 14:10:48

intrigeri wrote:
> * We use <vcpu placement='static'>, while <vcpu placement='auto'> would query numad; this is best combined with <numatune><memory mode='strict' placement='auto'/></numatune>, that queries numad too.

I’ve tried that (with numad running), let’s see how it goes.

#17 Updated by intrigeri 2018-08-18 09:28:04

intrigeri wrote:
> intrigeri wrote:
> > * We use <vcpu placement='static'>, while <vcpu placement='auto'> would query numad; this is best combined with <numatune><memory mode='strict' placement='auto'/></numatune>, that queries numad too.
>
> I’ve tried that (with numad running), let’s see how it goes.

I get better results than previously: all but one iso{builder,tester}s are correctly placed on a single NUMA node. So I’ll configure the same settings for all our other VMs and will call this done.