Tails

#5 Updated by intrigeri 2018-10-17 09:22:27

groente wrote:
> Apart from the systemd services that tried to bring md2_crypt back up again I already mentioned on xmpp, I think that pretty much covers it.

FTR that was fixed.

> Just to be safe, I would recommend running grub-install again on the remaining disks (sda — sdf), it should already be there, but with the occassional ‘grub not found’ during lizard reboots, it’s better to be safe than sorry before pulling disks out.

Good idea! I did sudo dpkg-reconfigure grub-pc, selected /dev/sd[c-f] and let it install GRUB on those drives. Note that we can’t install GRUB on /dev/sd[ab] because there’s simply no room for it (fully encrypted, no partition table nor filesystem that GRUB can use).

#8 Updated by intrigeri 2018-11-17 09:45:52

Our BIOS was still configured to start on the rotating drives. I’ve fixed that.

Pinged taggart on IRC today.

#9 Updated by groente 2018-11-17 19:11:45

due to md1 being degraded (see Bug #16131), the following LV’s will be moved from md1 to md4:

    root
    puppet-git-system   *
    apt-system
    apt-data
    rsync-system
    bittorrent-system
    apt-proxy-system
    apt-proxy-data
    whisperback-system
    bitcoin-data        **
    jenkins-system
    bridge-system
    www-system
    misc-system
    puppet-git-data
    bitcoin-system
    bitcoin-swap
    isos-www            **
    isotester1-system
    im-system
    monitor-system
    isotester2-system
    isotester3-system
    isotester4-system
    isotester4-data
    apt-snapshots       **
    isotester5-system
    isotester5-data
    isotester6-system
    isotester6-data
    translate-system    **
    isobuilder1-system
    isobuilder4-system
    isobuilder3-system
    isobuilder3-data    **
    isobuilder2-system 
    isobuilder2-libvirt **
    isobuilder3-libvirt **
    isobuilder4-libvirt **
    isobuilder1-libvirt **
    apt-proxy-swap

LV’s marked * also have a foot in md3, only the parts from md1 will be moved
LV’s marked were already partially on md4

#13 Updated by intrigeri 2018-11-27 18:30:23

Regarding spreading the I/O load again accross PVs aka. RAID arrays:

• this much seems obvious: spread the ISO builders & testers over at least 2 arrays; they don’t use that much I/O though (we’ve set up stuff & memory to minimize I/O needs here)
• top IOPS consumers (average IOPS over a week, max of read & write): jenkins-data (147.91), apt-snapshots (36.66), translate-system (10.90), apt-proxy-data (7.44), puppet-git-system (6.81), isos (4.86), bitcoin-data (3.80)
• ISO builders & testers, when busy, make other volumes busy (mainly jenkins-data, apt-snapshots, apt-proxy-data); let’s separate them if we can

So let’s try this:

• md3 (old, 500GB): translate-system, apt-proxy-data, puppet-git-system, bitcoin-data, half of Jenkins workers (isobuilders 1-2, isotesters 1-3)
• md4 (old, 2TB): jenkins-data, isos, 1/4 of ISO builders & testers (isobuilder3, isotester4)
• md5 (new, 4TB): apt-snapshots, 1/4 of Jenkins workers (isobuilder4, isotesters 5-6)

I’ll do this once the lower part of the stack is ready and a week or two later I’ll check latency and IOPS per PV, which should tell me how good or bad this first iteration was.

#16 Updated by intrigeri 2018-11-28 11:25:05

Amending the plan: md3 would be too full if we do exactly that so I’ll move isobuilder2 stuff to md5 instead.