Tails

#5 Updated by intrigeri 2019-10-04 04:43:32

In the last 8 test suite runs on devel, this problem happened 5 times.
In the first 2 test suite runs with Linux 5.3-rc5, this problem did not happen.
This makes me hopeful but that’s not enough data points to draw conclusions yet ⇒ I’ll keep crossing fingers and will keep an eye on this while working on Bug #17117.

#9 Updated by segfault 2019-10-07 09:19:00

intrigeri wrote:
> @segfault, oh oh, interesting! :)

Not really, I just increased the memory constants as a wild guess. It didn’t work, in both test runs for this commit the scenario failed again. I will delete the commit from the feature branch.

And I still don’t have a working setup to run the test suite locally, so I think I’m not in a good position to work on this.

#12 Updated by anonym 2019-10-07 14:59:31

I just noticed that this scenario lacks an “anti-test”, and I can see how it would be hard to implement: the point you want to dump the memory and verify that the patterns are there is right before OOM kills fillram which is pretty hard to pin-point to say the least. :) Still, it would inspire confidence in our testing method to have such a test!

Luckily, if we do a partial fill, like I suggest above, we will retain full control and can easily make the partial filler signal when done (e.g. create a file that we wait for via the remote shell) so we know when to dump and verify the pattern, and wait until we manually kill it after which we proceed to dump and verify that the pattern is gone.

So I am just wondering if I am missing something important about filling specifically all of RAM. Back when Tails was 32-bit (or mixed with 64-bit) there were some of uncertainty around the 4 GiB limit, which motivated us to fill a larger region than 4 GiB, so without much thought we might have just gone ahead and filled all of it. But, AFAICT, if we went with the approach I describe above and allocated >4 GiB then the test would still have done what we wanted.

Also note that in today’s 64-bit world we have no reason for allocating >4 GiB, so we could go with 128 MiB like the other tests arbitrarily (?) do.

#13 Updated by intrigeri 2019-10-09 15:10:51

Hi @anonym,

> I just noticed that this scenario lacks an “anti-test”, and I can see how it would be hard to implement: the point you want to dump the memory and verify that the patterns are there is right before OOM kills fillram which is pretty hard to pin-point to say the least. :) Still, it would inspire confidence in our testing method to have such a test!

Right, there’s a possibility that the fillram instances fail for unrelated reasons, and then we’re testing exactly nothing here.

> Luckily, if we do a partial fill, like I suggest above, we will retain full control and can easily make the partial filler signal when done (e.g. create a file that we wait for via the remote shell) so we know when to dump and verify the pattern, and wait until we manually kill it after which we proceed to dump and verify that the pattern is gone.

> So I am just wondering if I am missing something important about filling specifically all of RAM. Back when Tails was 32-bit (or mixed with 64-bit) there were some of uncertainty around the 4 GiB limit, which motivated us to fill a larger region than 4 GiB, so without much thought we might have just gone ahead and filled all of it. But, AFAICT, if we went with the approach I describe above and allocated >4 GiB then the test would still have done what we wanted.

> Also note that in today’s 64-bit world we have no reason for allocating >4 GiB, so we could go with 128 MiB like the other tests arbitrarily (?) do.

This proposal looks entirely good enough to me. And of course, it should solve this ticket :)

#20 Updated by intrigeri 2019-10-14 15:30:15

Hi @anonym,

> I tried $vm.has_process?, which is based on pidof -x, but it doesn’t match against the arguments, i.e. the script name used to identify the process at all. I didn’t dare switch implementation of $vm.has_process?.

OK!

> I was considering to actually make the remote shell record the PID of the last executed command and store it in the RemoteShell::ShellCommand object that is returned (like $! in bash etc.) but went with this simpler solution in the end.

In passing, for your curiosity mostly: I’ve played a bit with systemd-run a few weeks ago for other reasons (finding a way to avoid having to reboot isotesters between test suite runs, which would be desirable for a number of reasons). I think it would provide exactly what we want here: a way to reliably monitor and manage a process’s life cycle (we can give it a unique identifier with --unit; --collect might be useful too). It’s probably not worth the effort in this specific case where we already have something simple that works well, but now that I’ve re-discovered this tool, I’m pretty sure I’ll find cases where it’s exactly the one we need :)

> if you think it’s worth spending time on any of the above, let me know and I file a ticket.

Let’s not bother IMO. Good enough is good enough!