Feature #11581

Blocklist dangerous PCIe hotplugging modules that are not needed for supported use cases

Added by cypherpunks 2016-07-19 23:14:01 . Updated 2019-06-02 15:24:49 .

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Target version:
Start date:
2016-07-19
Due date:
% Done:

10%

Feature Branch:
Type of work:
Research
Blueprint:

Starter:
0
Affected tool:
Deliverable for:

Description

The shpchp kernel module enables PCIe hotplugging, which enables DMA attacks. These are commonly used in the wild by law enforcement in order to obtain forensically valid snapshots of memory. Tails users have no need for PCIe hotplugging, so the shpchp driver should be disabled.

--- /etc/modprobe.d/no-shpchp.conf
+++ /etc/modprobe.d/no-shpchp.conf
@@ -0,0 +1 @@
+blacklist shpchp

Subtasks


History

#1 Updated by intrigeri 2016-07-20 00:53:31

  • Assignee set to cypherpunks
  • QA Check set to Info Needed

Please explain what use cases blacklisting that module would break (e.g. docking stations?).

#2 Updated by cypherpunks 2016-07-20 19:12:16

intrigeri wrote:
> Please explain what use cases blacklisting that module would break (e.g. docking stations?).

High-availability servers that use PCIe-based SSDs or other peripherals will need to be shut down in order to have the devices replaced, rather than being opened up and serviced while still running. I don’t think that will affect Tails users, since they are not HA servers. I don’t believe it will affect an already installed ExpressCard (they have their own driver).

Examples of use cases of PCIe hotplugging (they’re all “enterprise solutions”):

http://electronicdesign.com/boards/pci-express-hot-plug-new-era-ras
https://docs.oracle.com/cd/E23824_01/html/821-1459/devconfig2-6.html
https://serverfault.com/questions/366651/can-i-pull-a-card-from-the-pci-x-bus-without-powering-down

Unfortunately it turns out Debian already has pciehp and acpiphp built into the kernel (which are also PCIe hotplugging drivers). I’m reading through the kernel source too and it seems that it disabling shpchp only disables one type of PCIe hotplugging. Blacklisting it won’t hurt, though, and will probably defeat some forensic tools. There might be a clever way to disable all PCI/PCIe hotplugging, even with the drivers being kernel builtins. I’m looking into that.

#3 Updated by emmapeel 2016-08-06 02:52:49

  • Assignee deleted (cypherpunks)
  • QA Check deleted (Info Needed)

information was provided… so changing QA check and assignee.

#4 Updated by emmapeel 2016-08-06 02:56:20

  • Status changed from New to Confirmed
  • Assignee set to Dr_Whax

(09:52:19 AM) drwhax: I guess blacklisting it doesn’t hurt, considering one cant normally hotplug PCIe in laptops
(09:52:34 AM) drwhax: and for desktops, I wouldnt understand why one would want that to happen

DrWhax seems interested on it….

#5 Updated by cypherpunks 2016-09-21 04:26:33

Just a small correction on my part. This actually disables hotplugging for legacy PCI on boards that support it, which should still be helpful to some older systems which do not yet have PCIe devices or have devices tightly occupied.

#6 Updated by intrigeri 2017-03-20 10:25:01

  • Subject changed from Blacklist the PCIe hotplugging module (shpchp) to Blacklist dangerous PCIe hotplugging modules that are not needed for supported use cases
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Wrt. acpiphp: https://bugs.debian.org/663433 explains why it can’t be autoloaded and needs to be built in. The use case that triggered that bug report and the corresponding fix (building the driver in the kernel) seems like something we should support. I’ve also found references that indicate this module is needed for docking some laptops (and it seems to be the case for PCI hotplug in general, e.g. some Thunderbolt docks apparently require shpchp). So it doesn’t look like we can disable PCI hotplug entirely. Is it worth doing anything at all if we can’t disable it fully? If yes, then what exactly should we disable while being confident we won’t break laptop docking support?

#7 Updated by intrigeri 2017-03-20 10:28:29

#8 Updated by cypherpunks 2017-03-30 04:08:14

intrigeri wrote:
> Wrt. acpiphp: https://bugs.debian.org/663433 explains why it can’t be autoloaded and needs to be built in. The use case that triggered that bug report and the corresponding fix (building the driver in the kernel) seems like something we should support. I’ve also found references that indicate this module is needed for docking some laptops (and it seems to be the case for PCI hotplug in general, e.g. some Thunderbolt docks apparently require shpchp). So it doesn’t look like we can disable PCI hotplug entirely. Is it worth doing anything at all if we can’t disable it fully? If yes, then what exactly should we disable while being confident we won’t break laptop docking support?

It is absolutely worth continuing to attempt to mitigate DMA even if PCI hotplug cannot be disabled entirely. Most motherboards do not support hotplugging for PCI, only PCIe, and attempting to hotplug PCI will likely fry both the PCI card and your board. On the majority of systems, even with acpiphp and shpchp loaded, the hardware itself will prevent the problematic situation we would like to avoid. The systems which do support PCI hotplugging, and which get the feature enabled by the modules, are simply out of luck for now.

If and when the rest of the common DMA issues are dealt with, I could try to get a patch upstream which provides a kernel boot parameter for the module which disables hotplugging, or sets the probe interval such that it never probes for new devices. As Debian has a reason to build it into the kernel, upstream Linux may be willing to accept such a feature and won’t shoot it down with the reasoning “just built it as a module and blacklist it”.

#9 Updated by intrigeri 2017-03-30 09:42:49

  • Assignee changed from Dr_Whax to cypherpunks
  • QA Check set to Info Needed

> It is absolutely worth continuing to attempt to mitigate DMA even if PCI hotplug cannot be disabled entirely.

Fair enough, so we’re now at the next question I’ve asked: “If yes, then what exactly should we disable while being confident we won’t break laptop docking support?”

#10 Updated by cypherpunks 2017-04-01 03:28:31

intrigeri wrote:
> > It is absolutely worth continuing to attempt to mitigate DMA even if PCI hotplug cannot be disabled entirely.
>
> Fair enough, so we’re now at the next question I’ve asked: “If yes, then what exactly should we disable while being confident we won’t break laptop docking support?”

I would imagine PCIe hotplugging can be disabled without breaking docking support, but I don’t know how the hardware architecture behaves in regards to laptops with docking support. Worst-case scenario, hotplugging can be re-enabled at runtime if the laptop is detected as one which supports docking.

#11 Updated by intrigeri 2017-04-01 06:48:33

> I would imagine PCIe hotplugging can be disabled without breaking docking support,

Well, at least acpiphp and shpchp can’t be disabled without breaking docking support.

>Worst-case scenario, hotplugging can be re-enabled at runtime if the laptop is detected as one which supports docking.

Indeed. We would need to do this early enough (e.g. initramfs) so that any device connected via the dock is available before the Greeter is started.

So, next questions are:

  • what exact kernel modules or module options should we disable while being confident we won’t break laptop docking support?
  • how do we detect that a dock is supported?

#12 Updated by intrigeri 2017-04-01 08:18:41

cypherpunk explained on Feature #11792#note-8 that this ticket “is primarily about internal PCI/PCIe devices and incorrectly marked as an external bus issue (even if external buses often interface with PCIe, themselves). An attack involving CardBus or Thunderbolt will not be exploiting PCIe hotplugging”. Indeed, we have other, dedicated tickets for Thunderbolt and ExpressCard. But I wonder about other kinds of external PCIe interfaces, such as:

Are these related in any way to hotplugging?

#13 Updated by cypherpunks 2017-04-03 01:40:40

intrigeri wrote:
> cypherpunk explained on Feature #11792#note-8 that this ticket “is primarily about internal PCI/PCIe devices and incorrectly marked as an external bus issue (even if external buses often interface with PCIe, themselves). An attack involving CardBus or Thunderbolt will not be exploiting PCIe hotplugging”. Indeed, we have other, dedicated tickets for Thunderbolt and ExpressCard. But I wonder about other kinds of external PCIe interfaces, such as:
>
> * ATI XGP
> * other kinds of external PCIe interfaces
>
> Are these related in any way to hotplugging?

True external PCIe interfaces are related to hotplugging and would not work with hotplugging disabled. The same goes for ATI XGP. Support for those are mutually exclusive with the ability to prevent a device from being trivially broken into by plugging a device into the exterior.

Those devices are very rare though, so I wasn’t really counting them. Does anyone who uses Tails rely on them?

#14 Updated by cypherpunks 2017-04-03 02:21:12

intrigeri wrote:
> So, next questions are:
>
> * what exact kernel modules or module options should we disable while being confident we won’t break laptop docking support?
> * how do we detect that a dock is supported?

If PCIe can be disabled early without breaking docking, then I believe it’s the pciehp driver. It’s currently a bulit-in and it doesn’t have any module parameters to disable it. Note that acpiphp and pciehp are mutually exclusive. The former deals with motherboards that support hardware PCI hotplugging and send out ACPI events, and the latter deals with PCIe devices, which natively support hotplugging. It might be possible to prevent PCIe hotplugging by activating acpiphp, even on a motherboard which does not support it and will never send a PCI hotplug ACPI event, but this is just a guess and I’ve never tested it before.

Detecting a dock would be done through sysfs, same way as detecting any other hardware from userspace. That’s the standard way to query hardware information. If done by the bootloader in order to pass on specific module parameters or something, then it’d be done through obtaining DMI info. I don’t know exactly how a dock presents itself and under what name, though.

#15 Updated by intrigeri 2017-04-05 08:04:16

> Those devices are very rare though, so I wasn’t really counting them.

Fair enough.

> Does anyone who uses Tails rely on them?

I have no idea. We could compute stats if we know what to grep from the logs (systemd Journal, lspci, etc.) attached to bug reports we receive.

#16 Updated by intrigeri 2017-04-05 08:08:38

  • Type of work changed from Code to Research

> If PCIe can be disabled early without breaking docking, then I believe it’s the pciehp driver. It’s currently a bulit-in and it doesn’t have any module parameters to disable it. Note that acpiphp and pciehp are mutually exclusive.

Both are built into the Debian kernel, so we can’t blacklist them.
Is there another way to disable this support at runtime?

(Meta: setting Type of work = Research, as it seems unclear at this point how exactly we could implement what this ticket is about.)

#17 Updated by cypherpunks 2018-06-04 04:30:24

intrigeri wrote:
> > If PCIe can be disabled early without breaking docking, then I believe it’s the pciehp driver. It’s currently a bulit-in and it doesn’t have any module parameters to disable it. Note that acpiphp and pciehp are mutually exclusive.
>
> Both are built into the Debian kernel, so we can’t blacklist them.
> Is there another way to disable this support at runtime?
>
> (Meta: setting Type of work = Research, as it seems unclear at this point how exactly we could implement what this ticket is about.)

I was unaware that Debian built two of the three hotplugging modules into the kernel when I first submitted this bug report. Where we can go from now… Submit an request upstream to create a boot parameter or sysctl to disable them at runtime, perhaps? Or go out with guns blazing and hook the kernel with kprobes!

#18 Updated by Anonymous 2018-08-17 15:34:00

ask upstream about it?

#19 Updated by mercedes508 2018-08-23 06:33:07

  • Subject changed from Blacklist dangerous PCIe hotplugging modules that are not needed for supported use cases to Blocklist dangerous PCIe hotplugging modules that are not needed for supported use cases

#20 Updated by cypherpunks 2019-03-07 12:23:57

I recall that there is now a way (added in the last year or two) to disable kernel drivers that are builtin, which would include those previously undisableable modules.

#21 Updated by intrigeri 2019-06-02 15:24:49

  • Assignee deleted (cypherpunks)
  • QA Check deleted (Info Needed)