CXL Driver Operation

The devices described in this section are present in

/sys/bus/cxl/devices/
/dev/cxl/

The cxl-cli library, maintained as part of the NDTCL project, may be used to script interactions with these devices.

Drivers

The CXL driver is split into a number of drivers.

  • cxl_core - fundamental init interface and core object creation

  • cxl_port - initializes root and provides port enumeration interface.

  • cxl_acpi - initializes root decoders and interacts with ACPI data.

  • cxl_p/mem - initializes memory devices

  • cxl_pci - uses cxl_port to enumerate the actual fabric hierarchy.

Driver Devices

Here is an example from a single-socket system with 4 host bridges. Two host bridges have a single memory device attached, and the devices are interleaved into a single memory region. The memory region has been converted to dax.

# ls /sys/bus/cxl/devices/
  dax_region0  decoder3.0  decoder6.0  mem0   port3
  decoder0.0   decoder4.0  decoder6.1  mem1   port4
  decoder1.0   decoder5.0  endpoint5   port1  region0
  decoder2.0   decoder5.1  endpoint6   port2  root0
Digraph of CXL fabric describing host-bridge interleaving

Diagraph of CXL fabric with a host-bridge interleave memory region

For this section we’ll explore the devices present in this configuration, but we’ll explore more configurations in-depth in example configurations below.

Base Devices

Most devices in a CXL fabric are a port of some kind (because each device mostly routes request from one device to the next, rather than provide a direct service).

Root

The CXL Root is logical object created by the cxl_acpi driver during cxl_acpi_probe - if the ACPI0017 Compute Express Link Root Object Device Class is found.

The Root contains links to:

  • Host Bridge Ports defined by CHBS in the CEDT

  • Downstream Ports typically connected to Host Bridge Ports.

  • Root Decoders defined by CFMWS the CEDT

# ls /sys/bus/cxl/devices/root0
  decoder0.0          dport0  dport5    port2  subsystem
  decoders_committed  dport1  modalias  port3  uevent
  devtype             dport4  port1     port4  uport

# cat /sys/bus/cxl/devices/root0/devtype
  cxl_port

# cat port1/devtype
  cxl_port

# cat decoder0.0/devtype
  cxl_decoder_root

The root is first logical port in the CXL fabric, as presented by the Linux CXL driver. The CXL root is a special type of switch port, in that it only has downstream port connections.

Port

A port object is better described as a switch port. It may represent a host bridge to the root or an actual switch port on a switch. A switch port contains one or more decoders used to route memory requests downstream ports, which may be connected to another switch port or an endpoint port.

# ls /sys/bus/cxl/devices/port1
  decoder1.0          dport0    driver     parent_dport  uport
  decoders_committed  dport113  endpoint5  subsystem
  devtype             dport2    modalias   ue