CXL-ClusterSim: Simulating CXL systems using gem5 and SST

45 minute read

Simulating multiple gem5 instances isn’t feasible. This documents how to use the composable memory simulation platform in a gem5, SST and gem5 + SST setup. The setup can be used in gem5 to fast-forward full-system simulation and then used in SST to simulate a multi-node system.

See the paper for more details CXL-ClusterSim: Modeling CXL-based Disaggregated Memory Cluster for Pooling and Sharing using gem5 and SST

TL;DR See Shared Memory Simulation in CXL 3.0 for simulating full-system shared memory workloads. TL;DR See CXL-ClusterSim to start using the code ASAP.

We started this project in 2023 when there were limited options in simulating multiple hosts to evaluate Now there are several tools that allows the simulation of CXL. This project simply helps scale existing gem5 implementation of CXL hosts using SST.

Alternately, you can also use pure-gem5 to do multi-host simulations. The fs.py script already supports bringing up two different gem5 instances on the same gem5 thread. The only downside (documented in the fs.py script) is that the simulation is slow (its super slow!). KVM doesn’t work in this case as both the gem5 instances have the exact memory ranges.

Currently we use classic caches in gem5. This can be extended to support directories.

Roadmap

D-Tours

Simple graph example for shared memory: simple-graphs
Extend the memory allocation of Shared-GAPBS to be a generic shared memory allocator for more workloads: disagg-shmem-alloc
Reimplement memcached with shared memory: memcached, YCSB
Reimplement TPC-C and/or TPC-H with shared memory
Reimplement some KV-cache for more shared memory workloads.

Source Code Changes

The code is mainly confined in the disaggregated_memory directory. The directory is divided into four subdirectories, similar to the structure of the gem5’s standard library:

boards: The disaggregated memory boards are inherited from the stdlib’s boards. Users can pass two memory ranges. The first one is to model the local memory and the second one is to model a remote memory. The remote memory may or may not be in gem5, as these boards can be used directly with SST. These ranges are exposed as NUMA and zNUMA nodes to the operating system. Currently the following boards are supported:
- ArmComposableMemoryBoard implemented in arm_main_board.py. Used for pooling out of the box.
- RiscvComposableMemoryBoard implemented in riscv_main_board.py. Used for pooling out of the box.
- X86ComposableMemoryBoard implemented in x86_main_board.py. Used for pooling with fake NUMA nodes for pooling.
- X86SharedMemoryBoard implemented in x86_shared_board.py. Uses DAX ranges (E820 type=12) enable memory sharing across hosts.
- X86AlternateSharedMemoryBoard removes the annoying 3 GiB limit on X86 systems. There is an additional kernel memory used as a hack.
memories: This directory contains ExternalRemoteMemory inherited from ExternalMemory. Users can use both gem5 and SST to model this remote memory. ExternalRemoteMemory is a gem5 stdlib component, that can be plugged into any gem5-side stdlib config script. Note that any other AbstractMemorySystem can be plugged in to simulate DAX ranges (cannot share).
cachehierarchies: gem5’s stdlib cachehierarchies were modified to handle more than one outgoing connection from the LLC. Currently the following cachehierarchies are supported:
- ClassicPrivateL1PrivateL2DMCache: A 2-level private classic cache hierarchy
- ClassicPrivateL1PrivateL2SharedL3DMCache: A 3-level classic cache hierarchy that has a shared LLC.
- Note ruby caches only work with the RiscvComposableMemoryBoard.
configs: Top-level gem5 scripts that can be used to take checkpoints or run SST simulations.
joblists: A folder that has some sample config scripts as json files.
unified_run.py: The end-user python script that starts the entire simulation.

Instructions on how to use this platform can be found in the following sections.

Workflow

In short, we use this setup to fast-forward simulations using gem5 to reach the ROI and take a checkpoint. We then end the simulation and start is again in SST while loading the checkpoint.

SST does not allow untimed memory accesses at runtime as different gem5 nodes might be reciding on different processes. Therefore, we split this simulation into two phases. The following diagram shows the workflow of the platform.

G t0 : starting simulation in gem5 (atomic/kvm)
E |
M |     t1 : simulation reached the start of ROI
5 |_____|____________________________________________________________ time ->
         |                                                  |
S        t2 : we start the simulation in SST (timing)       |
S                                                           |
T                                       end of simulation : t3

The first phase is entirely in gem5. This is represented by time t0 and t1. The objective here is to reach the ROI asap take a checkpoint.

The second phase starts by loading the checkpoint back into the system but using an SST-side script. The system remains identical except for the External Memory, which now sends requests and receives responses to and from SST’s memory.

This can be scaled into N differnt gem5 nodes. Checkpoints need to be taken for each of these nodes in their respective first phases.

See the paper link here for a better visualization.

Using the infrastructure

The setup of this infrastructure is fairly complicated. The main reason being two different softwares tied together. SST in theory can run gem5 in KVM creating just one simple run command however, it doesn’t support that at the moment. Therefore we came up with a simple python interface that allows users to quickly simulate multiple gem5 instances with or without sharing remote memory in SST.

To understand the required paramters of the infrastructure, you can:

python disaggregated_memory/unified_run.py -h

Please don’t get overwhelmed by the number of parameters the script accepts! To simplify the usability of the infrastructure, we let the user specify a JSON joblist. The configuration of a system is best written as a python script. However, since there are multiple systems now, we need a better and more structured way of denoting systems without code repetations. This can be further argued and I’d love to hear other ways of improving this.

Here are the required parameters to start a simulation:

--clock=[str]       # Specify the clock frequency in Hz. All the hosts are
                    # clocked at the same frequency. You can have variable
                    # frequency, however this feature is not implemented.
--count=[int]       # Specify the number of hosts in the system. The number of
                    # SST MPI processes is automatically count + 1, where the
                    # final process is for the remote memory.
--exp-name=[str]    # Specify the name of the output directory. All stats,
                    # checkpoints and output files will be stored here.
--systemd=[bool]    # Optionally specify if the simulation needs to boot up
                    # the OS using systemd. This is helpful if the user wants
                    # to fast forward without kvm.
--joblist=[str]     # Specify the path to a JSON file that describes a host.

There are other features to create the same system multiple times without creating a joblist. These features are not guaranteed to be supported at a later version.

I don’t know a better way to describe how to define multiple node specifications using a single script. I ended up using JSON as the most structured way to do the same. Shoot me an email if you have better ways to do this.

To describe a system, here is a simple joblist explaining each of the keys:

{
    "<node-id>": {
        "metadata": {
            "comment": "For better book-keeping of each host, the JSON allows commenting as a key-value.",
            "experiment": "The output directory of this host will be created using this string.",
            "checkpoint": "[optional] Specify the path to already load a checkpoint for this host.",
            "max-tics": "[optional] Specify an absolute maximum number of cycles to simulate across all the nodes. Enforced by SST."
        },
        "cpu": {
            "ff-core": "Specify the CPU to use for fast-forwarding [atomic/kvm]",
            "roi-core": "Specify a timing-based CPU for the ROI [timing/o3]",
            "isa": "Specify the ISA to simulate (can be mixed too!) [arm/riscv/x86]",
            "count": "Specify the maximum number of CPU cores to simulate."
        },
        "cache": {
            "type": "Specify the cache hierarchy [l1l2/l1l2l3]",
            "l1i-size": "Specify the amount of L1I cache in KiB/MiB",
            "l1d-size": "32KiB",
            "l2-size": "512KiB",
            "l3-size": "8MiB",
            "l3-assoc": "16"
        },
        "local-memory": {
            "type": "ddr4",
            "size": "2GiB"
        },
        "remote-memory": {
            "size": "",
            "shared": "false",
            "latency": "170ns",
            "start": "0x280000000",
            "end": "0x300000000"
        },
        "workitem": {
            "cmd": ["echo \"exp: starting STREAM remotely!\"\n",
                    "numastat;",
                    "numactl --membind=1 -- /home/ubuntu/simple-vectorizable-benchmarks/stream/stream.hw.m5 8388608;",
                    "numastat;",
                    "m5 --addr=0x10010000 exit;"],
            "disk": "/home/kaustavg/disk-images/arm/arm64-hpc-2204-numa-kvm.img-20240304",
            "kernel": "/home/kaustavg/vmlinux-5.4.49-NUMA.arm64",
            "bootloader": "/home/kaustavg/kernel/arm/bootloader/arm64-bootloader"
        }
    }

See the following sections on how to start the simulation.

Building

You need to build the infrastructure first! Follow the following steps if you’re building the infrastructure on a Linux based system:

git clone git@github.com:darchr/cxl-clustersim
cd cxl-clustersim
scons build/ALL/gem5.opt -j16
scons build/ALL/libgem5_opt.so -j16 --without-tcmalloc --duplicate-sources

# Start building SST now
cd ext/sst
export SST_CORE_HOME=`pwd`
wget https://github.com/sstsimulator/sst-core/releases/download/v14.0.0_Final/sstcore-14.0.0.tar.gz
tar xf sstcore-14.0.0.tar.gz
cd sstcore-14.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config
make all -j128
make install
cd ..
wget https://github.com/sstsimulator/sst-elements/releases/download/v14.0.0_Final/sstelements-14.0.0.tar.gz
tar xf sstelements-14.0.0.tar.gz
cd sst-elements-library-14.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
                                        --with-sst-core=$SST_CORE_HOME
make all -j128
make install
cd ..
# To avoid doing this everytime when compiling the gem5 shared object, put this lines in .bashrc
export PKG_CONFIG_PATH=`pwd`/lib/pkgconfig

# There are mac makefiles in case you want to test it with MacOS.
mv Makefile.linux Makefile
make clean ; make ARCH=ALL -j4

# go back to the top gem5 directory
cd ../..

The instructions for the dev branch may be different. Please see the local READMEs to keep up with the same.

Sample Example

There are sample joblist in the repository that is used to generate the results of the paper. To simulate STREAM in the remote memory with 170ns of remote memory latency across 4 nodes, you can:

python disaggregated_memory/unified_run.py --clock=4GHz \
                --count=4 \
                --exp-name=sample_STREAM_example_with_4_nodes \
                --joblist=disaggregated_memory/joblist/case-study-1/170ns/joblist_stream_remote_4_nodes.json \
                --systemd=True

This will start the fast-forward simulation in gem5. Then it’ll create checkpoints in sample_STREAM_example_with_4_nodes/exp_name_*/ directories. The simulation will then automatically switch to SST. The final statistics will then be written to sample_STREAM_example_with_4_nodes directory.

Gem5 resources

TODO. This section is under-contruction.

The setup requires specialized kernel and disk images. Instructions on how to compile the disk image, kernel, bootloader etc. can be found at https://github.com/kaustav-goswami/gem5-resources in the branch disaggregated

Changing the infrastructure to implement a research idea

Changing gem5 is fairly complicated in my opinion. There is a structured way of changing the infrastrcuture. This section explains most of the necessary changes.

Varying CXL latency

The host to device latency is a configurable parameter per host. On a joblist file, changing the [“remote-memory”][“latency”] in unit time (ps, ns, us, s) changes the CXL latency. Improving the device memory controller or QoS needs to be investigated on the SST-side.

Adding new host-side logic

Since hosts are implemented in gem5, the user only needs to add new SimObjects for timing, atomic and functional requests and responses. The LLC mem_side_ports (from a classic membus for this current version) needs to be connected to the new SimObject’s cpu_side_port and the SimObject’s mem_side_port needs to be connected to ExternalMemory’s cpu_side_ports aka port.

Adding new device-side logic

Device-side logic needs to be implemented in SST. The translator.h translates gem5 packets to SST standardmem requests. Device-side logic can be added at the gem5 subcomponent, that has access to the host’s memory range. In the case where a user wants to add new logic for shared memory across multiple hosts, a new SST component might be needed that accepts SST::StandardMem::Request after the gem5 subcomponent, byt before SST’s memHierarchy. See https://sst-simulator.org/sst-docs/docs/elements/memHierarchy/stdmem on how to get started.

The CXL BISnp is implemented in SST by extending the timingDRAM to send BISnp as an InvNotify* object from the StandardMem interface. The translator.h and sst_responder_subcomponent.cc is modified accordingly to accept incoming InvNotify requests back to gem5.

Shared Memory

CXL 3.0 allows simulating device-side shared memory. We implement memory sharing via DAX devices in CXL-ClusterSim. We demonstrate an example using GAPBS, a popular graph benchmark suite.

Prepare a new joblist where physical address ranges are shared across multiple hosts.

    ..
    "remote-memory": {
        "size": "",
        "shared": "true",
        "latency": "170ns",
        "start": "0x500000000",
        "end": "0x900000000"
    }
    ..

The start and end addresses of multiple hosts needs to be the same. There can be multiple shared memory ranges. SST automatically figures out the memory ranges.

In gem5, we make sure that the same range is mapped to the same AddrRange. The ExternalMemory class takes care of the rest. There exists an additional XBar after the membus, which is a proxy for adding additional lookup latencies in gem5. The CXL specification’s SAT, GDT, GFD tables etc. are not implemented. Instead additional latencies is added at the crossbar.

Shared Memory Workloads

We do not have a lot of shared memory workloads as of today. Disaggregated shared memory workloads is an active research topic (e.g. Tigon). We use the same foundation laid by FAMFS of using /dev/dax to enable physical address range sharing across hosts.

The shared memory programming model is another research topic. A simple single writer multiple reader for shared CXL memory is the easiest way to create such workloads. Such workloads are memory-decoupled. For instance, GAPBS, a popular graph processing benchmark, can be extended to share a graph across multiple hosts , where each host executes a different graph kernel (i.e. BFS) on the same graph. Similarly, memcached can be reconfigured to store both keys (yes, its slow) and the values on the shared memory such that multiple hosts can querry the same database. ML workloads can also share the weights across hosts running inference on the same model and these weights are usually not updated. I am working on a practical way to easily identify and create memory-decoupled workloads from existing workloads.

On the other hand, compute-decoupled workloads, where parts of the workload are distributed across hosts is a difficult performance problem to solve. Disaggregated memory does not use RPC to find a variable on the remote memory as there is no compute logic on the remote memory node. It uses LD/ST instructions directly to save time (1000x). The infrastrcuture currently supports software-based coherence with clwb and clflush instructions, enabling the exploration of multiple writer multiple reader workloads without any consistency guarantees.

Relevant resources include:

shared-memcached
YCSB
gem5 resources; branch: disaggregated

Building resources

For simplicity, we demonstrate a single writer host and multiple reader hosts programming model where a host allocates a graph in the shared memory and the rest of the hosts run a graph kernel from GAPBS. We used a special version of GAPBS that enables graph sharing between weighted and unweighted graphs.

Prepare the workload by building the gem5 resources:

git clone git@github.com:kaustav-goswami/gem5-resources
cd gem5-resources
git checkout disaggregated
cd src
cd shared-gapbs
./build-x86.sh 24.04
# artifacts will be created in the disk-images folder

cd ..
cd kernels
# see README.md on how to build the kernel
git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux
git checkout v6.9.9 # maybe
cp ../linux-configs/config.x86.6.9.9 .config
make -j`nproc`

Use these resources on your simulation script.

Running examples

To run this example from the repository:

python3 disaggregated_memory/unified_run.py --count=8 --exp-name=gapbs-shared-170ns --joblist disaggregated_memory/joblist/shared-gapbs/base-170ns.json
python3 disaggregated_memory/unified_run.py --count=8 --exp-name=gapbs-shared-250ns --joblist disaggregated_memory/joblist/shared-gapbs/base-250ns.json

Software-Enforced Back-Invalidate Snoops (BISnp)

The infrastructure currently supports BISnp. On a write seen by the shared memory, a BISnp via InvNotify request is sent to all the gem5s connected to the same port in SST. The gem5-bridge translates the InvNotify to TimingSnoop packet and forces the caches to invalidate the cache line, if cached. This is enabled via a new SST subcomponent that extends timingDRAM.

On a side note, I have been testing the infrastructure (and gem5 in general) with clwb and clflush instructions. From my understanding, these instructions are implemented in classic caches in gem5. However, these instructions are not sent to the downstream cache/memory immediately (see src/mem/cache/Cache.py params clusivity and writeback_clean).

Known Issues

The infrastructure does require the caches to be warmed up a bit before the ROI starts and ends.

Building with BISnp

https://github.com/sstsimulator/sst-core/releases/download/v15.0.0_Final/sstcore-15.0.0.tar.gz
tar xzf sstcore-15.0.0.tar.gz
cd sstcore-15.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
            --disable-mpi # optional, used when MPI is not available.
make all -j$(nproc)
make install
export PATH=$SST_CORE_HOME/bin:$PATH

https://github.com/kaustav-goswami/sst-elements.git
# make sure to use the correct branch
cd sst-elements
checkout cxl-bi
./autogen.sh
cd ..

cd sst-elements
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
            --with-sst-core=$SST_CORE_HOME
make all -j$(nproc)
make install

echo "export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$SST_CORE_HOME/lib/pkgconfig/" >> ~/.bashrc
scons build/RISCV/libgem5_opt.so -j $(nproc) --without-tcmalloc --duplicate-sources

Share on

X Facebook LinkedIn Bluesky

Kaustav Goswami