See the paper for more details CXL-ClusterSim: Modeling CXL-based Disaggregated Memory Cluster for Pooling and Sharing using gem5 and SST
TL;DR See Shared Memory Simulation in CXL 3.0 for simulating full-system shared memory workloads. TL;DR See CXL-ClusterSim to start using the code ASAP.
We started this project in 2023 when there were limited options in simulating multiple hosts to evaluate Now there are several tools that allows the simulation of CXL. This project simply helps scale existing gem5 implementation of CXL hosts using SST.
Alternately, you can also use pure-gem5 to do multi-host simulations.
The fs.py script already supports bringing up two different gem5 instances on the same gem5 thread.
The only downside (documented in the fs.py script) is that the simulation is slow (its super slow!).
KVM doesn’t work in this case as both the gem5 instances have the exact memory ranges.
Currently we use classic caches in gem5. This can be extended to support directories.
--numa-preferred calls.clflush on the hosts (WIP on the dev branch)OneShot API in SST to improve the performance of the simulation (WIP)The code is mainly confined in the disaggregated_memory directory.
The directory is divided into four subdirectories, similar to the structure of the gem5’s standard library:
boards: The disaggregated memory boards are inherited from the stdlib’s boards. Users can pass two memory ranges. The first one is to model the local memory and the second one is to model a remote memory. The remote memory may or may not be in gem5, as these boards can be used directly with SST. These ranges are exposed as NUMA and zNUMA nodes to the operating system. Currently the following boards are supported:
ArmComposableMemoryBoard implemented in arm_main_board.py. Used for pooling out of the box.RiscvComposableMemoryBoard implemented in riscv_main_board.py. Used for pooling out of the box.X86ComposableMemoryBoard implemented in x86_main_board.py. Used for pooling with fake NUMA nodes for pooling.X86SharedMemoryBoard implemented in x86_shared_board.py. Uses DAX ranges (E820 type=12) enable memory sharing across hosts.X86AlternateSharedMemoryBoard removes the annoying 3 GiB limit on X86 systems. There is an additional kernel memory used as a hack.memories: This directory contains ExternalRemoteMemory inherited from ExternalMemory. Users can use both gem5 and SST to model this remote memory. ExternalRemoteMemory is a gem5 stdlib component, that can be plugged into any gem5-side stdlib config script. Note that any other AbstractMemorySystem can be plugged in to simulate DAX ranges (cannot share).cachehierarchies: gem5’s stdlib cachehierarchies were modified to handle more than one outgoing connection from the LLC. Currently the following cachehierarchies are supported:
ClassicPrivateL1PrivateL2DMCache: A 2-level private classic cache hierarchyClassicPrivateL1PrivateL2SharedL3DMCache: A 3-level classic cache hierarchy that has a shared LLC.configs: Top-level gem5 scripts that can be used to take checkpoints or run SST simulations.joblists: A folder that has some sample config scripts as json files.unified_run.py: The end-user python script that starts the entire simulation.Instructions on how to use this platform can be found in the following sections.
In short, we use this setup to fast-forward simulations using gem5 to reach the ROI and take a checkpoint. We then end the simulation and start is again in SST while loading the checkpoint.
SST does not allow untimed memory accesses at runtime as different gem5 nodes might be reciding on different processes. Therefore, we split this simulation into two phases. The following diagram shows the workflow of the platform.
G t0 : starting simulation in gem5 (atomic/kvm)
E |
M | t1 : simulation reached the start of ROI
5 |_____|____________________________________________________________ time ->
| |
S t2 : we start the simulation in SST (timing) |
S |
T end of simulation : t3
The first phase is entirely in gem5. This is represented by time t0 and t1. The objective here is to reach the ROI asap take a checkpoint.
The second phase starts by loading the checkpoint back into the system but using an SST-side script. The system remains identical except for the External Memory, which now sends requests and receives responses to and from SST’s memory.
This can be scaled into N differnt gem5 nodes. Checkpoints need to be taken for each of these nodes in their respective first phases.
See the paper link here for a better visualization.
The setup of this infrastructure is fairly complicated. The main reason being two different softwares tied together. SST in theory can run gem5 in KVM creating just one simple run command however, it doesn’t support that at the moment. Therefore we came up with a simple python interface that allows users to quickly simulate multiple gem5 instances with or without sharing remote memory in SST.
To understand the required paramters of the infrastructure, you can:
python disaggregated_memory/unified_run.py -h
Please don’t get overwhelmed by the number of parameters the script accepts! To simplify the usability of the infrastructure, we let the user specify a JSON joblist. The configuration of a system is best written as a python script. However, since there are multiple systems now, we need a better and more structured way of denoting systems without code repetations. This can be further argued and I’d love to hear other ways of improving this.
Here are the required parameters to start a simulation:
--clock=[str] # Specify the clock frequency in Hz. All the hosts are
# clocked at the same frequency. You can have variable
# frequency, however this feature is not implemented.
--count=[int] # Specify the number of hosts in the system. The number of
# SST MPI processes is automatically count + 1, where the
# final process is for the remote memory.
--exp-name=[str] # Specify the name of the output directory. All stats,
# checkpoints and output files will be stored here.
--systemd=[bool] # Optionally specify if the simulation needs to boot up
# the OS using systemd. This is helpful if the user wants
# to fast forward without kvm.
--joblist=[str] # Specify the path to a JSON file that describes a host.
There are other features to create the same system multiple times without creating a joblist. These features are not guaranteed to be supported at a later version.
I don’t know a better way to describe how to define multiple node specifications using a single script. I ended up using JSON as the most structured way to do the same. Shoot me an email if you have better ways to do this.
To describe a system, here is a simple joblist explaining each of the keys:
{
"<node-id>": {
"metadata": {
"comment": "For better book-keeping of each host, the JSON allows commenting as a key-value.",
"experiment": "The output directory of this host will be created using this string.",
"checkpoint": "[optional] Specify the path to already load a checkpoint for this host.",
"max-tics": "[optional] Specify an absolute maximum number of cycles to simulate across all the nodes. Enforced by SST."
},
"cpu": {
"ff-core": "Specify the CPU to use for fast-forwarding [atomic/kvm]",
"roi-core": "Specify a timing-based CPU for the ROI [timing/o3]",
"isa": "Specify the ISA to simulate (can be mixed too!) [arm/riscv/x86]",
"count": "Specify the maximum number of CPU cores to simulate."
},
"cache": {
"type": "Specify the cache hierarchy [l1l2/l1l2l3]",
"l1i-size": "Specify the amount of L1I cache in KiB/MiB",
"l1d-size": "32KiB",
"l2-size": "512KiB",
"l3-size": "8MiB",
"l3-assoc": "16"
},
"local-memory": {
"type": "ddr4",
"size": "2GiB"
},
"remote-memory": {
"size": "",
"shared": "false",
"latency": "170ns",
"start": "0x280000000",
"end": "0x300000000"
},
"workitem": {
"cmd": ["echo \"exp: starting STREAM remotely!\"\n",
"numastat;",
"numactl --membind=1 -- /home/ubuntu/simple-vectorizable-benchmarks/stream/stream.hw.m5 8388608;",
"numastat;",
"m5 --addr=0x10010000 exit;"],
"disk": "/home/kaustavg/disk-images/arm/arm64-hpc-2204-numa-kvm.img-20240304",
"kernel": "/home/kaustavg/vmlinux-5.4.49-NUMA.arm64",
"bootloader": "/home/kaustavg/kernel/arm/bootloader/arm64-bootloader"
}
}
See the following sections on how to start the simulation.
You need to build the infrastructure first! Follow the following steps if you’re building the infrastructure on a Linux based system:
git clone git@github.com:darchr/cxl-clustersim
cd cxl-clustersim
scons build/ALL/gem5.opt -j16
scons build/ALL/libgem5_opt.so -j16 --without-tcmalloc --duplicate-sources
# Start building SST now
cd ext/sst
export SST_CORE_HOME=`pwd`
wget https://github.com/sstsimulator/sst-core/releases/download/v14.0.0_Final/sstcore-14.0.0.tar.gz
tar xf sstcore-14.0.0.tar.gz
cd sstcore-14.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config
make all -j128
make install
cd ..
wget https://github.com/sstsimulator/sst-elements/releases/download/v14.0.0_Final/sstelements-14.0.0.tar.gz
tar xf sstelements-14.0.0.tar.gz
cd sst-elements-library-14.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
--with-sst-core=$SST_CORE_HOME
make all -j128
make install
cd ..
# To avoid doing this everytime when compiling the gem5 shared object, put this lines in .bashrc
export PKG_CONFIG_PATH=`pwd`/lib/pkgconfig
# There are mac makefiles in case you want to test it with MacOS.
mv Makefile.linux Makefile
make clean ; make ARCH=ALL -j4
# go back to the top gem5 directory
cd ../..
The instructions for the dev branch may be different. Please see the local READMEs to keep up with the same.
There are sample joblist in the repository that is used to generate the results of the paper. To simulate STREAM in the remote memory with 170ns of remote memory latency across 4 nodes, you can:
python disaggregated_memory/unified_run.py --clock=4GHz \
--count=4 \
--exp-name=sample_STREAM_example_with_4_nodes \
--joblist=disaggregated_memory/joblist/case-study-1/170ns/joblist_stream_remote_4_nodes.json \
--systemd=True
This will start the fast-forward simulation in gem5.
Then it’ll create checkpoints in sample_STREAM_example_with_4_nodes/exp_name_*/ directories.
The simulation will then automatically switch to SST.
The final statistics will then be written to sample_STREAM_example_with_4_nodes directory.
TODO. This section is under-contruction.
The setup requires specialized kernel and disk images.
Instructions on how to compile the disk image, kernel, bootloader etc. can be found at https://github.com/kaustav-goswami/gem5-resources in the branch disaggregated
Changing gem5 is fairly complicated in my opinion. There is a structured way of changing the infrastrcuture. This section explains most of the necessary changes.
The host to device latency is a configurable parameter per host. On a joblist file, changing the [“remote-memory”][“latency”] in unit time (ps, ns, us, s) changes the CXL latency. Improving the device memory controller or QoS needs to be investigated on the SST-side.
Since hosts are implemented in gem5, the user only needs to add new SimObjects for timing, atomic and functional requests and responses.
The LLC mem_side_ports (from a classic membus for this current version) needs to be connected to the new SimObject’s cpu_side_port and the SimObject’s mem_side_port needs to be connected to ExternalMemory’s cpu_side_ports aka port.
Device-side logic needs to be implemented in SST.
The translator.h translates gem5 packets to SST standardmem requests.
Device-side logic can be added at the gem5 subcomponent, that has access to the host’s memory range.
In the case where a user wants to add new logic for shared memory across multiple hosts, a new SST component might be needed that accepts SST::StandardMem::Request after the gem5 subcomponent, byt before SST’s memHierarchy.
See https://sst-simulator.org/sst-docs/docs/elements/memHierarchy/stdmem on how to get started.
The CXL BISnp is implemented in SST by extending the timingDRAM to send BISnp as an InvNotify* object from the StandardMem interface.
The translator.h and sst_responder_subcomponent.cc is modified accordingly to accept incoming InvNotify requests back to gem5.
CXL 3.0 allows simulating device-side shared memory. We implement memory sharing via DAX devices in CXL-ClusterSim. We demonstrate an example using GAPBS, a popular graph benchmark suite.
Prepare a new joblist where physical address ranges are shared across multiple hosts.
..
"remote-memory": {
"size": "",
"shared": "true",
"latency": "170ns",
"start": "0x500000000",
"end": "0x900000000"
}
..
The start and end addresses of multiple hosts needs to be the same.
There can be multiple shared memory ranges.
SST automatically figures out the memory ranges.
In gem5, we make sure that the same range is mapped to the same AddrRange.
The ExternalMemory class takes care of the rest.
There exists an additional XBar after the membus, which is a proxy for adding additional lookup latencies in gem5.
The CXL specification’s SAT, GDT, GFD tables etc. are not implemented.
Instead additional latencies is added at the crossbar.
We do not have a lot of shared memory workloads as of today.
Disaggregated shared memory workloads is an active research topic (e.g. Tigon).
We use the same foundation laid by FAMFS of using /dev/dax to enable physical address range sharing across hosts.
The shared memory programming model is another research topic. A simple single writer multiple reader for shared CXL memory is the easiest way to create such workloads. Such workloads are memory-decoupled. For instance, GAPBS, a popular graph processing benchmark, can be extended to share a graph across multiple hosts , where each host executes a different graph kernel (i.e. BFS) on the same graph. Similarly, memcached can be reconfigured to store both keys (yes, its slow) and the values on the shared memory such that multiple hosts can querry the same database. ML workloads can also share the weights across hosts running inference on the same model and these weights are usually not updated. I am working on a practical way to easily identify and create memory-decoupled workloads from existing workloads.
On the other hand, compute-decoupled workloads, where parts of the workload are distributed across hosts is a difficult performance problem to solve.
Disaggregated memory does not use RPC to find a variable on the remote memory as there is no compute logic on the remote memory node.
It uses LD/ST instructions directly to save time (1000x).
The infrastrcuture currently supports software-based coherence with clwb and clflush instructions, enabling the exploration of multiple writer multiple reader workloads without any consistency guarantees.
Relevant resources include:
For simplicity, we demonstrate a single writer host and multiple reader hosts programming model where a host allocates a graph in the shared memory and the rest of the hosts run a graph kernel from GAPBS. We used a special version of GAPBS that enables graph sharing between weighted and unweighted graphs.
Prepare the workload by building the gem5 resources:
git clone git@github.com:kaustav-goswami/gem5-resources
cd gem5-resources
git checkout disaggregated
cd src
cd shared-gapbs
./build-x86.sh 24.04
# artifacts will be created in the disk-images folder
cd ..
cd kernels
# see README.md on how to build the kernel
git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux
git checkout v6.9.9 # maybe
cp ../linux-configs/config.x86.6.9.9 .config
make -j`nproc`
Use these resources on your simulation script.
To run this example from the repository:
python3 disaggregated_memory/unified_run.py --count=8 --exp-name=gapbs-shared-170ns --joblist disaggregated_memory/joblist/shared-gapbs/base-170ns.json
python3 disaggregated_memory/unified_run.py --count=8 --exp-name=gapbs-shared-250ns --joblist disaggregated_memory/joblist/shared-gapbs/base-250ns.json
The infrastructure currently supports BISnp.
On a write seen by the shared memory, a BISnp via InvNotify request is sent to all the gem5s connected to the same port in SST.
The gem5-bridge translates the InvNotify to TimingSnoop packet and forces the caches to invalidate the cache line, if cached.
This is enabled via a new SST subcomponent that extends timingDRAM.
On a side note, I have been testing the infrastructure (and gem5 in general) with clwb and clflush instructions.
From my understanding, these instructions are implemented in classic caches in gem5.
However, these instructions are not sent to the downstream cache/memory immediately (see src/mem/cache/Cache.py params clusivity and writeback_clean).
The infrastructure does require the caches to be warmed up a bit before the ROI starts and ends.
https://github.com/sstsimulator/sst-core/releases/download/v15.0.0_Final/sstcore-15.0.0.tar.gz
tar xzf sstcore-15.0.0.tar.gz
cd sstcore-15.0.0
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
--disable-mpi # optional, used when MPI is not available.
make all -j$(nproc)
make install
export PATH=$SST_CORE_HOME/bin:$PATH
https://github.com/kaustav-goswami/sst-elements.git
# make sure to use the correct branch
cd sst-elements
checkout cxl-bi
./autogen.sh
cd ..
cd sst-elements
./configure --prefix=$SST_CORE_HOME --with-python=/usr/bin/python3-config \
--with-sst-core=$SST_CORE_HOME
make all -j$(nproc)
make install
echo "export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$SST_CORE_HOME/lib/pkgconfig/" >> ~/.bashrc
scons build/RISCV/libgem5_opt.so -j $(nproc) --without-tcmalloc --duplicate-sources
TL;DR If you want to understand the code structure and data corruption, see understanding HammerSim.
TL;DR If you want to add a new mitigation, see adding new mitigation.
TL;DR If you’re interested to start using the infrastructure ASAP, see using HammerSim
TL;DR Changes to the gem5’s source is confined to the following files:
src/mem/DRAMInterface.pysrc/mem/mem_ctrl.ccsrc/mem/mem_ctrl.hhsrc/mem/packet.hhsrc/mem/dram_interface.hhsrc/mem/dram_interface.ccsrc/mem/mem_interface.hhsrc/mem/mem_interface.ccsrc/mem/SConscriptMost of the RowHammer parameters are defined in src/mem/DRAMInterface.py.
In the class DRAMInterface, we have defined the following parameters:
device_file - Absolute path to the device map file. The “device map” file
refers to a list of all weak cells in the DRAM device. Currently we only
only flip bits at the column level. The resolution of a bit flip can be
further tuned to be at the capacitor level. This file is a .json file with
the following format:
{
"rank_number": {
"bank_number": {
"row_number": ["(int)list_of_all_weak_columns"],
}
}
}
For getting started, you can use the map included in the repository under prob-005.json.zip.
This map is statistically generated using VARIUS
(S. Sarangi et al.) (see the abstract/writeup for details). You can also
generate this map from the hardware using a RowHammer software like TRRespass
(P. Frigo et al.) or Blacksmith (P. Jattke et al.).
rowhammer_threshold - This is the number of activates requires to trigger a
single bitflip in a victim row. This number is taken from previous research
(Y. Kim et al., J. S. Kim et al.) which states that the minimum activates
required for DDR3 DRAM DIMMs is 139,000 and DDR4 DRAM DIMMs is 50,000. LPDDR
numbers are even lower (~8,000 – 16,000).counter_table_length - This is a Target Row Refresh (TRR) specific
parameter. TRR is the mitigation mechanism present in all modern day DDR4
DRAM DIMMs. Most of these TRR parameters are either reverse-engineered via
previously mentioned RowHammer softwares or are taken from other reverse-
engineering papers including but not limited to (H. Hassan et al.).
counter_table_length is the total size of the main TRR table. TRR samples
frequently activated rows. This table keeps a track of these rows.trr_variant - [0 – 4]. We have implemented a version of the 2 TRR variants
out of the three major DRAM vendors (Samsung, SK Hynix and MICRON) based on
previous reverse-engineering techniques and also our own observations. This
is not a 1:1 implementation of the actual TRR as it is proprietary, however
we have tested for similar bitflips in same rows against real hardware.
Following are the four different trr_variants:
companion_table_length - Inserting a row into the companion table is tricky IMO.
Therefore, I have used another small table, similar to the work called ProHIT (M. Son et al.).
A row is initially inserted into the companion table first.
Then, it is promoted to the counter table.
This is specific to the TRR variant, which uses counter tables.companion_threshold - This is minimum number of activates required to make
an entry into the companion table. Understandably, the threshold for the
companion table is much lower than the actual TRR table (1024).trr_stat_dump - This is a boolean value ot dump all the actions of the TRR
mechanism. One can set this to true to do a post-runtime analysis of
RowHammer and TRR.rh_stat_dump - Similar to trr_stat_dump, you can also dump the stats of
the RowHammer triggers. This is helpful for post-runtime analysis.single_sided_prob - The number of bitflips observed with a single-sided
RowHammer attack is much lower than a double-sided rowhammer attack. We saw
that this drop is 1e7 times less probable than a double-sided RowHammer
attack.half_double_prob - Half-double (Google) is even more rare than a single
sided RowHammer attack. We could not reproduce this with our experimental
hardware setup. Therefore, we took this number from the Half-Double report.
We kept this probability at 1/1e9.Change to gem5 in HammerSim is quite invasive, meaning that dram_interface was directly modified without adding new SimObjects. The justification is that this is a separate fork of gem5, that models hardware equivalent of a DRAM DIMM within the simulator. Since RowHammer is baked into the DIMM directly, we decided to make changes directly into the dram_interface.
RowHammer is implemented using counters called rhTriggers[row][4] per row per bank.
Each of these counters count the likelyhood of the neighboring rows getting triggered.
Whenever there is an ACTIVATE for a row r, the rhTriggers[row] increment by 1.
ACTIVATE is implemented in the method
void
DRAMInterface::activateBank(Rank& rank_ref, Bank& bank_ref,
Tick act_tick, uint32_t row)
rhTriggers keeps a track of rows r - 2, r - 1, r + 1 and r + 2 using the four counter indices.
Since ACTIVATING the row and then PRECHARGING the same row nullifies the likelyhood of a bit flip, the rhTriggers of rows r - 2, r - 1, r + 1 and r + 2 are updated with respect to row r.
Whenever rhTrigger[row][index] crosses the rowhammer_threshold (defined via python), there is a non-zero probability of a bitflip on the same row row.
There are four probability distributions to make this decision:
device_map.RowHammer is checked in the method:
void
DRAMInterface::checkRowHammer(Bank& bank_ref, MemPacket* mem_pkt)
{
// each of the four conditions of having a bitflip is checked here.
}
This method checkRowHammer() is called after finishing a burst access to the DRAM:
std::pair<Tick, Tick>
DRAMInterface::doBurstAccess(MemPacket* mem_pkt, Tick next_burst_at,
const std::vector<MemPacketQueue>& queue)
This makes sure to cause RowHammer bitflips if triggered in the current ACTIVATE.
To make changes on how RowHammer behaves, start with checkRowHammer() method.
This method automatically calls doMemoryCorruption() with the aggressor row address, and, victim locations.
void
DRAMInterface::doMemoryCorruption(MemPacket* mem_pkt, uint8_t bank,
uint32_t victim_row, uint16_t col, int distance) {
The correct victim location (victim row, column, and capacitor) is computed each time when a new bit is flipped, which makes data corruption an expensive feature.
Currently, we only support DRAM address mapping with ro as the high bits and co as the low order bits.
See how different mapping in gem5 work: DRAM mappings in gem5.
We’ll support other memory mappings soon with multi-channeled memory.
If you’re using rowhammer-test, should be able to see your aggressor rows and victim rows correctly.
HammerSim implements TRR using trr_tables[TABLE_LENGTH][4 PARAMETERS] to keep track of highly activated rows per bank.
Each entry stores the rank, bank, row and a count of activates.
Some TRR implementations are per rank.
There are a lot of data structures that can be reused for keeping track of frequent aggressors.
See companion_table for a multi-table TRR/mitigation method.
In addition, there are flagged_entries to make sure the same capacitor cannot flip twice.
Aggressors are deterministically tracked using aggressor_rows.
Adding a new mitigation mechanism has to be done in the dram_interface.cc file.
This is done in:
// the sampler/counter mechanism is defined here.
void
DRAMInterface::activateBank(Rank& rank_ref, Bank& bank_ref,
Tick act_tick, uint32_t row) {
...
switch (trrVariant) {
...
case N: {
// write a new mitigation mechanism here.
}
...
}
...
}
// the inhibitor mechanism is implemented here. this is because the inhibitor
// mechanism is triggers when the DRAM device is locked for refreshing.
void
DRAMInterface::Rank::processRefreshEvent() {
...
switch(dram.trrVariant) {
...
case N: {
// write the inhibitor mechanism here to keep DRAM timing
// consistent.
}
...
}
...
}
RowHammer bitflips are checked in the following function:
void
DRAMInterface::checkRowHammer(Bank& bank_ref, MemPacket* mem_pkt) {
...
}
TWICE (E Lee et al.) is a good mitigation mechanism, easy to understand as a beginner to HammerSim. In this tutorial, we show how to add TWICE in HammerSim.
TWICE mathematically calculates the theoretical maximum number of aggressors possible during one tREFW. This is given by: \(N = \frac{tREFW}{RowHammer Threshold \times tRC}\)
For a simple DDR4 DIMM, $N = 25$ This can be further pruned via:
To implement tracking of these many aggressors per bank, we first define the trrTableLength in the python class for the DIMM.
class TwiceDIMM():
trr_table_length = N
# We'll define a unique trr_variant number for this mitigation
trr_variant = 10
We’ll use the above trr_variant to write our C++ changes.
For this, we’ll directly use trrTable that tracks rows with activate count.
TODO
HammerSim models a functional (not timing) ECC to detect and correct RowHammered bitflips. We implement SECDED within a simplified interface. ECC bits are computed and stored for each DRAM write. Data is corrected if corrupt when read. Since this is expensive to simulate in gem5 for full-system configureations, we selectively keep track of data that is corrupted. If the user wants to simulate ECC, then at every data corruption, we track the original data for the ECC bits calculation. At the time of reading the same data, we compute and use the ECC bits to correct up to 1 bit and detect up to 2 bits error.
There are pre-defined config scripts, that can be directly used with HammerSim.
There are located in configs/dram/rowhammer directory.
There are both traffic generators and also full system scripts.
Note that the disk image path need to be replaced.
Testing was done using Google’s rowhammer-test.
We ran one iteration of the hammering run for both single-sided and double-sided versions of RowHammer.
m5 exit is dropped before starting the workload and m5 exit is dropped at the end of the iteration (with or without bitflips).
To create the disk image, use:
git clone https://github.com/kaustav-goswami/gem5-resources.git
cd gem5-resources
git checkout rowhammer
cd src/rowhammer-fs
./build-x86.sh 22.04
To build the same kernel that we used for the full system simulation:
wget https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.4.49.tar.xz
tar xvf linux-5.4.49.tar.xz
cd linux-5.4.49.tar.xz
menu config # Use the default build
make -j32
Alternately, you can also build the same kernel from gem5-resources repository.
Use custom resource in gem5 to plug these artifacts.
board.set_kernel_disk_workload(
# The x86 linux kernel will be automatically downloaded to the if not
# already present.
kernel=CustomResource(
os.path.join(
os.path.expanduser("~"), ".cache/gem5/x86-linux-kernel-5.4.49"
)
),
# The x86 ubuntu image will be automatically downloaded to the if not
# already present.
disk_image=CustomDiskImageResource(
os.path.join(os.getcwd(),
"gem5-resources/src/rowhammer-fs/x86-disk-image-22-04/x86-ubuntu"),
root_partition="1"
),
readfile_contents=" ".join(command),
)
Scripts to run rowhammer-test is located in configs/dram/rowhammer/FSConfigs/rowhammer-test/x86-rowhammer-with-kvm.py (and there is a no cache version).
More on HammerSim can be found here: https://arch.cs.ucdavis.edu/memory/simulation/security/2023/03/20/yarch-hammersim.html
]]>