mutest-rs — Mutation testing tools for Rust

Robust, efficient, safe, and parallel mutation testing tools for Rust. Use mutest-rs to generate and evaluate program mutations to assess your test suite.

Get Started using mutest-rs >

Getting Started

To get started:

• first, install mutest-rs from source using the instructions in Installation,
• then, learn more about the basics of how to use the tool on your Cargo projects by reading Usage.

Installation

To install mutest-rs from source, first clone the source code repository at https://github.com/zalanlevai/mutest-rs.

git clone https://github.com/zalanlevai/mutest-rs

Then, build the mutest-runtime crate in release mode.

cargo build --release -p mutest-runtime

Finally, install mutest-driver and the Cargo subcommand cargo-mutest locally.

cargo install --force --path mutest-driver
cargo install --force --path cargo-mutest

Please make note of the directory where you checked out mutest-rs. When using mutest-rs, make sure that the MUTEST_SEARCH_PATH environment variable is set to point to the target/release directory inside. This is to ensure correct linking with the runtime crate. The easiest solution is to add the following to your shell’s init script (replacing <PATH_TO_MUTEST_RS_SRC_REPO> with a path pointing to your local mutest-rs repository):

export MUTEST_SEARCH_PATH=<PATH_TO_MUTEST_RS_SRC_REPO>/target/release

Currently, the only option to install and use mutest-rs is to compile it yourself.

Usage

To run mutest-rs on any Cargo package, use the cargo mutest run subcommand with the usual Cargo targeting options (see Cargo Package Selection, Cargo Target Selection, and Cargo Feature Selection):

cargo mutest -p example-package --lib run

NOTE: Currently, running the tool requires manually specifying the MUTEST_SEARCH_PATH environment variable to point to the local mutest-rs build artifacts (see Installation).

Prerequisites

The main cargo mutest subcommand provides a Cargo-compatible interface to mutest-rs for Cargo packages and workspaces. Generally speaking, as long as cargo test works for your package, then cargo mutest run will run the same test suite under mutation analysis.

Using cfg(mutest)

When running cargo mutest, the mutest cfg is set. This can be used to detect if code is running under mutest-rs, and enable conditional compilation based on it.

Starting with Rust 1.80, cfgs are checked against a known set of config names and values. If your Cargo package is checked with a regular Cargo command, it will warn you about the “unexpected” mutest cfg. To let rustc know that this custom cfg is expected, ensure that cfg(mutest) is present in the [lints.rust.unexpected_cfgs.check-cfg] array in the package’s Cargo.toml, like so:

[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ["cfg(mutest)"] }

mutest-rs and Integration Tests

Currently, mutest-rs does not support mutating integration tests (i.e. tests in a separate tests/ directory), and is unlikely to support mutating program code while evaluating an integration test. This is because rustc, and by extension cargo test, operates on a per-crate basis, meaning that all compilation and analysis is done separately for integration test cases.

If you would like to incorporate integration tests into the mutation analysis, then you have to integrate them into the program crate. This is not too difficult to do in most cases. By moving the tests from the tests/ directory into a new src/tests/ directory, and creating a new src/tests.rs module listing the test files, you can include the following lines in your src/lib.rs to retain similar functionality to before:

#[cfg(test)]
mod tests;

You may also have to add the following line to each of the moved integration test modules, or resolve path changes manually:

use crate as <crate_name>;

However, this workaround is admittedly not great.

Mutations

Mutations are the individual faults we inject into the program. Think of an accidental multiplication instead of an addition, a negated boolean expression, or something more complex. When a mutation is applied to the valid program, it effectively creates a mutant program, with slightly different behavior. These are used to evaluate the test suite, by seeing whether the test suite is able to “detect” this difference in behavior.

Mutations in mutest-rs are generated upfront, and combined into a single executable program for efficiency, alongside the generic mutation test harness (contained in the mutest-runtime crate).

The mutations mutest-rs produces can be controlled in various ways:

• by changing the mutation depth, the depth in the call graph up to which mutest-rs attempts to generate mutations, using the -d, or --depth argument (default: 3);
• by changing the set of mutation operators during the invocation of mutest-rs, changing the set of transformation rules applied to the program;
• by applying tool attributes to individual code elements (e.g. functions, statements, expressions), giving you fine-grained control over where and how mutation operators are applied.

Mutations in mutest-rs have a unique safety property, matching Rust’s notion of safety. Depending on the nature of your codebase, you may relax the safety requirements mutest-rs operates on, which influences how mutations are generated and evaluated. For more information, see Mutation Safety.

In addition to controlling the generation of mutations, mutest-rs also implements novel techniques for improving the efficiency of evaluating these mutations. By batching mutations, mutest-rs is able to improve the parallelism of mutation-related test case evaluation.

Mutation Operators

Mutation operators are the rules that match patterns of program code, and produce corresponding mutations.

The active set of mutation operators can be controlled using the --mutation-operators argument (default: all), by specifying a comma-seperated list of mutation operator names.

cargo mutest --mutation-operators call_delete,call_value_default_shadow run

The following list of mutation operators is currently implemented in mutest-rs, with detailed descriptions and examples below:

NOTE: The following replacements are illustrative and are meant to show how code behaviour effectively changes with each mutation.

arg_default_shadow

Replace the provided arguments of functions with Default::default() to check if each parameter is tested with meaningful values.

This is done by rebinding parameters at the beginning of the function.

Replaces

fn foo(hash: u64) {

with

fn foo(hash: u64) {
    let hash: u64 = Default::default();

bit_op_or_and_swap

Swap bitwise OR for bitwise AND and vice versa.

Replaces

byte & (0x1 << 2)

with

byte | (0x1 << 2)

bit_op_or_xor_swap

Swap bitwise OR for bitwise XOR and vice versa.

Replaces

bytes[i] |= 0x1 << 3

with

bytes[i] ^= 0x1 << 3

bit_op_shift_dir_swap

Swap the direction of bitwise shift operators.

Replaces

byte & (0x1 << i)

with

byte & (0x1 >> i)

bit_op_xor_and_swap

Swap bitwise XOR for bitwise AND and vice versa.

Replaces

byte & (0x1 << 2)

with

byte ^ (0x1 << 2)

bool_expr_negate

Negate boolean expressions.

Replaces

if !handle.is_active() {
    drop(handle);

with

if handle.is_active() {
    drop(handle);

call_delete

Delete function calls and replace them with Default::default() to test whether inner calls are meaningfully tested, without retaining any side-effects of the callees.

Replaces

let existing = map.insert(Id(123), 0);

with

let existing: Option<usize> = Default::default();

call_value_default_shadow

Replace the return value of function calls with Default::default() to test whether the return values of inner calls are meaningfully tested, while retaining expected side-effects of the callees.

Replaces

let existing = map.insert(Id(123), 0);

with

let existing: Option<usize> = {
    let _existing = map.insert(Id(123), 0);
    Default::default()
};

continue_break_swap

Swap continue expressions for break expressions and vice versa.

Replaces

for other in mutations {
    if conflicts.contains(&(mutation, other)) { continue; }

with

for other in mutations {
    if conflicts.contains(&(mutation, other)) { break; }

eq_op_invert

Invert equality checks.

Replaces

if buffer.len() == 0 {
    buffer.reserve(1024);

with

if buffer.len() != 0 {
    buffer.reserve(1024);

logical_op_and_or_swap

Swap logical && for logical || and vice versa.

Replaces

self.len() <= other.len() && self.iter().all(|v| other.contains(v))

with

self.len() <= other.len() || self.iter().all(|v| other.contains(v))

math_op_add_mul_swap

Swap addition for multiplication and vice versa.

Replaces

let offset = size_of::<DeclarativeEnvironment>() * index;

with

let offset = size_of::<DeclarativeEnvironment>() + index;

math_op_add_sub_swap

Swap addition for subtraction and vice versa.

Replaces

let center = Point::new(x + (width / 2), y + (height / 2));

with

let center = Point::new(x - (width / 2), y + (height / 2));

math_op_div_rem_swap

Swap division for modulus and vice versa.

Replaces

let evens = 0..100.filter(|v| v % 2 == 0);

with

let evens = 0..100.filter(|v| v / 2 == 0);

math_op_mul_div_swap

Swap multiplication for division and vice versa.

Replaces

let v = f64::sin(t * freq) * magnitude;

with

let v = f64::sin(t / freq) * magnitude;

range_limit_swap

Invert the limits (inclusivity) of range expressions.

Replaces

for i in 0..buffer.len() {

with

for i in 0..=buffer.len() {

relational_op_eq_swap

Include or remove the boundary (equality) of relational operators.

Replaces

if self.len() <= other.len() {

with

if self.len() < other.len() {

relational_op_invert

Completely invert relation operators.

Replaces

while i < buffer.len() {

with

while i >= buffer.len() {

Controlling Mutations through Attributes

Custom tool attributes can be used to control where and how mutation operators are applied by mutest-rs when generating mutations. These can be used to optionally annotate your code for use with mutest-rs.

Note, that these attributes are only available when running cargo mutest, so they need to be wrapped in #[cfg_attr(mutest, <MUTEST_ATTRIBUTE>)] to ensure that they are only present when you are running mutest-rs. Otherwise, regular Cargo commands will not run. See the prerequisites for using #[cfg(mutest)].

#[mutest::skip]

Tells mutest-rs to skip the function when applying mutation operators. Useful for marking helper functions for tests (test cases themselves are automatically skipped), or critical functions.

This attribute can only be applied to function declarations:

#[cfg_attr(mutest, mutest::skip)]
fn perform_tests() {
}

#[mutest::ignore]

Tells mutest-rs to ignore the statement or expression, including any subexpressions, or function parameter, when applying mutation operators. Useful if mutest-rs is trying to apply mutations to a critical piece of code that might be causing problems.

This attribute can be applied to

• statements (note that expression statements might have to be wrapped in {}, see this linked Rust issue):

  #[cfg_attr(mutest, mutest::ignore)]
  let buff_len = mem::size_of::<u16>() * 1024;

• expressions (wherever the compiler supports attrbiutes on expressions):

    fn foo() {
      #[cfg_attr(mutest, mutest::ignore)]
      Some(body)
  }

• and function parameters:

  fn foo(&self, #[cfg_attr(mutest, mutest::ignore)] experimental: bool) {
  }

Mutation Safety

Rust’s notion of safety is reflected in mutest-rs through a unique safety property that mutations have. In mutest-rs, we distinguish between safe and unsafe mutations, based on the safety of scope that they are introduced in, and the safety flag used to invoke mutest-rs with (see Safety Flags).

To illustrate how we have to take safety into account when producing program mutations, let’s use two valid, but incorrect example programs. This first one uses the unsafe get_unchecked function to perform an unchecked array lookup into an array xs of length 3. The index i comes from safe code, but its value of 100 will trigger undefined behavior in the unsafe code following it. If we imagine that the original program did not have this rogue value, but instead we introduced this behavior with a program mutation instead, then we would have generated a mutation that introduced undefined behavior into this program. Thus, we can see that modifying safe code in the context of unsafe code may lead to the introduction of undefined behavior.

let xs = [0; 3];
let i = 100;
let el = unsafe { xs.get_unchecked(i) };

Similarly, in this second example, we see the same scenario play out but with the incorrect out-of-bounds index coming from a safe function called form within the unsafe scope. Thus, we can see that modifying the body of a safe function may lead to the introduction of undefined behavior if the function is called from unsafe code.

fn size() -> usize {
    100
}

let xs = [0; 3];
let el = unsafe {
    let i = size() - 1;
    xs.get_unchecked(i)
};

To ensure safety of the mutated program, mutest-rs does two things when analysing the program and generating mutations:

• treats these possibly unsafe contexts as unsafe, and
• propagates potential unsafety through the call graph.

While this ensures safety in all contexts, depending on your codebase and use of unsafe, this behavior might be too eager. In addition, evaluating these unsafe mutations might be extremely valuable for uncovering inadequacies in the testing of unsafe code. As such, mutest-rs provides safety flags that relax some of the rules around which mutations are deemed safe, which context are mutations generated in, and whether unsafe mutations are evaluated.

Safety Flags in mutest-rs

The following table describes how each of the safety flags (default: --safe) influences what contexts mutations get generated in, and what safety the mutations in that scope get assigned. In the table, the ^ symbol signifies the place of the mutation, and M refers to a mutation in that scope.

Batching Mutations

Mutation batching is a novel, experimental technique for increasing the parallelism of mutation analysis. By ensuring that mutations appear in “distinct” parts of the program, mutest-rs is able to combine them into one, while ensuring that

• mutations do not influence each other’s behavior, and
• mutation detections can still be distinguished from test case outputs.

By combining mutations into bigger batches, mutest-rs is able to evaluate more test cases in each iteration, making better use of the parallel processors available, and ultimately speeding up mutation evalaution.

Mutation batching is based on, and described in detail in the following research paper:

Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust
Zalán Lévai, Phil McMinn

Using Mutation Batching in mutest-rs

To enable mutation batching, first change the batch size limit (default: 1) to a higher value using the --mutant-batch-size argument. Otherwise, no mutation batching method will be able to create batches.

cargo mutest --mutant-batch-size 100 run

Mutation batching is performed by various algorithms before the mutations are compiled and evaluated. To pick a mutation batching algorithm, pass its name with the --mutant-batch-algorithm argument (default: none). Note, that various mutation batching algorithms may take their own, additional set of optional arguments, with the --mutant-batch-<ALGORITHM>- prefix. See --help for more details.

cargo mutest --mutant-batch-algorithm greedy --mutant-batch-size 100 run

If you experience crashes with mutation batching during evaluation, then it is most likely due to a conflict not accounted for with the current settings. The conflict relationships used in mutation batching are based on static call graph construction, which has a depth limit for practical reasons. To increase the depth limit of the call graph construction, and thus increase mutest-rs’s ability to find all conflict relationships, pass a higher value to the --call-graph-depth argument (default: 3).

cargo mutest --call-graph-depth 10 --mutant-batch-algorithm greedy --mutant-batch-size 100 run

Research

mutest-rs is used in active research by the Software Testing Group in the School of Computer Science at The University of Sheffield.

mutest-rs has been used in the following research works:

• Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust
  Zalán Lévai, Phil McMinn
  

Cite

If mutest-rs was a significant part of your work, please consider citing it in your publications. Specifically, please cite our paper describing the tool, titled “Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust” (DOI 10.1109/ICST57152.2023.00014).

Recommended BibTex Entry

@inproceedings{levai-batching-2023,
  title = {Batching {{Non-Conflicting Mutations}} for {{Efficient}}, {{Safe}}, {{Parallel Mutation Analysis}} in {{Rust}}},
  author = {L{\'e}vai, Zal{\'a}n and McMinn, Phil},
  booktitle = {2023 {{IEEE Conference}} on {{Software Testing}}, {{Verification}} and {{Validation}} ({{ICST}})},
  pages = {49--59},
  publisher = {IEEE},
  address = {New York, NY, USA},
  doi = {10.1109/ICST57152.2023.00014},
  url = {https://ieeexplore.ieee.org/abstract/document/10132214},
  year = {2023},
  month = apr,
}