Criterion benchmarks¶
Nextest supports running benchmarks in "test mode" with Criterion.rs.
What is test mode?¶
Many benchmarks depend on the system that's running them being quiescent. In other words, while benchmarks are being run there shouldn't be any other user or system activity. This can make benchmarks hard or even unsuitable to run in CI systems like GitHub Actions, where the capabilities of individual runners vary or are too noisy to produce useful results.
However, it can still be good to verify in CI that benchmarks compile correctly, and don't panic when run. To support this use case, libraries like Criterion support running benchmarks in "test mode".
For criterion and nextest, benchmarks are run with the following settings:
- With the
testCargo profile. This is typically the same as thedevprofile, and can be overridden with--cargo-profile. - With one iteration of the benchmark.
Requirements¶
- Criterion 0.5.0 or above; previous versions are not compatible with nextest.
- Any recent version of cargo-nextest.
Running benchmarks¶
By default, cargo nextest run does not include benchmarks as part of the test run. (This matches cargo test.)
To include benchmarks in your test run, use cargo nextest run --all-targets.
This will produce output that looks like:
% cargo nextest run --all-targets
Finished test [unoptimized + debuginfo] target(s) in 0.05s
Starting 7 tests across 1 binaries
PASS [ 0.368s] my-benchmarks::bench/my_bench depends_on_cache
PASS [ 0.404s] my-benchmarks::bench/my_bench depends_on
PASS [ 0.443s] my-benchmarks::bench/my_bench into_ids
PASS [ 0.520s] my-benchmarks::bench/my_bench make_graph
PASS [ 0.546s] my-benchmarks::bench/my_bench resolve_package
PASS [ 0.588s] my-benchmarks::bench/my_bench make_cycles
PASS [ 0.625s] my-benchmarks::bench/my_bench make_package_name
------------
Summary [ 0.626s] 7 tests run: 7 passed, 0 skipped
To run just benchmarks in test mode, use cargo nextest run --benches.