Compare commits

...
Author SHA1 Message Date
Hui Xiao 9db54a4ba7 CPU corruption injector: robust seed handling, per-run command logging, and doc fixes (#14940)
Summary:

A few improvements to the CPU corruption injector:

- Seed handling: `--seed` is now validated against db_stress's accepted range and no longer silently wraps, so an out-of-range seed fails fast instead of producing misleading NO_INJECTION runs. Omit `--seed` for a fresh random base seed.
- Per-run reproducibility: each run's full db_stress command is recorded at the top of its `gdb.log`, so any run can be reproduced by copy-pasting one line.
- README: clarified the `--seed` semantics and examples.


**Test plan:**

All db_stress runs use a debug build: `make DEBUG_LEVEL=0 EXTRA_CXXFLAGS="-g -fno-inline" db_stress`.

## Seed handling: out-of-range seeds fail fast

Command:

```
python3 tools/cpu_corruption_injector/runner.py --op write --runs 2 --stress_cmd ./db_stress --report_dir /tmp/tp_badseed --seed 4294967295; echo "exit=$?"
python3 tools/cpu_corruption_injector/runner.py --op write --runs 1 --stress_cmd ./db_stress --report_dir /tmp/tp_badseed0 --seed 0; echo "exit=$?"
```

Output:

```
2026-07-15 01:26:14,408 ERROR --seed range [4294967295, 4294967296] is outside db_stress's accepted [1, 4294967295]; pass a smaller --seed
exit=2
2026-07-15 01:26:14,558 ERROR --seed range [0, 0] is outside db_stress's accepted [1, 4294967295]; pass a smaller --seed
exit=2
```

Both are rejected before any run starts (previously the seed silently wrapped and every run came back a misleading NO_INJECTION).

## Per-run command logged (full db_stress command in gdb.log)

Command:

```
python3 tools/cpu_corruption_injector/runner.py --op write --runs 2 --stress_cmd ./db_stress --report_dir /tmp/tp_seed --seed 424242
head -6 /tmp/tp_seed/run_00000/gdb.log
```

Output (the first line is the full command; the run's gdb/db_stress output follows it):

```
# /usr/bin/gdb --batch --nx -iex 'py import sys; sys.argv=["injector.py", "--op", "write", "--op_index", "259", "--entry_fn", "rocksdb::MemTable::Add", "--target_fn", "rocksdb::MemTable::Add", "--seed", "424242", "--dir", "/tmp/tp_seed/run_00000"]' -x /data/users/huixiao/fbsource/fbcode/internal_repo_rocksdb/repo/tools/cpu_corruption_injector/injector.py --args /data/users/huixiao/fbsource/fbcode/internal_repo_rocksdb/repo/db_stress --db=/tmp/tp_seed/run_00000/db --expected_values_dir=/tmp/tp_seed/run_00000/exp --seed=424242 --verify_cpu_corruption_dir=/tmp/tp_seed/run_00000 --ops_per_thread=1000 --max_key=1000 --value_size_mult=8 --column_families=1 --nooverwritepercent=0 --writepercent=50 --delpercent=40 --delrangepercent=10 --readpercent=0 --iterpercent=0 --prefixpercent=0 --flush_one_in=20 --compact_range_one_in=10 --compact_files_one_in=10 --target_file_size_base=65536 --reopen=0 --destroy_db_initially=1 --clear_column_family_one_in=0 --disable_wal=0 --threads=1 --num_dbs=1 --disable_auto_compactions=1 --write_buffer_size=1073741824 --remote_compaction_worker_threads=0 --subcompactions=1 --batch_protection_bytes_per_key=8 --memtable_protection_bytes_per_key=8 --block_protection_bytes_per_key=8 --paranoid_file_checks=1 --paranoid_memory_checks=1 --memtable_verify_per_key_checksum_on_seek=1 --detect_filter_construct_corruption=true --verify_checksum=1 --compression_checksum=1 --file_checksum_impl=crc32c --verify_checksum_one_in=1 --verify_file_checksums_one_in=1000 --read_fault_one_in=0 --write_fault_one_in=0 --metadata_read_fault_one_in=0 --metadata_write_fault_one_in=0 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=0 --secondary_cache_fault_one_in=0 --sync_fault_injection=0


This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.centos.org/>
Debuginfod has been disabled.
```

The first line of `gdb.log` is the complete `gdb` + `db_stress` command, including `--seed=424242` and `--max_key=1000`; the run's own output follows below it.

## The logged command reflects real flag values

Temporarily set `max_key` to `12345` in `runner_build.py`, then:

Command:

```
python3 tools/cpu_corruption_injector/runner.py --op write --runs 1 --stress_cmd ./db_stress --report_dir /tmp/tp_maxkey --seed 424242
head -1 /tmp/tp_maxkey/run_00000/gdb.log | tr ' ' '
' | grep -- --max_key
```

Output:

```
--max_key=12345
```

The logged command tracked the changed preset value. Reverted the change afterward.

Reviewed By: archang19

Differential Revision: D112075800
2026-07-21 07:08:05 -07:00
4 changed files with 51 additions and 14 deletions

No files matched your search

+12 -6
View File
@@ -49,10 +49,15 @@ bucket.
- `--op` is one of `write`, `flush`, `compaction`.
- `--parallel N` runs `N` runs at once (each `db_stress` run is itself
single-threaded, so raise this to use more cores).
- `--seed N` reproduces a set: run `i` uses seed `N+i`. With `--seed 0` (the
default) a fresh random `base_seed` is chosen and logged; pass that value back
via `--seed` to replay, and add `--runs 1` with `--seed=<base_seed+i>` to
replay a single run `i`.
- `--seed N` sets the campaign's `base_seed`: run `i` deterministically uses seed
`N+i`, which drives both the injection choices and `db_stress`'s own `--seed`, so
the whole set is reproducible from `base_seed` alone. Omit `--seed` to draw one
fresh random `base_seed` per campaign (chosen once per `runner.py` run and logged as `base_seed=N`); pass that `N` back via `--seed`
to replay the set, or `--seed=<N+i>` with `--runs 1` to replay just run `i`.
In the examples below, `<db_stress>` is the path to the `db_stress` binary you
built (see [Requirements](#requirements) above -- e.g. `./db_stress`), and
`<dir>` is a writable directory for the campaign's output.
```bash
# a compaction campaign of 100 runs
@@ -63,9 +68,10 @@ python3 tools/cpu_corruption_injector/runner.py --op compaction --runs 100 \
python3 tools/cpu_corruption_injector/runner.py --op flush --runs 200 \
--stress_cmd <db_stress> --report_dir <dir> --parallel 8
# reproduce an earlier set from its logged base_seed
# replay an earlier campaign: pass its logged base_seed back as --seed
# (the runner prints "reproduce with --seed=<base_seed>"); base_seed is in [1, 4294967295]
python3 tools/cpu_corruption_injector/runner.py --op write --runs 100 \
--stress_cmd <db_stress> --report_dir <dir> --seed 12345678
--stress_cmd <db_stress> --report_dir <dir> --seed 3141592653
```
Before doing any work, `runner.py` preflights the build: it confirms `gdb` can
+12 -7
View File
@@ -10,7 +10,6 @@ from __future__ import annotations
import argparse
import logging
import os
import random
import re
import subprocess
import sys
@@ -27,6 +26,7 @@ from runner_build import ( # noqa: E402
build_runs,
INJECTION_SITES,
REMOTE_COMPACTION_ENTRY_FN,
resolve_base_seed,
Run,
)
from runner_report import report # noqa: E402
@@ -39,8 +39,6 @@ def main(argv: list[str] | None = None) -> int:
report_dir = os.path.abspath(args.report_dir)
stress_cmd = os.path.abspath(args.stress_cmd)
parallel_runs = max(args.parallel, 1)
base_seed = args.seed or random.randint(1, 2**64) # 0 -> a fresh seed each run
os.makedirs(report_dir, exist_ok=True)
_setup_logging(report_dir)
@@ -48,6 +46,11 @@ def main(argv: list[str] | None = None) -> int:
if not os.path.exists(stress_cmd):
logger.error("stress_cmd not found at %s", stress_cmd)
return 2
try:
base_seed = resolve_base_seed(args.seed, args.runs)
except ValueError as e:
logger.error("%s", e)
return 2
try:
verify_injection_site(stress_cmd, args.op)
except RuntimeError as e:
@@ -109,10 +112,12 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser.add_argument(
"--seed",
type=int,
default=0,
help="master seed for the run set. 0 (default) picks a fresh random seed each "
"invocation (logged as base_seed=N); pass that N back via --seed to reproduce "
"the same runs. Run i uses seed N+i.",
default=None,
help="base_seed for the run set: run i uses seed base_seed+i (which drives both "
"the injection choices and db_stress's --seed), so the whole set is reproducible "
"from base_seed alone. Omit to draw one fresh random base_seed per campaign (once "
"per runner run; logged as base_seed=N); pass that N back "
"via --seed to replay the set, or --seed=<N+i> --runs 1 to replay just run i.",
)
parser.add_argument(
"--randomize_stress_flags",
+24 -1
View File
@@ -55,6 +55,11 @@ INJECTION_SITES: dict[str, InjectionSite] = {
# CompactionJob::Run entry_fn (see build_runs).
REMOTE_COMPACTION_ENTRY_FN = "rocksdb::CompactionServiceCompactionJob::Run"
# Largest value db_stress accepts for its --seed flag: it validates --seed against
# UINT32_MAX and refuses to start on anything larger, so every seed we pick or accept
# must stay within this bound.
_DB_STRESS_SEED_MAX = 2**32 - 1
# Small SDC-safe domain for the randomized max_key. db_crashtest's own range is far too
# large -- a single injected flip in a huge keyspace rarely lands on a multi-version key,
# washing out the SDC signal. Every choice here still produces a multi-version layout.
@@ -186,6 +191,24 @@ class Run:
inject_json: Optional[dict[str, object]] = None
def resolve_base_seed(requested_seed: int | None, run_count: int) -> int:
"""Pick the base seed for a run set and ensure the whole set fits db_stress's
accepted seed range. requested_seed None means "draw a fresh random one". Run i uses
base_seed + i, so the highest seed is base_seed + run_count - 1; that must not exceed
_DB_STRESS_SEED_MAX, or db_stress refuses to start and every run comes back a
misleading NO_INJECTION. Raises ValueError if an explicit --seed is out of range."""
if requested_seed is None:
# Draw a fresh base seed low enough that the whole run set stays in range.
return random.randint(1, _DB_STRESS_SEED_MAX - (run_count - 1))
highest_seed = requested_seed + run_count - 1
if requested_seed < 1 or highest_seed > _DB_STRESS_SEED_MAX:
raise ValueError(
f"--seed range [{requested_seed}, {highest_seed}] is outside db_stress's "
f"accepted [1, {_DB_STRESS_SEED_MAX}]; pass a smaller --seed"
)
return requested_seed
def build_runs(
op: str,
run_count: int,
@@ -202,7 +225,7 @@ def build_runs(
op_index_max = max(op_count_estimate(op) // 3, 1)
runs: list[Run] = []
for index in range(run_count):
seed = (base_seed + index) % 2**64
seed = base_seed + index
rng = random.Random(seed)
op_index = rng.randint(1, op_index_max)
target_fn = rng.choice(site.target_fns)
@@ -10,6 +10,7 @@ from __future__ import annotations
import json
import logging
import os
import shlex
import signal
import subprocess
@@ -147,6 +148,8 @@ def _injector_argv(run: Run) -> list[str]:
def _run_gdb(command: list[str], log_path: str) -> None:
with open(log_path, "wb") as log_file:
log_file.write(("# " + shlex.join(command) + "\n\n").encode("utf-8"))
log_file.flush()
proc = subprocess.Popen(
command,
stdout=log_file,