Compare commits

..
117 Commits
Author SHA1 Message Date
meta-rocksdb-release-automation[bot] ef3dd12e9a Update HISTORY.md for 11.12.0 release 2026-10-09 16:11:25 +00:00
Ranjan Banerjee dec67990f0 Add db_stress coverage for WriteBufferManager flush policies (#15048)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15048

Adds a validated db_stress flag for selecting the WriteBufferManager flush policy and randomizes it in db_crashtest. Crash-test generation covers all three policies; kFlushLargestAcrossDBs automatically uses two DBs sharing one manager so the cross-DB path is exercised.

Reviewed By: pdillinger

Differential Revision: D114638570

fbshipit-source-id: 00f664445614e083a4c59d4e2bb0896731d4bea7
2026-10-07 16:30:03 -07:00
Ranjan Banerjee 9b4c5422f0 Avoid WBM pressure delay after fast rejection (#15343)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15343

Problem

The cross-DB WriteBufferManager coordinator dispatches ranked flush candidates asynchronously. A candidate can reject its handoff immediately—for example because its DB mutex is busy, writes are stopped, or equivalent WBM work is already scheduled—and finish before the sorter reaches its pressure wait.

Before this change, completion called RequestRefresh(), but the sorter’s active-pressure branch did not consume that already-recorded notification. Condition-variable notifications are not retained, so if completion won the race, the sorter still slept for the full 10 ms pressure interval before trying the next ranked DB. With many quickly rejected candidates, that delay accumulated once per candidate; 10,000 rejected DBs ahead of a healthy DB could theoretically delay useful relief by roughly 100 seconds while shared memory remained above its limit.

Solution

Snapshot the existing refresh-request generation immediately before ProcessFlushHandoffRequest(). After processing, if the handoff is already idle, the current ranked candidate cycle remains active, and the generation advanced, the sorter knows that asynchronous work completed during the dispatch window. It consumes that persistent notification and immediately continues to the next candidate instead of entering the pressure backoff.

The generation check is deliberately narrower than checking the refresh Boolean alone: it reacts only to notifications that occurred during this handoff attempt and avoids treating an older unrelated refresh as completion of the current candidate. The existing 10 ms backoff remains after a complete no-progress candidate cycle, and queued or executing handoffs retain their existing lease/wait behavior.

Testing

A deterministic SyncPoint test pauses the sorter only after a handoff has actually been submitted, lets a rejecting executor callback finish first, and verifies that the healthy next candidate is reached without entering the pressure-wait point. The synchronization wait is bounded so a broken ordering fails the test rather than hanging CI.

Reviewed By: xingbowang

Differential Revision: D123926506

fbshipit-source-id: b057e5b9a41ee0abd415f72ef2067b0d3f718384
2026-10-07 16:03:20 -07:00
Davis Rollman 7292adcc9a AutoHeaders: migrate to explicit headers (batch 116)
Summary:
Finish OpenZL, the internal RocksDB source tree, and the released RocksDB copy, removing all three parent AutoHeaders opt-ins. Declare inferred headers while preserving aliases, globs, selectors, and visibility. RocksDB's generator lists library headers and passes test source headers through its template and shared wrapper. Header lists account for the source files compiled by each release, so the older released tree does not gain headers from newer internal test helpers.

Add `release_to_fbcode.py internal_repo_rocksdb --buck-only -y` to refresh the four Buck generator files and regenerate BUCK against the existing release's source lists. The command restores its five outputs if copying or generation fails, applies the existing generated-file warning, and rejects release-commit options. It does not copy runtime sources, change the release version, clone, commit, or deploy. Both RocksDB BUCK files are produced by the maintained generators.

Graft the OpenZL dev declarations into prod. Align the predictor's Folly include paths and dependencies with prod so promotion is reproducible. Add the IPNext controller transition header explicitly.

Authored with Codex.

Reviewed By: IanChilds

Differential Revision: D123482073

fbshipit-source-id: 05ab47e592d710834da56bbea57f5f93a0aea565
2026-10-07 15:43:34 -07:00
Peter Dillinger 6b15f1643e Give nightly-candidate workflow a distinct name (#15345)
Summary:
The workflow_dispatch-only nightly-candidate.yml shared the exact `name: facebook/rocksdb/nightly` with the scheduled nightly.yml. The name collision confuses GitHub's "list workflow runs by file name" REST query, so release tooling querying actions/workflows/nightly.yml/runs got a stale/partial set missing recent scheduled runs. Rename it to facebook/rocksdb/nightly-candidate, matching the pr-jobs-candidate convention, so every workflow name is unique.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15345

Test Plan: rety the integration after fix

Reviewed By: xingbowang

Differential Revision: D123973936

Pulled By: pdillinger

fbshipit-source-id: cde1c238e6a91444f7112848081e161f54f5bf24
2026-10-07 15:35:08 -07:00
Ranjan Banerjee a38cd3a439 Bypass stalled cross-DB WBM flush candidates (#15331)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15331

## Motivation

The parent diff already implements cross-DB flushable-memory accounting and a background sorter that ranks DBs outside the write path. This change does not alter how the best candidate is selected.

The remaining liveness problem occurs after the sorter selects a DB and hands it off for execution. Previously, the coordinator tracked one remote handoff as pending and waited for that DB to report completion before advancing. A selected DB could therefore monopolize coordinator progress if its job waited in Env::LOW, blocked on the DB mutex, waited behind an existing write leader or write stall, or otherwise failed to complete promptly. Other ranked DBs could be ready to reclaim memory but would not be attempted while the first handoff remained pending.

## Solution

Make cross-DB flush handoff execution bounded and bypassable.

Cross-DB attempts now run on a small four-worker executor owned by the WriteBufferManager instead of consuming Env::LOW compaction capacity. The executor is created only for kFlushLargestAcrossDBs and its workers are retired when that policy is disabled.

The coordinator distinguishes queued and executing handoffs. A candidate lease begins when an executor worker starts the callback, rather than while the callback is merely waiting in the executor queue. If an executing handoff exceeds its lease, the coordinator releases ownership of that attempt and advances to the next ranked candidate. The original callback is not cancelled and may still complete, so the executor and outstanding-handoff count provide a strict bound on concurrent stuck work.

Each registration has a per-DB outstanding-job reservation, preventing the sorter from dispatching the same DB again while an earlier callback remains alive. The coordinator also records the identity of the active candidate, so late completion from an expired attempt cannot clear or corrupt a newer candidate lease.

DB mutex acquisition, write-thread joining, memtable switching, and flush dispatch continue to happen outside registry and coordinator locks. The executor invokes the existing BackgroundCallWBMFlush path directly; actual immutable-memtable flushing still uses RocksDB normal flush infrastructure.

## Compatibility

This behavior is limited to kFlushLargestAcrossDBs. Existing policies retain their previous local flush behavior, and no WBM executor threads are started for them.

## Validation

Unit coverage exercises lease expiry, advancement to the next candidate, late completion, duplicate suppression, bounded worker capacity, policy changes with blocked callbacks, shutdown, and reranking after progress.

The end-to-end stack was also compared against the original local policy in one matched memory-pressure run with 1,000 DBs and 16 writers. The complete largest-across-DBs policy delivered 61.0% higher throughput, 55.5% fewer WBM flushes, 37.9% lower mean latency, 47.9% lower P99 latency, 5.17x larger average flushes, and 62.8% lower peak RSS. These measurements validate the full policy stack rather than isolating the performance contribution of this liveness follow-up.

Reviewed By: xingbowang

Differential Revision: D123461329

fbshipit-source-id: 4ad86102c2a43beb469f7672d722674dfbda9871
2026-10-07 14:47:58 -07:00
Anand Ananthabhotla 45203a867f Add retryable ExternalTableBuilder contract (#15342)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15342

Add an ExternalTableBuilderBase::TryAdd() contract so external table builders can distinguish accepted rows, real builder errors, and clean requests to cut the current output table and retry the same row.

Add external-table-facing result, reason, and context types. Existing external table builders inherit a default TryAdd() implementation that forwards to Add(), so current implementations keep their historical behavior unless they explicitly opt into retryable file cuts.

Reviewed By: xingbowang

Differential Revision: D123930815

fbshipit-source-id: b5701c704938759db68b7c0bf0ebeba19d176945
2026-10-07 13:54:38 -07:00
Ranjan Banerjee 0561153b73 Add configurable largest-memtable flush policies to WriteBufferManager (#15047)
Summary:
Motivation

A WriteBufferManager can be shared by many RocksDB instances and column families. Under the existing policy, the DB whose writer first observes memory pressure flushes its own oldest mutable memtable. That DB might own only a small fraction of the reclaimable memory, so workloads with many DBs or CFs can generate many small flushes while the largest memory consumer remains untouched. Scanning every registered DB while holding process-wide or DB mutexes would improve selection accuracy but would serialize unrelated write paths and is not viable for deployments with thousands of DBs.

Previous behavior

kFlushOldest was the only behavior: the write-group leader that noticed WBM pressure synchronously selected the oldest eligible mutable memtable in its own DB. The WBM had no per-DB view of reclaimable memory and could not direct reclamation to another DB sharing the same budget.

New behavior

This change adds configurable WriteBufferFlushPolicy modes:

- kFlushOldest preserves the existing behavior and remains the default.
- kFlushLargest selects the largest eligible flushable column family in the writing DB.
- kFlushLargestAcrossDBs selects the DB advertising the most reclaimable memory across all DBs sharing the WBM, then flushes its largest eligible CF. Atomic-flush DBs advertise and flush the aggregate eligible memory across their CFs.

High-level implementation

Each writable DB registers a lightweight FlushInitiator with the shared WBM. Per-DB accounting publishes two approximate values using atomics: total mutable memory and the largest flushable CF contribution, including eligible immutable memtables waiting for flush. Allocator reporting is batched, so the allocation hot path avoids registry scans and DB mutexes.

A dedicated WBM coordinator thread maintains an approximate ranked snapshot of registered DBs. Refreshes are driven by accounting changes, threshold crossings, and periodic pressure checks. Ranking and O(number of DBs) work happen on that coordinator thread; writers only update counters and signal the coordinator. Registration lifetime is protected independently so DB close can safely drain an in-flight handoff.

Threading and flush handoff

The coordinator reserves and pins one ranked candidate at a time, without holding a registry lock while invoking the DB. The target DB uses TryLock on its DB mutex and rejects the handoff if it is closing, paused, stopped, stalled, already handling a WBM job, or no longer has flushable memory. Rejected candidates do not block consideration of the next DB.

An accepted handoff queues one LOW-priority WBM job. The LOW worker joins the target DB write thread and revalidates eligibility before switching the memtable. The resulting ordinary flush is then scheduled through RocksDB's existing flush machinery on the HIGH-priority flush pool. Keeping the write-thread join off the requesting writer avoids making one DB's write path wait on another DB's mutex or write queue.

Only one manager-wide remote handoff is active at a time. Completion or cancellation clears the handoff and requests a rerank. If memory reaches the hard limit while remote coordination is delayed, a rate-limited local fallback lets a writer seal a local memtable to preserve forward progress. Write stalls, shutdown, WaitForCompact, policy changes, and DB deregistration explicitly drain or reject outstanding handoffs.

Compatibility

The legacy constructor and kFlushOldest behavior remain unchanged unless a new policy is explicitly selected. There are no on-disk format changes.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15047

Reviewed By: xingbowang

Differential Revision: D112396106

fbshipit-source-id: 3049adc7034dd4e00d74d7d86e1d19004292d299
2026-10-06 16:26:51 -07:00
Peter Dillinger 1c490a4a4d Fix "ancester" spelling; deprecate TablePropertiesNames (#15335)
Summary:
Follow-up to https://github.com/facebook/rocksdb/issues/15304, which renamed `TableProperties::creation_time` to `oldest_ancestor_time` but left the long-standing "ancester" misspelling everywhere else, so the same concept was spelled two ways (e.g. `TableProperties::oldest_ancestor_time` vs. `FileMetaData::oldest_ancester_time` and `SstFileMetaData::oldest_ancester_time`). This change makes the spelling consistent and addresses remaining review feedback on https://github.com/facebook/rocksdb/issues/15304.

Change:
* All internal identifiers and comments now use "ancestor": `FileMetaData::oldest_ancestor_time`, `CompactionServiceOutputFile::oldest_ancestor_time`, `kUnknownOldestAncestorTime`, `TryGetOldestAncestorTime()`, `Compaction::MinInputFileOldestAncestorTime()`, the `kOldestAncestorTime` manifest tag enum, and related locals and test sync point names.
* Public API: `SstFileMetaData::oldest_ancestor_time` is the new name; `oldest_ancester_time` remains as a deprecated alias sharing storage via an anonymous union (same approach as `TableProperties::creation_time`, so size, layout, and copy semantics are unchanged).
* Public API: `TablePropertiesNames::kOldestAncestorTime` is the new name; `kCreationTime` remains as a deprecated `static const std::string&` alias. A static reference member has no per-object cost and binds at constant initialization, so it adds no static-init-order exposure.
* `TablePropertiesNames` as a whole is marked DEPRECATED for public use. An fbsource-wide BigGrep found no references outside RocksDB and its vendored copies, and it is only meaningful to table format internals; users should read the corresponding `TableProperties` fields.

Fix:
* The `table_file_creation` event log entry had the `oldest_key_time` and `newest_key_time` values swapped since https://github.com/facebook/rocksdb/issues/13083 (v10.0). Now each label is paired with its own value.

Also: tools/c_api_gen/auto_simple_bindings.py resets the pending anonymous union field list after use and skips unnamed members inside the union (review suggestion); generated C API output is unchanged.

Compatibility: names that are persisted or externally parsed intentionally keep the old spelling: the on-disk property name `rocksdb.creation.time`, the `"oldest_ancester_time"` key in remote compaction serialization, and the `oldest_ancester_time` / `OldestAncesterTime` labels in `VersionEdit` DebugString/DebugJSON (ldb manifest_dump output). The manifest tag value is unchanged. HISTORY.md is left as historical record.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15335

Test Plan: DBTablePropertiesTest.CreationTimeAliasesOldestAncestorTime now also verifies `TablePropertiesNames::kCreationTime` and `kOldestAncestorTime` refer to the same object, and new DBTablePropertiesTest.SstFileMetaDataOldestAncesterTimeAlias verifies the two `SstFileMetaData` names share an address and that writes through the old name are visible through the new one. Remote compaction serialization through the unchanged `"oldest_ancester_time"` key is covered by compaction_service_test; manifest encode/decode through the renamed field by version_edit_test. `make check-c-api-gen` confirms the generated C API is up to date and backward compatible.

Reviewed By: jaykorean

Differential Revision: D123559299

Pulled By: pdillinger

fbshipit-source-id: 480106e8d539e654f31db210ef6e1de02325d604
2026-10-06 11:59:54 -07:00
generatedunixname3846135475516776 60f090647d Rocksdb Crash Test failed: Callstack - SharedState::SafeTerminate -> NonBatchedOpsStres (#15338)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15338

Reviewed By: jaykorean

Differential Revision: D122865369

fbshipit-source-id: 65a67955dd79a68fe1e4f15116d40dc669e1519e
2026-10-06 11:32:30 -07:00
Josh Kang 5b77e34985 Add range deletion support to external tables (#15334)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15334

External table formats cannot currently persist range tombstones, so SST ingestion and full-mode DB flushes or compactions fail when they contain `DeleteRange` entries. This change adds an opt-in contract for storing RocksDB’s canonical range-deletion block. RocksDB retains responsibility for sequence-number handling, fragmentation, and visibility through its native range-tombstone iterators.

Existing factories remain unsupported by default, and existing files without range tombstones are unaffected. Missing or invalid advertised tombstone blocks are treated as corruption. RocksDB also supplies logical entry counts and sizes when a format does not persist the properties block, while preserving format-specific properties.

Reviewed By: xingbowang

Differential Revision: D123475827

fbshipit-source-id: de8fec401bdc311282f5d4a48b248875c682f7c9
2026-10-05 18:39:42 -07:00
Andrew Chang 3ea1ec56df Fix MultiGet reading from disk with kBlockCacheTier (#15321)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15321

`MultiGet` with `read_tier = kBlockCacheTier` now returns `Incomplete` for data blocks that are not in the block cache. Before this fix, it read them from disk.

**Why it matters:** Callers use `kBlockCacheTier` to prevent disk I/O. `Get` obeyed it, but `MultiGet` did not.

**Problem**
- Without mmap reads, `MultiGet` sends cache misses to `RetrieveMultipleBlocks`.
- That path called `MultiRead` with no `kBlockCacheTier` check.
- `RetrieveBlock` (used by `Get`) has the check.

**Fix**
- `RetrieveMultipleBlocks`: if `read_tier == kBlockCacheTier`, set `Incomplete("no blocking io")` for each block and do no I/O.
- `MultiGet`: when a block is `Incomplete`, return that status for the key, the same as `Get`. Before, the key status stayed `OK` with no value.
- Release note in `unreleased_history/bug_fixes`.

**Tests**
- New `DBTest2.MultiGetBlockCacheTierSkipsColdSstData`: `Get` and `MultiGet` both return `Incomplete` for a key whose data block is not in the cache.
- `DBBlobBasicTest.MultiGetBlobsFromCache` relied on the bug. Its inline key passed only because `MultiGet` read the data block from disk. The test now loads the block cache first, so it still checks the blob cache.

Reviewed By: joshkang97

Differential Revision: D122431823

fbshipit-source-id: 262913930cb4b570ea0a4d8f4f92364de08a27f5
2026-10-05 15:13:28 -07:00
Jay Huh f16157cfc5 Fix ClipColumnFamily dropping the file that ends at begin_key (#15320)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15320

Why the change is required:
`DB::ClipColumnFamily(cf, begin_key, end_key)` keeps `[begin_key, end_key)`. It first drops whole non-L0 files outside the range with `DeleteFilesInRanges` on `(-inf, begin_key)` and `[end_key, +inf)`, but called it with the default `include_end = true`. The first range was therefore treated as `(-inf, begin_key]`, so a non-L0 file whose largest key is `begin_key` was deleted even though that key is inside the kept range. Clipping `[b, d)` on L1 files `[a, b]` and `[c, d]` lost `b`.

This showed up while smoke-testing `SplitDB` (D120757697): single-key files trivially moved to the bottom level made the split destination miss its first key.

New behavior:
- Pass `include_end = false`, so `DeleteFilesInRanges` skips files whose largest key equals `begin_key`. The second range has no end, so it is unaffected. The remaining keys below `begin_key` are still removed by the existing `DeleteRange` and compaction steps.

Reviewed By: xingbowang

Differential Revision: D123306376

fbshipit-source-id: 65cb36ae730a55750ea558a6cdcd48d8fc17f0e9
2026-10-05 12:48:29 -07:00
Peter Dillinger c83a84e98c Rename TableProperties::creation_time to oldest_ancestor_time (#15304)
Summary:
The `creation_time` table property has long stored the file's oldest ancestor time (the oldest key time among the file's ancestors, falling back to flush/compaction time), not the file's creation time, which is `file_creation_time`. A TODO in table_properties.h acknowledged the misleading name. This change starts the API migration: `TableProperties::oldest_ancestor_time` is the new name, and `creation_time` remains as a deprecated alias sharing storage via an anonymous union, so existing external reads and writes continue to compile and behave identically.

An anonymous union is used rather than a `uint64_t&` alias member. A reference member would delete TableProperties copy/move assignment, make the implicit copy constructor bind the copy's alias to the source object (dangling), permit writes through a const TableProperties, add 8 bytes, and break the uint64_t-range walk in TEST_SetRandomTableProperties. The union leaves size, layout, copying, constness, and offsetof unchanged. Reading the other union member is formally UB in ISO C++, but both members are uint64_t and this is supported in practice by GCC, Clang, and MSVC.

All in-repo uses of the old name now use `oldest_ancestor_time`. Names that are externally observable or persisted are intentionally unchanged:
* On-disk property name `rocksdb.creation.time` (`TablePropertiesNames::kCreationTime`)
* The `"creation_time"` key in TableProperties serialization (used by remote compaction)
* The event-log JSON key and `TableProperties::ToString()` label
* Java `TableProperties.getCreationTime()` and C `rocksdb_table_properties_creation_time()`

C API: the auto-binding generator (tools/c_api_gen/auto_simple_bindings.py) only considered direct FieldDecls and failed on the anonymous union's implicit unnamed field. It now treats anonymous union members as direct fields. Regenerating adds a new C getter
`rocksdb_table_properties_oldest_ancestor_time()` alongside the existing `rocksdb_table_properties_creation_time()`.

Follow-up: remove the deprecated `creation_time` alias in a future release.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15304

Test Plan: New unit test DBTablePropertiesTest.CreationTimeAliasesOldestAncestorTime verifies the two names share an address and that a write through the old name is visible through the new one. options_settable_test and compaction_service_test cover TableProperties serialization round trips through the unchanged "creation_time" key, now mapped to the renamed field.

Reviewed By: jaykorean

Differential Revision: D123079083

Pulled By: pdillinger

fbshipit-source-id: 18b07d91f303a8f6de58da0c730f80d8c371505c
2026-10-05 10:34:04 -07:00
Josh Kang c0512c4c34 Remove deprecated write_global_seqno ingestion support (#15287)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15287

`IngestExternalFileOptions::write_global_seqno` has been deprecated since RocksDB 8.0 and will soon be removed from the API. It requires ingestion to mutate SST files, sync them, regenerate checksums, and expose file-offset plumbing solely for that mutation.

Remove the option and its C, Java, and `ldb` APIs. Assigned sequence numbers now live only in the MANIFEST, leaving newly ingested SST bytes and whole-file checksums unchanged.

## Backward Compatibility

New readers continue to support existing SSTs containing a nonzero global sequence number written by legacy ingestion. Because that old write did not update the properties-block checksum, checksum verification retries against a temporary copy with only the global-seqno value restored to its original zero placeholder. The parsed sequence number and file contents remain unchanged.

This fallback only accepts mismatches explained by the legacy rewrite. Other checksum failures continue through normal corruption handling, and `verify_checksums=false` continues to bypass verification.

Reviewed By: xingbowang

Differential Revision: D122405856

fbshipit-source-id: 934e9bc755d58ea8d84e1c085e9769015bd8dfd7
2026-10-05 09:53:52 -07:00
Xingbo Wang 5f1132b884 Require DB mutex checks in AI code reviews (#15324)
Summary:
CONTEXT: The AI review workflow had generic concurrency and performance checks, but it did not require reviewers to determine whether changed code or transitive callees execute while DBImpl::mutex_ is held. Reviews could therefore miss added contention, blocking work, lifetime errors, or snapshot and lock-ordering violations.

PREVIOUS: Review agents could examine DB-mutex changes through general correctness and performance lenses, but there was no explicit call-path audit, before/after mutex-held work comparison, or evidence requirement for regressions.

NEW: Every non-trivial review must state whether the DB mutex is affected. When it is affected, reviewers must inspect lock boundaries and correctness invariants, compare CPU and I/O before and after the change, keep unavoidable locked work highly optimized, and require measurements for material regressions.

IMPLEMENTATION: Add a DB Mutex Review Perspective covering locking contracts, object lifetime, snapshot boundaries, lock ordering, hidden callee costs, blocking work, validation, and performance evidence. Assign its checks to the correctness and performance reviewers and add required context and review checklist entries.

SAFETY: This changes only the AI code-review instructions. It does not change RocksDB runtime behavior, APIs, file formats, or compatibility. Changes that cannot execute under the DB mutex only need to state that the mutex is not affected.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15324

Test Plan:
- `git show --check HEAD`
- `git diff --check upstream/main...2026_10_05_main`
- Documentation-only change; no runtime tests were run.

Reviewed By: joshkang97

Differential Revision: D123415306

Pulled By: xingbowang

fbshipit-source-id: 0f78e0b1d677edf003cd5dc6ed5ff8123aff4f98
2026-10-05 09:23:49 -07:00
Josh Kang c2b86f3ec0 Lazily allocate MultiGet coroutine scratch buffers (#15298)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15298

## Summary

BlockBasedTable::MultiGetCoroutine() embedded the synchronous 8 KiB scratch buffer in every coroutine frame, including cache-hit requests. The change uses zero coroutine stack capacity and allocates the required heap size only when I/O is needed, while preserving synchronous behavior.

## Benchmarks

| Variant | Coroutine frame allocation |
| Before | 21,048 bytes |
| After | 12,864 bytes |
| Change | -8,184 bytes (-38.9%) |

No API, format, or read-result changes.

Reviewed By: xingbowang

Differential Revision: D122810207

fbshipit-source-id: 003a8349b8da72de2eca2e62fb9f7b3d8802baed
2026-10-02 20:17:30 -07:00
Yoshinori Matsunobu 8585299552 Rate-limit DestroyDB cleanup of existing trash (#15285)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15285

Why the change is required:
MyRocks bulk-load v2 creates a temporary RocksDB and calls DestroyDB when the load is committed, rolled back, abandoned, or recovered. Successfully ingested SSTs are moved into the main DB, so leftover SST cleanup is primarily an error, abort, or recovery case; WALs, obsolete files, and compaction-created trash can still remain. All physical deletion performed by DestroyDB should honor the configured SstFileManager rate, including rocksdb_sst_mgr_rate_bytes_per_sec in MyRocks.

Previous behavior:
DestroyDB sent recognized table, blob, and WAL files through SstFileManager, but ignored filenames ending in .trash because ParseFileName rejects that suffix. The final best-effort DeleteDir then silently left the directory behind. DB::Open handled the same files through CleanupDirectory, forcing MyRocks cleanup code to discover and register trash itself. A global WaitForEmptyTrash workaround would also drain unrelated compaction, DROP TABLE, and DROP INDEX deletion work from every DB sharing the manager.

New behavior:
DestroyDB waits only for pending deletion work associated with each DB, data, WAL, and archive directory. It then schedules any remaining .trash file through the existing rate-limited scheduler and waits only for the private bucket created for this DestroyDB call. Directory keys are normalized for bookkeeping, so equivalent path spellings identify the same pending work while filesystem operations retain the original path.

Accounting and failure handling:
A failed accounted trash deletion retains its original file path and remaining trash-byte charge. If DB open or DestroyDB later discovers the file, the retry preserves the accounted classification, avoids charging the trash bytes twice, and clears SstFileManager tracked-size and trash-size accounting after successful deletion.

Scope and safety:
There is no new sysvar, public RocksDB API, or persisted-format change. Existing rate settings, hard-link handling, partial deletion, and separate-WAL foreground deletion remain unchanged. The bookkeeping is confined to file enqueue/completion and DestroyDB cleanup. A shared FIFO can still delay this DB behind unrelated work already ahead of its files, but unrelated work queued behind this DB no longer gates DestroyDB.

Reviewed By: xingbowang

Differential Revision: D122221503

fbshipit-source-id: ec106f57efa70e24827234b4173366e5dbcbda5c
2026-10-02 16:21:19 -07:00
Xingbo Wang 866e8e9f06 Make full error-recovery flush atomic (#15293)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15293

Why the change is required:
Full background-error recovery flushes all column families because WAL durability can be uncertain. A later recovery-flush failure or process crash must not persist different WAL replay boundaries for different column families. Otherwise reopen can recover only part of a write batch spanning those column families.

The risk includes non-file-scoped WAL write or sync errors. A failed write can have reached the WAL without reaching the memtables. An earlier unsynced write can instead remain in the memtables while its WAL bytes are not durable; retrying the fsync after resetting the writer error does not by itself prove those bytes survived. Sequential recovery can make either case visible in only some column families after reopen.

Previous behavior:
File-scoped WAL-error recovery already set FlushOptions::force_atomic_flush. Other full kErrorRecovery paths flushed column families sequentially when DBOptions::atomic_flush was false. Each successful per-column-family flush independently advanced its MANIFEST log number, so a later flush failure or process crash could leave mismatched replay boundaries.

New behavior:
Every full kErrorRecovery atomically commits the flush results and WAL replay boundaries for all participating column families. A successful recovery commits every column family; a failed recovery commits none. If an atomic recovery attempt fails, all of its flush output is discarded and recreated by a later attempt instead of retaining output from column families that finished first.

Implementation:
Set FlushOptions::force_atomic_flush for every full error-recovery FlushAllColumnFamilies call. Remove the now-unreachable recovery-only behavior from FlushMemTable and assert that recovery does not enter the non-atomic path. Keep the two-write-queue sequence fence at the atomic memtable-switch point.

Add an end-to-end fault-injection test for a retryable, non-file-scoped WAL sync error after a multi-column-family record is appended. It covers successful and failed recovery flushes, including a column family with no other memtable data, and verifies all-or-none behavior after reopen. Update the manual_wal_flush comment and add a release note.

Safety, compatibility, and limitations:
Only full kErrorRecovery changes. Normal, manual, shutdown, and kErrorRecoveryRetryFlush behavior is unchanged. File-scoped WAL-error recovery was already atomic. There are no API, ABI, option-default, or file-format changes.

The additional work is limited to the cold recovery path. A failed atomic recovery can redo more flush work than the previous sequential path, but this avoids partial multi-column-family recovery. Automatic recovery with manual_wal_flush remains disabled as fatal pending separate analysis and testing; this change does not remove that workaround. Validation covers local Linux builds and focused recovery tests; platform CI for this version is pending.

Reviewed By: pdillinger, rban1

Differential Revision: D122469648

fbshipit-source-id: f72845903d9b6bff70f5d881e02a4cb2e920ed72
2026-10-02 14:43:36 -07:00
Xingbo Wang 19b6f85266 Support async file open for SecondaryDB (#15294)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15294

Why the change is required:
`open_files_async` reduces startup latency by moving SST validation and table-reader opening to background work, but `DB::OpenAsSecondary()` still performed that work synchronously.

Previous behavior:
A secondary using `open_files_async=true` replayed its MANIFEST through `ReactiveVersionSet`, synchronously verified and opened every current SST, and only then returned. The option therefore provided no startup benefit for secondary instances.

New behavior:
`DB::OpenAsSecondary()` can now return after installing its initial read view and start initial SST validation and table-reader pinning through the existing asynchronous opener. With `max_open_files=-1`, the opener processes all current SSTs in the background; a finite table cache can stop the eager pass at capacity and validate remaining files on demand. Later `TryCatchUpWithPrimary()` rounds remain synchronous. Async-open failures notify listeners and are latched according to the existing severity and `paranoid_checks` policy. If an error is not latched, a later read retries the file access and reports a persistent failure. A latched error requires fixing the underlying problem and reopening the secondary. Read-only instances do not start write-oriented recovery.

Implementation:
Initial reactive MANIFEST replay can defer SST verification and handler loading. The deferral is immutable constructor state on `ReactiveVersionSet`, is cleared before catch-up mode, and is restricted to normal secondary recovery. `VersionBuilder::GetAndClearIntermediateFiles()` now moves out and clears its own historical add-then-delete bookkeeping so callers cannot accidentally retain stale state. Secondary open schedules the background opener after publishing the read view. Its context pins the captured Versions and column families, including retained secondary handles whose column family was subsequently dropped by the primary. Existing table-reader opening performs SST size and unique-ID validation, avoiding a separate serial metadata pass. Closing the DB or canceling background work safely cancels a queued opener or waits for a running opener. Read-only async-open errors use an ErrorHandler path that notifies listeners without automatic recovery. Secondary async open explicitly rejects FIFO compaction.

Safety, compatibility, and limitations:
The default `open_files_async=false` path and existing primary async-open behavior are unchanged. WAL and blob recovery, as well as later secondary catch-up, remain synchronous. Follower DB and `OpenAndCompact()` do not use this deferral. There are no file-format, ABI, or public API signature changes. Secondary async open requires `skip_stats_update_on_db_open=true` and does not support FIFO compaction. Applications must keep files accessible by coordinating deletion with the primary or by using an Env/FileSystem that delays physical deletion until every instance releases a file. With asynchronous opening, `max_open_files=-1` cannot guarantee every SST is pinned before `OpenAsSecondary()` returns; the table-files-only `max_open_files=-1` workaround applies only when asynchronous opening is disabled. No completion API is added.

Validation covers initial deferral and pinning, synchronous catch-up, table-reader detection of physical-size mismatch, FIFO rejection, listener-adjusted errors and synchronous read retry, retained handles after primary column-family deletion, close and explicit cancellation with no HIGH-priority worker, stopping new table opens during shutdown, primary async open, follower isolation, and remote compaction. Linux Buck builds and focused tests passed. Cross-platform CI is pending, and `make check-sources` cannot run successfully in this Sapling checkout because its script requires Git.

Reviewed By: pdillinger

Differential Revision: D122417547

fbshipit-source-id: 91d408469c35c4ea21165c5e7db75068e65c2389
2026-10-02 12:21:46 -07:00
xingbowang 4cd0450f31 Fix big-endian builds for the fault-injection filesystem (#15297)
Summary:
The persistent injected-error log writes `InjectedErrorLog::Entry` objects using their native in-memory layout, while the log format's consumer expects little-endian multi-byte fields. The existing `static_assert(port::kLittleEndian)` protected that format by rejecting every big-endian build that included the fault-injection filesystem, even though the log is an optional diagnostic.

This change keeps RocksDB and the fault-injection filesystem available on supported big-endian platforms while disabling only the incompatible persistent log. Log-file creation now returns an inactive descriptor when `port::kLittleEndian` is false, which makes `Record()` and `Flush()` remain safe no-ops. The fixed 256-byte entry-size assertion remains in place, and format-dependent tests are bypassed on big-endian targets.

The change also adds a release note and documents big-endian support plus the preferred capability-check pattern for optional architecture-specific features.

## Compatibility and limitations

- Little-endian behavior and the on-disk log layout are unchanged.
- On big-endian platforms, configuring an injected-error log path does not create or populate a log file; the rest of fault injection remains available.
- Windows already leaves this POSIX-backed diagnostic disabled.
- This does not introduce a portable log encoding. Supporting persistent injected-error logs on big-endian platforms would require a format change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15297

Test Plan:
- `build_tools/rockstest.sh fault_injection_fs_test '--gtest_filter=InjectedErrorLogTest.*'` (little-endian macOS): all 5 tests passed.
- `make check-sources`
- `git diff --check upstream/main...2026_10_01_endian_fix`

Native big-endian build and test execution were not available locally; that path relies on the compile-time `port::kLittleEndian` capability and should be covered by big-endian CI.

Reviewed By: jaykorean

Differential Revision: D122880760

Pulled By: xingbowang

fbshipit-source-id: 59dae9661a47608aa12f41078ad1b3ef0fa7231e
2026-10-01 15:15:13 -07:00
Xingbo Wang b58cbcf13c Apply transaction visibility to entity reads (#15269)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15269

Why the change is required:
Transaction entity reads must enforce the same commit visibility and read-your-own-write rules as value-only reads. Without the transaction-specific read callback, GetEntity-family APIs can return a value that is not visible to the caller.

Previous behavior:
TransactionBaseImpl handled GetEntity and MultiGetEntity through non-virtual helpers that called WriteBatchWithIndex without a ReadCallback. WritePrepared transactions could expose a prepared but uncommitted update. WriteUnprepared transactions could miss their own write after it had been flushed as an unprepared batch, although that write remained hidden from other transactions.

New behavior:
GetEntity, MultiGetEntity, and GetEntityForUpdate now follow the same snapshot visibility and read-your-own-write semantics as Get and MultiGet for WritePrepared and WriteUnprepared transactions.

Implementation:
Make the internal entity-read helper hooks virtual. Override them in WritePreparedTxn and WriteUnpreparedTxn to construct the existing policy-specific ReadCallback and pass it through WriteBatchWithIndex. The WritePrepared batched path validates the callback refreshed read sequence so reads without an explicit snapshot do not bypass eviction validation. Invalid callback or snapshot state returns TryAgain. Add shared transaction entity-read assertions and parameterized regression coverage for prepared visibility and unprepared read-your-own-write.

Safety boundaries and unsupported cases:
There is no public API or persisted-format change. Other transaction policies keep the existing base implementation. The change adds only the callback and validation already used by value-only reads. Validation covers synchronous snapshot and non-snapshot entity reads, prepared visibility, and unprepared read-your-own-write in normal and checked-Status builds. Coroutine-specific and named wide-column payload cases were not separately tested, and the full repository suite was not run.

Reviewed By: rban1

Differential Revision: D121793013

fbshipit-source-id: e300642ed360b640814056e4d95cced5f5c180d8
2026-10-01 15:04:26 -07:00
Josh Kang 4d180247ae Add PreparedFileInfo serialization APIs (#15274)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15274

Expose `SerializePreparedFileInfo()` and `DeserializePreparedFileInfo()` so opaque metadata produced by SST writers or DBs can be persisted and transferred across process boundaries. The representation uses RocksDB `OptionTypeInfo` adapters for `InternalKey` and `TableProperties` and includes a CRC32C integrity check.

Deserialization now rejects malformed nested string-map entries, including missing key/value separators and invalid hex, with `InvalidArgument` instead of asserting in debug builds or accepting malformed input in release builds. Valid serialized data and the wire format are unchanged. The `PreparedFileInfo` type metadata is also initialized on first use to avoid throwable namespace-scope initialization.

Reviewed By: xingbowang

Differential Revision: D121897852

fbshipit-source-id: 59a3d221bfd20e6c257bd60477823a12d05a35c5
2026-10-01 13:45:31 -07:00
Jay Huh ded3cb72a5 Fix trace replay on the default column family and error reporting in multi-threaded replay (#15296)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15296

**Why:** trace replay (`DB::NewDefaultReplayer()`, db_bench `replay`) cannot replay a trace of a DB with only the default column family, and multi-threaded replay reports success when records fail. Together these make db_bench replay look successful while replaying no reads.

**Previous behavior:**
- `TraceExecutionHandler` resolves a record's column family ID only against the `handles` passed to `NewDefaultReplayer()`. Records on a column family missing from `handles`, including the default one, fail with `Corruption: Invalid Column Family ID.`. An empty `handles` also trips `assert(!handles.empty())` in debug builds.
- db_bench fills its handle list (`db_.cfh`) only when `--num_column_families` > 1, so with the default of 1 it passes an empty list. Every Get, MultiGet and iterator record fails. Write records are not affected, because `DB::Write()` resolves the column family from the batch itself.
- In multi-threaded `Replay()` (`num_threads` > 1), `BackgroundWork()` sends only decode errors to the error callback. Execution errors are dropped, so `Replay()` returns OK. Single-threaded replay stops and returns the error, including a NotSupported from execution (for example `DB::Write()` of a DeleteRange with `row_cache` set). It skips only records whose trace type it cannot decode.
- Result in db_bench on a single column family DB: `--trace_replay_threads=1` prints `Replay failed. Error: Corruption: Invalid Column Family ID.`. `--trace_replay_threads=4` prints `Replay completed` with `rocksdb.number.keys.read` = 0.

**New behavior:**
- The replay handler always includes the DB's default column family, so `handles` may be empty or leave it out. A handle the caller passes for a column family is still the one used.
- Multi-threaded `Replay()` reports execution errors through the same callback as decode errors. It stops scheduling new records and returns the error of the failed record with the earliest trace timestamp, as the existing code already does for decode errors. Get records that find no key still count as success, since the handler treats NotFound as OK.
- As in single-threaded replay, NotSupported is skipped only when decoding returns it (a trace type the replayer does not support). NotSupported from executing a record stops the replay.
- The scheduling loop no longer reads the shared status `bg_s` while worker threads write it. It already did so for decode errors; with execution errors reported, TSAN flags it in the new test (see Test Plan).

**Implementation:**
- `TraceExecutionHandler` constructor: drop the non-empty assert; after the caller's handles, insert `db->DefaultColumnFamily()`. `unordered_map::insert` does not overwrite, so a caller's handle wins.
- `TraceExecutionHandler` constructor: skip the default column family if `DefaultColumnFamily()` returns nullptr (some test `DB` mocks do), so such DBs behave as before.
- `ReplayerImpl::BackgroundWork()`: call `error_cb` when `Accept()` returns an error. On a decode error, skip `error_cb` for NotSupported, matching single-threaded replay.
- `ReplayerImpl::Replay()`: `error_cb` no longer filters out NotSupported (the decode path now does that); it still keeps the error of the earliest record.
- `ReplayerImpl::Replay()`: `error_cb` also sets an `std::atomic<bool> bg_failed` while holding `mtx`, and the scheduling loop checks that flag instead of `bg_s.ok()`. `bg_s` is read only after `WaitForJobsAndJoinAllThreads()`.
- Document both contracts in `db.h` (`NewDefaultReplayer`), `trace_record.h` (`NewExecutionHandler`) and `replayer.h` (`Replay`).
- Release notes in `unreleased_history/bug_fixes/`.

**Safety, compatibility, limitations:**
- No API or ABI change.
- Callers that already pass the default column family handle see no change from the first fix.
- Behavior change: multi-threaded replay callers that got OK while records failed now get the error, including NotSupported from execution. That matches single-threaded replay and the new `Replay()` comment.
- A record read by the scheduling loop while a failing worker is setting `bg_failed` may still be scheduled. This is allowed by the `Replay()` comment (records already scheduled still run) and does not change the returned status, which is that of the earliest failed record.
- Cost: one extra map entry per replayer, and one branch per record in threaded replay. The loop's `bg_s.ok()` read becomes a relaxed atomic load.
- Read records on column families that are neither passed nor the default still fail with Corruption, as before. Write records never used `handles` and are unchanged.

Reviewed By: xingbowang

Differential Revision: D122626982

fbshipit-source-id: 9a0973448710bac25233feedbaf264696ac85035
2026-10-01 11:43:44 -07:00
Josh Kang aeebcbded4 Add coroutine external table reads (#15291)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15291

Add experimental coroutine interfaces for implementing asynchronous basic and full `ExternalTableReader`s. Implementations opt in through `CoroExternalTableReaderBase<Mode>` by providing `GetCoroutine()` and may override `MultiGetCoroutine()`; the default implementation runs the individual Get operations concurrently. The external-table adapter uses these APIs when supported and otherwise retains the existing synchronous behavior.

As a smaller supporting fix, Single Get coroutine reads using non-mmap, non-direct I/O on a kFSBuffer-capable `FileSystem` now use `FSRandomAccessFile::SubmitReadAsync()` instead of falling back to synchronous `Read()`.

Reviewed By: xingbowang

Differential Revision: D122187491

fbshipit-source-id: 217e2953756938bbd3f30358a997a7ca3ed5543e
2026-10-01 09:28:24 -07:00
Xingbo Wang 11516577ab Pass blob file checksums through blob file opens (#15268)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15268

Why this is required: remote file systems need an immutable blob-file checksum and checksum-function name when RocksDB opens a finalized blob file. This lets the file system distinguish file generations and safely validate or key cached extents.

Previous behavior: integrated BlobDB opened normal read-path blob files with the column family's ordinary FileOptions. Although finalized checksum metadata was stored in BlobFileMetaData and BlobFileAddition, it was not propagated through point reads, range reads, MultiGet, retry opens, or recovery verification. Delayed reads of direct-write files could also cache a finalized file reader opened without metadata.

New behavior: every finalized integrated BlobDB file open receives its checksum value and checksum-function name through FileOptions. Active direct-write files continue to use empty checksum fields until finalization. Delayed reads pinned to a pre-flush SuperVersion use the finalized metadata once the file has been sealed.

Implementation: add internal BlobFileOpenInfo transport across Version, BlobSource, and BlobFileCache. Populate it from version metadata and recovery additions. At the actual cache miss/open, copy the base FileOptions and set both checksum fields. Direct-write sealing registers owned checksum metadata with BlobFileCache, atomically evicts footer-less readers, and prevents a racing pre-seal refresh from repopulating the cache without metadata. Metadata is released after write-path ownership and old-memtable protection end.

Safety boundaries: this does not change public APIs, persisted formats, or blob cache keys. Cache hits do not copy checksum strings. Files with no generated checksum, including active direct-write files, retain empty checksum fields. Existing footer-skip, corruption, and reader-refresh behavior remains in place. Integration in downstream file systems is outside this change.

Tests and known limitations: filesystem-capture tests cover regular blob reads and delayed direct-write reads. After rebasing onto the latest master, blob_file_cache_test, blob_source_test, db_blob_basic_test, and db_blob_direct_write_test passed both normally and with ASSERT_STATUS_CHECKED=1 (95 tests in each configuration). Fast arc lint passed. make format-auto and make check-sources could not operate because their scripts assume a Git checkout while this checkout uses Sapling; all files in the patch were formatted directly with clang-format.

Reviewed By: chaitanyadatye, anand1976

Differential Revision: D121630172

fbshipit-source-id: 601d1097e4a3caa60f0d9ba1794e78e81855980c
2026-09-30 14:55:26 -07:00
Jay Huh 9d358e63d6 Widen compaction service counts to full kNumOfReasons (#14924)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14924

Widen the serialized "counts" array in CompactionServiceResult from kNumOfReasons - 1 back to the full kNumOfReasons, so the count for the newest CompactionReason is included in the remote compaction result.

The width was previously reduced by one to match readers that expected the narrower array; restoring it changes what is written. This is safe only because deserialization of "counts" is now size-tolerant (a reader expecting fewer elements drops the extra, one expecting more zero-fills the missing). **It must not land until that size-tolerant read D106321320 has been deployed to every primary, including rollback targets, so should be >= 2 releases ahead; otherwise a primary can still or fallback to running the strict reader would reject the wider result and fail deserialization of this field.**

== Test plan ==
**Legend**

CompactionReason is an enum whose last entry is kNumOfReasons (currently 21). CompactionStats::counts is an int[kNumOfReasons] holding one count per reason; it is serialized into the remote compaction result and read back by the primary.
- "narrow" = kNumOfReasons - 1 counts (omits the newest reason); what current code writes.
- "wide" = kNumOfReasons counts (includes the newest reason).
- strict reader: accepts only its exact expected element count; any other count fails deserialization (the plain OptionTypeInfo::Array behavior).
- tolerant reader: accepts any count -- more elements than expected are dropped, fewer are zero-filled; never fails on a count mismatch (this change, SizeTolerantDeserializeArray).

**Unit test**

CompactionServiceTest.CompatCheckCountsWidthTolerance -- verifies that deserializing "counts" tolerates a serialized element count equal to, one greater than, or one less than the reader's expected width (extras dropped, missing zero-filled), and that the binary serializes exactly the expected number of elements.

**Manual cross-version verification (ldb remote_compaction)**

Test 1: expect-narrow tolerant-read primary reads wide counts a worker write (extra reason dropped)
```
$ tolerant_wide remote_compaction_worker  --db=$DB --job_dir=$JOB &
$ tolerant_narrow remote_compaction_primary --db=$DB --job_dir=$JOB
remote_compaction_worker: OK (4915 bytes result)
remote_compaction_primary: OK
$ grep 'remote compaction result' $DB/LOG
[default] [JOB 3] Received remote compaction result, output path: /tmp/cs_run/output, files: 000022.sst
```
=> PASS: the primary parsed and installed the remote result.

Test 2: expect-wide tolerant-read primary reads narrow counts a worker writes (missing reason zero-filled)
```
$ strict_narrow remote_compaction_worker  --db=$DB --job_dir=$JOB &
$ tolerant_wide remote_compaction_primary --db=$DB --job_dir=$JOB
remote_compaction_worker: OK (4911 bytes result)
remote_compaction_primary: OK
$ grep 'remote compaction result' $DB/LOG
[default] [JOB 3] Received remote compaction result, output path: /tmp/cs_run/output, files: 000022.sst
```
=> PASS: the primary parsed and installed the remote result.

Test 3: expect-wide tolerant-read primary reads wide counts a worker writes
```
$ tolerant_wide remote_compaction_worker  --db=$DB --job_dir=$JOB &
$ tolerant_wide remote_compaction_primary --db=$DB --job_dir=$JOB
remote_compaction_worker: OK (4917 bytes result)
remote_compaction_primary: OK
$ grep 'remote compaction result' $DB/LOG
[default] [JOB 3] Received remote compaction result, output path: /tmp/cs_run/output, files: 000022.sst
```
=> PASS: the primary parsed and installed the remote result.

Reviewed By: hx235

Differential Revision: D111068809

fbshipit-source-id: d78dd4b7501cb74e60bcb27dccd869b73052ac6d
2026-09-30 12:39:00 -07:00
Xingbo Wang 31675ca6d0 Recover from poisoned WAL handles on remote file systems (#15193)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15193

Keep `DB::Write`'s existing immediate `IOError` contract while allowing RocksDB to recover safely when a remote file system reports that a WAL handle is permanently unusable. The public API documentation makes the persisted-state boundary, reopen behavior, and safe post-resumption retry point explicit. Normal operation remains unchanged; the additional protocol is entered only after a supported file-scoped WAL error.

### Motivation

A WAL write crosses an acknowledgement boundary: storage can accept the complete record and then fail before the caller receives confirmation. The resulting `IOError` therefore cannot tell RocksDB or the application whether the record is absent, partial, or complete. At the same time, the remote file handle can become permanently poisoned, so retrying that same handle can never succeed.

The previous recovery path retried the poisoned handle. Simply replacing it is not sufficient: RocksDB must also prevent reuse of an ambiguous sequence range, keep every column family on one durable boundary, and prevent WAL tailers from publishing a physically complete record that successful live resumption deliberately abandoned. The new protocol quarantines the affected WALs, reserves the ambiguous sequence range, creates a replacement WAL, atomically persists acknowledged state, retires the quarantined handles, and only then resumes writes.

### Behavior changes

#### 1. Recovering from a poisoned remote WAL handle

- **Why required:** A remote file system can permanently poison a WAL handle after an I/O failure. Retrying that handle cannot succeed and leaves the database write-stopped.
- **Previous behavior:** RocksDB retried the same WAL handle during recovery.
- **New behavior:** The remote file system marks the failure as file-scoped. RocksDB creates a replacement WAL, atomically persists all state that could depend on the failed WAL, and then removes the quarantined WAL handles from active use. Physical file deletion remains part of normal cleanup.

#### 2. Preserving an indeterminate WAL write outcome

- **Why required:** A WAL record can reach storage before the append returns `IOError`; neither RocksDB nor the application can know the outcome from that error alone.
- **Previous behavior:** Recovery could reuse a file-scoped failed write group's sequence numbers, allowing a later acknowledged write to overlap a record that might already be in the WAL. The public write API did not explain the acknowledgement ambiguity or distinguish live resumption from close/reopen recovery.
- **New behavior:** `DB::Write` still returns the original `IOError` immediately and does not retry the logical batch. Normal DB reads can continue, but the failed write is not visible in that DB session. The persisted DB remains indeterminate with respect to the failed write until supported file-scoped live resumption succeeds without closing or reopening the DB. On reopen, or for any other WAL error without the durable-abandonment guarantee, applications must resolve consistency within RocksDB's documented guarantees, including `WALRecoveryMode`, through idempotency, deduplication, reconciliation, or an equivalent mechanism. After successful supported file-scoped live resumption, RocksDB has durably abandoned the failed batch and reserved its sequence range; retrying the same logical write is then recommended to preserve one linear logical history.

#### 3. Keeping TransactionLogIterator consistent with live resumption

- **Why required:** A physically complete error-returned record can remain readable from its WAL even though supported file-scoped recovery excludes that batch from the database. Returning it from `GetUpdatesSince()` would make replication or change-data-capture consumers diverge from the recovered DB.
- **Previous behavior:** An iterator opened before recovery could return the abandoned batch. A new iterator could also return it while the physical WAL remained visible.
- **New behavior:** For file-scoped failures, RocksDB records the exact sequence range and WAL number of each indeterminate write. For the lifetime of that open DB, existing and new WAL iterators stop with `Status::NotFound` before the range, whether the physical record is complete or absent. A caller can begin a new iterator after the reserved range. Iterator creation snapshots the recovery-gap generation together with its WAL-file view so a failure on a later, unseen WAL returns `TryAgain` for the old iterator instead of skipping its valid prefix.

#### 4. Keeping all column families on one recovery boundary

- **Why required:** A write batch can update multiple column families. Retiring its WAL after independently persisting only some column families can expose a partially applied batch after reopen.
- **Previous behavior:** With `atomic_flush=false`, recovery could persist column families independently before abandoning the WAL.
- **New behavior:** File-scoped WAL recovery force-flushes every live column family atomically and verifies that each one advanced beyond the failed WAL before writers are retired or writes resume.

#### 5. Merging concurrent WAL failures

- **Why required:** Another WAL write can fail while an earlier recovery attempt temporarily releases the DB mutex.
- **Previous behavior:** A later failed WAL cutoff or ambiguous sequence could be overwritten or cleared by the earlier recovery attempt.
- **New behavior:** Each error episode monotonically merges the largest failed WAL cutoff and the sequence range of the latest ambiguous write group. Recovery clears the quarantine cutoff only if no concurrent failure extended it; otherwise the newer failure remains write-stopping for the next attempt. Iterator fencing tracks the actual ambiguous write range separately, so an older failed `SyncWAL()` and a newer ambiguous write are not incorrectly paired.

#### 6. Retaining file scope through `NoSpace` recovery

- **Why required:** `NoSpace` can still identify one permanently unusable WAL handle.
- **Previous behavior:** The `SstFileManager` recovery path entered generic recovery without carrying the failed WAL context.
- **New behavior:** File-scoped `NoSpace` carries the same cutoff and sequence evidence through the `SstFileManager` path as other file-scoped WAL errors.

#### 7. Scoping recovery context to one error episode

- **Why required:** A completed recovery must not affect a later unrelated background error.
- **Previous behavior:** WAL recovery context could remain latched after `ClearBGError` succeeded.
- **New behavior:** Successful background-error clearing resets the context. A failed recovery retains it so a later retry has the same safety information. The separate iterator gap record remains for the lifetime of the open DB because later WAL readers must continue to reject the abandoned physical record.

#### 8. Preserving successful `SyncWAL` metadata

- **Why required:** A healthy WAL can finish syncing while a separate recovery error already exists.
- **Previous behavior:** `SyncWAL` could return the existing background error before applying the successful WAL's `VersionEdit`, dropping its MANIFEST update.
- **New behavior:** `SyncWAL` skips quarantined writers, applies metadata for successfully synced healthy WALs, and only then returns any existing recovery error.

#### 9. Classifying only the intended remote-file-system failure

- **Why required:** File-scoped recovery is valid only for an actual WAL `IOError` from a handle whose ownership is still valid.
- **Previous behavior:** Poison state could be queried before ownership fencing was resolved, and file scope could be stamped onto a non-I/O status.
- **New behavior:** Ownership fencing is resolved first, poison state is queried afterward, and only an actual `IOError` can be classified as file-scoped. `IOFenced` remains fatal.

#### 10. Preserving `paranoid_checks=false` retry behavior

- **Why required:** Non-paranoid mode historically cleared a writer's sticky error so a later operation could retry after storage recovered.
- **Previous behavior:** Avoiding reclassification of a cached synthetic writer error also bypassed `WALIOStatusCheck`, so later `SyncWAL` calls kept returning the stale error.
- **New behavior:** A cached error still fails the current `SyncWAL` call, but the non-paranoid path resets the writer for a later retry without setting a background error or entering file-scoped recovery.

#### 11. Preserving async WAL precreation across recovery

- **Why required:** A precreated WAL is an unpublished replacement and is not the active WAL. Recovery can also consume the prepared slot while background work is stopped.
- **Previous behavior:** A file-scoped failure while creating or starting the unpublished replacement was attributed to the active WAL, unnecessarily quarantining it. After successful recovery consumed a prepared WAL, the next slot was not replenished, so the following rotation created its WAL synchronously.
- **New behavior:** Only an active WAL `WriteBuffer()` failure attaches the active-WAL cutoff. A failed unpublished candidate is discarded while the current WAL remains active, and successful recovery schedules the next async precreation after `ClearBGError()` re-enables background work.

### Implementation

- Carry `DBRecoverContext` through generic, `NoSpace`, and `SstFileManager` recovery, merging WAL cutoffs and failed write-group sequence ranges independently of error severity. Generic errors retain the range only so a later retry that establishes file scope can activate the stronger protocol; they do not persist a reservation or install an iterator gap on their own.
- Document the WAL acknowledgement ambiguity, persisted-state boundary, reopen behavior, application consistency requirements, and post-resumption retry guidance on the public `DB::Write` API.
- Publish a lock-free quarantine cutoff used by write and sync paths; replace even an empty failed current WAL; persist file-scoped ambiguous sequence reservations; force and verify an atomic all-column-family flush; and detach quarantined writers only after that durable boundary.
- Record each file-scoped ambiguous write group's exact WAL number and sequence range in a DB-lifetime tracker shared by `WalManager` and all `WalIteratorImpl` instances. Stop an iterator before a batch that contains or crosses the range, and use a generation snapshot to preserve the existing `TryAgain` behavior when an iterator's fixed WAL-file view predates the failure.
- Keep successful healthy-WAL tracking edits in `SyncWAL`, reset completed recovery context, and preserve legacy non-paranoid writer reset behavior.
- Distinguish failures on an unpublished replacement WAL from failures flushing the active WAL. Log the failed replacement identity, keep the active WAL out of quarantine, and replenish the async-precreation slot after successful recovery.
- Classify poisoned remote-file-system handles after ownership fencing and only for `IOError`.
- Add lifecycle logging and a state-machine comment covering quarantine, sequence reservation, iterator fencing, replacement, atomic persistence, retirement, and concurrent-error handoff.
- Add an opt-in `db_stress` mode that injects file-scoped faults only into WAL writes, disables WAL tracking in that mode, validates unsupported direct invocations, and sanitizes crash-test parameters to the supported recovery envelope. A small fraction of ordinary non-specialized crash-test runs select this mode automatically; existing stress behavior is unchanged when the flag is off.
- Make the concurrent-error regression test hold the initial background flush until `SyncWAL` records the intended older-WAL error, preventing scheduler-dependent retirement of the test WAL before fault injection.

### Safety boundaries and unsupported cases

- Automatic file-scoped WAL recovery fails closed with `NotSupported` for manual WAL flush, 2PC/transactions, two write queues, pipelined or unordered writes, WAL recycling, WAL TTL or size retention, either WAL-tracking mode, or blob direct writes.
- Unsupported recovery leaves the DB write-stopped. `DB::Resume()` returns `NotSupported`; after correcting the I/O condition, operational recovery requires close and reopen.
- The failed logical batch is never retried. Only supported file-scoped recovery durably reserves its sequence range and fences `GetUpdatesSince()`. Generic WAL errors retain the existing indeterminate outcome and can replay after close/reopen; applications must use idempotency or reconciliation unless successful file-scoped live resumption has provided the stronger guarantee.
- Failed recovery attempts retain their context. Only a successful `ClearBGError` resets it.
- Physical WAL deletion is not part of the recovery correctness boundary; normal obsolete-file cleanup removes files later.
- The TransactionLogIterator ambiguity fence is kept for the lifetime of the DB instance that observed the file-scoped error and is not persisted as new MANIFEST metadata. After a later close/reopen, normal WAL recovery and obsolete-WAL retention/deletion semantics apply.
- The in-memory iterator-gap tracker can grow with the number of recovered file-scoped failures during one DB lifetime. Safe pruning must account for retained WALs and existing iterator snapshots; it is left for follow-up.
- Async WAL precreation remains disabled with WAL recycling. A prepared WAL is unpublished until its start records succeed; failure before installation does not make the active WAL file-scoped.
- The new stress mode is test-only and defaults off, so normal `db_stress` and database operation without an error are unchanged.
- Automatic stress-profile selection is limited to fully parsed, supported configurations. Explicit targeted runs reject options files and other unsupported combinations.

### Tests and known limitations

- The affected recovery, WAL-manager, WAL-iterator, fault-injection, async-WAL-precreation, and secondary-close binaries build successfully.
- Checked-status suites pass for all 70 `error_handler_fs_test` cases, all 7 `wal_manager_test` cases, all 11 `db_log_iter_test` cases, all 8 `fault_injection_fs_test` cases, and all 5 `DBWALTest.AsyncWalPrecreate*` cases. `DBSecondaryTest.SecondaryCloseFiles` passes 20 consecutive repetitions.
- Focused regressions cover file-scoped gap fencing after an iterator snapshot, generic errors not installing a file-scoped sequence gap, and later recovery discovering that a generic error is file-scoped.
- Earlier validation on this diff also passed crash-test unit coverage, remote-file-system adapter tests, repeated concurrency tests, two bounded multi-column-family stress runs, and bounded black-box crash/reopen dry runs.
- The exact internal RocksDB format check and `arc f --take-formatters --enforce-lint-clean` report no formatting or lint issues.
- The full crash/stress matrix and TSAN were not run.

Reviewed By: pdillinger

Differential Revision: D119324616

fbshipit-source-id: 05779d2bff7018f0ae69c4cbbe5b6455a45675d6
2026-09-30 04:40:00 -07:00
Josh Kang ff29163e47 Fix GCC range-loop warning in table test (#15292)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15292

D120637416 introduced a range-for whose `const std::string&` loop variable binds to temporary strings created from a string-literal initializer list. GCC diagnoses this with `-Wrange-loop-construct`, breaking warnings-as-errors builds. Iterate by value to make the intended string construction explicit without changing test behavior.

___

Differential Revision: D122474818

fbshipit-source-id: c7f30a374dc7569ceb1a4ecc6f7f8c3c3637be00
2026-09-29 18:58:04 -07:00
Josh Kang 176d7c7ea1 Add full external table support and crash testing (#15271)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15271

## Summary

Extend the experimental external-table API with a full mode that preserves RocksDB internal keys, sequence numbers, and entry types. Unlike the existing ingestion-oriented mode, full external tables can represent normal flush and compaction output and participate in live databases with puts, merges, deletions, snapshots, `Get`, `MultiGet`, iteration, multiple LSM levels, and external-file ingestion.

The existing `ExternalTable*` interfaces remain aliases for the original sequence-zero, Put-only mode, preserving compatibility for current implementations.

## Implementation details

- Introduce `ExternalTableMode` and mode-specific reader and factory interfaces. Full-mode builders and iterators receive complete internal keys, while the original mode continues to receive user keys.
- Adapt full-mode point lookups to RocksDB’s `GetContext`, allowing readers to return multiple versions of a user key in internal-key order so RocksDB can resolve snapshots, merges, and deletions correctly.
- Add a batched `MultiGet` context that preserves per-request lookup state and maps external-reader results back to RocksDB’s original request range.
- Translate iterator seek boundaries and returned keys while ensuring file-wide global sequence numbers are applied exactly once.
- Track sequence-number bounds, deletion and merge counts, user-collected properties, file size, and unique IDs when building tables. Unsupported entries fail and abandon the builder.
- Forward checksum verification to the external reader while retaining compatibility with implementations that do not override it.
- Add a standalone test implementation shared by basic and full modes. It uses a simple length-prefixed file format with a RocksDB properties block and checksum, avoiding dependencies on native table implementations.

## Crash testing

Add `use_full_external_table` support to `db_stress` and a dedicated blackbox `db_crashtest.py` profile. The profile exercises randomized reads, writes, merges, ingestion, flushes, compactions, snapshots, checksum verification, and crash recovery.

The profile disables unsupported combinations, including range deletions, user timestamps, sequence-number time tracking, backward scans, multiple column families, backup reconstruction, remote compaction, dynamic table-factory replacement, randomized aggregate/all-table property probes, and global-sequence-number rewriting during ingestion.

Because the simple reader materializes each open table in memory, the profile limits open files and generated file sizes to bound total database memory usage.

Reviewed By: xingbowang

Differential Revision: D120637416

fbshipit-source-id: aa2cbd06999e4325a234210d9c45987addd37550
2026-09-29 17:57:17 -07:00
Omkar Gawde b4e068e1ed Reuse MANIFEST batch traversal after apply errors (#15288)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15288

Follow-up to D122242630 (https://github.com/facebook/rocksdb/pull/15286), addressing its review.

Previous behavior:
Before D122242630, when `VersionSet::ProcessManifestWrites` failed to build a batch's new `Version`, it returned without removing the batch from `manifest_writers_` (the two `FIXME? manifest_writers_ still has requested updates` comments). The writers are destroyed when their `LogAndApply` call returns, so later MANIFEST updates, including flushes and compactions, could hang behind the dangling entries or corrupt memory. It needs a fatal LSM consistency-check failure, or a `MemoryLimit` from charging `CacheEntryRole::kFileMetadata` (off by default) to a full block cache with `strict_capacity_limit`. It was found by reading the code.

D122242630 fixed that, but found the end of a failed batch with a second scan that copied the grouping loop's stop conditions, which can drift apart, and added no release note.

New behavior:
A failed batch completes the same writers as with D122242630, but only the grouping loop decides where it ends: `ProcessManifestWrites` records the first `LogAndApplyHelper` failure in `batch_apply_status`, later iterations skip version construction but still run the loop's end-of-batch checks, and the batch then completes through `finish_batch`. This also labels the inner-loop `break`s and adds a release note.

Safety, compatibility, and limitations:
No API, option, or format change. The success path and the `SaveTo` failure path are unchanged.

Reviewed By: pdillinger

Differential Revision: D122417768

fbshipit-source-id: c6d474ace32446d6b413b1a1b3cf2fa49a02447d
2026-09-29 17:31:56 -07:00
Xingbo Wang eed2f96676 Wait for in-flight secondary read-view reclamation (#15289)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15289

Why the change is required:

Secondary catch-up must release reader-free retired read views before scanning
for obsolete files. T290043099 and duplicate T290043100 exposed a race where
that ordering was lost, leaving two obsolete table files open and failing
DBSecondaryTest.SecondaryCloseFiles. The assertion failure also exposed unsafe
test cleanup because the secondary DB could outlive the test-local Env.

Previous behavior:

Retiring a read view always woke the background reclaimer. The worker could
remove a reader-free view from the retirement queue and pause before destroying
its SuperVersions. Synchronous cleanup then saw no queued work, ignored the
in-flight reclamation, and continued to obsolete-file discovery while the old
files were still referenced. The test also used a static lambda that captured
the first fixture across repeated runs and closed the secondary only on the
success path.

New behavior:

Catch-up synchronously reclaims reader-free views, or waits when a competing
background worker has already claimed them. Views still held by active readers
remain non-blocking and are reclaimed asynchronously after the last reader
releases them. The test now closes the secondary before its local Env on every
exit path and creates its verification lambda per test invocation.

Implementation:

Publishing a replacement view no longer wakes the background reclaimer; the
immediately following synchronous cleanup owns that work. Reader release still
wakes the worker. Cleanup waits on the existing in-flight counter when a worker
won an earlier race. The test uses a Defer guard for lifetime ordering and
removes the repeat-unsafe static lambda.

Safety, compatibility, and limitations:

This changes no public API, ABI, file format, recovery behavior, or read
consistency. Cleanup does not wait for views retained by active readers, so the
non-blocking multi-column-family iterator behavior is preserved. It waits only
for reclamation already in progress. Validation covered the internal Linux
ASAN/UBSAN target; the external GitHub Actions matrix was not run. Repository
make-based format/source checks require a Git worktree and were unavailable in
this Sapling checkout.

Reviewed By: jaykorean

Differential Revision: D121816425

fbshipit-source-id: a86cfff43a0cedd10b90e526a56e862f18d2a58d
2026-09-29 16:47:03 -07:00
Omkar Gawde 5fa16200aa Release MANIFEST writers after batch-build failures (#15286)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15286

**What**: `LogAndApply` completes every MANIFEST writer in a batch when version construction fails before persistence.

**How**: `VersionSet::ProcessManifestWrites` centralizes queue completion in `finish_batch`, invokes it for `VersionBuilder::Apply` and `SaveTo` failures, and extends assembly failures through the normal group boundary. Completion propagates the failure through callbacks, signals writers owned by other callers, and wakes the next leader.

**Why**: Stack-backed writers otherwise remain queued after their caller returns, leaving dangling pointers, blocked writes, and flush or compaction state that never receives its failure callback.

Reviewed By: xingbowang

Differential Revision: D122242630

fbshipit-source-id: bab4ac8f665269ba57dd86579299c1d5935fa907
2026-09-29 11:59:49 -07:00
Omkar Gawde 7510773057 Fix ClipColumnFamily key lifetime and atomic-flush wait alignment (#15273)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15273

**What**: `ClipColumnFamily` keeps boundary keys valid across unlocked writes, and atomic-flush waits pair every column family with the correct memtable bound.
**How**: `DBImpl::ClipColumnFamily` copies boundary keys from the current `Version` while holding the DB mutex, with a sync point that lets the regression test replace that version before deletion. `DBImpl::AtomicFlushMemTables` and `DBImpl::RetryFlushesForErrorRecovery` construct wait bounds in `cfds` order and supply no-bound fallbacks when a column family is absent from the flush-request map. Both wait sites gain a debug invariant asserting each id matches the map entry for the cfd at the same index.
**Why**: Background version replacement could leave `ClipColumnFamily` reading freed metadata, while unordered or missing atomic-flush map entries could cause early completion or out-of-bounds access for idle or duplicate column families.

Reviewed By: xingbowang

Differential Revision: D121840843

fbshipit-source-id: 8e49b5c953a1aa29a0244fc0aae2ca3ab84be74f
2026-09-28 14:31:40 -07:00
xingbowang 5eed79a0e1 Scale container shared memory by runner core count (#15284)
Summary:
GitHub Actions container jobs previously used a fixed 16 GB `/dev/shm` allocation regardless of runner size, while the clang-tidy container did not set one explicitly. This over-allocated shared memory on smaller runners and gave larger, more highly parallel runners no additional shared-memory capacity.

Scale container shared memory to 2 GB per runner core across the PR, nightly, nightly-candidate, weekly, and clang-tidy workflows:

- 4-core jobs use 8 GB.
- 16-core jobs use 32 GB.
- 32-core jobs use 64 GB.
- The nightly-candidate ARM job uses 8 GB, matching its four-way build.

The clang-tidy job now explicitly receives 8 GB. Existing unrelated container options, including the mini crash test's `nofile` limit, remain unchanged.

## Safety and compatibility

This only changes container resource configuration for GitHub Actions jobs. It does not change RocksDB code, APIs, persistent formats, or non-containerized jobs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15284

Test Plan:
- `make check-workflow-yaml`
- `git diff --check upstream/main...2026_09_28_shm`

Both checks pass. The CI workloads themselves were not executed locally, so the new shared-memory allocations will be exercised by PR and scheduled workflow runs.

Reviewed By: omkarhgawde

Differential Revision: D122218094

Pulled By: xingbowang

fbshipit-source-id: aa577c37683022ba01c4a073a011dbe0032abbf6
2026-09-28 14:10:43 -07:00
Omkar Gawde 3d381e1bd4 Fix heap-use-after-free in DBSecondaryTest.SecondaryCloseFiles Env teardown (#15283)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15283

**What**: `DBSecondaryTest.SecondaryCloseFiles` no longer calls into a destroyed `Env` when the test body returns early, fixing a ThreadSanitizer heap-use-after-free at secondary DB close.
**How**: The test opens `db_secondary_` (a fixture member) on a stack-local `traced_env`. A `Defer` now closes the secondary before `traced_env` is destroyed on every exit path, so the later `~DBSecondaryTestBase` close never reaches `DBImpl::CloseHelper`'s `env_->UnSchedule` on a freed Env. This mirrors the sibling `DBSecondaryCatchUpFaultTest` fixture, which already closes the secondary in its destructor before its Env member for the same reason.
**Why**: On any early `ASSERT_*` return before the explicit close, `traced_env` (a body local) was freed first while `db_secondary_` (a fixture member) was closed later in the base destructor, which made a virtual call on the freed Env. ThreadSanitizer flagged this as a land-blocking heap-use-after-free in the `build-linux-clang21-mini-tsan` CI job.

Failed CI run: https://fburl.com/jo6gtllf

___

Differential Revision: D122195601

fbshipit-source-id: 3d8b427ba50759c40f15d2b512379594b6057234
2026-09-28 13:06:36 -07:00
Steph Pontikes f8d3141e5c Thread column family names to table readers and add ExternalTableFactory recovery (#15265)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15265

This change adds column family name reflection to an external table reader and allows the user to extract an external table factory from a top level table factory, if it exists

Motivation

Adding column family metadata to the reader allows the reader to handle CF specific metadata, also add external table reflection

Previous/New Behavior

No change in behavior. Added a defaulted parameter threaded through many constructors to preserve existing API contract while introducing new capability. Added a function which allows unwrapping of an external table factory

Nothing much to test

Reviewed By: xingbowang

Differential Revision: D121335729

fbshipit-source-id: 74f9b1be6a8f301bdbe4540de0656b83e26f88ab
2026-09-28 12:04:07 -07:00
Jay Huh 4052fccde9 Enable SecondaryDB in CF-consistency crash tests (#15270)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15270

D121318547 added SecondaryDB batch-iterator coverage to CfConsistencyStressTest::ContinuouslyVerifyDb(), but the automated CF-consistency crash-test profile never selected test_secondary. Automated runs therefore exercised only the primary DB path while the targeted manual validation used test_secondary=true.

Randomize test_secondary between 0 and 1 in cf_consistency_params. Each generated run can now exercise either the existing primary-only path or the SecondaryDB path, preserving both configurations in the continuous matrix. The regression test deterministically verifies both random outcomes. Existing sanitization may still disable SecondaryDB when another selected option is incompatible, such as remote compaction or primary UDI mode.

Reviewed By: xingbowang

Differential Revision: D121813002

fbshipit-source-id: 4080e2131c1f063d590d24cc0bcee9b9096de63a
2026-09-25 16:27:49 -07:00
Xingbo Wang ac30c4d738 Add stress scan witness diagnostics (#15199)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15199

## Why
PrefixScan and MultiScan verification failures are difficult to reconstruct from the terminal assertion alone. The iterator options, bounds, seeks, preparation state, and immediately preceding movements are often the evidence needed to distinguish a database defect from an invalid test assumption.

## Why the existing recovery trace is not enough
The expected-state trace exists to reconstruct the expected database state after recovery. It preserves ordered write batches and deliberately filters out reads and iterator seeks, then recovery consumes a sequence-number-selected prefix. Consequently it contains none of the scan-specific evidence needed here: read options, bounds, seek targets, iterator validity, lazy-value preparation, movements, or the exact mismatch state. Scan witnesses capture that bounded human-readable context only when a verification failure needs it; they are best effort, are never replayed, and do not alter the recovery trace or its failure semantics.

## Previous behavior
The scan paths emitted generic failure output but did not retain a focused bounded witness of iterator setup and state transitions. Adding instrumentation naively would also format diagnostic strings on the normal path even when disabled and could create one file per failing operation.

## New behavior
An opt-in bounded ScanWitnessLog records scan configuration and recent iterator events for PrefixScan and MultiScan variants. On a qualifying failure it writes the latest complete witness for that DB, PID, thread, and scan type. The operation ordinal remains in the file contents. Repeated failures atomically replace the fixed witness rather than creating unbounded files.

## Implementation
The witness captures sanitized read options, input ranges, bounds, seeks, movement, PrepareValue results, iterator status, keys, value size and hash, and wide-column state. MultiScan tracks whether the current primary iterator value was actually prepared so failure snapshots report accurate state. Disabled witnesses avoid scan-label ownership, range formatting, expected-count formatting in release builds, and numeric iterator-label conversion. Witness writes share the per-thread diagnostic I/O-disable latch with breadcrumbs, write to a fixed temporary path, close, and rename over the final file. Entry counts are capped at 65,536. A formatting-only follow-up normalized all touched C++ lines with the repository clang-format rules.

## Safety and unsupported cases
The feature is stress-tool-only and raw db_stress keeps it disabled by default. It observes public iterator state and does not change database contents, expected-state tracing, recovery replay, iterator decisions, or fault-injection outcomes. Diagnostic I/O remains best effort and cannot fail the workload. Files are bounded per retained PID, thread, and scan type. The witness covers the instrumented PrefixScan and MultiScan implementations only and is not a general RocksDB trace. Files are not fsynced and therefore are not a power-loss durability mechanism.

## Tests and limitations
Validated with 47 tools/db_crashtest_test.py tests, exact-candidate debug and release db_stress builds, arc lint over the complete stack, oversized-entry rejection, and clean independent correctness reviews of the final integrated stack. make check-sources is unavailable because it invokes Git in this Sapling checkout; equivalent added-line hygiene checks pass. Only the newest complete witness per DB, PID, thread, and scan type is retained, so later failures intentionally replace earlier witnesses.

## Rebase update
Rebased onto fbsource master dc59f986b964. The logical diff is unchanged; the rebased stack passes debug and release db_stress builds, all 47 db_crashtest Python tests, and arc lint.

Reviewed By: jaykorean

Differential Revision: D119160911

fbshipit-source-id: 0e9b040c3bbb482e0a3b1b11070a7c5cc519f060
2026-09-25 16:20:54 -07:00
Josh Kang dbc2db5228 Make external table APIs more consistent with native tables (#15241)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15241

Native `TableFactory` implementations receive the RocksDB `WritableFileWriter` in `NewTableBuilder()` and `RandomAccessFileReader` in `NewTableReader()`. These wrappers provide buffering, checksums, rate limiting, statistics, listeners, tracing, and direct-I/O handling. External table implementations previously received a limited writable-file adapter and reopened readers from a path, bypassing parts of this wrapper logic.

This change makes `WritableFileWriter` and `RandomAccessFileReader` implement `FSWritableFile` and `FSRandomAccessFile`, respectively, so the same RocksDB wrappers can be passed to external tables. `ExternalTableFactory::NewTableReader()` now receives the already-opened reader and file size. The filesystem interfaces expose standard append, flush, sync, fsync, file metadata, synchronous reads, asynchronous reads, and batched reads. `RandomAccessFileReader::SubmitReadAsync()` schedules its wrapped coroutine read on the configured filesystem read executor, and the DB table-cache and `SstFileReader` construction paths pass that filesystem through. `SstFileReader` also supplies its configured statistics, listeners, rate limiter, and file temperature, removing the need for a separate external-table metrics filesystem wrapper.

For direct I/O, `MultiRead()` checks whether requests are already sorted. When alignment and merging are needed, it constructs a sorted index only for unsorted input, preserving caller request order and result mapping. Fully aligned requests using aligned caller buffers bypass this path and may be supplied in any order.

Reviewed By: xingbowang

Differential Revision: D121035539

fbshipit-source-id: 27cb897726c31cb1a4dc811c65c8d79e690d82b1
2026-09-24 11:46:13 -07:00
Xingbo Wang 07cbacadb4 Avoid materializing huge SSTs in size-boundary test (#15264)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15264

Why the change is required:
LargeSizeSstFileWriter creates approximately 12 GiB of SST output in the shared /dev/shm used by build-linux. With 32 concurrent test jobs in a 16 GiB tmpfs, this can exhaust the filesystem and make unrelated fault-injection and repair shards fail with ENOSPC.

Previous behavior:
The test wrote an approximately 8 GiB large-key SST and an approximately 4 GiB large-value SST even though it never read or ingested either file. It validated the size limits and successful Finish calls, but the physical output competed with every other test process.

New behavior:
The test still runs the real SstFileWriter and table builder for over-limit rejection and maximum-size acceptance, including Finish, sync, and close. The generated byte stream is counted and discarded instead of materialized in the shared tmpfs.

Implementation:
Add a test-local FileSystem wrapper whose writable file counts appended bytes and records sync and close calls. Inject it only into LargeSizeSstFileWriter and verify that the writer-reported logical size matches the observed append stream, the lifecycle completes, and no physical SST is created.

Safety, compatibility, and limitations:
Production code, shared filesystem implementations, serialized formats, and other tests are unchanged. The fake does not validate SST contents, but the previous test did not read the generated files either. Table construction still consumes substantial memory, so the existing HasBigMem guard remains; this change addresses tmpfs exhaustion only.

Reviewed By: rban1

Differential Revision: D121483663

fbshipit-source-id: 055372bc07e3a1f7289e34ebc35694bc3c4bfad9
2026-09-24 09:46:56 -07:00
Ranjan Banerjee 662cdbc9ec Update RocksDB main for 11.11 release (#15263)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15263

Why: keep internal RocksDB main ready for 11.12.0 development while the 11.11 release is cut.
Previous behavior: Conveyor opened a GitHub pull request for this update, which could be missed and was not stacked with the internal release diff.
New behavior: Conveyor owns an internal Phabricator diff that the release-import diff stacks on top of. The release manager exports this diff manually with ExportIt.

Implementation:
- Bump version.h from 11.11.0 to 11.12.0
- Add `11.11.fb` to `check_format_compatible.sh`
- Sync HISTORY.md `## 11.11.` section from `11.11.fb`
- Delete 19 consumed `unreleased_history/` note file(s) from main

Safety and unsupported cases:
- Only version.h, check_format_compatible.sh, HISTORY.md, and consumed release-note files are allowed.
- Open legacy GitHub update-main PRs block this flow; this automation never modifies, exports, or lands them.
- Concurrent direct CLI invocations are unsupported; Conveyor serializes the Release Cut node to one active run.
- The tool refuses unrelated commits, mismatched release metadata, and non-canonical GitHub repositories.

Known limitations:
- ExportIt remains manual. Bot create permission for internal_repo_rocksdb/repo is a rollout prerequisite.

Reviewed By: xingbowang

Differential Revision: D121434642

fbshipit-source-id: 3c6c90f425e043ff9c13318ecc671aedbdef0347
2026-09-23 21:51:29 -07:00
Jay Huh ffd4ac31af Make SecondaryDB multi-CF iterator reads non-blocking (#15259)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15259

Publish immutable SecondaryDB read views after open and successful catch-up so `NewIterators()` can acquire one cross-CF sequence and SuperVersion set without waiting for `DBImpl::mutex_`. Keep failed catch-ups on the previous view, reject refresh for batch iterators, and exercise concurrent catch-up plus batch scans in the CF-consistency stress profile.

Reviewed By: xingbowang

Differential Revision: D121318547

fbshipit-source-id: 3198f4766c90ab0bb39400963df3c329b4ecf6bf
2026-09-23 13:09:24 -07:00
Omkar Gawde 0675830967 Verify prefetched block checksums with retry-safe cleanup (#15254)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15254

**What**: `IODispatcher` verifies prefetched data blocks before serving or caching them and safely falls back when block construction fails.

**How**: `CreateAndPinBlockFromBuffer` validates every returned span and, when `ReadOptions::verify_checksums` is enabled, verifies its checksum and records the standard counters. `PollAndProcessAsyncIO` handles coalesced blocks independently and retires shared request state before failed entries fall back through `SyncRead`; `ReleaseFailedSyncPrefetchMemory` returns limiter capacity for unmaterialized synchronous blocks.

**Why**: Unverified prefetches could serve or cache bit rot, while newly detected failures could otherwise leave async state referencing released provider buffers or permanently consume bounded prefetch capacity.

Reviewed By: xingbowang

Differential Revision: D121277687

fbshipit-source-id: 65cc1d5b9dcc53bdc64f8ee5d672bfe2cda7b40b
2026-09-23 11:34:25 -07:00
Xingbo Wang 14a82121d4 Add shared diff and pull request summary guidance (#15261)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15261

Why the change is required:
Internal Phabricator diffs and external GitHub pull requests need the same substantive explanation of a RocksDB change. Existing local instructions focus on code generation and review but do not define what a summary must tell reviewers, so summaries can omit prior behavior, safety boundaries, compatibility, or validation evidence and can drift between the two review surfaces.

Previous behavior:
There was no CLAUDE.md at the internal_repo_rocksdb wrapper root, the RocksDB CLAUDE.md did not point authors to summary guidance, and there was no shared document covering both internal diff and external pull request summaries.

New behavior:
The wrapper-level CLAUDE.md requires authors to consult a shared summary guideline before writing or updating an internal diff summary. The repository CLAUDE.md applies the same guideline to external pull request summaries. The guideline requires motivation, previous and new behavior, implementation, safety and compatibility boundaries, and test evidence, with RocksDB-specific advice for correctness, persisted formats, concurrency, and performance changes. Examples are separated by access boundary: the internal wrapper contains Phabricator diff examples, while the repository guideline contains publicly accessible GitHub pull request examples.

Implementation:
Add a concise wrapper-level CLAUDE.md, add claude_md/pull_request_summary_guideline.md as the shared source of truth, and link it from repo/CLAUDE.md. Keep annotated internal diff references in the wrapper-level file and corresponding public pull request references in the repository guideline, including a recent change from Peter Dillinger.

Safety boundaries and unsupported cases:
This is documentation-only and changes no runtime, API, build, test, or CI behavior. The repository guideline contains no internal links, so external readers do not need Meta access. The guidance is manually followed by authors and agents; it does not mechanically validate summaries, replace reviewer judgment, or require a rigid section layout for small changes.

___

Differential Revision: D121388476

fbshipit-source-id: bb09ab7ba8ef6dc2b2830a42735322890536622f
2026-09-23 08:44:55 -07:00
Andrei Preda 507206e46b Fix stale readahead trim prefix on SeekToFirst (#15216)
Summary:
`BlockBasedTableIterator::SeekImpl()` sets
`seek_key_prefix_for_readahead_trimming_` only when given a seek key,
and `ResetBlockCacheLookupVar()` does not clear it. After a `Seek()`, a
`SeekToFirst()` on the same iterator trims readahead against the
previous seek key's prefix. `DBIter::SeekToFirst()` under
`prefix_same_as_start` takes its prefix from the first key found, so the
old prefix does not describe the keys being scanned.

With the stale prefix, `IsNextBlockOutOfReadaheadBound()` is true at
every block instead of only at the prefix boundary, and each prefetch is
clamped to the current block. The same predicate sets
`is_index_out_of_bound_`, which `FindBlockForward()` converts to
`is_out_of_bound_` and ends the scan. Under `ReadOptions::async_io` this
drops keys.

The bug needs `auto_readahead_size` and a preceding `Seek()` that reads
no data block. A `Seek()` that lands on a key leaves the prefetch buffer
anchored above the scan, and the trimming callback is not called. The
bug does not occur under `allow_mmap_reads`, where `FilePrefetchBuffer`
is constructed disabled.

The fix clears the prefix in `SeekToFirst()`, so prefix-based trimming
is disabled for that positioning. The `auto_readahead_size`
documentation in `include/rocksdb/options.h` already says prefix-based
trimming requires `Seek(key)`.

On buffered reads with `prefix_same_as_start`, `auto_readahead_size` and
`readahead_size=8192`, a cold-cache `SeekToFirst()` scan of 2M keys in a
single 222 MB file, preceded on the same iterator by a `Seek()` past the
last key, took 391 ms before and 308 ms after. The same scan with no
preceding `Seek()` takes 307 ms. Under direct IO on a local disk it went
from 33.1 s to 11.8 s. Data blocks read are unchanged; file reads drop
by 3x.

Also fixes a pre-existing test bug: `SeekParallelizationTestWithPosix`
asserts on `number_async_seek` without resetting that cumulative
thread-local, unlike its three siblings. The new tests move
`prefetch_test` from 11 shards to 13, so different tests share a
process, and the assertion fails at the parent commit under that
grouping.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15216

Test Plan:
New regression test in `PrefetchTrimReadaheadTestParam`, in sync and
`async_io` variants over the fixture's three `index_shortening` modes.
It runs two scans of the same keys from an equally cold block cache,
differing only in a preceding `Seek()`. Reverting the `clear()` fails 3
of 6 sync parameterizations on `READAHEAD_TRIMMED` (6 vs 40) and 3 of 6
`async_io` parameterizations on key count (40 vs 1).

`make check` and `ASSERT_STATUS_CHECKED=1 make check` show no new
failures. `prefetch_test` 122/122, `db_iterator_test` 249/249,
`table_test` 7629/7629, `block_based_table_reader_test` 5347/5347,
`db_basic_test` 300/300. New cases ran 1200 times under
`COERCE_CONTEXT_SWITCH=1` without failure.

Added db_stress coverage for `SeekToFirst()` after `Seek()` under
`prefix_same_as_start`. With the fix reverted and io_uring available,
the configuration in the second commit fails 8 of 8 seeds with
"Iterator diverged from control iterator"; with the fix, the same seeds
pass.

Reviewed By: rban1

Differential Revision: D120228466

Pulled By: xingbowang

fbshipit-source-id: a4935f03a0080cc8dd4c7a8c1438404e9b8f5c3a
2026-09-21 11:28:32 -07:00
zaidoon 3c30d43baf Promote IndexFactory as UDI SPI (#14962)
Summary:
Make `IndexFactory` the public interface for custom indexes. Keep the old `UserDefinedIndex` names, header, and `NewBuilder()` / `NewReader(Slice&)` overloads so existing implementations can still compile.

UDI and options tests pass on the full stack, both normally and with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`. The C API test and generation/compatibility checks pass. Full `make check` runs had test timeouts and hit the 30-minute local limit. These test results are from before the rebase.

Part 5 of 9. Previous: https://github.com/facebook/rocksdb/issues/14961 (landed). Next: https://github.com/facebook/rocksdb/issues/14963.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14962

Reviewed By: jaykorean

Differential Revision: D119960591

Pulled By: xingbowang

fbshipit-source-id: 9703950bd48723c2c524da89cbea63bf00254cc1
2026-09-21 09:28:49 -07:00
Denis Yaroshevskiy 96e7ef0269 Use memcpy instead of strncpy in Status::CopyState to speed up Status copies (#15238)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15238

Reviewed By: MatzeB

Differential Revision: D116157932

fbshipit-source-id: 56f52f7e5a92154be2be256151e4cc6f63bbbc05
2026-09-21 05:17:50 -07:00
Xingbo Wang 5d6f321b70 Document behavioral compatibility review requirements (#15218)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15218

Why the change is required:
Public and serialized RocksDB inputs can change observable performance or resource behavior without a signature or default change. These changes require deliberate human judgment, clear documentation, and research into downstream production impact. The automated CI review also needs an explicit perspective for identifying gaps in that analysis.

Previous behavior:
CLAUDE.md described source and ABI compatibility but did not explicitly guide authors or reviewers through behavioral changes to already accepted inputs. The CI review prompt had only a brief API-compatibility checklist item and no shared instructions for evaluating intentionality, release notes, downstream usage, or regression coverage.

New behavior:
CLAUDE.md states that behavior changes are human judgment calls that must be intentional, documented, and researched. It gives concrete guidance for existing accepted inputs while asking agents and reviewers to call out gaps or ambiguity. The CI review prompt applies the same perspective to every review agent and makes clear that a behavior change is not automatically a defect.

Implementation:
Revise the Behavioral Compatibility policy in CLAUDE.md, reduce its workflow entry to a concise pointer, add a dedicated Behavioral Compatibility Review Perspective to ci_review_prompt.md, and point the API and compatibility reviewer to that shared section.

Safety boundaries and unsupported cases:
This is documentation-only and changes no runtime, API, build, or CI execution behavior. The guidance supports human judgment; it does not automatically decide whether a behavior change is acceptable, guarantee exhaustive discovery of downstream consumers, or replace release-note, testing, and production-impact review.

Reviewed By: pdillinger

Differential Revision: D120344151

fbshipit-source-id: aa5bbb00fbbe167ef21b91b3551868c4d1b4046d
2026-09-18 11:28:27 -07:00
Josh Kang 256a699be3 Support MultiGet and global seqnos in external tables (#15237)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15237

Route `DB::MultiGet` batches through `ExternalTableReader::MultiGet` instead of `TableReader`'s per-key fallback, preserving per-key status and pinned-value handling.

Allow basic external tables to honor DB-assigned global sequence numbers across overlapping levels, snapshots, `Get`, `MultiGet`, and iteration. Preserve ingestion-only behavior by rejecting compaction output, and support both `allow_global_seqno` and deprecated `write_global_seqno` files.

Reviewed By: xingbowang

Differential Revision: D120629061

fbshipit-source-id: 366f48660e97a300afb59627b9a1a41fe3805590
2026-09-18 10:39:07 -07:00
Omkar Gawde 7530287bde Stop over-counting limited SstFileDumper reads (#15228)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15228

**What**: Correct the limited-read count and add regression coverage for recompression source-read errors.

**How**:
- `ReadSequential()` checks `read_num_limit` before consuming and counting the next entry.
- `RecompressReturnsSourceReadError` injects data-block read failures and verifies `ShowAllCompressionSizes()` returns the error.

**Why**: `sst_dump --read_num` reported one extra key, while the recompression input-materialization error path lacked regression coverage.

Reviewed By: joshkang97

Differential Revision: D120182510

fbshipit-source-id: d3b07794c9f7b74c29577242ad64c9c1f4a081cf
2026-09-17 15:31:21 -07:00
Peter Dillinger b03a058d61 Persist more SST table properties for compression analysis (#15231)
Summary:
These changes improve the block-based SST table properties available for compression benchmarking and optimization.

- Fix: `uncompressed_data_size` was computed while building a file but was never written to the properties block, so it always read back as 0. It is now persisted (written in AddTableProperty and read back in ParsePropertiesBlock).

- Add: a new packed `lsm_info_at_creation` property records the LSM level and bottommost status at the file's creation. A new `LsmInfoAtCreation` struct owns the (extensible) uint64 encoding via symmetric Encode()/DecodeFrom(), keeping TableProperties a bare data struct. It distinguishes three states: "unknown" (nothing recorded, e.g. older files), "not applicable" (no LSM position, e.g. SstFileWriter/misc output), and "applicable" (level + bottommost, with an unknown-level sentinel for the DB repairer). The level is stored as its natural value.

- Change: the `_compression_manager` detail in the `compression_options` property now records the manager's full serialized configuration (via ToString) instead of only its GetId().

New properties are additive and backward compatible: older files without them read back as "unknown"/0.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15231

Test Plan:
Added coverage in db/db_table_properties_test.cc: LsmInfoAtCreation encode/decode round-trips (natural levels, unknown level, not-applicable vs unknown); end-to-end persistence of uncompressed_data_size and lsm_info_at_creation after flush (L0, not bottommost) and a forced bottommost compaction; not-applicable recorded for an SstFileWriter file; and full compression manager configuration present in compression_options. Updated options/options_settable_test.cc for the new field.

Also made table_test's BlockBasedTableTest.CompressionRatioThreshold resilient to per-file overhead changes (such as these added table properties): it now verifies the compression decision via the persisted data_size vs uncompressed_data_size rather than modeling total file size against a fixed overhead constant, which had become too tight. Full table_test passes.

Reviewed By: rban1

Differential Revision: D120553038

Pulled By: pdillinger

fbshipit-source-id: 07ede62dbcbe31c040fd9b037c7636a49558c4d9
2026-09-17 13:18:07 -07:00
Xingbo Wang 8108b49e0c Add db_stress operation breadcrumbs (#15194)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15194

## Why
Intermittent db_stress and db_crashtest failures often report only the terminal assertion, after the operations that established the failing state have scrolled out of logs. The diagnostics must retain the latest operations even when the stress process is killed before it can run a failure handler.

## Why the existing recovery trace is not enough
The existing expected-state trace is a correctness oracle for recovery: after an expected-state checkpoint it records ordered write batches, and recovery replays the prefix implied by the database sequence number. Reads and iterator seeks are deliberately filtered out, trace-write failures are fatal, and the trace is machine-consumed rather than intended as a concise failure narrative. It therefore cannot show which read, scan, maintenance operation, or operation phase immediately preceded an assertion. Breadcrumbs fill that observability gap with a bounded, human-readable, per-thread history. They are best effort, never replayed, and remain independent of the recovery-validation contract.

## Previous behavior
db_stress did not retain a bounded per-thread operation history. db_crashtest did not automatically provision a local directory for these artifacts. The first version of this diff kept the ring in process-local memory and periodically wrote a complete temp-file snapshot, which could lose every operation since the last snapshot when the process crashed.

## New behavior
An opt-in per-thread memory-mapped breadcrumb ring records recent operation begin, context, and end events directly in a local file. Committed records therefore remain in the kernel page cache when only the db_stress process exits or is killed; no periodic flush is required. db_crashtest enables breadcrumbs by default, assigns a local diagnostics directory, retains the newest eight process groups after failed child runs, and removes managed artifacts after success.

## Implementation
Each ThreadState lazily creates a uniquely named, preallocated file and maps it through Env::NewMemoryMappedFileBuffer(). The file contains a fixed header and 512-byte text records. A record slot is marked I before its body is replaced and V only after the complete body is published; E denotes an unused slot. Each valid record carries a monotonically increasing sequence number so wrapped records can be ordered. Oversized context is bounded and ends with details_truncated=1. Lazy initialization avoids empty files for auxiliary ThreadState objects, and a process-wide file ID prevents collisions between ThreadState objects that share a logical thread ID.

The periodic-flush flag, temp-file/rename snapshot path, and verification-failure fallback flush are removed. Enabled operation-context formatting continues to reuse a per-thread string and uses std::to_chars for integers; disabled call sites avoid formatting. Diagnostic directory setup and mapping failures are reported but do not change the workload result. Auto-directory pruning and cleanup accept only generated filename schemas, reject symlink roots, preserve unrelated files and explicit user directories, and retain at most eight PID groups.

## Safety and unsupported cases
This is stress-tool-only diagnostic behavior. Raw db_stress keeps breadcrumbs disabled by default. The expected-state recovery trace and scan-witness contracts are unchanged. The mapping deliberately does not use msync or fsync: it is intended to survive a process crash while the kernel remains running, not power loss, host reset, or kernel failure. Filesystems or Env implementations without shared writable mappings disable breadcrumbs non-fatally. Replacing or truncating a live mapped file is unsupported. Records are fixed width, so unusually large operation context is explicitly truncated. Concurrent live readers are not promised a consistent snapshot; post-termination readers must ignore I and E records and order V records by sequence number. Nested operation scopes remain unsupported.

## Tests and limitations
Validated with all 47 tools/db_crashtest_test.py tests, exact integrated debug and release db_stress builds, complete-stack arc lint, lower/upper/combined clang-format checks, normal ring wrap, distinct files for four worker threads, an oversized 256-column-family context, and process-crash tests. A release process killed with SIGKILL retained all 16 committed ring records without an explicit flush. A debugger-forced SIGKILL after the I store but before V publication left the interrupted slot marked I. In five CPU-pinned alternating release pairs of 50,000 operations, enabled and disabled medians were both 2.15 seconds. This remains a bounded latest history rather than a complete log and provides no host-crash or power-loss durability.

## Rebase update
Rebased onto fbsource master dc59f986b964. The logical diff is unchanged; the rebased stack passes debug and release db_stress builds, all 47 db_crashtest Python tests, and arc lint.

## Review follow-up
The mapped-slot invalidation now uses an acquire-release atomic exchange so later body stores cannot become visible before the I marker on weakly ordered architectures.

Reviewed By: anand1976

Differential Revision: D119160910

fbshipit-source-id: ffe172af9ebc75c617196592806dc9641a539a25
2026-09-17 13:01:13 -07:00
Anand Ananthabhotla dd54c5abc9 Log DB session id for file deletion and manifest creation (#15229)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15229

Add `db_session_id` to obsolete file deletion logs and structured table/blob deletion event logs. Also include `db_session_id` when logging new MANIFEST creation so split-brain DBImpl sessions can be correlated without relying on process-local pointers.

Reviewed By: joshkang97

Differential Revision: D120413139

fbshipit-source-id: c175b70d9b1f93e3242ee9e3191c274dc51ef3bf
2026-09-17 12:46:43 -07:00
Omkar Gawde 7d721f398b Reject reopening an active SstFileWriter (#15232)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15232

**What**: Make `SstFileWriter::Open()` return `InvalidArgument` without replacing the active writer when a file is already open.
**How**: Check `rep_->builder` before constructing new output resources, matching `OpenWithEmbeddedBlobs()` and leaving the original file finishable.
**Why**: A second open otherwise destroys an unfinished `TableBuilder`, violating its lifecycle contract and triggering an assertion.

Reviewed By: pdillinger

Differential Revision: D120469611

fbshipit-source-id: 8daa4607725eea574fe80012f9df98132160ce3e
2026-09-17 11:33:08 -07:00
Jay Huh bcf8711165 Add per-column-family ReadOptions to NewIterators (#15230)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15230

Add a raw C++ `DB::NewIterators()` overload accepting parallel vectors of `ReadOptions` and column family handles. Each iterator receives its own options while all iterators share one consistent sequence snapshot. Require identical snapshot pointers and tailing modes across the options.

Implement the overload for primary, secondary, read-only, stackable, and write-prepared DBs. Building on D120048299, both secondary overloads share one helper that captures all SuperVersions and the sequence number under the DB mutex.

Preserve read-only `table_filter` support when range-tombstone conversion is configured, and reject that unsafe combination in write-prepared read-write iterators before allocating any iterator.

Add validation, per-CF prefix-seek coverage, a secondary catch-up regression test, table-filter contract tests, and a public API release note.

Reviewed By: xingbowang

Differential Revision: D120296578

fbshipit-source-id: 3775884537ea29a0d32bb3fe10330cf62ac5a406
2026-09-17 11:26:24 -07:00
Peter Dillinger 34d5074bfa Cross-key coalescing for lazy wide-column blob reads (#15178)
Summary:
Follow-up (Phase 3) to the experimental lazy wide-column read API (DB::GetEntityLazy / DB::MultiGetEntityLazy). A MultiGetEntityLazy batch now resolves blob references across keys with coalesced I/O instead of key-by-key.

- MultiGetEntityLazy acquires one SuperVersion and one consistent (implicit) sequence number for the whole batch via MultiCFSnapshot (as multi-CF iterators do) and transfers a single shared per-column-family pin into the LazyWideColumnsBatch, replacing the former per-key SuperVersion + explicit-snapshot loop. Single column family for now; a cross-CF ColumnFamily** overload remains a follow-up (the batch already models a per-CF pin map).
- LazyWideColumnsBatch::MultiResolve groups the batch's reads by (Version, blob file) for separate-file references and by SST for embedded references, and issues one coalesced read per group -- for whole-column and byte-range reads alike -- caching whole values in each entity's resolver and pinning partial reads directly.
- Adds the coalesced blob-read primitives this needs (none existed before): range-aware BlobFileReader::MultiGetBlobRange / BlobSource::MultiGetBlobRange / Version::MultiGetBlobLazy for separate-file references, and an embedded (SimpleGen2) batch read path (blob_gen2_format batch readers + BlobSource::MultiGetSimpleGen2Blob[Range] + BlockBasedTable::MultiGetSameFileBlob). Whole-value separate-file coalescing reuses the existing MultiGetBlob. The batch readers sort their requests by file offset before issuing the underlying MultiRead (which requires ascending offsets), and gen2 checksum verification is factored into one helper shared by the scalar and batch readers.
- The hot non-lazy Get / MultiGet / iterator / compaction paths are untouched: the new multis are lazy-only, and the GetImpl change is gated on a new opt-in GetImplOptions field (an injected shared SuperVersion + sequence number) used only by the batched lazy read.
- Blob read failures on these paths (short reads, invalid offsets, out-of-range requests, compression mismatches) now carry file/offset/size and expected-vs-actual context, mirroring the block-checksum diagnostics.
- db_bench gains a multireadrandomentitylazy benchmark for measuring the coalesced batch path.

No public API change, so this is a performance follow-up to the still-recent lazy API rather than a new feature. EXCEPT that `LazyColumnReadRequest::status == nullptr is now explicitly disallowed`.

Left for later follow-up:
- When a whole fetch is planned for an (entity, column), bind ordinary range requests for that column to the same fetch and slice them after adoption instead of issuing separate range I/O.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15178

Test Plan:
New unit tests in db/wide/db_lazy_entity_test.cc assert cross-key coalescing via a blob-file / SST MultiRead counter added to the test FileSystem wrapper:
- N whole-column separate-file reads across keys in one blob file collapse to a single coalesced MultiRead (and the fetched values are cached, so a repeat read does no further blob I/O);
- N byte-range separate-file reads collapse to one MultiRead, save the un-read bytes (rocksdb.blobdb.lazy.partial.bytes.saved), and never fill the blob cache;
- embedded (same-file) reads across keys in one SST collapse to one MultiRead over that SST;
- an out-of-order embedded batch (reads issued in descending record offset) still resolves correctly in one coalesced read, covering the batch readers' internal offset sort;
- a mixed batch (inline + whole + byte-range across keys) returns the correct bytes for each read;
- a batched MultiGetEntityLazy + one MultiResolve matches resolving each key via a separate GetEntityLazy. The existing db_stress lazy-vs-eager differential (MaybeTestMultiGetEntityLazy) exercises the new batched read and cross-key MultiResolve unchanged (same public API).

Reviewed By: xingbowang

Differential Revision: D118660682

Pulled By: pdillinger

fbshipit-source-id: 6a832175e730917eddeedab8e0b9cb2b8d6ae7b4
2026-09-16 18:10:21 -07:00
Jay Huh 79b58b0dd6 Keep secondary multi-CF iterators consistent (#15211)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15211

Capture the secondary DB sequence number and all requested column-family SuperVersions while holding the DB mutex, then construct the iterators after releasing it. This keeps every child on one state even when `TryCatchUpWithPrimary()` concurrently installs file-backed SuperVersions.

Add a file-only catch-up regression test and release note.

Reviewed By: xingbowang

Differential Revision: D120048299

fbshipit-source-id: 89f4de533c4fbc1c6e86656845bda4bbb9da4d2c
2026-09-16 15:27:11 -07:00
Omkar Gawde 613a642b5c Fix null dereferences in SstFileDumper error paths (#15215)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15215

**What**: Stop `SstFileDumper` from dereferencing a null file handle on two error paths, both of which crash `sst_dump` today.

**How**: Both sites create a file and then use the handle without checking whether creation succeeded. `Env::NewWritableFile()` and `FileSystem::NewRandomAccessFile()` each reset the caller's handle before they can fail, so on failure the handle is null.
- `DumpTable()` fell through to a shared error path that closed the handle. It now returns as soon as `NewWritableFile()` fails.
- `GetTableReader()` re-opens plain and cuckoo tables with mmap enabled and discarded the returned status entirely, then wrapped the null handle in a `RandomAccessFileReader` that the next property read dereferenced. It now propagates the status.

**Why**: Both are reachable from the `sst_dump` tool. The output file cannot be created on an unwritable directory, a missing parent, or a full filesystem; the mmap re-open fails when mmap is unavailable or the file changed between the two opens. In each case the tool crashed instead of reporting the error. The plain and cuckoo branch had no test coverage at all, which is why its unchecked status never tripped an `ASSERT_STATUS_CHECKED` build.

Reviewed By: xingbowang

Differential Revision: D120182509

fbshipit-source-id: dbe9b7f84140ed11dea395f4192b895e0a7bc70d
2026-09-15 15:54:16 -07:00
Peter Dillinger e231c1826c Revamp sst_dump recompress for serious benchmarking (#15195)
Summary:
Extends `sst_dump --command=recompress` from a basic evaluation tool to a serious compression benchmarking tool, also able to evaluate interactions with table format options affecting uncompressed sizes and compressibility.

For each input file it now emits a benchmark "baseline" entry for the file as stored, followed by one entry per compression type / strategy / level, reporting compressed and uncompressed data-block sizes, index sizes, compression ratio, and both wall-clock and CPU time for the write and read tests. All data/index sizes are block payload excluding block trailers, so compressed vs uncompressed figures are on a consistent basis. The input's uncompressed data size is inferred from decompression statistics, so it is available even for files that don't persist it. That inference is exact when the input's index blocks are stored uncompressed (the default); with enable_index_compression it can be slightly overstated because index-block decompression is also counted (see the CAVEAT in GetInputBenchmarkMeasurement).

New capabilities:
* `--json` for machine-readable output. It is self-contained: each measurement carries the compression manager id and the full CompressionOptions, and every baseline/measurement entry is tagged with its input file path so multiple input files produce one valid document.
* `--compression_strategy=` to sweep a list of CompressionOptions strategy values (multiplied with levels), useful for side-channel configuration of custom compressors.
* `--show_properties` additionally dumps the input and last recompressed output TableProperties.

To keep measurements meaningful, a quiet throw-away priming build runs whenever the compression type changes (warming up the compressor), CPU time is measured process-wide so parallel compression worker threads (compression_parallel_threads > 1) are included rather than undercounted, and the timed windows are kept tight: the write build is fed from an in-memory copy of the input so its timing excludes reading and decompressing the input (and is independent of block-cache state), and the read timing covers only the block reads. The timings for compression and decompression do include BlockBasedTable overheads but this is considered reasonable--if not preferable--to the intended application of finding the overall most efficient settings related to compression.

Also fixes a latent bug exposed by measuring several compression types in one run: the recompressed table was read back through the shared block cache under a common (unknown) cache-key identity, so a later type could get a cache hit on a stale block from an earlier, differently-compressed table and fail with "Corruption: bad entry in block". The read-back now bypasses the block cache.

Also fixes a typo in CLAUDE.md.

Intended follow-up: persist TableProperties::uncompressed_data_size. It is already computed in memory at build time but is never written to the properties block (PropertyBlockBuilder::AddTableProperty has no entry for it), so it reads back as 0 for on-disk files. Persisting it (backward compatible; old files read 0) would fix `data uncompressed size: 0` for all property readers and let this tool read the value directly, removing the decompression-ticker inference above and its index-compression caveat for all but pre-existing files.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15195

Test Plan:
Added sst_dump_test cases exercising recompress with --show_properties and with --json.

Ad hoc manual testing against real SST files: verified the text benchmark table, and piped `--json` output through `json_pp` to confirm well-formed JSON, for a single file, a directory of multiple SST files, and combined with --show_properties, --output_hex, --compression_strategy, and --block_based_table_options. Confirmed process CPU time exceeds wall-clock write time with --compression_parallel_threads=4 (and matches at =1).

Reviewed By: joshkang97

Differential Revision: D119536444

Pulled By: pdillinger

fbshipit-source-id: 30dbabc14872b9c6f7d8ca11227e762674dcef02
2026-09-15 12:40:17 -07:00
Peter Dillinger 1419ae41b1 Add Compressor::MaybeOverrideCompressionOptions() omnibus hook (#15177)
Summary:
Compressor::GetRecommendedParallelThreads() was a narrow hook for the one CompressionOptions field (parallel_threads) that the SST builder, rather than the Compressor, is responsible for applying. Other such options exist (max_compressed_bytes_per_kb, auto_skip, auto_skip_min_sample_every), and more may be added. This replaces the narrow hook with an omnibus MaybeOverrideCompressionOptions(CompressionOptions*) that a Compressor can override to steer any of them. GetRecommendedParallelThreads() is now deprecated; the default implementation of the new method still honors an overridden GetRecommendedParallelThreads(), so existing custom Compressors keep working. CompressorWrapper forwards the new method; built-in compressors are unchanged (picked up via the default bridge).

The block-based table builder now consults the data-block compressor (falling back to the base compressor before the data-block compressor exists, e.g. during dictionary sampling, where parallel compression does not start anyway) and applies the possibly-overridden options. While here, the builder records the effective (post-override) options in the rocksdb.compression_options table property instead of the pre-override configured options, and the two affected Rep members (compression_parallel_threads, max_compressed_bytes_per_kb) are set exactly once from the effective options -- with poisoned defaults so any use before they are computed surfaces -- rather than initialized in the member list and overwritten later.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15177

Test Plan:
Added DBCompressionTest.CompressorOverridesCompressionOptions, which installs a Compressor that overrides MaybeOverrideCompressionOptions() to force parallel_threads and verifies via the builder's sync point that parallel compression starts with the overridden thread count, then reads the data back. The existing GetRecommendedParallelThreads and
ConfiguredCompressionTypeRecordedInProperties tests continue to cover the deprecated bridge and the table-property recording respectively.

Reviewed By: joshkang97

Differential Revision: D118656406

Pulled By: pdillinger

fbshipit-source-id: 8a5d7cde90ca38913525a7ac11650607fa0bce58
2026-09-15 10:38:56 -07:00
Peter Dillinger db114777b1 Optimize Slice::difference_offset with word-wise shared-prefix scan (#15198)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15198

Reviewed By: cnli87

Differential Revision: D119236575

fbshipit-source-id: e8bd20b0a1bbc04414883a08c5463b4087d08e3d
2026-09-14 14:10:13 -07:00
zaidoon 7a90dc5ba8 Add built-in index factory wrappers (#14961)
Summary:
Let the built-in binary, hash, and partitioned indexes be created through `IndexFactory`. The wrappers preserve the existing internal-key handling and multi-block finish path.

Pass the common-prefix flag to every builder, including both levels of a partitioned index. The regression test checks the prefix encoding and decoded keys with the flag on and off.

The factory tests pass on the original PR plus the fix. Focused tests on the full stack also pass with `ASSERT_STATUS_CHECKED=1` and `COERCE_CONTEXT_SWITCH=1`. Full `make check` runs had test timeouts and hit the 30-minute local limit.

Part 4 of 9. Previous: https://github.com/facebook/rocksdb/issues/14960. Next: https://github.com/facebook/rocksdb/issues/14962. The first three parts have landed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14961

Reviewed By: jaykorean

Differential Revision: D119493650

Pulled By: xingbowang

fbshipit-source-id: 7d43cc456de86e8bd35e4b0689fb3d256608cf2f
2026-09-11 16:37:12 -07:00
Xingbo Wang a4e298b8c4 Validate block entry bounds in release builds (#15200)
Summary:
Why this change is required:

Block iterators parse persisted SST bytes. Truncated headers, malformed restart arrays, and invalid key or value lengths are runtime corruption, not private control-flow failures. They must be rejected safely in release builds before any out-of-range read, pointer arithmetic, or Slice construction.

Previous behavior:

DecodeEntry relied on an assertion before reading its three-byte fast-path header. Strict entry-span validation was enabled only for selected meta-block paths. Restart offsets were not all validated up front, and several separated-key/value bounds assumptions were asserted only after pointers or slices had already been constructed. Because assertions disappear under NDEBUG, malformed blocks could trigger out-of-bounds access or incorrect iteration in release builds.

New behavior:

All block iterators now reject truncated entry headers, keys or inline values extending past the key section, invalid separated-value offsets or lengths, and malformed restart arrays with Status::Corruption. Block construction validates every restart offset for section bounds and strict ordering before an iterator can use it.

Implementation:

Remove the StrictCheck template distinction and apply the span checks to every BlockIter. Validate restart keys before constructing their Slice, validate separated values before pointer arithmetic, and scan the restart array once during Block construction. Mark corruption-only branches UNLIKELY and combine the inline key/value span into one overflow-safe uint64_t comparison so valid data remains on the compiler hot path.

Safety boundaries and unsupported cases:

These checks are unconditional because they are required for safe parsing of persisted bytes; they are not gated by paranoid_checks or checksum verification. Existing assertions covering private iterator state and post-validation relationships remain. The change does not alter the SST format or repair corrupted blocks; it detects malformed blocks and reports corruption.

Performance:

Both revisions were explicitly cleaned and rebuilt with DEBUG_LEVEL=0, then measured on one pinned AMD EPYC Genoa CPU using 2 million 16-byte keys, 32-byte values, 4 KiB blocks, restart interval 16, and no compression. Hardware cycle counts are candidate versus parent, with lower being better:

    - Cached random reads: 47.68B versus 47.69B cycles (-0.03%).
    - Random seeks plus forward/reverse scans and batched MultiGet: 70.26B versus 69.96B cycles (+0.43%).
    - mmap random reads without block cache: 16.92B versus 17.19B cycles (-1.55%).
    - Separated-KV random plus sequential reads: 30.44B versus 31.20B cycles (-2.43%).

No tested workload regressed. These are targeted single-thread CPU microbenchmarks rather than a guarantee for every production workload.

Tests performed and known limitations:

The corruption tests cover truncated headers, entries extending past key or value sections, invalid separated-value offsets, and out-of-range or non-increasing restart offsets in both inline and separated-KV layouts. They verify detection rather than recovery or repair of a corrupted block.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15200

Test Plan:
Built block_test and ran the 14 BlockEntryCorruption and separated-KV corruption tests under ASSERT_STATUS_CHECKED=1 and for 100 repetitions with COERCE_CONTEXT_SWITCH=1. Ran make check-sources.

Ran make clean and built db_bench with DEBUG_LEVEL=0 for both this commit and its parent. Compared cached random reads, random seeks, forward and reverse scans, batched MultiGet, mmap reads without block cache, and separated-KV variants using the configuration documented above.

Reviewed By: pdillinger

Differential Revision: D119668019

Pulled By: xingbowang

fbshipit-source-id: 1594379acefa047f9598df341e675340831066ee
2026-09-11 13:36:51 -07:00
Ranjan Banerjee 576dc39899 Fix destination type check in lossless_cast for pointers (#15188)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15188

In the pointer branch of `lossless_cast`, the destination type check tested
`std::is_enum_v<To>` instead of `std::is_enum_v<ToDeref>`. Since `To` is a
pointer type in that branch, `std::is_enum_v<To>` is always false, so the
check collapsed to `std::is_integral_v<ToDeref>` and rejected casting to a
pointer-to-enum, e.g. `lossless_cast<MyEnum8*>(&some_uint8)`.

Reviewed By: xingbowang

Differential Revision: D119422904

fbshipit-source-id: b1f4e205d536ba3d10563fa62adb04867fae02ac
2026-09-11 08:31:56 -07:00
Xingbo Wang 4c2a977aa5 Track newer snapshot versions in RocksDB metadata reads (#15086)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15086

Why: An application built on top of RocksDB could use cache population reads through explicit snapshots and needs to know when the returned value has already been superseded, without a second point read.

Previous behavior: Get and MultiGet returned the snapshot-visible value and optional timestamp but could not report a newer point update or covering range deletion.

New behavior: experimental C++, C, and Java metadata-read APIs report newer committed puts, deletes, merges, and covering range tombstones; ordinary primary-DB writes completed before the read are guaranteed to be reported, racing writes remain best-effort, and true has no false positives.

Implementation: metadata reads widen lookup behind read-callback hooks while keeping metadata-only state off passive common object layouts; per-key state stays in KeyContext/GetContext, ordinary MultiGet row-cache replay semantics are preserved, BlobDB/TTL use shared output helpers, dead single-CF plumbing and an invalid cross-key stress assertion are removed, and the range-tombstone ordering invariant is documented without changing behavior.

Safety and unsupported cases: tracking is opt-in and requires an explicit snapshot; persisted-tier reads, custom transactional visibility callbacks, secondary DBs, and unsupported subclasses return NotSupported, while read-only/compacted DBs return false because their in-process views are static.

Tests and limitations: 1,265 Buck tests plus focused metadata, 100x coerced-context-switch row-cache, write-committed/write-prepared crash tests, builds, and lint passed. Current-parent release db_bench found no material passive regression: readrandom +0.65%, MultiGet batch 4 -1.40%, and batch 32 +0.48%.

Reviewed By: pdillinger

Differential Revision: D104182397

fbshipit-source-id: 36f7be1cd50fe2383689adaae31a6bb7d7e462dc
2026-09-11 04:44:59 -07:00
Omkar Gawde 88a8fdf849 Compare excluded-file lookup against the engine it searched (#15197)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15197

**What**: Compare the excluded-file lookup iterator against the engine it was obtained from, in `BackupEngineImpl::RestoreDBFromBackup()`.

**How**: The loop walks `locked_restore_from_dirs` calling `find()` on each engine's `backuped_file_infos_`, but tested the result against `this` engine's `end()`. Test against the same engine's `end()`.

**Why**: Comparing iterators obtained from different containers is undefined. It is inert here rather than a live bug: on libstdc++ and libc++ an `unordered_map` end iterator is a null sentinel that is not container-specific, so a miss compares equal and a hit does not. On an implementation with a container-specific end iterator - MSVC, a required CI axis - a miss would read as found and dereference the sentinel node. No test can pin this; see the test plan.

Reviewed By: xingbowang

Differential Revision: D119585045

fbshipit-source-id: 9ff0e770b426f6a2c4a0fdad261910a927d1bfa7
2026-09-11 04:35:03 -07:00
Josh Kang a844cbf5b7 Add coroutine reads for SstFileReader (#15189)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15189

Add `CoroDB::CoGet` and `CoMultiGet` overloads for standalone `SstFileReader` instances. Reuse a shared sync/coroutine implementation on the private reader `Rep`, configure and validate the read executor at `Open`, and retain synchronous fallback when unavailable.

Reviewed By: xingbowang

Differential Revision: D119431444

fbshipit-source-id: 1ac1926b6a2ffc11cd3e91a438a8c62cff5d57ec
2026-09-10 14:56:12 -07:00
Omkar Gawde 3d8e4be2f5 Fix SstFileReader heap corruption past MAX_BATCH_SIZE keys (#15196)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15196

**What**: Fix heap corruption and out-of-bounds reads in `SstFileReader::MultiGet()` when the request exceeds `MultiGetContext::MAX_BATCH_SIZE` keys.

**How**:
- Batch the lookup in `MAX_BATCH_SIZE` slices over the sorted keys, with a fresh `MultiGetContext` and `TableReader::MultiGet()` call per slice. Caller ordering survives because each `KeyContext` already points into the caller's status and value vectors.
- `reserve()` the `key_context`, `get_ctx` and `merge_ctx` autovectors, as `DBImpl::MultiGetCommon` does: the setup loop takes addresses of their elements, and past its inline capacity an `autovector` spills into a `std::vector` that reallocates as later keys are appended.
- Only the slicing is borrowed from `DBImpl::MultiGetImpl`. Its cross-batch value-size accumulation and per-batch deadline check are deliberately not replicated: value size is accumulated only under `db/` and never by a `TableReader`, and the deadline already reaches every read through the table reader's I/O options.

**Why**: `MultiGetContext` caps at `MAX_BATCH_SIZE` keys but only asserts it, so release builds hit a bad `delete[]` from its mismatched `LookupKey` construct/destroy bounds, an out-of-bounds read of its fixed-size `sorted_keys_` array, and an out-of-range shift once the key count exceeds the width of its range mask; `include/rocksdb/sst_file_reader.h` documents no key-count limit, so any caller of this public API could reach it.

Reviewed By: xingbowang

Differential Revision: D119401930

fbshipit-source-id: 3bbafd72badaa944dbb94f4ab6ecb7f14a11f6d0
2026-09-10 12:18:35 -07:00
Xingbo Wang 7d3150ffce Report RocksDB benchmark interval latency and perf metrics (#15101)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15101

This diff contains all changes under fbcode/internal_repo_rocksdb/repo for the RocksDB regression benchmark observability work.

It adds opt-in per-interval latency percentile reporting to db_bench. When --report_interval_percentiles is enabled with histogram and a positive stats interval, each worker records into pre-created per-operation interval histograms and counters. A background reporter periodically merges and resets those thread-owned stats, skips excluded background-writer stats, performs a final partial-interval flush, and emits IntervalPercentiles lines with sample count, operation count, and P50/P75/P99/P99.9/P99.99 latency. Worker updates and reporter snapshots use a double-buffer handoff, so the reporter never reads or clears a histogram still being updated. The legacy histogram timing path remains unchanged when interval reporting is disabled.

The interval implementation uses RocksDB UnorderedMap aliases for the hot lookup maps and UnownedPtr for non-owning reporter references.

It also aggregates each worker thread PerfContext and IOStatsContext into the final merged benchmark Stats and emits final PERF_CONTEXT and IOSTATS_CONTEXT blocks, so multi-thread benchmark metrics represent all workers rather than one thread.

For options-file runs, explicit benchmark-critical CLI overrides are applied after loading the file. This includes --disable_auto_compactions, fixing an existing precedence bug where disable_auto_compactions=false in the options file silently overrode an explicit CLI true value. The regression harness relies on that CLI setting, so the bug allowed automatic compactions during origin construction and write-heavy benchmarks. The regression test persists false in an options file, passes --disable_auto_compactions=true, reloads the opened DB options, and verifies the effective value is true.

One shared WriteBufferManager is created after option loading and reused by all DBs, enforcing the intended aggregate memory cap for multi-DB benchmarks.

The compactall benchmark now accepts --compact_all_parallelism. It compacts independent DBs through a bounded worker set, applies --subcompactions to each DB, emits per-DB progress and elapsed time, and propagates the first CompactRange failure after joining all workers. Its default parallelism remains 1 for backward compatibility.

The dependent harness diff enables these features, parses their output, and publishes the resulting interval and final metrics.

Reviewed By: pdillinger

Differential Revision: D116025321

fbshipit-source-id: d44185ba2110aacbe9fadc65a59799bf8c1c54f0
2026-09-10 08:08:38 -07:00
Jay Huh 91b96f41cf Retry the OpenAndCompact secondary open when the primary replaces CURRENT (#15179)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15179

A remote compaction worker (`DB::OpenAndCompact()`) opens the *live* primary's DB directory read-only, and that open reads `CURRENT`. The primary publishes each MANIFEST rotation by writing a new `CURRENT` under a temp name and renaming it over the old `CURRENT`. POSIX keeps an already-opened `CURRENT` readable after that replacement, but file systems without read-after-unlink semantics (remote or object stores) do not: the replaced object is reported as gone, surfacing in RocksDB as `IO error: No such file or directory` (`Status::PathNotFound`). The secondary open was not retried, so this transient race failed the whole compaction. There is an equivalent, already-recognized race one step later for the MANIFEST file (`ReactiveVersionSet::MaybeSwitchManifest()` converts it into `Status::TryAgain`), which was fatal here for the same reason.

`DB::OpenAndCompact()` now retries the secondary open (up to 3 attempts) when it fails with `TryAgain` / `PathNotFound` / `NotFound` -- exactly the statuses a concurrent `CURRENT`/MANIFEST replacement produces. The retry limit is configurable via `OpenAndCompactOptions::max_secondary_open_retries`. Genuine failures are still returned after the bounded retries, and no safety check is weakened or skipped.

Found by a `db_stress` crash test failure. Two non-default options in that run make this reachable and fatal:
* `remote_compaction_worker_threads=8` with `remote_compaction_failure_fall_back_to_local=0` -- the latter was only recently randomized to 0; before that a remote compaction failure was always silently retried locally. Now `DbStressCompactionService::Wait()` maps the failure to `kFailure`, which becomes a sticky background error on the primary, so every subsequent `Flush()` returns it and `StressTest::ProcessStatus()` aborts with `assert(false)`.
* `max_manifest_file_size=8192` -- the primary rotates MANIFEST (and therefore replaces `CURRENT`) constantly, which is what makes the race window hit in practice.

Full chain: primary renames a new `CURRENT` over the old one -> the file system reports the replaced object as gone for the reader that already resolved it -> `GetCurrentManifestPath()` inside the worker's secondary open fails with `IO error: No such file or directory` -> `DB::OpenAndCompact()` returns that error with no retry -> `kFailure` (no local fallback) -> primary background error -> `Flush failed: IO error: No such file or directory: ... /CURRENT ...` -> assertion failure in `db_stress`.

Reviewed By: joshkang97

Differential Revision: D118480192

fbshipit-source-id: a2618ee24e582e2e713af04464ee1e4038925ccb
2026-09-09 19:32:00 -07:00
Omkar Gawde dfde6a60b0 Fix out-of-bounds read in SstFileWriter::DeleteRange
Summary:
**What**: Fix an out-of-bounds read and a forged end timestamp in the timestamped `SstFileWriter::DeleteRange()` overload.

**How**:
- `Rep::DeleteRange()` now builds each boundary through a new `Rep::AppendTimestamp()` helper, which tests one key's adjacency to `timestamp` and either aliases the two or materializes `key + timestamp` into a caller-owned buffer.
- Drops the fast path's `assert`. It was never a guard: it performed the same out-of-bounds read it was meant to police, and `DeleteRangeImpl()` already asserts the two boundary timestamps match.
- Behavior change for callers whose two keys each carry their own trailing timestamp: they now take one extra copy. Aliasing is still used for whichever key is genuinely adjacent.

**Why**: A single `timestamp` buffer can be byte-adjacent to at most one of two distinct keys, but the old branch tested `end_key` using `begin_key`'s length and then widened both slices whenever either test passed - so an asymmetric caller, one key contiguous with the timestamp and the other bare, read past an allocation and persisted a range tombstone carrying foreign heap bytes as its end timestamp. Nothing forbids that layout: the public contract requires only that the keys carry no timestamp.

Reviewed By: xingbowang

Differential Revision: D119401932

fbshipit-source-id: 4c842d4afbe7bd7c4eabc68c6aee0543cdee2c9c
2026-09-09 19:26:02 -07:00
Peter Dillinger bcda8c4b1a Fix Cache::ApplyToAllEntries ignoring average_entries_per_lock (#15186)
Summary:
`ShardedCache::ApplyToAllEntries()` clamped `average_entries_per_lock` with `std::min(aepl, 1)`, forcing every positive value (including the default 256) down to 1. The intent was a floor of 1 (`std::max`); `ApplyToSomeEntries` even asserts the value is > 0. As a result a full cache traversal advanced a single hash-table slot per iteration step instead of batching ~256, so the documented 256 default never actually took effect. The bug dates to the original API (https://github.com/facebook/rocksdb/issues/8225) and was carried through the ShardedCache refactor (https://github.com/facebook/rocksdb/issues/10801) unchanged, so the intended default was never exercised: in hindsight the "gather stats" latency in the https://github.com/facebook/rocksdb/issues/8225 test plan is a batch-size-1 measurement.

Despite its name, `average_entries_per_lock` is really a batch size for iterating the cache, not something specific to locking: it bounds the work done per uninterrupted step. For a lock-based cache (LRUCache) a shard mutex is held for each batch, so single-entry batches mean many extra lock acquire/release cycles; but lock-free implementations (HyperClockCache) also pay fixed per-batch overhead and benefit from the fix as well. This is a control-path cost (e.g. periodic block cache stats via DumpStats -> CollectCacheEntryStats), not hot-path read/write, so the practical impact is usually minor. The option's doc comment is updated to describe it as a batch size; the field is not renamed given how obscure the option is.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15186

Test Plan:
No unit test: asserting on batch/lock-acquisition counts would be intrusive/awkward, and LRUCache is disfavored (though not deprecated).

Measured with release-mode cache_bench, comparing the binary built before (`std::min`) vs after (`std::max`); 4 GiB cache, 1 KiB values, ~3.7M entries, default gather_stats_entries_per_lock=256. Average full ApplyToAllEntries traversal latency:

| cache_type              | before  | after   | speedup |
|-------------------------|---------|---------|---------|
| lru_cache               | 2.69 s  | 0.64 s  | ~4.2x   |
| fixed_hyper_clock_cache | 0.441 s | 0.138 s | ~3.2x   |
| auto_hyper_clock_cache  | 0.264 s | 0.136 s | ~1.9x   |

All implementations improve; LRUCache most, due to per-batch mutex acquire/release. HCC traversals are already ~6-10x cheaper than LRU in absolute terms, and entries visited are identical before and after.

Reviewed By: rban1

Differential Revision: D119256456

Pulled By: pdillinger

fbshipit-source-id: e58f7f97d47d8970ebc8c3ab786199132ce6340d
2026-09-09 09:41:06 -07:00
Josh Kang 37234200b5 Handle benign io_uring stderr in crash tests (#15181)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15181

A crash test failed when `use_async_db_api=1` created read-executor workers on a host unable to allocate io_uring rings. Folly logged the setup failure and threw; RocksDB caught it and successfully fell back to synchronous reads, but the harness failed because stderr was non-empty.

Representative stderr:

```
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0903 14:05:39.470695 286445 IoUringBackend.cpp:527] io_uring_queue_init_params(512,1024) failed errno = 0:"Success" 0x7d2e06295280 retrying with capacity = 512
E0903 14:05:39.470759 286445 IoUringBackend.cpp:537] io_uring_queue_init_params(512,512) failed ret = -12:"Unknown error -12" 0x7d2e06295280
```

Initialize glog in the Folly-enabled OSS `db_stress` entry point and centralize known stderr sanitization. Only `IoUringBackend` ring-initialization failures are moved to stdout when the async DB API is enabled; existing expected post-SIGTERM errors use the same sanitizer. All unrelated stderr remains fatal.

___

Differential Revision: D118836521

fbshipit-source-id: c9ce05896f417c7723c5cd73d293fbb1cd14ee1e
2026-09-04 12:22:30 -07:00
Josh Kang ac0f4ae69f Isolate coroutine statistics across async reads (#15142)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15142

Coroutine statistics use TLS, but coroutine reads can interleave on the same executor thread. Previously, when a stats-enabled request suspended, it saved its counters without disabling the executor's TLS configuration. A stats-disabled request running next could inherit those enabled settings and collect statistics unexpectedly. Stats setup also lived in individual DB implementations, so wrapper early returns and future stackable DB implementations could bypass it.

Move statistics ownership to the public `CoroDB` and callback-based async read boundaries, before virtual dispatch. Each operation captures its caller's configuration, enabled requests preserve their counters across suspensions, and every suspension or completion leaves executor TLS disabled. Because the scope wraps stackable DB dispatch, current and future `CoroStackableDB` implementations automatically inherit the correct behavior without adding their own stats reset or scope.

For consistency between coroutine and callback-based reads, the experimental callback API now uses TLS for both configuring and consuming statistics. `AsyncCallback::OnComplete()` no longer receives context arguments; RocksDB publishes the completed per-operation counters to TLS before invoking it.

```
Before

time -------------------------------------------------------------->

stats-on A:   onSet(enable A) -- work -- onUnset(save A) ...... onSet(A)
executor TLS: [A enabled] ------------> [still enabled] ------> [A enabled]
stats-off B:                                  run
                                               ^ inherits A's enabled TLS

After

time -------------------------------------------------------------->

stats-on A:   onSet(enable A) -- work -- onUnset(save A, disable) ... onSet(A)
executor TLS: [A enabled] ------------> [disabled] ----------------> [A enabled]
stats-off B:                                  run
                                               ^ remains stats-disabled
```

Reviewed By: xingbowang

Differential Revision: D117017089

fbshipit-source-id: 8a85f7ba43e3b73225995f5d0b79da3892baafb9
2026-09-04 11:19:16 -07:00
Jay Huh bec9dcec7c Make db_stress actually tolerate transient remote IO errors (#15170)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15170

`warm_storage_tsan_liveness_crash_test` fails with

```
error : Get() returns IO error: Failed to read 1032 bytes from file
ws://ws.zdb.ldc0zdbtest1/.../000093.sst from offset 5461, message
[0 code=DEADLINE_EXCEEDED isRetryable=1 ... error=WSE_BLOCK_READ_TIMEOUT
classification=TRANSIENT_ERROR ...] for key: 0000...78 (30280).
Verification failed :(
```

The Warm Storage error is explicitly `classification=TRANSIENT_ERROR isRetryable=1`, and the DB artifact confirms there is no RocksDB problem: `000093.sst` is intact (37760 bytes, downloaded successfully after the run) and the DB `LOG` reaches `Shutdown complete` with zero IO-error or corruption entries. This is exactly the class of failure `--tolerate_non_injected_io_errors_for_remote_dbs` was added for (D112136438), but that mechanism never actually took effect. Two independent defects:

**1. The triggering non-default option: `tolerate_non_injected_io_errors_for_remote_dbs=0` on a remote DB.**
`db_crashtest.py` hard-coded the parameter to `0` in `default_params` and only ever *force-disabled* it (for local DBs), leaving each remote job to opt in on its own command line. `rocks/sandcastle/test-driver.sh`'s `warm_storage_*crash*` branch never passed it, so every Warm Storage crash-test job ran with the flag off and a single transient read timeout failed the run. Fix: the parameter now defaults to `1` whenever `--env_uri` / `--fs_uri` names a remote backend, so no remote job has to remember to opt in. A caller can still pass `0` explicitly, and `finalize_and_sanitize()` still force-disables it for local DBs so genuine local IO errors (e.g. io_uring failures) are never masked.

**2. `injected_error_count == 0` made the flag a no-op in `no_batched_ops_stress.cc`.**
All five read-path guards were
`injected_error_count == 0 || !IsErrorInjectedAndRetryable(s)`.
`IsErrorInjectedAndRetryable()` covers *non-injected* remote IO errors, which by definition have no injected fault to correlate with — and the `liveness` profile forces every `*_fault_one_in` to `0`, pinning `injected_error_count` to `0` for the whole run. So the left operand short-circuited to true and the error was reported as a verification failure regardless of the flag. Fix: extract `StressTest::IsTolerableNonInjectedRemoteIOError()` and add `StressTest::IsRetryableOperationError(injected_error_count, s)`, which gates the injected-error branch on `injected_error_count > 0` but lets a tolerated non-injected remote IO error through unconditionally. The five call sites now use the helper.

With the tolerate flag off (all local runs), the new guard reduces algebraically to the old expression, so local behaviour is unchanged. The genuine consistency-check gate at `no_batched_ops_stress.cc:1146` (`do_consistency_check && injected_error_count == 0`) is intentionally left alone.

Fixes T286575055.

[Session trajectory link](https://www.internalfb.com/intern/devai/devmate/inspector/?id=9646c050-6791-4ac3-9abb-3ee110dcb54e)

Reviewed By: pdillinger

Differential Revision: D117886823

fbshipit-source-id: d7591472db8ee6e03c9d9f018480409b6d802838
2026-09-03 10:43:54 -07:00
Jay Huh 897162a6bc Add session-scoped temporary directory for remote compaction (#15169)
Summary:
Remote compaction writes output SSTs into caller-selected job directories. If the primary crashes before installation or falls back to local compaction, those directories can outlive the DB session and accumulate under the DB directory.

This change defines `kSessionTmpDirName` as `session_tmp`, the canonical name for a DB-local temporary directory whose contents have the lifetime of a DB session. The immutable boolean option `use_session_tmp_dir_for_remote_compaction` opts remote compaction into this directory. When enabled, an `OpenAndCompact()` directory name is resolved as `<dbname>/session_tmp/<job>`.

During DB open, RocksDB creates the session temporary directory, completes recovery, and then synchronously deletes contents left by the previous DB session before scheduling compactions. `DestroyDB()` removes it as part of database deletion, and normal completion, failure, and cancellation paths clean the resolved per-job directory.

The option defaults to false. Disabled callers retain the existing `<dbname>/<job>` layout and no session-directory cleanup. For rollout compatibility, legacy callers that already pass a DB-rooted output path are accepted unchanged only while the option is false. Directory creation and invalid-path errors are delegated to the configured filesystem rather than prevalidated by RocksDB.

After D118301531 landed, per-job directory creation and resumption cleanup are kept together in `InitializeCompactionWorkspace()`. `OpenAndCompact()` resolves the client-provided name and expects DB open to have initialized the session temporary directory when the feature is enabled.

The boolean is wired through C++ and generated C APIs, option serialization and reconstruction, `OpenAndCompact()`, ldb, db_bench, and db_stress. Tests cover path resolution, cross-session deletion, ownership boundaries, disabled behavior, `DestroyDB()`, configured compaction output, resumption fallback, and option round trips.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15169

Reviewed By: pdillinger

Differential Revision: D118227112

fbshipit-source-id: 01aa834b0b3f75638102130a86d95d537a2797bd
2026-09-02 21:46:24 -07:00
rocksdb-release-bot 1ca38e1442 Update version to 11.11.0 for 11.10 release (#15157)
Summary:
- Bump version.h from 11.10.0 to 11.11.0
- Add `11.10.fb` to `check_format_compatible.sh`
- Sync HISTORY.md `## 11.10.` section from `11.10.fb`
- Delete 13 consumed `unreleased_history/` note file(s) from main

Part of 11.10 release workflow.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15157

Reviewed By: jaykorean

Differential Revision: D118302742

Pulled By: xingbowang

fbshipit-source-id: 751fb106cc5a4f5dd502c5838c5653278b5a8529
2026-09-02 09:07:42 -07:00
Josh Kang da3a0961fa Add custom read context to ReadOptions (#15173)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15173

Add an opaque, non-owning `ReadOptions::custom_context` pointer that is passed to table implementations so external table readers can consume application-defined read options. Document its lifetime and synchronization contract, and keep the unsupported pointer out of generated C bindings.

Reviewed By: xingbowang

Differential Revision: D118341098

fbshipit-source-id: e51185f58b5ce80d8fd07efab02a05dbc8c7bd12
2026-09-01 18:51:16 -07:00
Josh Kang 96ed085402 Blog post for coroutines (#15143)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15143

## Summary

Adds a RocksDB blog post describing the native async/coroutine read APIs introduced in RocksDB 11.10.0. It covers the callback and `folly::coro::Task` interfaces, differences from `ReadOptions::async_io`, executor and I/O-completion handoffs, filesystem integration, request-local statistics across suspension, and API lifetime constraints.

The post includes three explanatory diagrams and benchmark results comparing synchronous and coroutine point reads across task counts and `MultiGet` batch sizes.

https://www.internalfb.com/intern/px/p/cDGqv/

Reviewed By: xingbowang

Differential Revision: D117159178

fbshipit-source-id: 5452e6b07e29abb5594505021995e9b5bac9d198
2026-09-01 15:55:37 -07:00
Peter Dillinger 2c1c7ea2f1 Surface remote cx crash test failures and fix resumption output-dir cleanup (#15168)
Summary:
Follow-up to the "UPDATE" note on D116738823 (PR https://github.com/facebook/rocksdb/issues/15133, "Faster and safer MANIFEST recovery for remote compaction (OpenAndCompact)"). That change observed that the crash test does not really cover remote compaction because `remote_compaction_failure_fall_back_to_local` is pinned to 1 in db_crashtest.py, so any remote compaction failure is silently retried locally instead of failing the test -- masking existing failures. Addressing that was called out as a high-priority follow-up (tracked under T232588967, "Rocksdb Crash Test failed: Remote Compaction").

This does the cited enablement and fixes the first masked failure it surfaces:

* Crash test: randomize `remote_compaction_failure_fall_back_to_local` between 0 and 1 (the TODO(jaykorean)), so remote compaction failures actually surface instead of always being hidden by a local fallback.

* Fix a real OpenAndCompact bug the enablement immediately exposed. When a compaction is scheduled with `allow_resumption=true` but resumption is internally disabled because it is incompatible with output-hash verification (`paranoid_file_checks=true`, or `verify_output_flags` requesting iteration verification), the worker started a fresh compaction but did NOT clean the output directory. The caller, having requested resumption, does not empty the directory either, so a compaction that was canceled after writing some output and then retried reused the same output file numbers and collided with the leftover files -- surfacing under the crash test's fault-injection FS as `NewWritableFile violates no-reopen-for-write contract`, which became a background error and failed subsequent writes. The fix honors the documented `OpenAndCompactOptions::allow_resumption=true` fallback contract: when resumption is requested but cannot be fulfilled, clean any leftover progress and output files before starting fresh (reusing the existing Scan/Cleanup helpers), so the fresh compaction always begins from a clean directory. When `allow_resumption=false`, the caller still owns emptying the directory (unchanged).

No known production exposure: this bug can only bite a caller that requests `allow_resumption=true` on a column family where output-hash verification is enabled (`paranoid_file_checks=true`, or `verify_output_flags` including `kVerifyIteration`), because only that combination disables resumption while the (uncleaned) output directory is left to the caller. Auditing how the option is used across internal deployments found nothing in that state: the deployments that enable resumption for remote compaction do not enable those iteration/paranoid verification modes, and where remote-compaction output verification is enabled at all it is block-checksum-only, which keeps resumption active and never hits the buggy path. So this is effectively a latent-bug fix -- it prevents a failure that would appear if iteration/paranoid output verification were later turned on for a resumption-enabled remote-compaction fleet.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15168

Test Plan:
Added compaction_service_test.cc unit test
`CleansOutputDirWhenResumptionDisabled`, which drives OpenAndCompact with `allow_resumption=true` and `paranoid_file_checks=true` after seeding a leftover output SST, and asserts the leftover is cleaned and the compaction succeeds. It was confirmed to FAIL without the fix (leftover retained) and pass with it.

Local crash test runs (blackbox, `remote_compaction_worker_threads=8`, `remote_compaction_failure_fall_back_to_local=0`):
- Without the fix, the collision-trigger config (`allow_resumption_one_in=1 --paranoid_file_checks=1`) fails within the first ~60s iteration on the no-reopen-for-write contract violation.
- With the fix, that same config ran ~10 min clean, and a full-random config (everything else randomized) ran ~15 min clean, both with 10-15 kill/reopen cycles plus a final crash-recovery verification. More local crash test runs are planned to shake out any remaining, rarer masked failures now that the fallback no longer hides them.

Reviewed By: jaykorean

Differential Revision: D118301531

Pulled By: pdillinger

fbshipit-source-id: ceb9683ae06ce930ea78bafa3585ad616012243f
2026-09-01 12:05:27 -07:00
Anand Ananthabhotla fe6406f7b2 Skip recovery SetNextFile for OPTIONS-only advances (#15163)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15163

Reintroduces the `optimize_manifest_for_recovery` optimization for `OPTIONS-*` and temp files without letting `VersionSet::NewFileNumber()` collide with files already present in the DB directory.

Recovery now distinguishes between files that require a persisted `SetNextFile` edit and files that only need the in-memory allocator reserved. If only `OPTIONS-*` or temp files are above the recovered `next_file_number_`, recovery advances `next_file_number_` in memory and still skips the MANIFEST edit.

Reviewed By: jaykorean

Differential Revision: D117887276

fbshipit-source-id: f211fa90303904befdf331242ba0d97553f84169
2026-08-31 20:25:51 -07:00
Anand Ananthabhotla 0df7ed563b Pass SST block CRC32C checksums to file writer (#15155)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15155

SST block writing now hands checksum coverage across the table-builder to file-writer boundary when the block checksum algorithm is CRC32C.

The block builder passes the raw CRC32C of the block contents to `WritableFileWriter::Append()`, which validates the copied destination-buffer contents and accumulates that checksum into the filesystem handoff checksum instead of recomputing it after the copy. The trailer is still appended separately; table building computes and passes the trailer handoff checksum independently, while preserving the existing persisted block-checksum encoding. Non-CRC32C block checksum algorithms continue to use the existing file-writer checksum path.

Destination-buffer validation failures return a stage-specific corruption `Status` that includes the file path. Flush and compaction propagate that status through their existing background error handling, so the same message is logged without adding local duplicate log sites.

The SST tests record the builder append stream through sync-point callbacks while writing to a null sink, then replay those exact append calls into a file-backed writer to cover successful handoff, block-content corruption, compression-type corruption, trailer-checksum corruption, non-CRC32C fallback behavior, and a small randomized boundary-injection loop.

Reviewed By: xingbowang

Differential Revision: D116969191

fbshipit-source-id: f75dcb37936106b7f6f14cf1ed999c81a0879d7b
2026-08-31 17:20:01 -07:00
Xingbo Wang 068aeda9d8 Account partitioned filter reads in PerfContext timing (#15164)
Summary:
Instrument partitioned Bloom-filter block retrievals with read_filter_block_nanos so PerfContext includes time spent looking up these blocks in the block cache or reading them from SST files.

Root cause:
The top-level filter-partition index is read through ReadFilterBlock, which already owns the timing guard. Individual filter partitions bypass that helper and call RetrieveBlock directly. Once the top-level index is cached, workloads can therefore report filter block reads and cache hits while read_filter_block_nanos remains zero.

Implementation:
Place the timer at the partition retrieval boundary, after the pinned filter-map fast path and immediately around RetrieveBlock. This keeps the metric aligned with actual cache or SST retrieval work without charging accesses to already-pinned partition objects.

Testing:
Extend the partitioned MultiGet test to enable time-level PerfContext collection and verify that partition reads populate read_filter_block_nanos across whole-key and prefix-filter configurations and table format versions.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15164

Test Plan:
- `make check-sources`
- `build_tools/rocksptest.sh db_bloom_filter_test`
- `ASSERT_STATUS_CHECKED=1 build_tools/rockstest.sh db_bloom_filter_test --gtest_filter=*PartitionedMultiGet*`

Reviewed By: rban1

Differential Revision: D118052826

Pulled By: xingbowang

fbshipit-source-id: e4467b107ba13e5cfe4826a8a93834b319a131a5
2026-08-31 14:49:48 -07:00
Xingbo Wang a0966fd94e Validate remote compaction snapshot ordering (#15160)
Summary:
Remote compaction passes snapshot sequence numbers through serialized `CompactionServiceInput`. Downstream compaction logic assumes the sequence numbers are strictly increasing when selecting the earliest snapshot and using `std::lower_bound()` for snapshot visibility. Previously, malformed input could bypass the debug-only ordering assertion in release builds and select an incorrect visibility boundary.

- Validate decoded snapshot sequence numbers before remote compaction setup and return `InvalidArgument` for duplicate or descending values.
- Document the strict ordering contract while retaining the `CompactionIterator` assertion as defense in depth for internally constructed inputs.
- Canonicalize randomized serialization-test snapshots and add coverage for malformed snapshot lists through `DB::OpenAndCompact()`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15160

Test Plan:
- Built `compaction_job_test` and ran `CompactionJobTest.OpenAndCompactRejectsUnorderedSnapshots` in the normal configuration.
- Ran the focused test with `ASSERT_STATUS_CHECKED=1`.
- Ran the focused test for 100 repetitions.

Reviewed By: rban1

Differential Revision: D117888826

Pulled By: xingbowang

fbshipit-source-id: 2312205acfe17271c2083808d372c93a5914114c
2026-08-31 12:56:55 -07:00
Ranjan Banerjee fe5123681e Fix GetEntity snapshot isolation for WritePrepared transactions (#15136)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15136

`WritePreparedTxnDB` overrides `Get` and `MultiGet` to install a
`WritePreparedTxnReadCallback` that performs visibility checking for
prepared-but-uncommitted data. However, `GetEntity` (both the single-CF
`PinnableWideColumns` variant and the multi-CF `PinnableAttributeGroups`
variant) was never overridden, causing it to fall through to
`StackableDB::GetEntity` which delegates to the inner `DBImpl` without
any WritePrepared visibility callback.

This means `GetEntity` could read prepared-but-uncommitted data, violating
snapshot isolation. The bug surfaces in the crash test with the combination:
`--use_txn=1 --txn_write_policy=1 --use_attribute_group=1 --use_get_entity=1
--unordered_write=1 --two_write_queues=1 --test_batches_snapshots=1`

The fix adds `GetEntity` overrides to `WritePreparedTxnDB` that create a
`WritePreparedTxnReadCallback` and pass it via `GetImplOptions::callback`,
matching the pattern used by the existing `Get` override.

Reviewed By: xingbowang

Differential Revision: D116324596

fbshipit-source-id: faf636cfbf093f47a645c1a5d5037baa02ff594a
2026-08-31 12:45:16 -07:00
zaidoon f9c117e203 Add optional UDI builder protocols (#14960)
Summary:
Part 3 of 13 in the UDI split.

Stack order:
1. https://github.com/facebook/rocksdb/issues/14954 Add IndexFactory compatibility names
2. https://github.com/facebook/rocksdb/issues/14959 Add UDI index mode API vocabulary
3. https://github.com/facebook/rocksdb/issues/14960 Add optional UDI builder protocols
4. https://github.com/facebook/rocksdb/issues/14961 Add built-in index factory wrappers
5. https://github.com/facebook/rocksdb/issues/14969 Add built-in index factory tests
6. https://github.com/facebook/rocksdb/issues/14962 Promote IndexFactory as UDI SPI
7. https://github.com/facebook/rocksdb/issues/14963 Wire IndexFactory through block-based tables
8. https://github.com/facebook/rocksdb/issues/14964 Add IndexFactory table routing tests
9. https://github.com/facebook/rocksdb/issues/14970 Add IndexFactory parallel routing tests
10. https://github.com/facebook/rocksdb/issues/14965 Account UDI blocks as index blocks
11. https://github.com/facebook/rocksdb/issues/14966 Support parallel AddIndexEntry in trie index
12. https://github.com/facebook/rocksdb/issues/14967 Cover trie index modes
13. https://github.com/facebook/rocksdb/issues/14968 Add IndexFactory stress and benchmark flags

Previous: https://github.com/facebook/rocksdb/issues/14959.
Next: https://github.com/facebook/rocksdb/issues/14961.

Depends on https://github.com/facebook/rocksdb/issues/14959. Until https://github.com/facebook/rocksdb/issues/14954 and https://github.com/facebook/rocksdb/issues/14959 land, GitHub shows the cumulative diff against main. After those land, the intended review diff is:

`zaidoon/udi-02-index-mode-api..zaidoon/udi-03-index-builder-protocols`

That final review delta is 1 file with 62 insertions.

What changed:
- Add optional UserDefinedIndexBuilder protocols with default no-op behavior.
- Keep the API additive. Runtime wiring comes in later PRs.

Validation:
- `git diff --check`
- `make check-sources`

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14960

Reviewed By: jaykorean

Differential Revision: D117536541

Pulled By: xingbowang

fbshipit-source-id: affe3e49e7185af6cfdf85314bb6f2e002c84c08
2026-08-31 09:26:43 -07:00
xingbowang 82dfd6a510 Fix Folly build due to libaio dead download link (#15159)
Summary:
Failures in GitHub Actions CI workflows attempting to download assets from releases.pagure.org occur because the Pagure platform has been decommissioned. Maintainers must update their configurations to retrieve packages from active mirrors, alternative repositories.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15159

Reviewed By: rban1

Differential Revision: D117863179

Pulled By: xingbowang

fbshipit-source-id: e54e80aa6f1e2e0503dbd7ec16e67b86c4bad7ad
2026-08-28 09:01:50 -07:00
Anand Ananthabhotla 80a5a6e4b2 Revert PR 15106 (Ignore OPTIONS files for manifest recovery advance) (#15158)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15158

Reverts `https://github.com/facebook/rocksdb/pull/15106`, which ignored `OPTIONS-*` and temp files when deciding whether recovery must advance `next_file_number_`.

Crash-test failures show backup/checkpoint failing to copy the current `OPTIONS-*` file after options cleanup, because `DeleteObsoleteOptionsFiles()` ignores `next_file_number_` and `options_file_number_` when choosing which files to keep. The failing commands use `optimize_manifest_for_recovery=1`, so restore the previous conservative file-number reservation behavior while we investigate a narrower fix.

Reviewed By: xingbowang

Differential Revision: D117807052

fbshipit-source-id: 89f8656f045be4ac36d2ec851980573ed83370fe
2026-08-27 21:08:29 -07:00
Omkar Gawde d699ffbd5c Support partial atomic range replacement (#15154)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15154

**What**: Let `IngestExternalFileArg::atomic_replace_range` replace a half-open `[start, limit)` range that only partially overlaps existing SSTs, by committing a generated range tombstone alongside the replacement files.

**How**:
- Tombstone: `NumFilesToPrepare` reserves one extra file number and `PrepareAtomicReplaceRangeTombstone` writes the SST only when the pinned `Version` holds a straddling file (`HasPartialOverlap`), so the no-straddle case pays no extra write; run rescans and returns `TryAgain` if a straddler appeared since. On activation it is rotated to the front of `files_to_ingest_` so one `VersionEdit` lands `old < tombstone < data` seqnos, straddlers are retained, contained files metadata-deleted, and `generated_for_ingestion` keeps `move_files` from unlinking it.
- Half-open plumbing: `RangesOverlapWithMemtables`, `RangeOverlapWithCompaction`, `OverlapProximalLevelOutputRange`, and `IsRangeOverlapped` take a required limit-exclusivity argument, and `largest_internal_key` now comes from `RangeTombstone::SerializeEndKey()`. `OverlapWithLevelIterator` stays inclusive: over-counting can only push the tombstone one level shallower, which is safe.
- Eligibility: `SupportsAtomicReplaceRangeTombstone` requires universal compaction, a delete-range-capable table factory, `allow_global_seqno`, `!allow_db_generated_files`, `!snapshot_consistency`, and no UDT. `fail_if_not_bottommost_level=true` rejects partial overlap because the retained file forces the tombstone and replacements above Lmax. The `atomic_replace_range` doc comment and a new `unreleased_history/public_api_changes/` note state the half-open contract.
- `ResetTableReader` passes `kNoFileChecksumFuncName` in `FileOptions` for external files, making "no checksum metadata" explicit rather than empty.
- Stress: new `ingest_external_file_atomic_replace_one_in` gflag. `PickAtomicReplaceRange` picks a run of overwriteable keys strictly inside an existing SST so every replacement straddles, writes even-indexed keys and marks the interleaved odd-indexed keys deleted so the tombstone must clear the holes, and rolls back expected values on `TryAgain`. `db_stress_tool.cc` and `db_crashtest.py` gate the flag on the same eligibility set and force `acquire_snapshot_one_in=0`.

**Why**: Atomic replacement previously required range bounds to align with SST bounds, and the public doc admitted the upper bound "may be interpreted as inclusive or exclusive". The tombstone overlay removes the alignment restriction and pins the contract to half-open without slowing the Nimble path or the no-straddle case.

Reviewed By: pdillinger

Differential Revision: D117155307

fbshipit-source-id: 65f45463e8082a7db7a9d44c8dc50debe95e36e7
2026-08-26 21:52:40 -07:00
Peter Dillinger e4a7136d3e Preferred-phase scheduling for periodic (time-based) compaction
Summary:
`periodic_compaction_seconds` softly guarantees every SST is rewritten at least
once per interval. Across a fleet of similarly-configured DBs, many events line
those intervals up in lockstep -- a shared enable, a coordinated restart (e.g.
DC power-outage recovery), or just similar write history -- so the periodic
compactions all come due together, a recurring "thundering herd" that is
especially painful with remote compaction (SEV S698635).

The obvious fixes trade one problem for another. Spreading the herd out means
some files get compacted off their natural schedule, which is wasted work; and a
clumsy spread just relocates the herd or spawns a new one. So the real goal is
narrow: **de-herd while keeping "badness" -- extra and/or late compaction --
small, and without inducing fresh herding at any step.**

This change adds an EXPERIMENTAL, opt-in mechanism that gives each `(DB, column
family)` a stable, well-spread *preferred phase* within the interval and steers
periodic compactions toward it. It is randomized and *uncoordinated* (DBs never
talk to each other), which both keeps it simple and, as it turns out, de-herds
far more cheaply than adding jitter to the interval. It is controlled by a single
mutable DBOption and is **off by default**:

- `periodic_compaction_phase_recovery_percent` (default `0` = disabled / kill
  switch; recommended `33`; clamped to `[0, 100]`) -- how aggressively each
  trigger converges toward the phase, as a percent of the remaining gap, only
  ever pulling the trigger *earlier*. `0` reproduces the classic scheduler
  exactly; `100` jumps to the phase in one cycle; lower values approach gradually
  with much less extra work.

The preferred phase itself needs no tuning: it is derived from the DB ID (stable
across re-open and physical cloning/migration, but distinct across DBs) and each
column family is offset from the DB's base phase, so the CFs of one DB don't fire
together. (An earlier draft exposed a `compaction_schedule_seed` string option
for choosing the phase source; it was over-engineered for the one behavior anyone
needed, and as a mutable string it also broke OPTIONS-file round-trip
verification in crash testing. It has been removed in favor of always using the
DB ID.)

## The two kinds of badness

"Badness" here is the fraction of compaction work that is ideally unnecessary:
over-compaction (rewriting a file earlier than the interval required) plus a
penalty for lateness (going overdue past the interval). Steady state is 0; both
"never compact" and "compact twice as often as needed" approach 1. De-herding
inherently spends a little badness to spread load out; every design choice below
exists to bound one of those two costs *without* re-clustering the fleet.

The scenario names cited in the design bullets below (`upgrade_dynamic_r33`,
`naive_upgrade_r25`, `burst_r*`, `turndown_r33`, `turnup_r33`, ...) all come from
the out-of-tree simulator; see its results page (the "Simulator and
visualization" section below, with an interactive plot per scenario) for the
evidence behind each claim.

## Design

- **A stable per-(DB, CF) preferred phase = one fixed attractor per DB.** Each DB
  gets a base phase from its DB ID; each CF is offset from it. Because the target
  is fixed (not re-rolled each cycle) and unique per DB, the fleet converges to a
  uniform spread and then compacts *on time* -- badness decays toward ~0. This is
  the crucial contrast with naive per-cycle jitter, which de-herds but never stops
  jittering and so pays a permanent over-compaction cost (see
  `naive_upgrade_r25`: flat ~12-14%, only diffusive spreading) vs.
  `upgrade_dynamic_r33`, whose badness decays to ~0 for the same anti-herding.

- **Converge gradually, and only ever earlier.** Each trigger closes a
  `recovery_percent` *fraction* of the gap to the phase, reaching it over a few
  cycles rather than in one burst, and never later than
  `periodic_compaction_seconds` in steady state. A lower rate does less extra
  work, which matters when the phase keeps moving (write-driven / bursty
  workloads), where a high rate wastefully "chases" it -- the burst sweep shows
  `burst_r25 < burst_r33 < burst_r50 < burst_r100` in sustained badness. `33` is
  the default balance: quick spreading, low extra work.

- **Fractional recovery (not "snap") is latency-tolerant and doesn't re-herd.**
  Closing a fraction of the gap gives a single stable convergence point, so
  real-world scheduling/queueing latency -- the delay between a file becoming due
  and its compaction actually running -- merely shifts the whole fleet uniformly
  instead of re-clustering it. A tried "snap onto the phase when in range" variant
  converged a bit faster with no latency but re-herded around the snap
  threshold under latency; it was rejected. Graceful degradation beat peak speed.

- **Spread a suddenly-due cohort instead of bursting it (bounded lateness).**
  Enabling the feature, turning `periodic_compaction_seconds` down, or catching up
  after downtime can leave a whole cohort past due at the same instant. Firing it
  all at once would just be the herd again, so such a cohort is spread over the
  first quarter-interval (~N/4) after the phasing anchor on an *absolute*,
  epoch-aligned per-(DB, CF) grid. This trades a bounded lateness (<= ~N/4 past
  the deadline, so the interval is a *soft* upper bound during these transients)
  for anti-herding, and the absolute grid keeps a rapidly restarting DB from being
  perpetually re-deferred and starved. `turndown_r33` shows the past-due cohort
  smeared over N/4 rather than bursting; `turnup_r33` confirms a turn-up creates
  no past-due cohort (deadlines only grow) and so no burst. For immediate
  enforcement at such a transition, trigger a manual compaction.

- **Off by default, with a kill switch.** Inert unless
  `periodic_compaction_phase_recovery_percent > 0`; at `0` behavior is identical
  to the classic scheduler.

## Implementation notes

- Phase math is integer-only (`Hash64` + `FastRange64`); floating point is avoided
  for its pitfalls. The per-CF offset is a golden-ratio (Weyl/Kronecker) additive
  sequence -- low-discrepancy for any number of CFs.
- Logic lives in `db/periodic_compaction_phaser.{h,cc}`
  (`PeriodicCompactionPhaser`), which owns the recovery percent and the phasing
  anchor and exposes the stateless trigger-time math; `VersionSet` holds one and
  caches the per-CF result on each `Version`. The anchor is (re)set to now on DB
  open, on a recovery-percent change, and on any `periodic_compaction_seconds`
  change -- the three events that can create a past-due cohort.
- Plumbed through the C API, `db_bench`, `db_stress`, and `db_crashtest`.

## Simulator and visualization (out-of-tree, not part of this change)

A standalone simulator drives 40 real RocksDB instances through a real
periodic-task scheduler, records every compaction (periodic vs write-driven), and
scores the badness metric above across ~17 scenarios: dynamic/restart enable,
interval turn-down/turn-up via `SetOptions`, on-cadence writes, random write
bursts with a recovery-rate sweep, and the naive-jitter baseline. It is the basis
for the numbers cited above. Interactive HTML (**NOTE** fb-internal URL sanitization might require you to copy-paste the URL yourself):
https://htmlpreview.github.io/?https://github.com/facebook/rocksdb/blob/5759c6a62d9ff36cecf030b80453cc819b2f66ae/pc_phase_sim.html

Reviewed By: jaykorean

Differential Revision: D117487278

fbshipit-source-id: a62439548b7dd6fe7dfdbbe21b69531d8b57a1e8
2026-08-26 20:14:26 -07:00
Peter Dillinger f7b791b213 Fix bottommost-files infinite compaction loop with preserve/preclude (#15146)
Summary:
Leveled compaction combined with preserve_internal_time_seconds or preclude_last_level_data_seconds is a known-broken combination, but it is easy to enable by accident -- for example a fleet-wide config rollout that turns on the preserve time for CFs that happen to be leveled. Rather than degrading gracefully, it triggers an infinite kBottommostFiles compaction loop: a bottommost file gets marked, rewritten with no progress, and immediately re-marked, continuously rewriting the file and growing the MANIFEST so the file set never quiesces. This change makes such a misconfiguration a harmless no-op instead.

Before this change, the marking logic
(ComputeBottommostFilesMarkedForCompaction) marked a bottommost file whenever its largest seqno was below the oldest snapshot, without checking whether a compaction could actually zero that seqno. Recently written seqnos stay within the seqno->time "preserve" window and cannot be zeroed, so the rewrite never made progress. The same futile marking also occurred with user-defined timestamps when full_history_ts_low is unset. The underlying issue is that the "can this bottommost seqno be zeroed?" question was answered independently by the marking logic and by the compaction output path (CompactionIterator::PrepareOutput), and the two had drifted apart.

With this change, both paths answer that question through one shared predicate (BottommostSeqnoCanBeZeroed), and the preserve-window boundary that marking needs is plumbed into VersionStorageInfo from DBImpl, mirroring the existing oldest-snapshot plumbing. Snapshot visibility stays a per-site check. On CFs without preserve/preclude the predicate is a no-op. Confirmed that caller invariants ensure that relocating the `ikey_.sequence <= preserve_seqno_after_` check in CompactionIterator::PrepareOutput() will not trip the ROCKSDB_LOG_FATAL.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15146

Test Plan:
- compaction_picker_test: new CompactionPickerTest BottommostFileNotMarkedWithinPreserveWindow / BottommostFileMarkedBelowPreserveWindow cover the preserve-window boundary; new CompactionPickerU64TsTest BottommostNotMarkedWithEmptyFullHistoryTsLowAndUnknownMaxTs covers the UDT edge case.
- tiered_compaction_test: new PrecludeBottommostLoopTest LeveledPreserveTimeNoBottommostLoop is an end-to-end regression for the loop and the DBImpl plumbing: a bottommost file with a nonzero largest seqno inside the preserve window, made a marking candidate via a snapshot release, must not be marked. It reproduces the loop (hangs) when the plumbing is neutralized and passes with the fix.

Reviewed By: xingbowang

Differential Revision: D117299979

Pulled By: pdillinger

fbshipit-source-id: 51e8e2d4e2116984d16d3c7ef54a9635d5aded74
2026-08-26 19:44:47 -07:00
Josh Kang 6dbb6f30e6 Avoid restoring coroutine stats on foreign threads (#15145)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15145

## Summary

RocksDB coroutine statistics are stored in `RequestContext`, but their `RequestData` is thread-affine. A filesystem implementation may propagate the context while completing `SubmitReadAsync` on another thread. Installing that context invokes `CoroutineStatsRequestData::onSet()`, whose `AssertSameThread()` detects the thread hop and aborts.

```
Before:

Thread A                                      Thread B
RequestContext
├── caller tracing/attribution data
└── RocksDB coroutine stats ── propagated ──> setContext()
                                                │
                                                ▼
                                      CoroutineStatsRequestData::onSet()
                                                │
                                                ▼
                                      AssertSameThread(A != B)
                                                │
                                                ▼
                                              abort
```

Keep propagating the complete `RequestContext`, including the coroutine-stats token, so filesystem callbacks retain caller tracing and attribution. `CoroutineStatsRequestData` now records its owner using `Env::GetThreadID()` and ignores `onSet()` and `onUnset()` on foreign threads. When `ViaIfAsync` schedules the continuation back onto the RocksDB read EventBase, it restores the saved context on the owner thread before the coroutine continues, so the stats are safely loaded back into RocksDB TLS.

```
After:

Thread A                                      Thread B
caller RequestContext
├── caller data ───────── propagated ───────> caller data
└── RocksDB coroutine stats ─ propagated ───> onSet()/onUnset()
                                                │
                                                ▼
                                      owner thread != current thread
                                                │
                                                ▼
                                               no-op

continuation queued back onto Thread A
        │
        ▼
saved RequestContext restored on owner thread
        │
        ▼
RocksDB coroutine stats reinstalled in TLS
```

Reviewed By: xingbowang

Differential Revision: D117274942

fbshipit-source-id: 9995cc3053a0a1d3a63d0cb2f4dd9320ab553005
2026-08-25 13:00:55 -07:00
Kunal Pathak bd16531142 Hoist top-level filter-partition index iterator out of MultiGet per-key loop (#15137)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15137

Reviewed By: pdillinger

Differential Revision: D115489931

fbshipit-source-id: 9d5b607c834a6e8754d55619309ec3705094cc76
2026-08-25 09:13:49 -07:00
Xingbo Wang e6a2ee0bd2 Fix subset checkpoints with incomplete MANIFEST tails (#15123)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15123

Subset checkpoint post-processing appends column-family drop records to a copied MANIFEST. Recovery can tolerate an incomplete atomic group at the tail while retaining the reader high-water size, so appending at that size can produce a checkpoint that fails recovery.

Track the valid logical append boundary together with its MANIFEST file number, refresh it after recovery and successful writes, reset it across reactive MANIFEST switches, and trim only subset-checkpoint copies before appending drop records. Whole-DB checkpoints and the source MANIFEST remain unchanged.

Reviewed By: anand1976

Differential Revision: D116449394

fbshipit-source-id: a162dd6a2558f0f9ec40d878ee83956a89df698b
2026-08-22 12:17:33 -07:00
Andrei Preda ad81b654a4 Prune resolved 2PC recovery state on secondaries (#15121)
Summary:
Previously, during WAL replay, a secondary stored each 2PC prepare in
`recovered_transactions_`. Replaying the matching commit or rollback marker
deletes it. If the primary purges that marker's WAL before the secondary reads
it, the prepared `WriteBatch` and its tracker entry remain until the secondary
closes. This retention case was flagged by xingbowang in https://github.com/facebook/rocksdb/issues/15066. Even when
the marker is replayed, the secondary records completion but previously did not
prune the completed prefix from `logs_with_prep_tracker_`.

This change deletes recovered transactions whose prepare WALs are all below
`min_log_number_to_keep` after opening every discovered WAL reader but before
replaying records. The primary keeps that watermark at or below every
unresolved prepare and every prepare with unflushed committed data, so an entry
below it is committed and flushed, or rolled back. Deleting before replay lets
a reused transaction name start a new generation instead of merging into stale
state. If a reader cannot be opened, no recovered transaction is deleted.

After a round successfully replays every discovered WAL, the secondary also
prunes the completed prefix from `logs_with_prep_tracker_`. This covers
completions from normal commit or rollback replay and from the watermark-based
cleanup. The oldest outstanding prepare stops the pruning scan.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15121

Test Plan:
Added coverage for missed-marker cleanup, transaction-name reuse before replay,
the exact watermark boundary, a catch-up round that reports success after
replaying no WALs, and tracker pruning after normal commit replay. The
missed-marker case verifies cleanup of both retained structures.

Reviewed By: pdillinger

Differential Revision: D116620460

Pulled By: xingbowang

fbshipit-source-id: baca7faf53f414507957f2afdcc4ca67133c5669
2026-08-21 11:39:01 -07:00
Peter Dillinger 8f492b2f47 Store common user-key prefix once per block (format_version 8) (#15117)
Summary:
Building on D116105558, this change finishes and publishes `format_version = 8` as
production-ready.

The headline feature is a tweaked block encoding that stores the common user-key
prefix shared by all keys in a block **once**, in the block's leading bytes
`[0, restarts[0])`, instead of repeating it at every restart point. This shrinks
prefix-heavy blocks (and their block-cache footprint) and, for bytewise-ordered
keys, speeds up `Seek` by comparing prefix-stripped suffixes. It is applied to
**data blocks** and **every index tier** (leaf/partition index, the
partitioned-index top-level, and the partitioned-filter top-level).

Motivation: MyRocks/ZippyDB/timeseries-style keys carry a long shared prefix
(table/index id, partition, etc.). Repeating it at each restart wastes space and
cache, and every Seek re-compares those bytes. Profiles show index-block seek CPU
is comparable to data-block seek CPU (partitioned index does two index seeks per
data seek), so index blocks are a first-class target, not just data blocks.

### Format (self-describing, no new footer bit)

- The common prefix occupies the block's leading bytes; a non-zero `restarts[0]`
  (the offset of the first entry, i.e. the prefix length) self-signals its
  presence. `restarts[0]` is 0 in every block that does not use the feature (all
  versions, all block types), so a single fv8 reader transparently handles any
  mix of stripped and unstripped blocks -- no footer bit is consumed.
- Previous versions would misinterpret this encoding by quietly ignoring the
  common prefix, which (along with the lack of extensibility in data-block
  metadata) is why it is gated behind format_version 8.
- Only restart-point entries change on disk; non-restart entries are byte
  identical to fv7, so the writer produces the exact same bytes regardless of
  implementation.

### Public option

New `BlockBasedTableOptions::optimize_key_common_prefix`
(`enum class OptimizeKeyCommonPrefix`), default `kIfFastSeek`:

- `kDisabled` -- never use it.
- `kIfFastSeek` (default) -- use it only where it also enables the Seek speedup:
  format_version >= 8 with a built-in (reverse-)bytewise comparator.
- `kEnabled` -- use it for all comparators at fv8; (reverse-)bytewise get space +
  fast seek, other comparators get the space savings only (the reader
  reconstructs full keys and compares with the real comparator).

Expressed as a criteria so it need not be toggled per CF based on the comparator
(which can vary by CF). It additionally requires `use_delta_encoding` (the
feature builds on delta encoding) and `ts_sz == 0`; when `use_delta_encoding` is
false -- the setting that also preserves zero-copy key pinning -- it is a no-op.
Default format_version is unchanged (still 7); the feature is opt-in via fv8.

### Scope: every index tier

The reader handles all index-like blocks uniformly (a non-zero `restarts[0]`
self-signals a stripped block regardless of block type), so the writer strips the
common prefix from every tier:

- **Data blocks** and **leaf / partition index blocks**: enabled.
- **Partitioned-index top-level block**: enabled. Under the default
  `index_shortening = kShortenSeparatorsAndSuccessor` the file's last separator
  would otherwise be a short successor sharing no leading bytes, collapsing the
  single-block top-level's common prefix to 0. When common-prefix is on the
  builder suppresses that one successor (keeps the full last key -- i.e.
  `kShortenSeparators` behavior for that one entry), so the top-level strips
  uniformly under every shortening mode; the full last key is also a tighter file
  upper bound, so it is not worse.
- **Partitioned-filter top-level index**: enabled (same uniform treatment). Its
  space/CPU effect is small at default partitioning (the top-level filter index
  is tiny) but grows with many small partitions -- a filter miss short-circuits
  at, and is dominated by, the top-level filter-index seek (see Performance).
  Included for a uniform on-disk surface across tiers.

Bloom filter *contents* still hash full user keys, so prefix/whole-key filtering
is unaffected.

### Implementation

- Writer (`block_builder.{h,cc}`): keys are encoded incrementally in the fv7
  layout while the running common prefix is tracked cheaply (reusing the delta
  `shared`); at `Finish()` only the restart-point entries are rewritten in place
  with the prefix stripped, and the prefix is written once at the block start.
  The rewrite is generalized to the value-delta (fv4) index layout used by index
  blocks (no `value_size` header; bulk-copy the inline BlockHandle and
  non-restart entries verbatim).
- Reader (`block.{h,cc}`, `dbformat.h`): `Block` decodes the prefix from
  `restarts[0]`; `BlockIter::ParseNextKey` prepends it on a restart point
  (`shared == 0`). The suffix-seek helpers (strip target once, compare shortened
  suffixes) live in the templated `BlockIter` base and are shared by data and
  index iterators (binary search, templated on `DecodeKey`/`DecodeKeyV4`); for
  non-bytewise comparators (data) and the hash `PrefixSeek` refine step, the
  reader reconstructs the probed restart key and compares full keys.
- Gating (`block_based_table_builder.cc`, `index_builder.{h,cc}`,
  `partitioned_filter_block.{h,cc}`): the criteria is threaded to the data-block
  builder and to every index tier -- leaf/partition index, the partitioned-index
  top-level (`use_common_prefix_top`), and the partitioned-filter top-level --
  together with the last-separator successor suppression
  (`ShortenedIndexBuilder::GetSeparatorWithSeq`) that lets the top-level strip
  under the default shortening mode. The index base gate also requires
  `super_block_alignment_size == 0` (block alignment can trigger
  `skip_delta_encoding`, producing non-restart `shared == 0` entries the reader
  could not distinguish from stripped restart keys).
- Memory: the per-block/per-iterator prefix is stored as a `uint32_t` length
  (the prefix is always at offset 0) rather than a full `Slice`, and `Block` /
  `BlockBuilder` members were repacked so the feature adds little or no struct
  size.
- Java: a new `OptimizeKeyCommonPrefix` enum and a `BlockBasedTableConfig`
  getter/setter, plumbed through JNI in both directions (the
  `newTableFactoryHandle` writer and the options-reconstruction constructor).

### Whole-file (table-level) prefix: investigated and rejected

A natural extension is to store the whole-file common prefix once per *file*
(when an `SstPartitioner` guarantees every key in the file shares a fixed prefix)
instead of once per *block* -- e.g. the MyRocks index/table id, or a future
128-bit global id. It was prototyped end-to-end (partitioner-provided prefix
captured at file open, stored in a `user_key_common_prefix` table property, and
stripped from every data/index block; the reader re-prepends it) and then
dropped. The `user_key_common_prefix` table property reserved in the
previous diff is dropped in this one. The rationale, in case it is revisited:

- Marginal benefit over per-block. Per-block already strips the full shared
  prefix from every key; the only thing whole-file removes on top is the single
  per-block copy of that prefix, i.e. `~= prefix_len / block_size` of the
  uncompressed (block-cache) footprint, with no meaningful read-CPU change
  (per-block already delivers strip-once + suffix compares). Measured at
  prefix_len=12, 4 KB blocks this was ~-0.3% incremental space; at the 8-16 KB
  data blocks typical of the space-constrained target (MyRocks/UDB) it is
  ~0.1-0.2% even with a 16-byte global id -- a fraction of a percent, seen only
  by partitioner users. Small block-size workloads are becoming more
  niche, and even where they exist, they are also CPU-sensitive (see next
  paragraph).

- Real, if small, CPU tax on the *per-block* path. Reconstructing a key from two
  non-contiguous parts (file prefix + block residual) cost ~0.5-1.0% additional
  retired instructions on the per-block read path (deterministic across gcc and
  clang), wall-clock-neutral in cache-resident tests but present even when no
  whole-file prefix is configured. Attempts to remove it did not pan out: a
  compile-time specialization of the reconstruction *increased* instructions
  (+2.9%) by defeating inlining; a per-block combined-prefix buffer trades the
  CPU cost for a memory cost on every (default) block; and a per-iterator buffer
  was the original larger regression. Only invasive iterator-level
  monomorphization would truly zero it, which is not justified on this hot path.

- Decision. A fraction-of-a-percent, partitioner-only space win does not justify
  a permanent per-block-read-path CPU tax plus additional on-disk format surface
  (a table property that changes whole-file interpretation). Dropped. The generic
  `SstPartitioner::ShouldPartitionByPrefix` hook is kept (cheap, and useful
  independently), and the reserved `user_key_common_prefix` table property is
  removed since no writer will emit it.

### Performance (db_bench, release/native, cache-warm, best-of-3, pinned)

Three arms: (A) fv7 on the base/main branch, (B) fv7 with this change (should be
neutral / byte-identical writing), (C) fv8 with this change and
`optimize_key_common_prefix=enabled` (the optimization is active). Partitioned
index + filters; `compression_type=none`, so SST bytes are the uncompressed
block-cache footprint. `int` = db_bench default keys (variation-early, near worst
case); `pfxN` = an N-byte shared prefix + varying suffix (MyRocks-like).
Setup: `num=3,000,000`, `reads=1,500,000`, single thread,
`metadata_block_size=4096`, cache holds the working set, pinned to one core.

| shape (key/val) | arm | SST bytes | readrandom us/op | seekrandom us/op |
|---|---|---:|---:|---:|
| int (32/8)    | A fv7 main | 138,983,658 | 3.484 | 4.995 |
|               | B fv7 ours | 138,983,146 | 3.518 | 5.024 |
|               | C fv8 ours | 137,681,420 | 3.490 | 4.986 |
| pfx24 (32/8)  | A fv7 main |  71,697,531 | 3.417 | 5.561 |
|               | B fv7 ours |  71,698,043 | 3.448 | 5.579 |
|               | C fv8 ours |  65,578,236 | 3.284 | 5.386 |
| pfx56 (64/16) | A fv7 main | 102,942,266 | 3.624 | 5.830 |
|               | B fv7 ours | 102,942,266 | 3.678 | 5.835 |
|               | C fv8 ours |  90,554,642 | 3.444 | 5.626 |

fv8 vs fv7-main (C vs A) -- space / readrandom / seekrandom:

- int (32/8):    **-0.9%**  / ~neutral  / ~neutral
- pfx24 (32/8):  **-8.5%**  / **-3.9%** / **-3.1%**
- pfx56 (64/16): **-12.0%** / **-5.0%** / **-3.5%**

fv7-ours vs fv7-main (B vs A): SST bytes byte-identical (within 512 B; the fv8
paths are gated off), read/seek us/op within run-to-run noise (~1%, best-of-3 on
a shared host). Net: fv8 is a space **and** read/seek-CPU win on prefix-heavy
data and neutral on variation-early (`int`) data, so `kIfFastSeek` is safe on by
default at fv8.

### Space by tier (compression=none, num=3M, mdb=4096; fv7 -> fv8)

Where the win comes from -- the index tiers strip the most:

| shape | data | leaf/partition index | top-level index | filter |
|---|---:|---:|---:|---:|
| int   | -0.8%  | -29% | -45% | -0.2% |
| pfx24 | -8.3%  | -69% | -89% | -0.8% |
| pfx56 | -11.1% | -83% | -96% | -1.6% |

Data bytes dominate the absolute total, but every index tier is stripped:
leaf/partition index -29% to -83% and the partitioned-index top-level -45% to
-96%. Filter *blocks* barely move (-0.2% to -1.6%) because filter bits hash full
user keys; only the small top-level filter index is stripped.

### Point-miss lookups (readmissing)

A partitioned-filter miss short-circuits at the filter (no data-block read), so
it is dominated by the top-level filter-index seek plus the partition bloom. At a
typical `metadata_block_size=4096` (few partitions, tiny top-level filter index)
fv8 is neutral on misses (int +0.9%, pfx24 -1.0%, pfx56 -0.7% us/op -- all
noise). The top-level-filter stripping pays only with many small partitions: at
`metadata_block_size=256` it is pfx24 1.166->1.138 (**-2.4%**) and pfx56
1.273->1.188 (**-6.7%**) us/op.

### Backward / forward compatibility

- Default `format_version` remains 7; the feature is inert unless a CF opts into
  fv8. Older RocksDB rejects fv8 files at footer decode (fail-safe).
- Within fv8, the stripped-block set a writer can emit is fixed and every fv8
  reader can read it (the `restarts[0]` signal plus uniform reader handling), so
  the on-disk surface is self-consistent across builds/downgrade.

### Follow-ups (not in this change)

- Suffix-aware interpolation / hash index seek (currently correct via the
  reconstruction fallback, not perf-tuned).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15117

Test Plan:
- New focused unit tests (`table_test`), run at the full `format_version` matrix
  including the now-supported fv8:
  `GeneralTableTest.OptimizeKeyCommonPrefix` (data-block space+read+Seek
  correctness: existing and missing keys straddling the prefix, before/after-all,
  SeekForPrev), `OptimizeKeyCommonPrefixNonBytewise` (reconstruction path), and
  `IndexBlockCommonPrefixEquivalence` across `{kBinarySearch,
  kBinarySearchWithFirstKey, kHashSearch, kTwoLevelIndexSearch}` x
  `{partitioned filters}` x `{bytewise, reverse-bytewise}` (so both top-level
  tiers -- partitioned-index and partitioned-filter -- are exercised), asserting
  identical reads and a shrunken fv8 index.
- The randomized `ParameterizedHarnessTest` now runs at fv8 (reverse-bytewise,
  restart intervals `{1,16,1024}`, special/empty keys fuzzed against the feature).
- `db_crashtest.py` randomizes `format_version` including 8 and
  `optimize_key_common_prefix` in {0,1,2}.
- Removed the reserved-property carve-out: dropped the `ReservedUserKeyCommonPrefix`
  reader test and the C API getter for the property; C API regenerated via
  `tools/c_api_gen/` (`make regen-c-api`), and the intentional public-C-API
  removal is recorded in `tools/c_api_gen/api_compatibility_allowlist.json`.
- Java: `BlockBasedTableConfigTest.optimizeKeyCommonPrefix` (default
  `kIfFastSeek`, plus set/get of every value).
- `table_test` `FooterTests` updated for the now-published fv8 footer, whose
  metaindex-gap encoding stores the metaindex block size in 16 bits (the test
  caps that size for fv >= 8).
- Manual verification: the db_bench runs in Performance above (three-arm
  space + read/seek CPU, per-tier byte breakdown via `sst_dump`, and
  `readmissing`); and `readrandom`/`seekrandom` with
  `--block_protection_bytes_per_key=8` find 100% of keys at both fv7 and fv8
  (reconstructed keys pass per-key checksum).

Reviewed By: xingbowang

Differential Revision: D116365016

Pulled By: pdillinger

fbshipit-source-id: 66f4b5c60ea4fe0ff48d34c98d191ba8ebdf0665
2026-08-20 17:32:41 -07:00
Peter Dillinger e8554f1b11 Faster and safer MANIFEST recovery for remote compaction (OpenAndCompact) (#15133)
Summary:
Faster and safer MANIFEST recovery for remote compaction (OpenAndCompact)
`DB::OpenAndCompact` (remote/distributed compaction) opens the target DB as a
secondary and reconstructs the LSM state from the MANIFEST before running the
requested compaction. Previously, point-in-time recovery would stat/open the
SST (and blob) files referenced by the MANIFEST to classify them found/missing
and materialize candidate versions, and eagerly load their table handlers.
That is (a) a lot of unnecessary file operations and open latency, especially
on remote/distributed storage, and (b) fragile: when a file referenced by the
MANIFEST but not an input to the compaction was momentarily unavailable to the
worker -- e.g. an obsolete file the primary had already deleted -- recovery
could fail the job (surfacing as an incomplete "no valid result") or roll back
to an earlier version, even though the compaction only needs its own input
files, which the primary keeps referenced for the job's duration. This was seen
as a flaky failure of CompactionServiceTest.BasicCompactions on aarch64, and
reproduces on x86 under COERCE_CONTEXT_SWITCH.

Efficiency + reliability: recovery for the OpenAndCompact path now trusts the
MANIFEST. It reconstructs the current version from FileMetaData without
stat-ing/opening files to classify them and without eagerly opening
candidate-version table handlers, so it does far fewer file operations and
never fails or rolls back over a file the compaction does not need. The
compaction's input files are opened (and unique-id verified) on demand as it
reads them.

Correctness hardening: trusting the MANIFEST means the worker's own (possibly
eventually-consistent) view of CURRENT/MANIFEST drives the reconstructed LSM
shape, which feeds correctness-critical decisions (Compaction::IsBottommostLevel
/ KeyNotExistsBeyondOutputLevel). A worker that read a stale CURRENT (older
MANIFEST), a truncated MANIFEST, or a MANIFEST with a corrupt/torn tail could
reconstruct against an older LSM shape and silently drop keys that should be
kept. To prevent this, the primary records the MANIFEST position (file number
and size) it scheduled the compaction from and passes it in
CompactionServiceInput as a floor. After recovery the worker compares the size
of the maximal *valid* prefix it reconstructed -- which stops at any corrupt or
torn record and at a partial atomic group at EOF, so tolerated tail garbage
cannot inflate it -- against that floor, and refuses to compact against an older
view (an older file number, or the same file number whose valid prefix is
shorter), returning Status::Incomplete so the CompactionService can fall back to
a local compaction. Accept-equal-or-later is safe by monotonicity: while the
inputs stay locked (being_compacted), a later view can only have moved data
further down, never resurrecting a shadowed value, so only an older view is
unsafe.

Single kill switch: both behaviors are bundled behind one signal and one
option. Presence of the floor on the request (min_manifest_file_number != 0)
couples on both trust-manifest recovery and the floor check in the worker;
absence reverts the worker to the original point-in-time recovery with no floor
check. A single mutable DBOption, remote_compaction_manifest_floor (default
on), gates whether the primary populates the floor, so it is the one escape
hatch that disables the entire new recovery path on a running DB via
SetDBOptions if it ever misbehaves. The new CompactionServiceInput fields are
wire-compatible in both directions (0 = not provided), so an old worker or an
old primary simply behaves as if the switch were off.

Follow-up: recovery is not yet strictly "open only the compaction's inputs". A
limited number of non-input files are still opened by the bounded file-property
sampling in Version::PrepareAppend -> UpdateAccumulatedStats (tolerating any
that are missing), which today populates input FileMetaData::num_entries for
compaction input-record-count verification. A natural follow-up is to record
the needed per-file info (e.g. num_entries) in the MANIFEST so the worker can
skip these auxiliary opens entirely; alternatively, source input num_entries
from the worker's own read of its input table properties. Relatedly, the floor
comparand is the recovered "maximal valid prefix" size, which today equals the
MANIFEST file size but is defined to stay correct if a future MANIFEST format
writes trailing padding/footer past the last record.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15133

Test Plan:
Added tests in compaction_service_test.cc:
- RemoteCompactionMissingLowerFileTrustsManifest: a non-input file referenced
  by the MANIFEST is unavailable during a remote compaction; the job completes
  with correct results (fails without trust-manifest recovery). The underlying
  race also reproduces under COERCE_CONTEXT_SWITCH.
- RemoteCompactionManifestFloorRejectsStaleView: inflate the floor so the
  worker's recovered MANIFEST is below it; assert the worker rejects, the job
  falls back to local, and a below-output deleted key stays deleted (no silent
  resurrection).
- RemoteCompactionManifestFloorSendsPosition: a non-zero floor is attached and
  the worker accepts its equal-or-later view.
- RemoteCompactionManifestFloorKillSwitchOff: with the option off, no floor is
  sent, the worker uses the original recovery, and a transiently-missing
  non-input file makes the job fail/roll back (the escape hatch reverts
  behavior).
- RemoteCompactionManifestFloorMutable: SetDBOptions flips the option; the
  floor stops being attached and the worker skips the check.
options_settable_test covers the new mutable DBOption.

Crash test: remote compaction is already exercised (db_crashtest randomly
enables remote_compaction_worker_threads, driving DB::OpenAndCompact), so with
remote_compaction_manifest_floor defaulting on, the new recovery path is
covered by the crash test by default -- no new crash-test plumbing is needed.
The kill switch is intentionally left un-randomized (always on): turning it off
re-enables the original, occasionally-broken recovery, which would only add
noisy failures for no benefit.

UPDATE: crash test is not well covered because of `"remote_compaction_failure_fall_back_to_local": 1` which is masking other existing failures. Addressing this is a high-priority follow-up item. (Not practical to fix in this diff.)

Reviewed By: jaykorean

Differential Revision: D116738823

Pulled By: pdillinger

fbshipit-source-id: c02e0a437d62fc526dbbfcf70edf43434dfc7c1f
2026-08-20 16:07:48 -07:00
rocksdb-release-bot 330ded5129 Update version to 11.10.0 for 11.9 release (#15135)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15135

- Bump version.h from 11.9.0 to 11.10.0
- Add `11.9.fb` to `check_format_compatible.sh`
- Sync HISTORY.md `## 11.9.` section from `11.9.fb`
- Delete 17 consumed `unreleased_history/` note file(s) from main

Part of 11.9 release workflow.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15096

Reviewed By: anand1976

Differential Revision: D116833037

Pulled By: joshkang97

fbshipit-source-id: 04ebda8db5b2c4fdae71a2cf107784b6abdc0f3c
2026-08-20 15:13:31 -07:00
Anand Ananthabhotla 4b41ccd2a7 Fix full_history_ts_low race for long-running snapshots in db_stress (#15129)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15129

When `persist_user_defined_timestamps=false` and `long_running_snapshots=1`, `AssertSame` in `MaybeReleaseSnapshots` reuses the timestamp saved at snapshot acquisition time. That timestamp can become stale because concurrent flushes advance `full_history_ts_low`, and long-running snapshots can be held much longer than the 10-second headroom added by `GetReadTimestamp()`.

The snapshot validation is intended to check that a held snapshot still exposes the same sequence-number view. It should not require the acquisition-time read timestamp to remain valid after history below `full_history_ts_low` has been collapsed.

The fix keeps the timestamped read API contract intact. Column families with user-defined timestamps still require `ReadOptions.timestamp`, even when timestamps are memtable-only. For `persist_user_defined_timestamps=false`, `AssertSame` now uses a fresh validation timestamp so the read remains above `full_history_ts_low` while the snapshot sequence number continues to pin data visibility. For persisted timestamps, it continues reusing the original timestamp to validate the exact timestamped snapshot view.

Reviewed By: joshkang97

Differential Revision: D116110124

fbshipit-source-id: 9cf6cae6e7123f8453b7065565e9e2e0eb34321a
2026-08-20 12:32:59 -07:00
Ranjan Banerjee 2846163d9e Fix remote compaction input record count mismatch due to uninitialized FileMetaData stats (#15130)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15130

## Summary

Fix a corruption error in remote compaction where `FilterInputsForCompactionIterator` produces different results on the primary host vs. the remote worker.

**Root Cause:**

When a standalone range tombstone file completely shadows another input file, `FilterInputsForCompactionIterator()` removes the shadowed file from `input_levels_`. This filtering relies on `FileIsStandAloneRangeTombstone()`, which checks `num_entries == num_range_deletions == 1`.

On the primary host, these `FileMetaData` fields are always populated (set during flush/compaction when the file was created). On the remote worker, however, these fields are NOT persisted in the MANIFEST and are only lazily loaded from table properties via `UpdateAccumulatedStats()` during DB open, which is capped at 20 files (`kMaxInitCount`).

If the standalone range tombstone file is not among those first 20 initialized files, its `num_entries` remains 0 on the remote worker. This causes `FileIsStandAloneRangeTombstone()` to return false, so the remote worker does NOT filter the shadowed file and processes all its keys. Meanwhile the primary's expected record count is computed from its (filtered) view, leading to:

```
Corruption: Compaction number of input keys does not match number of keys
processed. Expected 0 but processed 12.
```

**Fix:**

Before constructing the `Compaction` object on the remote worker, explicitly initialize `FileMetaData` stats from table properties for all input files. This ensures `FilterInputsForCompactionIterator` produces consistent filtering results on both sides.

**Introduced by:** D83375779 (2025-09-29), which enabled `FilterInputsForCompactionIterator` on the remote worker by passing `earliest_snapshot` to `PickCompactionForCompactFiles`.

Reviewed By: xingbowang

Differential Revision: D116252973

fbshipit-source-id: 2cb2d61dae1f3982edeaf41f9a0f39d5aef1e789
2026-08-20 10:54:16 -07:00
Josh Kang 03b8d1afa0 Prevent normal flushes from starving error recovery (#15132)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15132

This fixes a liveness bug in background-error recovery. Retryable soft errors allow foreground writes to continue, but recovery must wait for every scheduled background flush worker to exit before it can submit the recovery flush.

While recovery was active, normal flush requests could still enter `flush_queue_`. Atomic flush requests bypass per-column-family deduplication, so writes could accumulate a backlog of normal requests.

For each request, the background worker:

  1. Pops and drops the normal request because BG work is stopped
  2. Sleeps for the one-second background-error backoff
  3. Decrements `bg_flush_scheduled_`: 1 -> 0
  4. Calls `MaybeScheduleFlushOrCompaction()`
  5. Schedules the next queued request: 0 -> 1
  6. Only then signals the recovery thread

The rescheduling happens under the DB mutex before recovery is signaled:

  Queued requests: R1, R2, R3, ...

  Flush worker                      Recovery thread
  ------------                      ---------------
  drop R1
  sleep 1 second                    waiting for scheduled == 0
  scheduled: 1 -> 0
  schedule R2: 0 -> 1
  signal -------------------------> wake, observe scheduled == 1
                                    wait again

  drop R2
  sleep 1 second
  scheduled: 1 -> 0
  schedule R3: 0 -> 1
  signal -------------------------> wake, observe scheduled == 1
                                    wait again

The dropped request is not requeued. Instead, the worker exit path guarantees that the next queued request is scheduled while `unscheduled_flushes_ > 0`.

A finite backlog therefore delays recovery by roughly one second per request. If writes keep producing requests as quickly as workers discard them, the queue may never drain and recovery is starved indefinitely. Recovery can complete after writes become quiescent, so this is a liveness failure rather than a lock deadlock or data-corruption bug.

There is also a queue-accounting edge. Scheduling consumes `unscheduled_flushes_`, but a worker does not own its `FlushRequest` until it pops `flush_queue_`. A worker can exit on the background error before popping, leaving:

  flush_queue_.size()  > 0
  unscheduled_flushes_ == 0

Clearing only unscheduled requests therefore misses scheduled-but-unclaimed requests.

The fix:

- Reject new non-recovery requests while background work is stopped.
- Allow error-recovery flush reasons to bypass that guard.
- Clear the entire flush queue before recovery waits for scheduled workers.
- Retain the worker-side reason check for requests queued before the error.
- Rebuild recovery flushes and schedule a catch-up flush for writes made during recovery.

Reviewed By: pdillinger

Differential Revision: D116708546

fbshipit-source-id: 6e29fd3653ef72d047c64ca89d311a7bee51ab84
2026-08-20 10:54:15 -07:00
Xingbo Wang 42fca1bb39 Enable core dumps for sanitizer crash tests (#15131)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15131

ASAN and other sanitizer runtimes can disable coredumps or terminate without a core after the test driver sets its shell core limit. This leaves sanitizer stress-test failures with an LLDB stack but no core for the DB artifact.

Configure ASAN_OPTIONS, TSAN_OPTIONS, and UBSAN_OPTIONS during sanitizer crash-test steps so reports terminate with a core-generating abort and core dumping remains enabled. Preserve caller-provided options and the existing Make fallback semantics. Keep the behavior scoped to test steps and crash targets.

Enabling sanitizer cores raises a concern that the generated core-dump artifacts can be large because they include the process working set. Reduce RocksDB's cache and memtable contribution to the core size without replacing the normal randomized test configuration. Add a db_crashtest cache/write-buffer size multiplier that is applied after each iteration chooses its values, and pass 0.5 from ASAN, TSAN, and UBSAN crash lanes. For the simple profile, cache_size remains randomized with 8/32 MiB becoming 4/16 MiB, while the 32 MiB write buffer becomes 16 MiB. Preserve RocksDB's 64 KiB minimum write-buffer size and leave non-sanitizer lanes unchanged.

Reviewed By: rban1

Differential Revision: D116666171

fbshipit-source-id: aa157cfa255bb637d4a68e54ea2039db779fdc53
2026-08-20 04:59:40 -07:00
Peter Dillinger d3a82f28ce Optimize SstPartitioner overhead in compaction via ShouldPartitionByPrefix() (#15118)
Summary:
When a column family configures an SstPartitioner, compaction paid a per-key cost for every key-value written to an output SST: a full user-key copy (into `last_key_for_partitioner_`) plus a virtual `SstPartitioner::ShouldPartition()` call (which also builds a PartitionerRequest and truncates/compares two Slices).

This is motivated by an internal user whose custom SstPartitioner parses a variable-length prefix (a prefix code) out of each key to decide partition boundaries; today that prefix parsing runs on every key written during compaction, on top of the copy and virtual call above.

This wires up the `SstPartitioner::ShouldPartitionByPrefix()` API (added, but unused, alongside the format_version=8 groundwork) so compaction can skip that per-key work. When a partitioner can express its next boundary as a key prefix, `CompactionOutputs` caches that prefix once per output file (re-derived from the file's first key) and replaces the per-key copy + virtual call with an inlined `Slice::starts_with()` check. That also lets a partitioner like the one above do its prefix parsing once per output file instead of once per key. When the partitioner returns no prefix (ambiguous, e.g. `SstPartitionerFixedPrefix` for keys shorter than its length), compaction falls back to the previous per-key `ShouldPartition()` path. Output file boundaries are byte-for-byte identical to before. `SstPartitionerFixedPrefix` implements `ShouldPartitionByPrefix()`; the base-class default returns "no prefix", so existing custom partitioners are unaffected until they opt in.

Also:
- db_bench: new `--sst_partitioner_fixed_prefix_len` to configure a fixed-prefix partitioner (enables benchmarking; no default behavior change).
- `ShouldPartitionByPrefix()` API doc updated now that it is used and recommended for compaction CPU efficiency (any prefix code works, not just fixed-size).

No new option and no behavior or format change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15118

Test Plan:
Added db_compaction_test cases covering the new fast path and its fallbacks: many keys sharing a prefix (one output file per prefix, no spurious cuts), keys shorter than the fixed prefix length (fallback to ShouldPartition), the zero-length-prefix "never partition" case, and short<->long key transitions across output-file boundaries. Existing SstPartitioner tests and the range tombstone + partitioner tests (db_range_del_test) provide regression coverage that file boundaries are unchanged, including the fast path with range tombstones. The crash test already exercises the partitioner via `enable_sst_partitioner_factory`.

Benchmark (db_bench `compact` of a pre-built 50M-key DB; 16B keys / 8B values, 8-byte shared key prefixes with FixedPrefix length 8, no compression, single subcompaction). A/B compares this change against the same binary with only the compaction hot-path change reverted. Means of 6 trials, optimized faster on every trial and every metric; both variants produced identical output (50 files, i.e. identical partition boundaries):

| Metric                          | Baseline | Optimized | Delta  |
|---------------------------------|----------|-----------|--------|
| Instructions (whole process)    | 92.46 B  | 90.09 B   | -2.6%  |
| CPU cycles                      | 46.20 B  | 45.28 B   | -2.0%  |
| CPU time (task-clock)           | 18.74 s  | 18.39 s   | -1.9%  |
| `compact` wall time             | 20.58 s  | 20.08 s   | -2.4%  |

DB-internal compaction stats corroborate: CompMergeCPU 15.84s -> 15.42s (-2.6%), compaction write throughput 38.9 -> 39.9 MB/s. The relative gain shrinks with larger values or compression enabled, where the partitioner is a smaller fraction of total compaction work; it is a pure win whenever a prefix partitioner is configured.

Reviewed By: rban1

Differential Revision: D116372363

Pulled By: pdillinger

fbshipit-source-id: 63de190329112aa2ce5b48da9fc77ef386494558
2026-08-19 11:43:10 -07:00
Xingbo Wang d8536d3274 Enable continuous verification for secondary stress tests (#15114)
Summary:
PR https://github.com/facebook/rocksdb/issues/15066 fixed the bug that has caused this combination fails in the past. Re-enable this combination in the stress test.

db_crashtest.py currently disables continuous_verification_interval whenever test_secondary=1. This change removes that quarantine and enables continuous verification for secondary DB stress tests.

The disable predates auto_refresh_iterator_with_snapshot. When auto-refresh was later added, its stress-test integration mechanically added snapshot setup to several iterator paths, including the secondary-
only NonBatchedOpsStressTest::ContinuouslyVerifyDb() path:

snapshot = std::make_unique<ManagedSnapshot>(db_);
read_opts.snapshot = snapshot->snapshot();

This creates a snapshot from the primary db_ and passes it to an iterator created from secondary_db_. That was never supported: a ReadOptions snapshot must belong to the DB being read, and secondary DB
iterators explicitly reject non-null snapshots.

The incompatibility went unnoticed because the existing sanitizer made this combination unreachable in recurring crash tests. It was therefore a stress-test integration bug, not evidence that snapshot-
preserving auto-refresh was supported by secondary DBs.

This change:

- Stops forcing continuous_verification_interval=0 for secondary tests.
- Avoids attaching a primary-owned snapshot to secondary iterators.
- Continues exercising auto_refresh_iterator_with_snapshot through primary iterator paths.

Secondary iterators still support ordinary Iterator::Refresh(), which advances them to the latest caught-up secondary state. Snapshot-preserving automatic refresh remains unsupported.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15114

Test Plan:
- python3 tools/db_crashtest_test.py
- make check-sources
- AUTO_CLEAN=1 make -j192 db_stress
- Targeted blackbox crash test with:
- test_secondary=1
- continuous_verification_interval=1000
- auto_refresh_iterator_with_snapshot=1

Reviewed By: jaykorean

Differential Revision: D116381630

Pulled By: xingbowang

fbshipit-source-id: a162ab357a537c2e5047c1eda85d0c2138b56ed1
2026-08-19 11:34:01 -07:00
zaidoon 136d0d1fa1 Add UDI index mode API vocabulary (#14959)
Summary:
Part 2 of 13 in the UDI split.

Stack order:
1. https://github.com/facebook/rocksdb/issues/14954 Add IndexFactory compatibility names
2. https://github.com/facebook/rocksdb/issues/14959 Add UDI index mode API vocabulary
3. https://github.com/facebook/rocksdb/issues/14960 Add optional UDI builder protocols
4. https://github.com/facebook/rocksdb/issues/14961 Add built-in index factory wrappers
5. https://github.com/facebook/rocksdb/issues/14969 Add built-in index factory tests
6. https://github.com/facebook/rocksdb/issues/14962 Promote IndexFactory as UDI SPI
7. https://github.com/facebook/rocksdb/issues/14963 Wire IndexFactory through block-based tables
8. https://github.com/facebook/rocksdb/issues/14964 Add IndexFactory table routing tests
9. https://github.com/facebook/rocksdb/issues/14970 Add IndexFactory parallel routing tests
10. https://github.com/facebook/rocksdb/issues/14965 Account UDI blocks as index blocks
11. https://github.com/facebook/rocksdb/issues/14966 Support parallel AddIndexEntry in trie index
12. https://github.com/facebook/rocksdb/issues/14967 Cover trie index modes
13. https://github.com/facebook/rocksdb/issues/14968 Add IndexFactory stress and benchmark flags

Previous: https://github.com/facebook/rocksdb/issues/14954.
Next: https://github.com/facebook/rocksdb/issues/14960.

Depends on https://github.com/facebook/rocksdb/issues/14954. Until https://github.com/facebook/rocksdb/issues/14954 lands, GitHub shows the cumulative diff against main. After that lands, the intended review diff is:

`zaidoon/udi-01-index-factory-compat-shim..zaidoon/udi-02-index-mode-api`

That final review delta is 4 files with 52 insertions.

What changed:
- Add inert IndexMode and ReadIndex vocabulary to the public options headers.
- Block those experimental enum fields from automatic C API generation until the explicit C API mapping lands later in the stack.
- Keep behavior unchanged. Runtime wiring comes in later PRs.

Validation:
- `git diff --check`
- `make check-sources`
- `python3 tools/c_api_gen/verify_generated_up_to_date.py`
- `AUTO_CLEAN=1 make check-c-api-gen`
- `./c_test`

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14959

Reviewed By: jaykorean

Differential Revision: D116211178

Pulled By: xingbowang

fbshipit-source-id: 42edd3f0991d90b0b5c90232e5023de6e3740dac
2026-08-19 10:31:27 -07:00
Anand Ananthabhotla 49faa51e11 Fix crash test Restore() failure with empty trace when replay_write_ops == 0 (#14840)
Summary:
`ExpectedState::Restore()` always attempted to open and replay the post-`saved_seqno_` trace. In the crash-test failure, recovery had already returned the DB exactly to `saved_seqno_`, so `replay_write_ops == 0` and the trace was legitimately empty. `FileTraceReader` reported `Status::Incomplete()` on that empty trace, which incorrectly failed restore.

Skip the trace replay work when `replay_write_ops == 0`, but still run the normal state promotion, cleanup, and `saved_seqno_` reset path so later saves continue to work.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14840

Reviewed By: xingbowang

Differential Revision: D107968797

fbshipit-source-id: b4024e1ff329aa3bd47f0307e67d9a4f224b66d4
2026-08-18 21:09:30 -07:00
Anand Ananthabhotla 7d18ffe316 Ignore OPTIONS files for manifest recovery advance (#15106)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15106

When `optimize_manifest_for_recovery` is enabled, `MaybeUpdateNextFileNumber` uses on-disk files to decide whether recovery must append a `SetNextFile` edit. `OPTIONS-*` and temp options files are written as sidecar option snapshots during open, after recovery, so their presence alone should not force MANIFEST growth on a later clean reopen.

Exclude `kOptionsFile` and `kTempFile` from the trigger that requires a MANIFEST file-number advance. If any real file requires an advance, the code still advances past the largest observed file number, including OPTIONS files.

Also tighten db_stress strict MANIFEST verification so fatal zero-growth checks only run when all fault injection is disabled. Open write/sync fault injection can legitimately make `reuse_manifest_on_open` fall back to a fresh MANIFEST.

Reviewed By: xingbowang

Differential Revision: D116092705

fbshipit-source-id: 65b57b0b7f57b2cef42efc4df353fb2218688d34
2026-08-18 17:53:22 -07:00
Xingbo Wang db1adf47bb Add bootstrap configuration for RocksDB external tables
Summary:
Add an opaque external_table_config option to the external table adapter. RocksDB persists the string in OPTIONS files and passes it to ExternalTableFactory::Configure during bootstrap, allowing implementations such as Nimble to parse implementation-specific settings without adding RocksDB-specific fields. The option is immutable after DB open, and attempted runtime updates fail safely for non-clonable table factories. A reopen test verifies that callers can supply a different valid configuration between DB opens. Initialize the adapter option metadata lazily as const to avoid namespace-scope dynamic initialization and mutable-global lint issues.

PR https://github.com/facebook/rocksdb/pull/15119

Reviewed By: anand1976

Differential Revision: D116362444

fbshipit-source-id: 225a4aedc455f320b5c90ac2f0862917da477ccb
2026-08-18 10:12:47 -07:00
Xingbo Wang bbdcd2825f Add remote storage liveness crash test signal (#15102)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15102

Add a dedicated Skycastle signal that invokes liveness through the existing remote-storage test-driver path.

For remote DB runs, double default wrapper/no-progress timeouts while preserving explicit overrides. Default remote liveness `expected_values_dir` to local storage so `UniqueIdVerifier` never treats a remote DB path as a local path; local liveness behavior remains unchanged.

Normalize both equals and spaced `--env_uri`/`--fs_uri` forms once so remote detection, workload execution, and cleanup share the same arguments. Track the crash harness and internal release scripts as CI source triggers, scoped so release-script changes select only the remote-storage signal. Add command-level regression coverage and failure-safe expected-values cleanup. Use generic remote DB/storage terminology throughout public RocksDB code and comments.

Reviewed By: pdillinger

Differential Revision: D115936164

fbshipit-source-id: e4997bf19eec18e4914626d479690b78c6cd3ad2
2026-08-17 19:30:37 -07:00
Peter Dillinger 495c21dc5c Unify lazy wide-column blob read verify policy; tool coverage (#15115)
Summary:
Follow-ups to the lazy wide-column read API (DB::GetEntityLazy / MultiGetEntityLazy). Three related, low-risk changes under one theme; no public API change.

1. Internal read-API cleanup (retires TODO(lazy-blob-resolution-cleanup)). The lazy byte-range work had left a whole/range/force-verify method fan-out on the blob read path: VersionBlobFetcherBase::FetchBlobRange plus a separate FetchBlobForceVerify, and SameFileBlobReader::GetSameFileBlob paired with GetSameFileBlobRange, with the (verify_checksums, force_verify) decision duplicated across branches. This folds them into a single 3-valued BlobVerifyPolicy (kVerifyIfPresent / kVerifyIfNoAmplification / kSkip, added to db/blob/blob_constants.h) derived once at the resolver boundary and threaded down. FetchBlobRange and GetSameFileBlob now take a range descriptor (range_length == kWholeBlobLength means the whole value) plus the policy; FetchBlobForceVerify and the *Range twins are removed. The public LazyColumnReadRequest::force_verify bool is unchanged. The widely-called lower-layer whole-value primitives (BlobFileReader / BlobSource / Version GetBlob) are intentionally left as delegated primitives to bound blast radius; folding those is a later cleanup.

2. Embedded block-cache-tier quirk fix (retires TODO(lazy-blob-resolution-quirks)). ValidateEmbeddedBlobIndex rejected read_tier == kBlockCacheTier before the blob-cache probe, so a same-file (embedded) column already in the blob cache returned Incomplete under kBlockCacheTier -- unlike the separate-file path, which probes the cache first. The up-front rejection moves to the two no-cache paths (ResolveEmbeddedBlob / ResolveEmbeddedBlobPinned); the cache-probing paths now serve a cached record under kBlockCacheTier, matching separate-file behavior. This is a shared lower-layer change: the non-lazy Get / GetEntity and iterator embedded read paths (get_context / MaybeResolveEmbeddedValue -> ResolveEmbeddedBlobCached) get the same corrected behavior, not just the lazy resolver.

3. Coverage. db_bench gains a readrandomentitylazy benchmark (GetEntityLazy + MultiResolve of --lazy_entity_read_length bytes per column) for measuring byte-range (partial) blob read savings; it requires open_files == -1. The crash test now also fuzzes LazyColumnReadRequest::force_verify in the lazy-vs-eager differential (force_verify affects only verification, not the returned bytes, so the comparison is unchanged).

This also sets up the next planned step -- cross-key coalescing for MultiGetEntityLazy / LazyWideColumnsBatch::MultiResolve. The internal BlobReadRequest already carries a {offset, len} range descriptor, and with the verify decision now a single BlobVerifyPolicy rather than a bool plus a separate force-verify method, a batched resolve can group a vector of range+policy requests per (CF, Version, blob file) and coalesce them onto the existing MultiGetBlob machinery -- without re-introducing whole/range/force-verify method twins in the multi-read path.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15115

Test Plan:
- New unit test DBLazyEntityTest.EmbeddedBlockCacheTierServedFromCachedValue: a cached embedded column serves whole- and sub-range reads under kBlockCacheTier instead of returning Incomplete. Existing force_verify / BlockCacheTier tests are unchanged (public behavior preserved).
- New unit test DBBlobIndexTest.EmbeddedBlobWideColumnGetEntityBlockCacheTierServesCached: covers the non-lazy GetEntity path -- a cached embedded column is served under kBlockCacheTier; the existing ...GetEntityBlockCacheTier test still asserts a genuine miss yields Incomplete.
- Crash test: extended the existing lazy differential to fuzz force_verify (reuses --lazy_entity_read_one_in).
- db_bench: readrandomentitylazy over 2000 entities x 2 uncompressed embedded blob columns (4000 B each), reading 100 B per column:

| metric | value |
|---|---|
| rocksdb.blobdb.lazy.partial.read.count | 4000 |
| rocksdb.blobdb.lazy.read.bytes | 400000 |
| rocksdb.blobdb.lazy.partial.bytes.saved | 15600000 |

i.e. 100 B read per column instead of the 4000 B whole-column value.

Reviewed By: xingbowang

Differential Revision: D116334969

Pulled By: pdillinger

fbshipit-source-id: cebe90456b23be0f5be60cdc4bff85c8a37b5442
2026-08-17 17:06:32 -07:00
Anand Ananthabhotla 575b6df1b8 Force immediate creation of a new MANIFEST if the current one cannot be reused (#15103)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15103

The crash artifact showed `MANIFEST-000005` was physically well-formed but contained an incomplete atomic group that had been tolerated at EOF. `reuse_manifest_on_open` then appended ordinary records after that tail, turning the tolerated crash tail into mid-file `corrupted atomic group` corruption seen by OpenAndCompact and backup restore.

Record whether MANIFEST recovery ended with a pending atomic group. When that marker is set, skip `ReopenManifestForAppend()` so the next manifest write creates a fresh MANIFEST instead of appending after the partial group. The existing physical-size guard remains as a separate torn-tail defense.

This replaces the earlier keep-size preallocation hypothesis with the artifact-backed root cause.

Reviewed By: xingbowang

Differential Revision: D116062618

fbshipit-source-id: fb55ce2891a6fd50fa2c670dc720e91bad27e94d
2026-08-17 16:18:06 -07:00
Peter Dillinger f0c1e3d7d9 Fix cleanup-on-failure and flaky liveness check in point_lock_manager_stress_test (#15116)
Summary:
The point_lock_manager_stress_test (via PointLockValidationTestRunner) was seen flaky under USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1, manifesting as a heap-use-after-free plus "terminate called without an active exception" / SIGABRT. There were two distinct problems.

Cleanup on failure: run() uses gtest ASSERT_* macros, which return early from run() on failure. That early return skipped the shutdown flag set and the join of the worker threads, so the worker threads kept running and accessed members of the (now destroyed) test runner object (heap-use-after-free), and destroying the still-joinable threads_ vector called std::terminate. Fixed by adding a destructor plus a StopAndJoinThreads() helper that always signals shutdown and joins the worker threads before any member is destroyed, no matter how run() exits. Copy/move are deleted since the object owns worker threads.

Underlying flaky failure: the per-second "No locks were acquired in the last 1 second" liveness assertion was too strict. Under sanitizer builds combined with very short lock timeout/expiration and post-acquisition sleeps, all threads can occasionally fail to acquire a single lock within a given one-second window without the lock manager actually being stuck. The per-thread variant of this check was already disabled for this reason; the aggregate one is now relaxed to tolerate a few consecutive no-progress seconds, which still reliably detects a genuinely stuck (dead/livelocked) lock manager. Also fixed the end-of-run check from `>= 0` (always true) to `> 0` so it actually catches a fully stuck manager.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15116

Test Plan:
Verified the cleanup path by temporarily forcing the liveness assertion to fail while worker threads were active: the test now reports a normal gtest failure with clean teardown, with no heap-use-after-free and no std::terminate/SIGABRT.

I don't have a consistent reproducer for the failure but ran
* 3x `USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1 make check`
* `USE_CLANG=1 COMPILE_WITH_ASAN=1 COMPILE_WITH_UBSAN=1 rocksptest point_lock_manager_stress_test -r100`

Reviewed By: xingbowang

Differential Revision: D116349420

Pulled By: pdillinger

fbshipit-source-id: 104784c5c59770414224ac4751eabf69f65e0958
2026-08-17 16:01:35 -07:00
Andrei Preda b616b203b8 Drop flushed WAL data from the secondary's active memtable (#15066)
Summary:
Fix https://github.com/facebook/rocksdb/issues/15051: a secondary serves stale reads because entries replayed from a WAL the primary has since flushed are never evicted from the active memtable. During TryCatchUpWithPrimary(), seal the active memtable once the installed Version provably covers everything it holds, so the following RemoveOldMemTables() call drops it. Every memtable-dropping decision takes that same watermark, so a file set the secondary cannot read never turns the stale read into a vanished key.

## Why and How

A secondary replays the primary's WALs into its active memtable and stops replaying a WAL once the primary flushes it. Entries already replayed stayed behind, and since reads consult the memtable first, they shadowed the newer flushed values indefinitely. Iterators go stale too once bottommost compaction rewrites the flushed entry's sequence number to 0; until then Get() and an iterator disagree.

Seal the active memtable during catch-up, gated on the installed Version: each point-in-time Version now carries the column family's log number it reflects, exposed via the new
ReactiveVersionSet::GetInstalledVersionLogNumber(). Sealing requires cf_id_to_current_log_[cfd->GetID()] < that number, which guarantees the primary flushed everything the memtable holds (unflushed WAL data is kept) and that the flushed data is readable through the installed Version. The gate asks the Version directly because proxies fail: the log number alone advances even when no Version was installed, and "no missing files" misses atomic_flush groups parked by a sibling's missing file and corrupt-but-present files with
verify_sst_unique_id_in_manifest=false. In those cases the memtable holds the only readable copy, so dropping it would turn a stale read into a vanished key; the gate fails closed and keeps it.

Reconcile every initialized column family, not just cfds_changed: a round that failed after advancing the log number leaves a stale memtable that later rounds see no new MANIFEST record for. It can also leave a collectible immutable memtable, so the loop consults the new O(1) MemTableList::HasOldMemTablesToRemove() before skipping a column family. The predicate keeps the skip precise: reconciling any column family with a non-empty immutable list would be correct but wasteful, installing a fresh SuperVersion every catch-up call for memtables legitimately waiting on a primary flush and costing readers a mutex acquisition each time. With it, no extra super versions are installed in the steady state. That predicate and the RemoveOldMemTables() cutoff take the installed Version's log number as well, not cfd->GetLogNumber(): that advances as soon as the flush record is read, so an immutable memtable sealed on a WAL switch could otherwise be freed while the flushed file superseding it is unreadable. A sealed memtable is stamped with cf_id_to_current_log_[cfd->GetID()] + 1, the log number following its contents.

cf_id_to_current_log_ is keyed by column family id rather than ColumnFamilyData*, because ids are never reused while a dropped column family's pointer can be recycled, and it is recorded before WriteBatchInternal::InsertInto() rather than after: a batch stops at its first failure with earlier entries already inserted, so recording afterwards could leave the map naming an older WAL than the memtable holds. While the watermark lags and uncollectible memtables accumulate, the secondary warns, since it has no write path to enforce max_write_buffer_number.

The seal sequence, shared with RecoverLogFiles() and the follower's TryCatchUpWithLeader(), is extracted into
DBImplSecondary::SealActiveMemtable(). The follower is refactored onto it with no behaviour change: it keeps cfd->GetLogNumber() as its cutoff because that catch-up path never replays WALs, so its memtables hold nothing that only a not-yet-installed Version could make readable.

## Testing

New tests in db_secondary_test cover: the stale read via both Get() and an iterator, before and after bottommost compaction; unflushed WAL-only data surviving sealing, including two column families sharing a WAL; no data loss with a missing, atomic_flush-sibling-missing, or corrupt flushed file; an immutable memtable sealed on a WAL switch surviving an unreadable flushed file; a batch rejected partway through replay leaving the recorded WAL no older than the memtable's contents; and reconciliation after a failed round (injected via FaultInjectionTestFS at a sync point). Each mechanism was individually neutralized and confirmed to fail its test.

Verified with build_tools/rocksptest.sh over db_secondary_test, db_follower_test, memtable_list_test, version_set_test and version_edit_test (all pass, also under ASSERT_STATUS_CHECKED=1), plus db_basic_test, corruption_test, repair_test and db_flush_test; the secondary tests also ran 20x under COERCE_CONTEXT_SWITCH=1 with no flakes. make format-auto and make check-sources are clean.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15066

Reviewed By: jaykorean, anand1976

Differential Revision: D115317523

Pulled By: xingbowang

fbshipit-source-id: a1d4615eb951d75e69ed242731336485c0dfa137
2026-08-17 13:47:30 -07:00
Anand Ananthabhotla 92a6308300 Verify compressed block bytes after checksum computation (#15098)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15098

Compute the base block checksum in `CompressAndVerifyBlock()` before `verify_compression` decompresses the compressed output. Parallel compression stores that base checksum on the block rep so worker threads can keep the CPU-heavy checksum and verification work off the ordered write path; the writer only applies the offset context modifier before writing the trailer.

Update the regression test to cover both serial and parallel compression paths.

Reviewed By: pdillinger

Differential Revision: D115950785

fbshipit-source-id: b5e47f7d7c41680e7e7907443fc805bc0d043d6e
2026-08-17 12:34:23 -07:00
Peter Dillinger 021374c62c Start format_version=8 development toward new features (#15104)
Summary:
format_version background: SST files are shared across RocksDB versions. format_version is a critical tool in ensuring prior (and later) versioned readers do not misinterpret a file created by some version. Opt-in features can sometimes rely on previously-rejected corners of the schema space, but not always.

This change: lands much of the format_version=8 (fv8) format surface as an internal-only draft -- readable and testable but not user-writable -- so upcoming features can build on a stable base. fv8 is NOT published: kLatestBbtFormatVersion stays 7, and reading or writing fv8 is gated behind the existing TEST-only TEST_AllowUnsupportedFormatVersion() hook. Publishing fv8 (raising the ceiling) is deferred to a later change.

The pieces of fv8 added here, and what each unlocks:

- Index value-delta escape. Existing behavior: index blocks (format_version >= 4) store each block handle as a delta against the previous entry's handle, and signal "this value is a full handle instead" by reusing the key sentinel `shared == 0`. Because the two are coupled, an index entry whose handle is not contiguous with the previous one must be forced to a restart, costing a full key as well as a full value. fv8 decouples them with an in-value escape codepoint: varint 0 means "a full BlockHandle follows," and genuine size deltas are remapped to zigzag(delta)+1 so that 0 stays free. A non-contiguous entry can then remain an ordinary key-delta-encoded, on-cadence entry. The escape applies to every index-shaped block, which includes the top-level index of index partitions and the index of filter partitions, not just the index over data blocks. Unlocks: the upcoming optimize_key_common_prefix index feature (removes its need to special-case super_block_alignment) and value-delta encoding for embedded-blob SSTs (previously disabled outright because interleaved blob records break handle contiguity). Guarded strictly on fv8; format_version <= 7 encoding and decoding are byte-for-byte unchanged. Note that restart points must not delta-encode, so they still have full values without a sentinel.

- Data block footer bit 30 is reserved as an "extended metadata present" escape. Decode rejects a set bit as unsupported/corrupt at all versions. Unlocks: adding block-level metadata in the future through the one shared footer decoder, without another format_version bump or plumbing format_version into every block parser. (Older reader versions relied on this bit being zero for some integrity/consistency checking; format_version bump now makes it safe to use.)

- Footer metaindex gap. For fv8, the top 16 bits of the footer's 4-byte metaindex-size field record the gap (in bytes) between the end of the metaindex block and the start of the footer; the low 16 bits hold the metaindex block size. Unlocks: a space very near the end of the file to save full file checksum state, for a "files carry their own checksums" feature that doesn't require SST parsing to extract the checksum states (to run them to completion on the trivial file tail).

- Reserved "user_key_common_prefix" table property. The reader returns NotSupported when it is present and non-empty. Unlocks: a future whole-file optimization that stores the common prefix of all user keys once instead of in every restart key; until implemented, fv8 readers refuse such a file rather than mis-read it.
  - Because of the problem of inferring such a prefix at the start of constructing an SST file, we expect it will rely on SstPartitioner. Here we've added SstPartitioner::ShouldPartitionByPrefix() (not yet used) that will enable this optimization, and avoid the overhead of a virtual callback to SstPartitioner on every key written to an SST file.

Supporting cleanups: the Footer's "unset" format_version sentinel is named (kInvalidFormatVersion) and the FormatVersionUses*() predicates assert against it, as each is a `version >= N` test that an unpopulated Footer would silently satisfy. Index block reader entry points (Block::NewIndexIterator() and friends) drop their defaulted trailing arguments so each caller states which codec it expects, and BlockBuilder::Add() takes delta_value by const reference rather than a pointer whose null case no caller used.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15104

Test Plan:
New unit tests, all exercising fv8 through TEST_AllowUnsupportedFormatVersion():

- block_test: value-delta escape codec round-trip (delta 0, negative, large, and the escape; asserts format_version <= 7 bytes match the legacy encoding and a legacy 0x00 still decodes as delta 0), and rejection of footer bit 30 at all versions.

- db_flush_test: fv7-vs-fv8 read equivalence for super_block_alignment, including the separate_key_value_in_data_block intersection, confirming padding events (hence escape entries) occur and fv8 index values are delta encoded.

- sst_file_reader_test: fv7-vs-fv8 read equivalence for embedded-blob SSTs, confirming fv8 (unlike fv7) delta-encodes their index values and actually emits escape entries (counted through a sync point).

- block_based_table_reader_test: a builder+reader test that injects a footer metaindex gap via a sync point (verifies the gap bytes are written and the reader locates the metaindex via the recorded gap, and that too large a gap surfaces as a builder error), and that a non-empty reserved user_key_common_prefix property makes the reader return NotSupported.

- table_test: two-level index with partitioned filters and index_block_restart_interval > 1, the configuration in which both top-level indexes delta-encode their values; verifies scan and point lookup round trips at each tested format_version including fv8.

- (Not strictly fv8) Basic SstPartitionerFixedPrefix tests in slice_test (ShouldPartition, ShouldPartitionByPrefix, their mutual consistency, and CanDoTrivialMove), which had no non-DB coverage before.

fv8 is also added to the table_test harness (GenerateArgList) and the shared footer version set (test::kFooterFormatVersionsToTest), so the existing table-level and DB-level property/checksum suites now also run at fv8. And added to crash test (minor revisions to db_stress).

## Performance details
Release db_bench (PORTABLE=0, -march=native) A/B/C on the read/write fast path,
comparing (a) parent, (b) this commit @ fv7, (c) this commit @ fv8. All DBs on
tmpfs, 20M keys, block_size=2048, compression=none. Note index_block_restart_
interval=16 (not the default 1) is required to actually exercise the fv7<->fv8
index value-delta codec -- with restart interval 1 every index entry is a full-
handle restart and the changed path never runs (confirmed delta-value=1 via
sst_dump). Reads single-thread + core-pinned, warm cache, interleaved/rotated
rounds, paired deltas.

Reads (readrandom + seekrandom, non-partitioned single ~1.9MB index block and
partitioned index+filters): fv7->fv8 in the *same* binary is within noise on all
four workloads (largest -1.0%, CIs cross zero). before->after (fv7 path) is also
within noise; perf shows a deterministic +0.4-0.6% instructions/op from the added
branches (the format_version>=8 value_delta_escape guard + refactored data-block
footer decode), which is below the throughput noise floor.

Writes (fillseq + compact total CPU): fv7->fv8 null. compact (which rewrites all
index blocks) is null before->after. fillseq showed +3.5% CPU before->after, but
perf attributes it entirely to cycles/IPC with *identical* instructions/op
(+0.01-0.04%) -- a code-layout effect on the untouched memtable-insert loop, not
this change.

Sanity: any single throughput/CPU delta <~1% is within the build-to-build noise
floor, measured directly by comparing two byte-identical binaries (swings up to
~0.9% reads; the pre-native/PORTABLE=1 build showed a spurious -1.85% that
vanished under -march=native).

Net: no measurable regression from fv8 draft; fv7 encoding/decoding byte-for-byte
unchanged, only the noted sub-1% instruction bump on the shared read path.

Reviewed By: xingbowang

Differential Revision: D116105558

Pulled By: pdillinger

fbshipit-source-id: ead776cdd396d497d47872c0956ea8c651c0d13f
2026-08-17 12:08:59 -07:00
Josh Kang 9475798205 Refresh recovery plan after error escalation (#15111)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15111

T284514290 exposed an online-recovery race between the automatic recovery coordinator and a foreground synchronous write. A retryable no-WAL flush error selected lightweight retry-flush recovery. While that recovery attempt waited for the failed background flush, `Delete(sync=true)` appended and synced sequence N to the active WAL, then failed while persisting WAL-tracking metadata to the MANIFEST. Because memtable insertion and sequence publication happen after that MANIFEST step, the record was durable but logically unpublished.

OLD: the concurrent error triggered another attempt, but the attempt reused a stale recovery plan. That preserved both ingredients needed for a duplicate: the first `Delete(key, N)` remained in the active WAL, while the published sequence remained `N-1`.

```
Recovery coordinator                         Foreground sync-write thread
------------------------------------         --------------------------------
[db_mutex held]
plan = RetryFlush
(copied once before the retry loop)

attempt 1: ResumeImpl(plan)
  WaitForBackgroundWork()
  bg_cv_.Wait()
  [atomically releases db_mutex]  ---------> assign sequence N
                                             append + sync Delete(key, N)
                                             [db_mutex held]
                                             MANIFEST WAL edit fails
                                             no memtable insert
                                             no LastSequence publication
                                             write returns I/O error

                                             recovery_error_ = Hard
                                               (requests another attempt)
                                             bg_error_ = Hard
                                             recover_context_ eventually
                                               becomes FullRecovery
                                             [db_mutex released]
  [db_mutex reacquired]          <---------
  ClearBGError() sees recovery_error_
  attempt 1 cannot clear the error

attempt 2:
  recovery_error_ = OK
  ResumeImpl(plan)
             ^
             +-- still the cached RetryFlush plan

  RetryFlushesForErrorRecovery()
    retries the failed immutable flush only
    does not call SwitchMemtable()
    does not switch the active WAL
  ClearBGError() succeeds
                                             caller retries Delete(key)
                                             LastSequence is still N-1
                                             => assigns sequence N again
                                             appends Delete(key, N) to same WAL
                                             memtable insert succeeds
                                             publishes LastSequence = N

Reopen / WAL replay:
  first  (key, N, deletion) -> memtable insert succeeds
  second (key, N, deletion) -> TryAgain("key+seq exists")
  normal WAL recovery has seq_per_batch_ == false
  assert(seq_per_batch_) fails
```

The cached argument to `ResumeImpl()` is therefore the direct bridge to the assertion: lightweight retry-flush recovery can report success without retiring the WAL that contains the unpublished record. Since it also does not publish sequence N, the successful caller retry writes the identical internal key into that same WAL.

NEW: the same recovery coordinator consumes the concurrent error through a mutex-ordered handoff.

```
Recovery coordinator                         Foreground sync-write thread
------------------------------------         --------------------------------
[db_mutex held]
attempt 1:
  plan = recover_context_  // RetryFlush
  ResumeImpl(plan)
  WaitForBackgroundWork()
  bg_cv_.Wait()
  [atomically releases db_mutex]  ---------> append + sync Delete(key, seq=N)
                                             [db_mutex held]
                                             MANIFEST WAL edit fails
                                             CheckAndSet...(Hard, FullRecovery):
                                               recovery_error_ = Hard
                                               bg_error_ = Hard
                                               recover_context_ = FullRecovery
                                             [db_mutex released]
  [db_mutex reacquired]          <---------
  ClearBGError() sees recovery_error_
  current attempt cannot clear the error

next retry, with db_mutex held:
  recovery_error_ = OK
  plan = recover_context_  // FullRecovery
  ResumeImpl(plan)
    SwitchMemtable()
    switch to a new WAL
    clear the background error
```

The foreground unlock of `db_mutex_` happens before the recovery coordinator reacquires it and snapshots the next plan. Publishing `bg_error_` and `recover_context_` together under that mutex prevents the next attempt from observing a new severity with an old recovery requirement. `recovery_error_` is the per-attempt latch: a concurrent I/O error cannot be cleared as part of the in-flight attempt; it either stops recovery or forces another attempt to consume the updated context. If another error arrives while `ResumeImpl()` releases the mutex again, the same handoff repeats. No second recovery coordinator is needed.

Manual `Resume()` now uses the retained recovery context instead of reconstructing one from the background-work scheduling flag, and the soft-error clear-only shortcut is allowed only when that flag shows there is no pending recovery work. The write ordering and `assert(seq_per_batch_)` detector are unchanged.

Reviewed By: anand1976, xingbowang

Differential Revision: D116106177

fbshipit-source-id: a78450c0924a326b5302ae6709e367f07bd3d5a6
2026-08-17 09:04:06 -07:00
413 changed files with 44799 additions and 6364 deletions

No files matched your search

+1
View File
@@ -12,6 +12,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
with:
+2 -2
View File
@@ -1,4 +1,4 @@
name: facebook/rocksdb/nightly
name: facebook/rocksdb/nightly-candidate
on: workflow_dispatch
permissions: {}
jobs:
@@ -10,7 +10,7 @@ jobs:
labels: arm64large
container:
image: ubuntu-2004:202111-02
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
+8 -8
View File
@@ -11,7 +11,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
with:
@@ -33,7 +33,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
env:
TEST_TMPDIR: "/tmp/rocksdb_test_tmp"
steps:
@@ -47,7 +47,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=32gb
env:
CC: clang-21
CXX: clang++-21
@@ -84,7 +84,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -103,7 +103,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -161,7 +161,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -174,7 +174,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -189,7 +189,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
+18 -18
View File
@@ -92,7 +92,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -110,7 +110,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -135,7 +135,7 @@ jobs:
labels: 32-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=64gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -160,7 +160,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -180,7 +180,7 @@ jobs:
labels: 32-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=64gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -209,7 +209,7 @@ jobs:
shard: [0, 1, 2, 3]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -232,7 +232,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -252,7 +252,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/setup-ccache"
@@ -306,7 +306,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -373,7 +373,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -394,7 +394,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: gcc:latest
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- run: apt-get update -y && apt-get install -y libgflags-dev
@@ -422,7 +422,7 @@ jobs:
crash_duration: 240
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb --ulimit nofile=1048576:1048576
options: --shm-size=8gb --ulimit nofile=1048576:1048576
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -457,7 +457,7 @@ jobs:
shard: [0, 1, 2]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=64gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -484,7 +484,7 @@ jobs:
shard: [0, 1, 2]
container:
image: ghcr.io/facebook/rocksdb_ubuntu:24.1
options: --shm-size=16gb
options: --shm-size=64gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -507,7 +507,7 @@ jobs:
labels: 16-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -614,7 +614,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -637,7 +637,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
@@ -740,7 +740,7 @@ jobs:
labels: 4-core-ubuntu
container:
image: evolvedbinary/rocksjava:alpine3_x64-be
options: --shm-size=16gb
options: --shm-size=8gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/install-maven"
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
timeout-minutes: 840
container:
image: ghcr.io/facebook/rocksdb_ubuntu:22.1
options: --shm-size=16gb
options: --shm-size=32gb
steps:
- uses: actions/checkout@v4.1.0
- uses: "./.github/actions/pre-steps"
+68 -4
View File
@@ -96,6 +96,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
"db/merge_operator.cc",
"db/multi_scan.cc",
"db/output_validator.cc",
"db/periodic_compaction_phaser.cc",
"db/periodic_task_scheduler.cc",
"db/range_del_aggregator.cc",
"db/range_tombstone_fragmenter.cc",
@@ -205,6 +206,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
"table/block_based/block_cache.cc",
"table/block_based/block_prefetcher.cc",
"table/block_based/block_prefix_index.cc",
"table/block_based/builtin_index_factory.cc",
"table/block_based/data_block_footer.cc",
"table/block_based/data_block_hash_index.cc",
"table/block_based/filter_block_reader_common.cc",
@@ -239,6 +241,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
"table/plain/plain_table_index.cc",
"table/plain/plain_table_key_coding.cc",
"table/plain/plain_table_reader.cc",
"table/range_del_block.cc",
"table/sst_file_dumper.cc",
"table/sst_file_reader.cc",
"table/sst_file_writer.cc",
@@ -413,13 +416,27 @@ cpp_library_wrapper(name="rocksdb_test_lib", srcs=[
"table/mock_table.cc",
"test_util/mock_time_env.cc",
"test_util/secondary_cache_test_util.cc",
"test_util/simple_external_table_factory.cc",
"test_util/testharness.cc",
"test_util/testutil.cc",
"tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
"tools/trace_analyzer_tool.cc",
"utilities/agg_merge/test_agg_merge.cc",
"utilities/cassandra/test_utils.cc",
], deps=[":rocksdb_lib"], headers=[], link_whole=False, extra_test_libs=True)
], deps=[":rocksdb_lib"], headers=[
"db/db_test_util.h",
"db/db_with_timestamp_test_util.h",
"table/mock_table.h",
"test_util/mock_time_env.h",
"test_util/secondary_cache_test_util.h",
"test_util/simple_external_table_factory.h",
"test_util/testharness.h",
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/trace_analyzer_tool.h",
"utilities/agg_merge/test_agg_merge.h",
"utilities/cassandra/test_utils.h",
], link_whole=False, extra_test_libs=True)
cpp_library_wrapper(name="rocksdb_with_faiss_test_lib", srcs=[
"db/db_test_util.cc",
@@ -427,13 +444,27 @@ cpp_library_wrapper(name="rocksdb_with_faiss_test_lib", srcs=[
"table/mock_table.cc",
"test_util/mock_time_env.cc",
"test_util/secondary_cache_test_util.cc",
"test_util/simple_external_table_factory.cc",
"test_util/testharness.cc",
"test_util/testutil.cc",
"tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
"tools/trace_analyzer_tool.cc",
"utilities/agg_merge/test_agg_merge.cc",
"utilities/cassandra/test_utils.cc",
], deps=[":rocksdb_with_faiss_lib"], headers=[], link_whole=False, extra_test_libs=True)
], deps=[":rocksdb_with_faiss_lib"], headers=[
"db/db_test_util.h",
"db/db_with_timestamp_test_util.h",
"table/mock_table.h",
"test_util/mock_time_env.h",
"test_util/secondary_cache_test_util.h",
"test_util/simple_external_table_factory.h",
"test_util/testharness.h",
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/trace_analyzer_tool.h",
"utilities/agg_merge/test_agg_merge.h",
"utilities/cassandra/test_utils.h",
], link_whole=False, extra_test_libs=True)
cpp_library_wrapper(name="rocksdb_tools_lib", srcs=[
"test_util/testutil.cc",
@@ -442,7 +473,12 @@ cpp_library_wrapper(name="rocksdb_tools_lib", srcs=[
"tools/simulated_hybrid_file_system.cc",
"tools/tool_hooks.cc",
"tools/trace_analyzer_tool.cc",
], deps=[":rocksdb_lib"], headers=[], link_whole=False, extra_test_libs=False)
], deps=[":rocksdb_lib"], headers=[
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/simulated_hybrid_file_system.h",
"tools/trace_analyzer_tool.h",
], link_whole=False, extra_test_libs=False)
cpp_library_wrapper(name="rocksdb_cache_bench_tools_lib", srcs=["cache/cache_bench_tool.cc"], deps=[":rocksdb_lib"], headers=[], link_whole=False, extra_test_libs=False)
@@ -466,10 +502,28 @@ rocks_cpp_library_wrapper(name="rocksdb_stress_lib", srcs=[
"db_stress_tool/expected_value.cc",
"db_stress_tool/multi_ops_txns_stress.cc",
"db_stress_tool/no_batched_ops_stress.cc",
"test_util/simple_external_table_factory.cc",
"test_util/testutil.cc",
"tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
"tools/trace_analyzer_tool.cc",
], headers=[])
], headers=[
"db_stress_tool/db_stress_common.h",
"db_stress_tool/db_stress_compaction_service.h",
"db_stress_tool/db_stress_compression_manager.h",
"db_stress_tool/db_stress_driver.h",
"db_stress_tool/db_stress_filters.h",
"db_stress_tool/db_stress_listener.h",
"db_stress_tool/db_stress_shared_state.h",
"db_stress_tool/db_stress_test_base.h",
"db_stress_tool/db_stress_wide_merge_operator.h",
"db_stress_tool/expected_state.h",
"db_stress_tool/expected_value.h",
"db_stress_tool/multi_ops_txns_stress.h",
"test_util/simple_external_table_factory.h",
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/trace_analyzer_tool.h",
])
cpp_binary_wrapper(name="ldb", srcs=["tools/ldb.cc"], deps=[":rocksdb_tools_lib"], extra_preprocessor_flags=[], extra_bench_libs=False)
@@ -4613,6 +4667,12 @@ cpp_unittest_wrapper(name="bloom_test",
extra_compiler_flags=[])
cpp_unittest_wrapper(name="builtin_index_factory_test",
srcs=["table/block_based/builtin_index_factory_test.cc"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
cpp_unittest_wrapper(name="cache_reservation_manager_test",
srcs=["cache/cache_reservation_manager_test.cc"],
deps=[":rocksdb_test_lib"],
@@ -4753,6 +4813,7 @@ cpp_unittest_wrapper(name="compression_test",
cpp_unittest_wrapper(name="configurable_test",
srcs=["options/configurable_test.cc"],
source_headers=["options/configurable_test.h"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
@@ -5457,6 +5518,7 @@ cpp_unittest_wrapper(name="periodic_task_scheduler_test",
cpp_unittest_wrapper(name="persistent_cache_test",
srcs=["utilities/persistent_cache/persistent_cache_test.cc"],
source_headers=["utilities/persistent_cache/persistent_cache_test.h"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
@@ -5475,6 +5537,7 @@ cpp_unittest_wrapper(name="point_lock_manager_stress_test",
cpp_unittest_wrapper(name="point_lock_manager_test",
srcs=["utilities/transactions/lock/point/point_lock_manager_test.cc"],
source_headers=["utilities/transactions/lock/point/point_lock_manager_test.h"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
@@ -5685,6 +5748,7 @@ cpp_unittest_wrapper(name="trace_analyzer_test",
cpp_unittest_wrapper(name="transaction_test",
srcs=["utilities/transactions/transaction_test.cc"],
source_headers=["utilities/transactions/transaction_test.h"],
deps=[":rocksdb_test_lib"],
extra_compiler_flags=[])
+24 -6
View File
@@ -8,12 +8,18 @@ This document provides guidance for generating and reviewing code in the RocksDB
### Code Quality and Maintainability
**Clarity and Readability:** Write clear, self-documenting code. Use meaningful variable names, add comments for complex logic, and structure code to minimize cognitive load. Avoid clever tricks that sacrifice readability for marginal performance gains unless absolutely necessary. Avoid static_cast, reinterpret_cast, and C-style casts; static_cast_with_check, up_cast, and lossless_cast from cast_util.h are preferred.
**Clarity and Readability:** Write clear, self-documenting code. Use meaningful variable names, add comments for complex logic, and structure code to minimize cognitive load. Avoid clever tricks that sacrifice readability for marginal performance gains unless absolutely necessary.
**Consistent Style:** Follow existing code style conventions. RocksDB uses `.clang-format` for formatting, specific naming conventions, and structural patterns. Deviations from these patterns are frequently flagged in reviews.
**Error Handling:** Ensure robust error handling throughout the codebase. Use RocksDB's `Status` type consistently, propagate errors appropriately, and avoid silently ignoring failures. Reviewers pay close attention to edge cases and failure modes.
**Refactoring traps:** At least in production code, avoid constructs that could allow existing code to unexpectedly, quietly change meaning.
Specifically:
* Avoid new defaulted parameters. This is the #1 trap on refactoring!
* Avoid static_cast, reinterpret_cast, and C-style casts; static_cast_with_check, up_cast, and lossless_cast from cast_util.h are preferred.
* Avoid declared type `auto` (`auto&`, `auto*` OK).
### Testing Philosophy
**Comprehensive Coverage:** Every change should include appropriate test coverage. This includes unit tests for isolated functionality, integration tests for component interactions, and stress tests for concurrency and performance validation. Reviewers will ask for additional tests if coverage is insufficient.
@@ -32,7 +38,7 @@ This document provides guidance for generating and reviewing code in the RocksDB
**Memory Copy:** Avoid unnecessary memory copies. Use move semantics, `std::string_view`, `Slice`, and pass-by-reference where appropriate. Be aware of implicit copies in STL containers and function returns. Prefer in-place operations over copy-and-modify patterns.
**CPU Cache Efficiency:** Design data structures and access patterns to be cache-friendly. Keep frequently accessed data together (data locality). Prefer sequential memory access over random access. Be mindful of cache line sizes (typically 64 bytes) and avoid false sharing in concurrent code. Consider struct packing and field ordering to improve cache utilization.
**CPU Cache Efficiency:** Design data structures and access patterns to be cache-friendly. Keep frequently accessed data together (data locality). Prefer sequential memory access over random access. Be mindful of cache line sizes (CACHE_LINE_SIZE, typically 64 bytes) and avoid false sharing in concurrent code (see CacheAlignedWrapper). Especially when adding new data members, be mindful of ordering within structs and classes to reduce unnecessary padding (special care with bool members) and avoid larger-than-necessary types (e.g. prefer OptSlice to std::optional<Slice>).
**Loop Optimization:** Look for opportunities to collapse nested loops, reduce loop overhead, and minimize branch mispredictions. Hoist invariant computations out of loops. Consider loop unrolling for tight inner loops. Batch operations when possible to amortize per-operation overhead.
@@ -47,12 +53,14 @@ This document provides guidance for generating and reviewing code in the RocksDB
- **Cold path** (executed rarely, e.g., DB open, configuration parsing, error handling): Maintainability and clarity are more important. Prefer readable code over micro-optimizations. Complex optimizations here add maintenance burden with negligible performance benefit.
- **Warm path** (moderate frequency): Balance both concerns. Use profiling data to guide optimization decisions.
**Avoid Premature Optimization:** While performance is critical, focus on correctness first, then optimize based on profiling data. However, be performance-aware from the start—choosing the right algorithm and data structure upfront is not premature optimization. Use the hot path analysis above to decide how much optimization effort is warranted.
**Avoid Premature Optimization:** Performance is known critical in places like inner loops of compaction, read path and write path. Elsewhere, try to find a "holy trinity" solution of *correct, simple, and reasonably efficient*. More complex solutions are only appropriate when profiling/benchmarking data shows a measurable payoff.
### API Design and Compatibility
**Backwards Compatibility:** RocksDB maintains strong backwards compatibility guarantees. Breaking changes are rare and require extensive justification. When deprecating features, follow the project's deprecation policy (typically spanning multiple releases).
**Behavioral Compatibility:** Behavior changes are human judgment calls that must be intentional, well-documented (including release notes), and well researched for potential downstream production impact. Agents and reviewers must call out potential gaps, ambiguity, or adverse impacts in intentionality, documentation, and production impact. For an existing accepted input—including an undocumented, discouraged, or historically ignored value—the analysis should compare old and new observable behavior, including performance and resource impact, identify known direct callers and stored configurations, and add or update regression coverage as appropriate.
**API Consistency:** New APIs should be consistent with existing patterns. Use similar naming conventions, parameter ordering, and return types. Reviewers will suggest changes to improve consistency with the broader codebase.
**Documentation:** Public APIs must be thoroughly documented, without unnecessary embelishment nor dwelling on implementation details nor project planning. When non-obvious, include usage examples, parameter descriptions, known bugs or limitations, notes on thread safety, performance characteristics, and compatibility considerations. Re-read comments for ambiguous terminology and phrasing, such as ambiguously re-purposed programming jargon.
@@ -73,7 +81,7 @@ The database core handles write-ahead logging (WAL), memtables, compaction, and
**Testing:** Database core changes require extensive testing, including unit tests, integration tests, and stress tests. Test with various configurations, compaction styles, and concurrent workloads.
### Public Headers (`include`)
### Public Headers / "Public API" (`include/rocksdb/`)
Public headers define RocksDB's API surface. Changes here have the highest compatibility impact.
@@ -236,6 +244,11 @@ from an implementation detail instead of an explicit option.
## Important tips
### Pull request and diff summaries
* Before writing or updating an external RocksDB pull request summary or its
corresponding internal Phabricator diff summary, read and follow
`claude_md/pull_request_summary_guideline.md`.
### Build system
* There are 3 build system. Make for git clones, BUCK (meta internal) for hg
clones, and CMake for some special cases.
@@ -373,6 +386,9 @@ architectures.
### Adding new public API
Refer to claude_md/add_public_api.md
### Changing existing public API behavior
Follow the Behavioral Compatibility requirements above.
### Adding new option
Refer to claude_md/add_option.md
@@ -402,7 +418,7 @@ architectures.
* Release note should be kept short at high level for external user consumption.
Release notes identify what users might care about most in a release. They
are not exhaustive and are not a guide. PLEASE learn from past agents who
ried to build elaborate release notes with implementation details and
tried to build elaborate release notes with implementation details and
elsewhere-documented nuance. That wastes time. Fight the bias that
"my change" is important so must be worthy of release note mention.
* If more than single markdown line, consider how their formatting will be
@@ -418,7 +434,9 @@ architectures.
multiple minutes.
* Run `AUTO_CLEAN=1 ASSERT_STATUS_CHECKED=1 make check` to verify all Status
objects are properly checked. This catches missing error handling that can
lead to silent data corruption.
lead to silent data corruption. Merely building with ASSERT_STATUS_CHECKED=1
accomplishes nothing; it enables a runtime check. New unit tests missing
Status checks is a common failure point, even from agents.
### Monitoring make check progress
* Use `make check-progress` to get machine-parseable JSON progress while
+5
View File
@@ -838,6 +838,7 @@ set(SOURCES
db/merge_operator.cc
db/multi_scan.cc
db/output_validator.cc
db/periodic_compaction_phaser.cc
db/periodic_task_scheduler.cc
db/range_del_aggregator.cc
db/range_tombstone_fragmenter.cc
@@ -937,6 +938,7 @@ set(SOURCES
table/block_based/block_cache.cc
table/block_based/block_prefetcher.cc
table/block_based/block_prefix_index.cc
table/block_based/builtin_index_factory.cc
table/block_based/data_block_hash_index.cc
table/block_based/data_block_footer.cc
table/block_based/filter_block_reader_common.cc
@@ -970,6 +972,7 @@ set(SOURCES
table/plain/plain_table_index.cc
table/plain/plain_table_key_coding.cc
table/plain/plain_table_reader.cc
table/range_del_block.cc
table/sst_file_dumper.cc
table/sst_file_reader.cc
table/sst_file_writer.cc
@@ -1594,6 +1597,7 @@ if(WITH_TESTS)
options/options_test.cc
table/block_based/block_based_table_reader_test.cc
table/block_based/block_test.cc
table/block_based/builtin_index_factory_test.cc
table/block_based/data_block_hash_index_test.cc
table/block_based/full_filter_block_test.cc
table/block_based/partitioned_filter_block_test.cc
@@ -1685,6 +1689,7 @@ if(WITH_TESTS)
db/db_with_timestamp_test_util.cc
monitoring/thread_status_updater_debug.cc
table/mock_table.cc
test_util/simple_external_table_factory.cc
utilities/agg_merge/test_agg_merge.cc
utilities/cassandra/test_utils.cc
)
+85 -1
View File
@@ -1,10 +1,94 @@
# Rocksdb Change Log
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
## 11.9.1 (08/18/2026)
## 11.12.0 (10/07/2026)
### New Features
* Added `FullExternalTableFactory` and related full-mode external table APIs that preserve RocksDB internal keys, enabling external tables to support nonzero sequence numbers, point and single deletions, merge operands, blob indexes, wide-column entities, snapshots, live DB writes, and multi-level compactions.
### Public API Changes
* Added experimental coroutine capability interfaces for basic and full `ExternalTableReader` implementations. Coroutine-capable external readers can initiate file reads through `FSRandomAccessFile::SubmitReadAsync()`.
* `ExternalTableFactory::NewTableReader()` now receives an already-opened `FSRandomAccessFile` and its file size instead of requiring implementations to reopen `file_path`. The supplied file preserves RocksDB's direct-I/O alignment handling, rate limiting, read statistics and listener notifications, prefetching, and synchronous, asynchronous, and batched reads.
* Added `PutRangeDeletionBlock()`, `GetRangeDeletionBlock()`, and `IsDeleteRangeSupported()` to the external table interfaces so formats can opt in to range deletions using RocksDB's raw range-deletion metadata block.
* Opaque `PreparedFileInfo` metadata can now be serialized to a checksummed representation with `SerializePreparedFileInfo()` and restored with `DeserializePreparedFileInfo()` for reuse across processes.
* Removed the deprecated `IngestExternalFileOptions::write_global_seqno` option and its C, Java, and `ldb` APIs, `TableProperties::external_sst_file_global_seqno_offset`, and the `file_offset` output from `ExternalTableReader::GetPropertiesBlock()`. Existing external SST files whose global sequence number was rewritten without updating the properties-block checksum remain readable.
* `TablePropertiesNames` has no apparent use in the public API so has been deprecated there.
* `TableProperties::creation_time` is renamed to `oldest_ancestor_time`, which better describes its meaning, along with `TablePropertiesNames::kCreationTime` to `kOldestAncestorTime`. The misspelled `SstFileMetaData::oldest_ancester_time` is renamed to `oldest_ancestor_time`. The old names remain available as deprecated aliases and will be removed in a future release.
* Added `WriteBufferFlushPolicy` and `WriteBufferManager` policy selection for flushing the oldest or largest column family, including configurable batched flushing of the largest column families across DBs sharing a manager, with C and Java bindings.
### Behavior Changes
* When `ExternalTableBuilderBase::PutPropertiesBlock()` returns `NotSupported`, RocksDB now overrides `num_entries`, `raw_key_size`, and `raw_value_size` from `ExternalTableBuilderBase::GetTableProperties()` with the logical totals tracked by the external-table adapter. Format-specific properties such as `data_size` remain unchanged.
* The `FSWritableFile` passed to `ExternalTableFactory::NewTableBuilder()` now reports logical file size including RocksDB-buffered data and exposes the underlying file's direct-I/O mode and required buffer alignment.
* `Iterator::Refresh()` now returns `NotSupported` on iterators from a secondary DB's `DB::NewIterators()` only, because those iterators are pinned to one cross-column-family read view. Iterators from `DB::NewIterator()` are unaffected; to advance a batch, call `DB::NewIterators()` again after `TryCatchUpWithPrimary()`.
* `SstFileReader` now reports standard SST read statistics and file I/O listener notifications, and honors the configured read rate limiter.
### Bug Fixes
* Fixed full recovery from retryable background errors to atomically advance WAL replay boundaries for all column families, preventing partial recovery of multi-column-family write batches after a failed recovery flush.
* Fixed RocksDB builds on big-endian architectures such as s390x by disabling the optional persistent fault-injection log, whose binary format is little-endian-only.
* Fixed `DB::ClipColumnFamily()` deleting a non-L0 SST file whose largest key equals `begin_key`, which lost keys at the start of the kept range.
* Fixed swapped `oldest_key_time` and `newest_key_time` values in the `table_file_creation` event log entry.
* Fixed `ExternalTableReader::VerifyChecksum()` results not being honored by the external table adapter.
* Fixed a bug in manifest handling that could lead to a hang or memory corruption in the case of LSM sanity checks being fatally violated (exceptional; see `force_consistency_checks`).
* Fixed `MultiGet()` with `ReadOptions::read_tier == kBlockCacheTier` reading data blocks from disk when `allow_mmap_reads` is false. Keys whose data block is not in the block cache now return `Status::Incomplete()`, the same as `Get()`.
* Fixed trace replay (`DB::NewDefaultReplayer()`, db_bench `replay`) failing every read record (Get, MultiGet, iterator seek) on the default column family with `Corruption: Invalid Column Family ID` when the default column family handle was not passed, which is always the case in db_bench with `--num_column_families=1`.
* Fixed multi-threaded `Replayer::Replay()` ignoring errors from executing trace records and returning OK. It now stops and returns the error, including `Status::NotSupported()` from executing a record, as single-threaded replay does.
* Fixed secondary DB `NewIterators()` returning iterators from different database states or blocking for `TryCatchUpWithPrimary()` by serving the last completed catch-up view.
### Performance Improvements
* Single Get coroutine reads using non-mmap, non-direct I/O on a `kFSBuffer`-capable `FileSystem` now use `FSRandomAccessFile::SubmitReadAsync()` instead of falling back to synchronous `Read()`. When rate-limited, these reads acquire their full token budget before submission to preserve a single filesystem-owned buffer, which can make physical I/O burstier than reads using caller-provided buffers.
## 11.11.0 (09/21/2026)
### New Features
* External tables now support DB-assigned global sequence numbers when `IngestExternalFileOptions::allow_global_seqno` is enabled, allowing overlapping external tables to be ingested across multiple levels when compactions are disabled. This includes the deprecated `write_global_seqno` path when the reader supplies the properties block's file offset.
* Added checksum handoff support for SST file writes when `BlockBasedTableOptions::checksum` is `kCRC32c`, allowing block and trailer CRC32C checksums to be passed from the table builder to the file writer for buffered-data validation and filesystem in-flight write protection.
* Enhanced `sst_dump --command=recompress` to be more powerful for benchmarking, including through automation with a new `--json` option for machine-readable output.
### Public API Changes
* Changed the experimental async read statistics API by removing `DB::AsyncCallback::EnableStats()` and changing `OnComplete(const PerfContext*, const IOStatsContext*)` to `OnComplete()`. Callers must configure TLS statistics before each async or coroutine read. `DB::GetAsync()` and `DB::MultiGetAsync()` callbacks must copy per-operation statistics from TLS during `OnComplete()`; coroutine users must copy them from inside a `CoroStackableDB` hook before suspending again. Each operation leaves TLS statistics collection disabled.
* Added `Compressor::MaybeOverrideCompressionOptions()` (EXPERIMENTAL), an omnibus hook letting a custom compressor adjust the subset of `CompressionOptions` that the SST builder applies rather than the compressor itself: `parallel_threads`, `max_compressed_bytes_per_kb`, `auto_skip`, and `auto_skip_min_sample_every` (and potentially more in the future). This deprecates `Compressor::GetRecommendedParallelThreads()`, which it subsumes; the default implementation still honors an overridden `GetRecommendedParallelThreads()` for compatibility.
* Added experimental `CoroDB::CoGet()` and `CoroDB::CoMultiGet()` overloads for point reads from an `SstFileReader`.
* Added experimental `DB::GetWithMetadata()` and `DB::MultiGetWithMetadata()` APIs with opt-in output metadata for key timestamps and explicit-snapshot newer-version detection.
* Added a `DB::NewIterators()` overload that accepts parallel vectors of `ReadOptions` and column family handles, allowing each returned iterator to use independent read settings while sharing a consistent database state. The vectors must have the same size, all snapshots must be null or identical, and all `tailing` settings must match.
* Added `ReadOptions::custom_context`, an opaque, non-owning pointer passed through table reads so external table implementations can consume application-defined read options.
### Behavior Changes
* `ExternalTableReader::MultiGet()` is now used by RocksDB's `DB::MultiGet()` path. Previously, the external table adapter used `TableReader`'s fallback, which called `ExternalTableReader::Get()` separately for each key.
* `DB::OpenAndCompact()` now requires `output_directory` to be a simple name for a new subdirectory when the new `use_session_tmp_dir_for_remote_compaction` DB option is enabled. The option places that subdirectory under the canonical `<dbname>/session_tmp/` root, and `DB::Open()` deletes stale contents under the root after recovery. The disabled default preserves the existing `<dbname>/<output_directory>` layout without automatic cleanup. Previously, remote-compaction output directories were created directly under `<dbname>`, which made it challenging to safely identify and delete obsolete files.
* Several block-based SST table properties now better support compression benchmarking and optimization: `uncompressed_data_size` is now persisted in the file (previously it was computed only at write time and read back as 0); a new `lsm_info_at_creation` property records the LSM level and "bottommost" status at the file's creation, distinguishing "not applicable" (e.g. SstFileWriter files) from "unknown" (older files); and the `compression_options` property now records the full compression manager configuration instead of only its name.
### Bug Fixes
* Fixed a usually-minor performance bug where `Cache::ApplyToAllEntries()` used single-entry batches instead of the intended 256-entry default for `ApplyToAllEntriesOptions::average_entries_per_lock`. All of the Cache implementations (LRU and HCC) benefit from the fix, but the disfavored LRU cache was especially impacted by repeated mutex acquisition and release.
* Fixed stats-disabled coroutine and asynchronous reads inheriting enabled TLS statistics state when interleaved with stats-enabled reads on the same executor. Each read now uses its captured per-call configuration, while enabled request counters remain isolated across suspensions.
* Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) requested with `OpenAndCompactOptions::allow_resumption=true` but for which resumption was internally disabled (incompatible with `paranoid_file_checks` or output-iteration verification) could fail a retried attempt, because the output directory was not cleaned before starting fresh and output files left by a previously interrupted attempt collided with the reused file numbers.
* Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) could fail when the primary rotated its MANIFEST while the worker was opening the primary's directory read-only. The open reads `CURRENT`, and the primary publishes a new MANIFEST by renaming a freshly written `CURRENT` over the old one; on file systems without POSIX read-after-unlink semantics (remote/object stores) the replaced object is then reported as gone (`PathNotFound`/`NotFound`, or `TryAgain` for the MANIFEST equivalent) instead of remaining readable. `DB::OpenAndCompact()` now retries the secondary open a bounded, configurable number of times for those transient statuses, so a concurrent rotation no longer fails the compaction (which, for a `CompactionService` that does not fall back to a local compaction, became a background error on the primary).
* Fixed secondary DB `NewIterators()` returning iterators from different database states when `TryCatchUpWithPrimary()` runs concurrently.
* Fixed a bug where a `SeekToFirst()` on an iterator previously used for a `Seek()` could end the scan early and silently omit keys with `ReadOptions::async_io`, or otherwise run with readahead effectively disabled. Affects scans using `ReadOptions::prefix_same_as_start` together with `ReadOptions::auto_readahead_size` (enabled by default); memory-mapped reads are unaffected.
### Performance Improvements
* `DB::MultiGetEntityLazy()` now reads all keys of a batch under a single shared per-column-family SuperVersion pin, and a single `LazyWideColumnsBatch::MultiResolve()` coalesces the resulting blob reads across keys: one read per blob file (and one per SST for embedded blobs), for whole-column and byte-range reads alike, instead of reading each key's blobs separately.
## 11.10.0 (08/28/2026)
### New Features
* Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
* A new experimental feature spreads out periodic (time-based) compaction trigger times across a fleet of similarly-configured DBs to avoid recurring "thundering herd" triggers, especially with remote compaction. See the new experimental DB option `periodic_compaction_phase_recovery_percent`.
### Public API Changes
* `IngestExternalFileArg::atomic_replace_range` now supports a range that only partially overlaps existing files, instead of failing. Requires universal compaction and a table factory that supports range deletion; other configurations still fail on partial overlap. The range is also now specified as half-open (`start` inclusive, `limit` exclusive), resolving a previously documented ambiguity in the upper bound.
* Added an immutable `external_table_config` option that is persisted in OPTIONS files and passed to external table factories during bootstrap.
### Bug Fixes
* Fixed a liveness bug where RocksDB could remain indefinitely in recovery from a soft background error when concurrent writes continuously generated normal flush requests.
* Fixed a race in online error recovery where a concurrent higher-severity I/O error could leave a stale retry-flush plan active, prevent the required WAL switch, and create duplicate internal WAL history detected on DB reopen.
* Fix "bottommost files" infinite compaction loops (`kBottommostFiles`). This happens with the unsupported combination of leveled compaction and `preserve_internal_time_seconds` / `preclude_last_level_data_seconds`, and in an edge case with user-defined timestamps when `full_history_ts_low` is unset. The underlying logic is now more unified to reduce risk of a similar divergence in logic in the future.
* Fixed `reuse_manifest_on_open` to create a fresh MANIFEST instead of appending after recovery sees an incomplete atomic group at the MANIFEST tail.
* Starting remote compaction (`CompactionService` / `DB::OpenAndCompact`) is now more efficient and more reliable, by avoiding unnecessary file operations (latency) while using more accurate consistency / precondition checks, both avoiding some spurious failures and rejecting more legitimate inconsistencies. New DBOption `remote_compaction_manifest_floor` can be disabled in case of unexpected problems with the change.
* Fixed unbounded retention of recovered transaction state by a long-lived secondary opened with `allow_2pc`.
* Fixed stale reads on secondary instances, where entries replayed from a WAL that the primary had already flushed could shadow the newer values in the flushed files indefinitely. Also fixed a case where a secondary could stop returning such a key altogether while the primary's flushed file holding it was unreadable.
* Fixed subset checkpoints to discard tolerated MANIFEST tail corruption before appending column-family drop records.
### Performance Improvements
* Improved compaction CPU efficiency for column families using an `SstPartitioner` (such as the built-in `SstPartitionerFixedPrefix`): when the partitioner can express its next boundary as a key prefix via the new `SstPartitioner::ShouldPartitionByPrefix()` API, compaction now checks that cached prefix inline instead of copying every user key and making a virtual `ShouldPartition()` call for each key-value written to an SST.
## 11.9.0 (08/14/2026)
### New Features
* Added an integrated and improved compression auto-skip feature (new CompressionOptions, still experimental) that saves CPU by skipping compression when and where the overall ratio is inadequate. This improves on the previous auto-skip compression manager by (a) taking achieved compression ratios into account, (b) using a continuously evolving payoff estimate, and (c) combining natively with parallel compression.
+4
View File
@@ -591,6 +591,7 @@ ifneq ($(filter check-headers, $(MAKECMDGOALS)),)
DEV_HEADER_DIRS := $(sort include/ $(dir $(ALL_SOURCES)))
FOLLY_DEPENDENT_HEADERS := \
include/rocksdb/coro_db.h \
include/rocksdb/coro_external_table.h \
include/rocksdb/utilities/coro_stackable_db.h
# Some headers like in port/ are platform-specific
DEV_HEADERS_TO_CHECK := $(filter-out $(FOLLY_DEPENDENT_HEADERS), $(shell $(FIND) $(DEV_HEADER_DIRS) -type f -name '*.h' | grep -E -v 'port/|plugin/|range_tree/|secondary_index/'))
@@ -2018,6 +2019,9 @@ block_fetcher_test: table/block_fetcher_test.o $(TEST_LIBRARY) $(LIBRARY)
block_test: $(OBJ_DIR)/table/block_based/block_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
builtin_index_factory_test: $(OBJ_DIR)/table/block_based/builtin_index_factory_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
data_block_hash_index_test: $(OBJ_DIR)/table/block_based/data_block_hash_index_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
+77 -11
View File
@@ -35,6 +35,49 @@ from util import ColorString
# tests to export as libraries for inclusion in other projects
_EXPORTED_TEST_LIBS = ["env_basic_test"]
_TEST_LIB_HEADERS = [
"db/db_test_util.h",
"db/db_with_timestamp_test_util.h",
"table/mock_table.h",
"test_util/mock_time_env.h",
"test_util/secondary_cache_test_util.h",
"test_util/simple_external_table_factory.h",
"test_util/testharness.h",
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/trace_analyzer_tool.h",
"utilities/agg_merge/test_agg_merge.h",
"utilities/cassandra/test_utils.h",
]
_TEST_HEADERS = {
"configurable_test": ["options/configurable_test.h"],
"persistent_cache_test": ["utilities/persistent_cache/persistent_cache_test.h"],
"point_lock_manager_test": [
"utilities/transactions/lock/point/point_lock_manager_test.h",
],
"transaction_test": ["utilities/transactions/transaction_test.h"],
}
_STRESS_LIB_HEADERS = [
"db_stress_tool/db_stress_common.h",
"db_stress_tool/db_stress_compaction_service.h",
"db_stress_tool/db_stress_compression_manager.h",
"db_stress_tool/db_stress_driver.h",
"db_stress_tool/db_stress_filters.h",
"db_stress_tool/db_stress_listener.h",
"db_stress_tool/db_stress_shared_state.h",
"db_stress_tool/db_stress_test_base.h",
"db_stress_tool/db_stress_wide_merge_operator.h",
"db_stress_tool/expected_state.h",
"db_stress_tool/expected_value.h",
"db_stress_tool/multi_ops_txns_stress.h",
"test_util/simple_external_table_factory.h",
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/trace_analyzer_tool.h"
]
# Parse src.mk files as a Dictionary of
# VAR_NAME => list of files
def parse_src_mk(repo_path):
@@ -183,27 +226,34 @@ def generate_buck(repo_path, deps_map):
":rocksdb_lib",
],
)
# rocksdb_test_lib
BUCK.add_library(
"rocksdb_test_lib",
test_lib_sources = (
src_mk.get("MOCK_LIB_SOURCES", [])
+ src_mk.get("TEST_LIB_SOURCES", [])
+ src_mk.get("EXP_LIB_SOURCES", [])
+ src_mk.get("ANALYZER_LIB_SOURCES", []),
+ src_mk.get("ANALYZER_LIB_SOURCES", [])
)
# Older release source lists may not compile every test helper yet.
test_lib_headers = [
header for header in _TEST_LIB_HEADERS
if header[:-2] + ".cc" in test_lib_sources
]
# rocksdb_test_lib
BUCK.add_library(
"rocksdb_test_lib",
test_lib_sources,
[":rocksdb_lib"],
extra_test_libs=True,
headers=test_lib_headers,
)
# rocksdb_with_faiss_test_lib
BUCK.add_library(
"rocksdb_with_faiss_test_lib",
src_mk.get("MOCK_LIB_SOURCES", [])
+ src_mk.get("TEST_LIB_SOURCES", [])
+ src_mk.get("EXP_LIB_SOURCES", [])
+ src_mk.get("ANALYZER_LIB_SOURCES", []),
test_lib_sources,
deps=[
":rocksdb_with_faiss_lib",
],
extra_test_libs=True,
headers=test_lib_headers,
)
# rocksdb_tools_lib
BUCK.add_library(
@@ -212,6 +262,12 @@ def generate_buck(repo_path, deps_map):
+ src_mk.get("ANALYZER_LIB_SOURCES", [])
+ ["test_util/testutil.cc"],
[":rocksdb_lib"],
headers=[
"test_util/testutil.h",
"tools/block_cache_analyzer/block_cache_trace_analyzer.h",
"tools/simulated_hybrid_file_system.h",
"tools/trace_analyzer_tool.h",
],
)
# rocksdb_cache_bench_tools_lib
BUCK.add_library(
@@ -225,12 +281,19 @@ def generate_buck(repo_path, deps_map):
src_mk.get("POINT_LOCK_BENCH_LIB_SOURCES", []),
[":rocksdb_lib"],
)
stress_sources = (
src_mk.get("ANALYZER_LIB_SOURCES", [])
+ src_mk.get("STRESS_LIB_SOURCES", [])
+ ["test_util/testutil.cc"]
)
# rocksdb_stress_lib
BUCK.add_rocksdb_library(
"rocksdb_stress_lib",
src_mk.get("ANALYZER_LIB_SOURCES", [])
+ src_mk.get("STRESS_LIB_SOURCES", [])
+ ["test_util/testutil.cc"],
stress_sources,
headers=[
header for header in _STRESS_LIB_HEADERS
if header[:-2] + ".cc" in stress_sources
],
)
# ldb binary
BUCK.add_binary(
@@ -353,6 +416,7 @@ def generate_buck(repo_path, deps_map):
test_src,
deps=json.dumps(deps["extra_deps"] + [":" + test_library]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
source_headers=_TEST_HEADERS.get(test, []),
)
else:
if with_faiss:
@@ -361,6 +425,7 @@ def generate_buck(repo_path, deps_map):
test_src,
deps=json.dumps(deps["extra_deps"] + [":rocksdb_with_faiss_test_lib"]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
source_headers=_TEST_HEADERS.get(test, []),
)
else:
BUCK.register_test(
@@ -368,6 +433,7 @@ def generate_buck(repo_path, deps_map):
test_src,
deps=json.dumps(deps["extra_deps"] + [":rocksdb_test_lib"]),
extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
source_headers=_TEST_HEADERS.get(test, []),
)
BUCK.export_file("tools/db_crashtest.py")
BUCK.export_file("tools/fault_injection_log_parser.py")
+6 -1
View File
@@ -162,12 +162,17 @@ add_c_test_wrapper()
).encode("utf-8")
)
def register_test(self, test_name, src, deps, extra_compiler_flags):
def register_test(self, test_name, src, deps, extra_compiler_flags, source_headers):
with open(self.path, "ab") as targets_file:
targets_file.write(
targets_cfg.unittests_template.format(
test_name=test_name,
test_cc=str(src),
source_headers=(
f" source_headers=[{pretty_list(source_headers, 16)}],\n"
if source_headers
else ""
),
deps=deps,
extra_compiler_flags=extra_compiler_flags,
).encode("utf-8")
+1 -1
View File
@@ -30,7 +30,7 @@ cpp_binary_wrapper(name="{name}", srcs=[{srcs}], deps=[{deps}], extra_preprocess
unittests_template = """
cpp_unittest_wrapper(name="{test_name}",
srcs=["{test_cc}"],
deps={deps},
{source_headers} deps={deps},
extra_compiler_flags={extra_compiler_flags})
"""
@@ -142,6 +142,11 @@ uint64_t rocksdb_table_properties_column_family_id(
return props->rep.column_family_id;
}
uint64_t rocksdb_table_properties_oldest_ancestor_time(
const rocksdb_table_properties_t* props) {
return props->rep.oldest_ancestor_time;
}
uint64_t rocksdb_table_properties_creation_time(
const rocksdb_table_properties_t* props) {
return props->rep.creation_time;
@@ -172,11 +177,6 @@ uint64_t rocksdb_table_properties_fast_compression_estimated_data_size(
return props->rep.fast_compression_estimated_data_size;
}
uint64_t rocksdb_table_properties_external_sst_file_global_seqno_offset(
const rocksdb_table_properties_t* props) {
return props->rep.external_sst_file_global_seqno_offset;
}
uint64_t rocksdb_table_properties_tail_start_offset(
const rocksdb_table_properties_t* props) {
return props->rep.tail_start_offset;
@@ -212,6 +212,11 @@ uint64_t rocksdb_table_properties_separate_key_value_in_data_block(
return props->rep.separate_key_value_in_data_block;
}
uint64_t rocksdb_table_properties_lsm_info_at_creation(
const rocksdb_table_properties_t* props) {
return props->rep.lsm_info_at_creation;
}
const char* rocksdb_table_properties_db_id(
const rocksdb_table_properties_t* props, size_t* size) {
*size = props->rep.db_id.size();
@@ -102,6 +102,10 @@ rocksdb_table_properties_fixed_key_len(const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t rocksdb_table_properties_column_family_id(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_oldest_ancestor_time(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_creation_time(const rocksdb_table_properties_t* props);
@@ -122,10 +126,6 @@ extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_fast_compression_estimated_data_size(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_external_sst_file_global_seqno_offset(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t rocksdb_table_properties_tail_start_offset(
const rocksdb_table_properties_t* props);
@@ -151,6 +151,10 @@ extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_separate_key_value_in_data_block(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API uint64_t
rocksdb_table_properties_lsm_info_at_creation(
const rocksdb_table_properties_t* props);
extern ROCKSDB_LIBRARY_API const char* rocksdb_table_properties_db_id(
const rocksdb_table_properties_t* props, size_t* size);
@@ -816,6 +816,26 @@ const char* rocksdb_options_get_db_host_id(rocksdb_options_t* opt,
return opt->rep.db_host_id.data();
}
void rocksdb_options_set_remote_compaction_manifest_floor(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.remote_compaction_manifest_floor = v;
}
unsigned char rocksdb_options_get_remote_compaction_manifest_floor(
rocksdb_options_t* opt) {
return opt->rep.remote_compaction_manifest_floor;
}
void rocksdb_options_set_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.use_session_tmp_dir_for_remote_compaction = v;
}
unsigned char rocksdb_options_get_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt) {
return opt->rep.use_session_tmp_dir_for_remote_compaction;
}
void rocksdb_options_set_lowest_used_cache_tier(rocksdb_options_t* opt, int v) {
opt->rep.lowest_used_cache_tier =
static_cast<decltype(opt->rep.lowest_used_cache_tier)>(v);
@@ -856,6 +876,16 @@ uint64_t rocksdb_options_get_max_compaction_trigger_wakeup_seconds(
return opt->rep.max_compaction_trigger_wakeup_seconds;
}
void rocksdb_options_set_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt, int v) {
opt->rep.periodic_compaction_phase_recovery_percent = v;
}
int rocksdb_options_get_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt) {
return opt->rep.periodic_compaction_phase_recovery_percent;
}
void rocksdb_options_set_follower_refresh_catchup_period_ms(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.follower_refresh_catchup_period_ms = v;
@@ -2027,16 +2057,6 @@ unsigned char rocksdb_ingestexternalfileoptions_get_ingest_behind(
return opt->rep.ingest_behind;
}
void rocksdb_ingestexternalfileoptions_set_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
opt->rep.write_global_seqno = v;
}
unsigned char rocksdb_ingestexternalfileoptions_get_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt) {
return opt->rep.write_global_seqno;
}
void rocksdb_ingestexternalfileoptions_set_verify_checksums_before_ingest(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
opt->rep.verify_checksums_before_ingest = v;
@@ -2117,6 +2137,16 @@ int rocksdb_ingestexternalfileoptions_get_file_opening_threads(
/* OpenAndCompactOptions */
void rocksdb_open_and_compact_options_set_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt, uint32_t v) {
opt->rep.max_secondary_open_retries = v;
}
uint32_t rocksdb_open_and_compact_options_get_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt) {
return opt->rep.max_secondary_open_retries;
}
unsigned char rocksdb_open_and_compact_options_get_allow_resumption(
rocksdb_open_and_compact_options_t* opt) {
return opt->rep.allow_resumption;
@@ -2514,6 +2544,17 @@ double rocksdb_block_based_options_get_uniform_cv_threshold(
return opt->rep.uniform_cv_threshold;
}
void rocksdb_block_based_options_set_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt, int v) {
opt->rep.optimize_key_common_prefix =
static_cast<decltype(opt->rep.optimize_key_common_prefix)>(v);
}
int rocksdb_block_based_options_get_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt) {
return static_cast<int>(opt->rep.optimize_key_common_prefix);
}
void rocksdb_block_based_options_set_enable_index_compression(
rocksdb_block_based_table_options_t* opt, unsigned char v) {
opt->rep.enable_index_compression = v;
@@ -247,6 +247,21 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_db_host_id(
extern ROCKSDB_LIBRARY_API const char* rocksdb_options_get_db_host_id(
rocksdb_options_t* opt, size_t* size);
extern ROCKSDB_LIBRARY_API void
rocksdb_options_set_remote_compaction_manifest_floor(rocksdb_options_t* opt,
unsigned char v);
extern ROCKSDB_LIBRARY_API unsigned char
rocksdb_options_get_remote_compaction_manifest_floor(rocksdb_options_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_options_set_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt, unsigned char v);
extern ROCKSDB_LIBRARY_API unsigned char
rocksdb_options_get_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt);
extern ROCKSDB_LIBRARY_API void rocksdb_options_set_lowest_used_cache_tier(
rocksdb_options_t* opt, int v);
@@ -267,6 +282,14 @@ extern ROCKSDB_LIBRARY_API const char*
rocksdb_options_get_daily_offpeak_time_utc(rocksdb_options_t* opt,
size_t* size);
extern ROCKSDB_LIBRARY_API void
rocksdb_options_set_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt, int v);
extern ROCKSDB_LIBRARY_API int
rocksdb_options_get_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_options_set_follower_refresh_catchup_period_ms(rocksdb_options_t* opt,
uint64_t v);
@@ -793,14 +816,6 @@ extern ROCKSDB_LIBRARY_API unsigned char
rocksdb_ingestexternalfileoptions_get_ingest_behind(
rocksdb_ingestexternalfileoptions_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_ingestexternalfileoptions_set_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v);
extern ROCKSDB_LIBRARY_API unsigned char
rocksdb_ingestexternalfileoptions_get_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_ingestexternalfileoptions_set_verify_checksums_before_ingest(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v);
@@ -863,6 +878,14 @@ rocksdb_ingestexternalfileoptions_get_file_opening_threads(
/* OpenAndCompactOptions */
extern ROCKSDB_LIBRARY_API void
rocksdb_open_and_compact_options_set_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt, uint32_t v);
extern ROCKSDB_LIBRARY_API uint32_t
rocksdb_open_and_compact_options_get_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt);
extern ROCKSDB_LIBRARY_API unsigned char
rocksdb_open_and_compact_options_get_allow_resumption(
rocksdb_open_and_compact_options_t* opt);
@@ -1170,6 +1193,14 @@ extern ROCKSDB_LIBRARY_API double
rocksdb_block_based_options_get_uniform_cv_threshold(
rocksdb_block_based_table_options_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_block_based_options_set_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt, int v);
extern ROCKSDB_LIBRARY_API int
rocksdb_block_based_options_get_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt);
extern ROCKSDB_LIBRARY_API void
rocksdb_block_based_options_set_enable_index_compression(
rocksdb_block_based_table_options_t* opt, unsigned char v);
+39 -6
View File
@@ -151,6 +151,12 @@ static void rocksdb_roundtrip_test_rocksdb_block_based_options(void) {
rocksdb_block_based_options_set_optimize_filters_for_memory(obj, 0);
CheckCondition(
rocksdb_block_based_options_get_optimize_filters_for_memory(obj) == 0);
rocksdb_block_based_options_set_optimize_key_common_prefix(obj, 1);
CheckCondition(
rocksdb_block_based_options_get_optimize_key_common_prefix(obj) == 1);
rocksdb_block_based_options_set_optimize_key_common_prefix(obj, 0);
CheckCondition(
rocksdb_block_based_options_get_optimize_key_common_prefix(obj) == 0);
rocksdb_block_based_options_set_partition_filters(obj, 1);
CheckCondition(rocksdb_block_based_options_get_partition_filters(obj) == 1);
rocksdb_block_based_options_set_partition_filters(obj, 0);
@@ -550,12 +556,6 @@ static void rocksdb_roundtrip_test_rocksdb_ingestexternalfileoptions(void) {
rocksdb_ingestexternalfileoptions_set_verify_file_checksum(obj, 0);
CheckCondition(
rocksdb_ingestexternalfileoptions_get_verify_file_checksum(obj) == 0);
rocksdb_ingestexternalfileoptions_set_write_global_seqno(obj, 1);
CheckCondition(
rocksdb_ingestexternalfileoptions_get_write_global_seqno(obj) == 1);
rocksdb_ingestexternalfileoptions_set_write_global_seqno(obj, 0);
CheckCondition(
rocksdb_ingestexternalfileoptions_get_write_global_seqno(obj) == 0);
rocksdb_ingestexternalfileoptions_destroy(obj);
}
@@ -586,6 +586,20 @@ static void rocksdb_roundtrip_test_rocksdb_livefiles_storage_info_options(
rocksdb_livefiles_storage_info_options_destroy(obj);
}
static void rocksdb_roundtrip_test_rocksdb_open_and_compact_options(void) {
rocksdb_open_and_compact_options_t* obj =
rocksdb_open_and_compact_options_create();
rocksdb_open_and_compact_options_set_max_secondary_open_retries(obj, 1);
CheckCondition(
rocksdb_open_and_compact_options_get_max_secondary_open_retries(obj) ==
1);
rocksdb_open_and_compact_options_set_max_secondary_open_retries(obj, 0);
CheckCondition(
rocksdb_open_and_compact_options_get_max_secondary_open_retries(obj) ==
0);
rocksdb_open_and_compact_options_destroy(obj);
}
static void rocksdb_roundtrip_test_rocksdb_optimistictransactiondb_options(
void) {
rocksdb_optimistictransactiondb_options_t* obj =
@@ -833,6 +847,12 @@ static void rocksdb_roundtrip_test_rocksdb_options(void) {
CheckCondition(rocksdb_options_get_paranoid_memory_checks(obj) == 1);
rocksdb_options_set_paranoid_memory_checks(obj, 0);
CheckCondition(rocksdb_options_get_paranoid_memory_checks(obj) == 0);
rocksdb_options_set_periodic_compaction_phase_recovery_percent(obj, 1);
CheckCondition(
rocksdb_options_get_periodic_compaction_phase_recovery_percent(obj) == 1);
rocksdb_options_set_periodic_compaction_phase_recovery_percent(obj, 0);
CheckCondition(
rocksdb_options_get_periodic_compaction_phase_recovery_percent(obj) == 0);
rocksdb_options_set_persist_stats_to_disk(obj, 1);
CheckCondition(rocksdb_options_get_persist_stats_to_disk(obj) == 1);
rocksdb_options_set_persist_stats_to_disk(obj, 0);
@@ -859,6 +879,12 @@ static void rocksdb_roundtrip_test_rocksdb_options(void) {
CheckCondition(rocksdb_options_get_read_io_executor_threads(obj) == 1);
rocksdb_options_set_read_io_executor_threads(obj, 0);
CheckCondition(rocksdb_options_get_read_io_executor_threads(obj) == 0);
rocksdb_options_set_remote_compaction_manifest_floor(obj, 1);
CheckCondition(rocksdb_options_get_remote_compaction_manifest_floor(obj) ==
1);
rocksdb_options_set_remote_compaction_manifest_floor(obj, 0);
CheckCondition(rocksdb_options_get_remote_compaction_manifest_floor(obj) ==
0);
rocksdb_options_set_reuse_manifest_on_open(obj, 1);
CheckCondition(rocksdb_options_get_reuse_manifest_on_open(obj) == 1);
rocksdb_options_set_reuse_manifest_on_open(obj, 0);
@@ -905,6 +931,12 @@ static void rocksdb_roundtrip_test_rocksdb_options(void) {
CheckCondition(rocksdb_options_get_use_fsync(obj) == 1);
rocksdb_options_set_use_fsync(obj, 0);
CheckCondition(rocksdb_options_get_use_fsync(obj) == 0);
rocksdb_options_set_use_session_tmp_dir_for_remote_compaction(obj, 1);
CheckCondition(
rocksdb_options_get_use_session_tmp_dir_for_remote_compaction(obj) == 1);
rocksdb_options_set_use_session_tmp_dir_for_remote_compaction(obj, 0);
CheckCondition(
rocksdb_options_get_use_session_tmp_dir_for_remote_compaction(obj) == 0);
rocksdb_options_set_verify_manifest_content_on_close(obj, 1);
CheckCondition(rocksdb_options_get_verify_manifest_content_on_close(obj) ==
1);
@@ -1363,6 +1395,7 @@ static void rocksdb_run_generated_roundtrip_tests(void) {
rocksdb_roundtrip_test_rocksdb_flushwaloptions();
rocksdb_roundtrip_test_rocksdb_ingestexternalfileoptions();
rocksdb_roundtrip_test_rocksdb_livefiles_storage_info_options();
rocksdb_roundtrip_test_rocksdb_open_and_compact_options();
rocksdb_roundtrip_test_rocksdb_optimistictransactiondb_options();
rocksdb_roundtrip_test_rocksdb_options();
rocksdb_roundtrip_test_rocksdb_readoptions();
+1 -1
View File
@@ -245,7 +245,7 @@ class ShardedCache : public ShardedCacheBase {
std::unique_ptr<size_t[]> states(new size_t[num_shards]{});
size_t aepl = opts.average_entries_per_lock;
aepl = std::min(aepl, size_t{1});
aepl = std::max(aepl, size_t{1});
bool remaining_work;
do {
+130 -2
View File
@@ -4,6 +4,122 @@
Thorough multi-agent code review for RocksDB commits following CLAUDE.md guidelines,
with structured codebase exploration, inter-agent debate, and verification.
## Behavioral Compatibility Review Perspective
Behavior changes are human judgment calls. They must be intentional,
well-documented (including release notes), and researched for potential
downstream production impact. Agents and reviewers must call out concrete gaps,
ambiguity, or adverse impacts in the change's intent, documentation, and
production-impact analysis.
When a diff changes how an existing accepted public or serialized input is
interpreted—including an undocumented, discouraged, or historically ignored
value—reviewers must:
- Compare the old and new observable behavior, including performance and
resource impact.
- Check that the rationale and user-facing documentation clearly describe the
change.
- Identify known direct callers and stored configurations that may depend on
the previous behavior.
- Check for appropriate regression coverage, or a documented reason why such
coverage is not practical.
A behavior change is not automatically a defect. Report a finding when the
intent, documentation, downstream-impact analysis, or mitigation has a concrete
gap or ambiguity that could lead to an unintended compatibility regression.
## DB Mutex Review Perspective
For every non-trivial change, determine whether the changed code, or any
transitive callee it adds or changes, can run while `DBImpl::mutex_` is held.
State the result explicitly, including when the DB mutex is not affected.
`DBImpl::mutex_` is the DB-wide metadata and coordination mutex. It protects
cross-object invariants involving `ColumnFamilyData`, memtable and
`SuperVersion` publication, `VersionSet`, snapshots, flush/compaction queues,
background-job state, and file lifetime. It is shared by all column families
and can be a hot lock. It is not the filesystem `db_lock_`, and it should not
replace data-path synchronization already provided by `WriteThread`,
`wal_write_mutex_`, atomics, referenced `SuperVersion`s, or component-local
synchronization.
When a DB-mutex critical section is affected:
### Locking and correctness
- Trace the full call path and mark the exact lock and unlock boundaries. Do
not inspect only the lines that directly acquire the mutex.
- State the locking contract. Helpers that require the lock should document it
and normally use `mutex_.AssertHeld()`. Check whether each caller expects the
lock to be held on return.
- Treat calls that can release and re-acquire the mutex as concurrency
boundaries. Examples include `VersionSet::LogAndApply()`, condition-variable
waits, write-thread coordination, and notification helpers. Revalidate state
inspected before such a call.
- Keep atomic metadata transitions under the mutex. Examples include moving a
memtable between mutable and immutable state, reserving compaction inputs,
installing a `Version` or `SuperVersion`, updating scheduling counters and
queues, and changing file-lifetime protection.
- For expensive work, prefer a prepare/run/install structure: reserve and pin
the required state while locked, do CPU or I/O work while unlocked, then
re-lock, validate, and publish. Before unlocking, copy mutable options and
acquire every required object reference or lifetime reservation.
- Check object lifetime across every unlocked region. Do not use a raw
`ColumnFamilyData*`, `Version*`, memtable, file metadata pointer, or DB-owned
container element after unlocking unless an explicit reference,
`SuperVersion`, job object, or pending-output reservation keeps it alive.
- Preserve the snapshot/input-selection boundary. Flush and compaction must not
miss a snapshot whose visible key versions they may remove. Missing a newly
created snapshot is a correctness bug; retaining a recently released
snapshot is conservative but safe.
- Never invoke untrusted or re-entrant code while holding the mutex. The mutex
is non-reentrant, so a callback into RocksDB can self-deadlock.
- Check lock ordering. `options_mutex_` is acquired before the DB mutex. When
both the DB mutex and `wal_write_mutex_` are needed, the DB mutex is acquired
first. Establish and document the order for other locks.
- Keep cleanup outside the critical section when possible. File deletion,
`JobContext::Clean()`, expensive `SuperVersion` destruction, logging, and
listener callbacks should normally run after protected state is detached or
snapshotted.
- Do not add the DB mutex to steady-state point reads or the normal write data
phase without strong justification. Reads normally use a referenced
`SuperVersion`; writes use `WriteThread` and component-specific
synchronization.
### Critical-section performance
- Compare the CPU and I/O performed under the mutex before and after the
change. Include call frequency, loop bounds, input-size scaling, hidden work
in callees, and worst-case cost.
- Account for filesystem or network I/O, waits, callbacks, logging and
formatting, allocation, container growth, copies, sorting, scans, reference
release and destructors, cache misses, and repeated metadata lookups.
- Do not allow a significant regression in mutex-held work without clear
justification, hold-time or contention measurements, and representative
benchmarks.
- Treat unavoidable work under the mutex as hot-path code and keep it highly
optimized. Precompute, allocate, copy, and perform I/O outside the lock;
remove repeated scans and lookups; cache or batch work where practical; and
keep locked operations O(1) or tightly bounded when possible.
- Treat new blocking I/O, waits, callbacks into user or plugin code, unbounded
loops, or work that scales with DB size, files, column families, snapshots,
or pending jobs as a blocker unless it is unavoidable and supported by strong
measurements and explicit design justification.
### Validation and evidence
- Test unlock windows with sync points rather than sleeps. Cover relevant races
with memtable switches, snapshot create/release, column-family drop,
flush/compaction install, shutdown, and background errors. Run TSAN where
appropriate.
- Distinguish wait time from hold time. `PerfContext::db_mutex_lock_nanos` and
`DB_MUTEX_WAIT_MICROS` primarily expose acquisition wait; use contention
profiles or scoped instrumentation to find long critical sections.
- Report affected critical sections, before/after cost, workload assumptions,
optimization choices, and remaining risk. If exact measurement is not
practical, provide a conservative static bound and do not claim no regression
without evidence.
IMPORTANT — Where the review goes: Your FINAL RESPONSE TEXT is the review.
The PR comment is generated verbatim from your final response, so the
complete, polished review MUST appear there in full. Do NOT end your turn with
@@ -283,6 +399,9 @@ in isolation may be a critical bug when you consider the full call chain.
- **Callee side effects**: Does the change call functions that mutate shared
state (counters, seqnos, metadata)? Are the mutations correct for the new
calling context? (Use the callee-chain analysis from context.md)
- Apply the DB Mutex Review Perspective above to locking contracts, unlocked
object lifetimes, stale-state validation, lock ordering, and snapshot
boundaries.
#### Agent: Cross-Component & Adversarial Reviewer
@@ -386,6 +505,9 @@ Write findings to `findings-caller-audit.md`.
- Loop optimization opportunities
- Branch prediction (LIKELY/UNLIKELY)
- Zero-overhead design verification
- Apply the DB Mutex Review Perspective above, with special focus on transitive
CPU and I/O, before/after cost, critical-section hold time, and benchmark
evidence
#### Agent: API & Compatibility Reviewer
- Public API backwards compatibility
@@ -393,8 +515,8 @@ Write findings to `findings-caller-audit.md`.
- Documentation completeness
- Deprecation policy compliance
- Forward compatibility (struct extensibility)
- **Behavioral compatibility**: Do changed semantics (e.g., kOutOfBound vs
kUnknown) break implicit contracts with callers?
- Apply the Behavioral Compatibility Review Perspective above to changes in
observable semantics and implicit caller contracts.
#### Agent: Serialization & Deserialization Reviewer
- Format correctness and versioning
@@ -569,6 +691,8 @@ Rules for this structure:
- [ ] Multi-component interactions (compaction, recovery, snapshots, iterators)
- [ ] Configuration dependencies and unexpected option combinations
- [ ] Potential pitfalls from caller-chain analysis
- [ ] DB-mutex reachability checked through transitive callees; affected lock
and unlock boundaries documented
### Review Phase (verified by agents)
- [ ] Database semantics preserved (snapshot isolation, key ordering)
@@ -582,6 +706,10 @@ Rules for this structure:
- [ ] Code follows RocksDB style conventions
- [ ] Behavioral contracts with upstream callers preserved
- [ ] All callers enumerated with parameter range table
- [ ] Before/after CPU and I/O under the DB mutex compared, with call frequency,
scaling, hidden callee work, and worst-case cost included
- [ ] No significant DB-mutex-held work regression; unavoidable work is highly
optimized, bounded, and supported by measurements when material
## Output Structure
Your **final response text is the primary output** — it becomes the PR comment
+106
View File
@@ -0,0 +1,106 @@
# RocksDB Diff and Pull Request Summary Guidance
This guidance applies to both internal Phabricator diff summaries and external
GitHub pull request summaries for changes under `fbcode/internal_repo_rocksdb/`.
## Goal
Write each diff or pull request summary for a reviewer who has not read the
patch. The summary must explain the behavioral story and the evidence for it,
not merely inventory the edited files or symbols.
Every new or updated diff or pull request summary must clearly cover:
1. Why the change is required.
2. What the previous behavior was, including the failure mode or limitation.
3. What the new behavior is.
4. What changed in the implementation.
5. Safety boundaries, compatibility, and unsupported or intentionally deferred
cases.
6. Tests performed and known limitations of the validation.
Use explicit headings for a broad, subtle, or high-risk change. A small,
focused change may combine these points into a few paragraphs, but none may be
left implicit when it affects a review decision.
## What Good RocksDB Summaries Make Concrete
- Name the exact conditions that activate a bug or behavior: relevant options,
call ordering, storage mode, concurrency mode, file format, platform, or
workload shape. Also state important cases that are not affected.
- Explain the causal chain from the old behavior to its externally observable
result. For correctness fixes, distinguish a symptom from the underlying
invariant violation.
- Describe the new contract before implementation mechanics. Then identify the
essential implementation choices and why they are sufficient.
- Call out API, ABI, serialized-format, behavioral, and performance
compatibility where relevant. For additive persisted fields, state how older
data is interpreted and whether older readers remain supported.
- State safety boundaries precisely: defaults, feature gates, fallback paths,
ownership and lifetime constraints, memory-ordering assumptions, and cases
deliberately postponed to a follow-up.
- Support performance claims with a baseline, workload and configuration,
measurement method, and results. Report meaningful tradeoffs, not only the
winning metric.
- Make the test plan evidence-based. Name the tests and configurations, say
what behavior each proves, and report failures, timeouts, untested platforms,
or stale pre-rebase results honestly. For a regression test, say how it fails
without the fix when that was verified.
- Separate incidental cleanup or a pre-existing test fix from the primary
change, and explain why it belongs in the diff.
Avoid vague claims such as "improves performance," "fixes an issue," or "tests
pass." Avoid copying the patch into prose, exhaustive file-by-file narration,
raw command output, and claims broader than the evidence.
## Suggested Structure
Adapt this structure to the change; do not keep empty or irrelevant sections.
```text
Summary:
Why the change is required:
<User or system impact, and why the current behavior is inadequate.>
Previous behavior:
<Old contract or mechanism, triggering conditions, and observable result.>
New behavior:
<New contract and externally visible effect.>
Implementation:
<Essential design and code changes; omit routine edit-by-edit narration.>
Safety, compatibility, and limitations:
<Defaults, unaffected paths, compatibility, unsupported cases, and follow-ups.>
Test Plan:
<Exact tests/benchmarks, configurations, what they demonstrate, and gaps.>
```
For a focused bug fix, a cohesive narrative can be clearer than six headings:
problem and root cause, trigger and unaffected cases, fix, then quantified or
counterfactual validation. For a multi-part change, labeled bullets such as
`Fix`, `Add`, and `Change` can make the implementation and review surface easy
to scan.
## Reference Pull Request Summaries
- [PR #15216](https://github.com/facebook/rocksdb/pull/15216) is a strong
focused bug-fix example. It traces stale iterator state to readahead behavior
and dropped keys, identifies the exact prerequisites and unaffected mmap
case, gives before/after measurements, and explains how the regression test
fails when the fix is reverted.
- [PR #15218](https://github.com/facebook/rocksdb/pull/15218) is a concise
example of explicit `Why`, `Previous behavior`, `New behavior`,
`Implementation`, and `Safety boundaries and unsupported cases` sections.
- Peter Dillinger's [PR #15231](https://github.com/facebook/rocksdb/pull/15231)
is a strong recent multi-part example. Its `Fix`/`Add`/`Change` bullets
separate the behaviors, its compatibility sentence explains how old files
read back, and its test plan maps each new persisted property to concrete
coverage while explaining an adjacent test adjustment.
Use these pull requests as models for reasoning, specificity, and evidence. Do
not copy their length or headings mechanically; scale the summary to the risk
and complexity of the change.
+11
View File
@@ -25,6 +25,8 @@ CRASHTEST_PY=$(PYTHON) -u tools/db_crashtest.py --stress_cmd=$(DB_STRESS_CMD) --
crash_test_with_multiops_wup_txn \
crash_test_with_optimistic_txn \
crash_test_with_tiered_storage \
crash_test_with_full_external_table \
liveness_crash_test \
blackbox_crash_test blackbox_crash_test_with_atomic_flush \
blackbox_crash_test_with_wc_txn blackbox_crash_test_with_wp_txn \
blackbox_crash_test_with_wup_txn \
@@ -35,6 +37,7 @@ CRASHTEST_PY=$(PYTHON) -u tools/db_crashtest.py --stress_cmd=$(DB_STRESS_CMD) --
blackbox_crash_test_with_multiops_wup_txn \
blackbox_crash_test_with_optimistic_txn \
blackbox_crash_test_with_tiered_storage \
blackbox_crash_test_with_full_external_table \
whitebox_crash_test whitebox_crash_test_with_atomic_flush \
whitebox_crash_test_with_wc_txn whitebox_crash_test_with_wp_txn \
whitebox_crash_test_with_wup_txn \
@@ -85,6 +88,8 @@ crash_test_with_tiered_storage: $(DB_STRESS_CMD)
$(CRASHTEST_MAKE) whitebox_crash_test_with_tiered_storage
$(CRASHTEST_MAKE) blackbox_crash_test_with_tiered_storage
crash_test_with_full_external_table: blackbox_crash_test_with_full_external_table
crash_test_with_multiops_wc_txn: $(DB_STRESS_CMD)
$(CRASHTEST_MAKE) blackbox_crash_test_with_multiops_wc_txn
@@ -94,6 +99,9 @@ crash_test_with_multiops_wp_txn: $(DB_STRESS_CMD)
crash_test_with_multiops_wup_txn: $(DB_STRESS_CMD)
$(CRASHTEST_MAKE) blackbox_crash_test_with_multiops_wup_txn
liveness_crash_test: $(DB_STRESS_CMD)
$(CRASHTEST_PY) liveness $(CRASH_TEST_EXT_ARGS)
blackbox_crash_test: $(DB_STRESS_CMD)
$(CRASHTEST_PY) --simple blackbox $(CRASH_TEST_EXT_ARGS)
$(CRASHTEST_PY) blackbox $(CRASH_TEST_EXT_ARGS)
@@ -128,6 +136,9 @@ blackbox_crash_test_with_multiops_wup_txn: $(DB_STRESS_CMD)
blackbox_crash_test_with_tiered_storage: $(DB_STRESS_CMD)
$(CRASHTEST_PY) --test_tiered_storage blackbox $(CRASH_TEST_EXT_ARGS)
blackbox_crash_test_with_full_external_table: $(DB_STRESS_CMD)
$(CRASHTEST_PY) --test_full_external_table blackbox $(CRASH_TEST_EXT_ARGS)
blackbox_crash_test_with_optimistic_txn: $(DB_STRESS_CMD)
$(CRASHTEST_PY) --optimistic_txn blackbox $(CRASH_TEST_EXT_ARGS)
+41
View File
@@ -5,7 +5,9 @@
#pragma once
#include <cstddef>
#include <cstdint>
#include <limits>
#include "rocksdb/rocksdb_namespace.h"
@@ -31,4 +33,43 @@ constexpr uint64_t kInvalidBlobFileNumber = 0;
constexpr uint64_t kCurrentFileBlobIndexFileNumber = kInvalidBlobFileNumber;
static_assert(kCurrentFileBlobIndexFileNumber == kInvalidBlobFileNumber);
// Sentinel range length meaning "the whole blob value" for the unified
// blob-read primitives, which take a range descriptor {range_offset,
// range_length}; range_length == kWholeBlobLength selects the whole value and
// any other value selects the strict sub-range
// [range_offset, range_offset + range_length).
//
// This is the internal counterpart of the public kLazyWholeColumn
// (include/rocksdb/lazy_wide_columns.h); both are the maximum size_t. The lazy
// resolver does not forward the public value as a range descriptor -- it
// normalizes a whole-column request to kWholeBlobLength -- so the two are not
// required to be equal, but both rely on the sentinel exceeding any real
// blob/column size so a "whole" request is never mistaken for a strict
// sub-range. Keep them in sync.
inline constexpr size_t kWholeBlobLength = std::numeric_limits<size_t>::max();
// How aggressively a blob read verifies the record's checksum. Derived once at
// the lazy read-path boundary (ReadPathBlobResolver) from
// (ReadOptions::verify_checksums, LazyColumnReadRequest::force_verify) and
// threaded down through the blob-read primitives, replacing the earlier mix of
// a verify_checksums bool and a separate force-verify method. (Not literally a
// constant, but kept here as a small shared blob-read enum rather than its own
// header.)
//
// - kVerifyIfPresent: verify the whole-record checksum whenever the format
// provides one, even if ReadOptions::verify_checksums is off. A byte-range
// request under this policy is escalated to a whole-record read (a strict
// sub-range cannot cover the checksum). This is the public force_verify.
// - kVerifyIfNoAmplification: verify when the checksummable unit is already
// being read -- a whole-value read verifies (the check is "free"), while a
// strict sub-range read skips verification rather than amplify the read to
// the whole record. Corresponds to verify_checksums on, force_verify off.
// - kSkip: never verify. Corresponds to verify_checksums off, force_verify
// off.
enum class BlobVerifyPolicy : uint8_t {
kVerifyIfPresent,
kVerifyIfNoAmplification,
kSkip,
};
} // namespace ROCKSDB_NAMESPACE
+41 -36
View File
@@ -44,43 +44,41 @@ Status VersionBlobFetcherBase::FetchBlob(const Slice& user_key,
prefetch_buffer, blob_value, bytes_read);
}
Status VersionBlobFetcherBase::FetchBlobRange(const Slice& user_key,
const BlobIndex& blob_index,
uint64_t range_offset,
size_t range_length,
PinnableSlice* blob_value,
uint64_t* bytes_read) const {
// Only ever called when SupportsRangeRead() holds (a Version to read through,
// no direct-write fallback); the direct-write path resolves whole records and
// has no range variant.
assert(SupportsRangeRead());
assert(version_);
return version_->GetBlobRange(read_options(), user_key, blob_index,
range_offset, range_length, blob_value,
bytes_read);
}
Status VersionBlobFetcherBase::FetchBlobForceVerify(
const Slice& user_key, const BlobIndex& blob_index,
FilePrefetchBuffer* prefetch_buffer, PinnableSlice* blob_value,
uint64_t* bytes_read) const {
if (read_options().verify_checksums) {
// Verification already happens on the normal path.
Status VersionBlobFetcherBase::FetchBlobRange(
const Slice& user_key, const BlobIndex& blob_index, uint64_t range_offset,
size_t range_length, BlobVerifyPolicy verify_policy,
PinnableSlice* blob_value, uint64_t* bytes_read) const {
if (range_length == kWholeBlobLength) {
// Whole-value read. Delegate to FetchBlob (which handles the direct-write
// fallback). For kVerifyIfPresent, force whole-record checksum verification
// even when ReadOptions::verify_checksums is off, via a transient fetcher
// over a verify-enabled copy of the read options (both locals outlive the
// borrowed-options VersionBlobFetcher).
constexpr FilePrefetchBuffer* prefetch_buffer = nullptr;
if (verify_policy == BlobVerifyPolicy::kVerifyIfPresent &&
!read_options().verify_checksums) {
ReadOptions verify_read_options = read_options();
verify_read_options.verify_checksums = true;
VersionBlobFetcher verifying_fetcher(version_, verify_read_options,
blob_file_cache_,
allow_write_path_fallback_);
return verifying_fetcher.FetchBlob(user_key, blob_index, prefetch_buffer,
blob_value, bytes_read);
}
return FetchBlob(user_key, blob_index, prefetch_buffer, blob_value,
bytes_read);
}
// Force verification via a transient fetcher over a verify-enabled copy of
// the read options. `verify_read_options` outlives `verifying_fetcher` (both
// are local to this call), so the borrowed-options VersionBlobFetcher is
// safe.
ReadOptions verify_read_options = read_options();
verify_read_options.verify_checksums = true;
VersionBlobFetcher verifying_fetcher(version_, verify_read_options,
blob_file_cache_,
allow_write_path_fallback_);
return verifying_fetcher.FetchBlob(user_key, blob_index, prefetch_buffer,
blob_value, bytes_read);
// Strict sub-range read: value bytes only, no whole-record checksum, no cache
// fill. Only ever chosen when SupportsRangeRead() holds (a Version to read
// through, no direct-write fallback) and verification is not forced (a strict
// sub-range cannot cover the whole-record checksum).
assert(SupportsRangeRead());
assert(version_);
assert(verify_policy != BlobVerifyPolicy::kVerifyIfPresent);
return version_->GetBlobRange(read_options(), user_key, blob_index,
range_offset, range_length, blob_value,
bytes_read);
}
Status EmbeddedAwareBlobFetcher::FetchBlob(const Slice& user_key,
@@ -94,9 +92,16 @@ Status EmbeddedAwareBlobFetcher::FetchBlob(const Slice& user_key,
// Same-file ("embedded") blob records live in the current SST and are read
// by the SameFileBlobReader (the BlockBasedTable). A separate blob-file
// prefetch buffer and bytes_read counter do not apply here (embedded reads
// account their own BLOB_DB_* stats via BlobSource).
return same_file_reader_->GetSameFileBlob(read_options(), blob_index,
blob_value);
// account their own BLOB_DB_* stats via BlobSource). This is a whole-value
// read; the verify policy follows read_options().verify_checksums (the
// decorator carries no separate force-verify signal).
const BlobVerifyPolicy verify_policy =
read_options().verify_checksums
? BlobVerifyPolicy::kVerifyIfNoAmplification
: BlobVerifyPolicy::kSkip;
return same_file_reader_->GetSameFileBlob(
read_options(), blob_index,
/*range_offset=*/0, kWholeBlobLength, verify_policy, blob_value);
}
if (base_ == nullptr) {
+23 -27
View File
@@ -7,6 +7,7 @@
#include <utility>
#include "db/blob/blob_constants.h"
#include "rocksdb/options.h"
#include "rocksdb/status.h"
@@ -63,37 +64,26 @@ class VersionBlobFetcherBase : public BlobFetcher {
PinnableSlice* blob_value,
uint64_t* bytes_read) const override;
// Fetches a byte sub-range [range_offset, range_offset + range_length) of an
// *uncompressed*, separate-file blob's value into *blob_value, reading only
// those bytes on a cache miss (see Version::GetBlobRange /
// BlobSource::GetBlobRange). Only valid when SupportsRangeRead() is true (a
// Version to read through and no direct-write fallback); callers check that
// first and take the whole-value FetchBlob path otherwise. Not routed through
// the same-file (embedded) decorator -- same-file range reads are a later
// phase.
// Fetches the value of a separate-file blob reference into *blob_value,
// applying `verify_policy` (see BlobVerifyPolicy). range_length ==
// kWholeBlobLength reads the whole value (delegating to FetchBlob, which also
// handles the direct-write fallback, and forcing whole-record checksum
// verification for kVerifyIfPresent even when ReadOptions::verify_checksums
// is off). Any other length reads only the sub-range [range_offset,
// range_offset + range_length) directly from the blob file, reading just
// those bytes on a cache miss -- no record-header/key read, no whole-record
// checksum, no cache fill (see Version::GetBlobRange /
// BlobSource::GetBlobRange). A strict sub-range is only valid when
// SupportsRangeRead() is true and verify_policy != kVerifyIfPresent
// (verifying a strict sub-range would have to amplify the read to the whole
// record); callers check that first and take the whole-value path otherwise.
// Not routed through the same-file (embedded) decorator -- same-file
// references are resolved by the caller through a SameFileBlobReader.
Status FetchBlobRange(const Slice& user_key, const BlobIndex& blob_index,
uint64_t range_offset, size_t range_length,
BlobVerifyPolicy verify_policy,
PinnableSlice* blob_value, uint64_t* bytes_read) const;
// Fetches the whole blob with checksum verification forced on, regardless of
// this fetcher's ReadOptions::verify_checksums. Used to honor a lazy read's
// force_verify when verify_checksums is off: the caller wants the record's
// checksum checked even though the global option would skip it. When
// verify_checksums is already on this is just FetchBlob.
//
// TODO(lazy-blob-resolution-cleanup): fold FetchBlobRange /
// FetchBlobForceVerify (and the whole-vs-range GetBlob twins throughout the
// blob stack) into one parameterized primitive taking a BlobReadRequest -- a
// range descriptor ({offset, len}; whole == [0, npos)) plus a 3-valued verify
// policy (must-verify-if-present / verify-if-no-amplification / ignore). That
// retires this method and the range/whole method pairs. See the lazy blob
// resolution plan (deferred for feature velocity).
Status FetchBlobForceVerify(const Slice& user_key,
const BlobIndex& blob_index,
FilePrefetchBuffer* prefetch_buffer,
PinnableSlice* blob_value,
uint64_t* bytes_read) const;
// True if this fetcher has enough context to resolve a (non-inlined,
// non-same-file) blob reference: either a Version to read through, or the
// direct-write fallback (a blob file cache) enabled. Callers that may hold a
@@ -111,6 +101,12 @@ class VersionBlobFetcherBase : public BlobFetcher {
return version_ != nullptr && !allow_write_path_fallback_;
}
// The Version this fetcher reads separate-file blob references through (may
// be null for a direct-write-fallback-only fetcher). Used by the cross-key
// lazy coalescing path to group and dispatch reads via
// Version::MultiGetBlobLazy.
const Version* version() const { return version_; }
protected:
VersionBlobFetcherBase(const Version* version, BlobFileCache* blob_file_cache,
bool allow_write_path_fallback)
+87 -23
View File
@@ -38,12 +38,14 @@ BlobFileCache::BlobFileCache(Cache* cache,
}
Status BlobFileCache::GetBlobFileReader(
const ReadOptions& read_options, uint64_t blob_file_number,
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
CacheHandleGuard<BlobFileReader>* blob_file_reader,
bool allow_footer_skip_retry) {
assert(blob_file_reader);
assert(blob_file_reader->IsEmpty());
const uint64_t blob_file_number = blob_file.file_number;
// NOTE: sharing same Cache with table_cache
const Slice key = GetSliceForKey(&blob_file_number);
@@ -74,18 +76,8 @@ Status BlobFileCache::GetBlobFileReader(
std::unique_ptr<BlobFileReader> reader;
{
assert(file_options_);
Status s = BlobFileReader::Create(
*immutable_options_, read_options, *file_options_, column_family_id_,
blob_file_read_hist_, blob_file_number, io_tracer_,
/*skip_footer_validation=*/false, &reader);
if (!s.ok() && s.IsCorruption() && allow_footer_skip_retry) {
reader.reset();
s = BlobFileReader::Create(
*immutable_options_, read_options, *file_options_, column_family_id_,
blob_file_read_hist_, blob_file_number, io_tracer_,
/*skip_footer_validation=*/true, &reader);
}
Status s = OpenBlobFileReader(read_options, blob_file,
allow_footer_skip_retry, &reader);
if (!s.ok()) {
ROCKS_LOG_WARN(immutable_options_->logger,
"BlobFileCache open failed for blob file %" PRIu64
@@ -114,7 +106,7 @@ Status BlobFileCache::GetBlobFileReader(
}
Status BlobFileCache::OpenBlobFileReaderUncached(
const ReadOptions& read_options, uint64_t blob_file_number,
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
std::unique_ptr<BlobFileReader>* blob_file_reader,
bool allow_footer_skip_retry) {
assert(blob_file_reader);
@@ -123,21 +115,77 @@ Status BlobFileCache::OpenBlobFileReaderUncached(
Statistics* const statistics = immutable_options_->stats;
RecordTick(statistics, NO_FILE_OPENS);
Status s = OpenBlobFileReader(read_options, blob_file,
allow_footer_skip_retry, blob_file_reader);
if (!s.ok()) {
RecordTick(statistics, NO_FILE_ERRORS);
}
return s;
}
void BlobFileCache::RegisterBlobFileChecksum(
const BlobFileOpenInfo& blob_file) {
assert(blob_file.file_checksum.empty() ==
blob_file.file_checksum_func_name.empty());
if (blob_file.file_checksum.empty()) {
return;
}
const Slice key = GetSliceForKey(&blob_file.file_number);
MutexLock cache_lock(&mutex_.Get(key));
{
MutexLock checksum_lock(&blob_file_checksums_mutex_);
BlobFileChecksum& checksum = blob_file_checksums_[blob_file.file_number];
checksum.value = blob_file.file_checksum.ToString();
checksum.function_name = blob_file.file_checksum_func_name.ToString();
}
// Readers opened while the file was active did not carry finalized metadata
// and may also have observed a footer-less file size.
cache_.get()->Erase(key);
}
void BlobFileCache::UnregisterBlobFileChecksum(uint64_t blob_file_number) {
MutexLock checksum_lock(&blob_file_checksums_mutex_);
blob_file_checksums_.erase(blob_file_number);
}
Status BlobFileCache::OpenBlobFileReader(
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
bool allow_footer_skip_retry,
std::unique_ptr<BlobFileReader>* blob_file_reader) {
assert(file_options_);
assert(blob_file.file_checksum.empty() ==
blob_file.file_checksum_func_name.empty());
FileOptions file_options = *file_options_;
if (!blob_file.file_checksum.empty()) {
file_options.file_checksum = blob_file.file_checksum.ToString();
file_options.file_checksum_func_name =
blob_file.file_checksum_func_name.ToString();
} else {
MutexLock checksum_lock(&blob_file_checksums_mutex_);
const auto it = blob_file_checksums_.find(blob_file.file_number);
if (it != blob_file_checksums_.end()) {
file_options.file_checksum = it->second.value;
file_options.file_checksum_func_name = it->second.function_name;
} else {
file_options.file_checksum.clear();
file_options.file_checksum_func_name.clear();
}
}
Status s = BlobFileReader::Create(
*immutable_options_, read_options, *file_options_, column_family_id_,
blob_file_read_hist_, blob_file_number, io_tracer_,
*immutable_options_, read_options, file_options, column_family_id_,
blob_file_read_hist_, blob_file.file_number, io_tracer_,
/*skip_footer_validation=*/false, blob_file_reader);
if (!s.ok() && s.IsCorruption() && allow_footer_skip_retry) {
blob_file_reader->reset();
s = BlobFileReader::Create(
*immutable_options_, read_options, *file_options_, column_family_id_,
blob_file_read_hist_, blob_file_number, io_tracer_,
*immutable_options_, read_options, file_options, column_family_id_,
blob_file_read_hist_, blob_file.file_number, io_tracer_,
/*skip_footer_validation=*/true, blob_file_reader);
}
if (!s.ok()) {
RecordTick(statistics, NO_FILE_ERRORS);
}
return s;
}
@@ -178,7 +226,7 @@ Status BlobFileCache::InsertBlobFileReader(
}
Status BlobFileCache::RefreshBlobFileReader(
uint64_t blob_file_number,
const BlobFileOpenInfo& blob_file,
std::unique_ptr<BlobFileReader>* blob_file_reader,
CacheHandleGuard<BlobFileReader>* cached_blob_file_reader) {
assert(blob_file_reader);
@@ -186,10 +234,26 @@ Status BlobFileCache::RefreshBlobFileReader(
assert(cached_blob_file_reader);
assert(cached_blob_file_reader->IsEmpty());
const Slice key = GetSliceForKey(&blob_file_number);
const Slice key = GetSliceForKey(&blob_file.file_number);
MutexLock lock(&mutex_.Get(key));
TypedHandle* handle = cache_.Lookup(key);
if (blob_file.file_checksum.empty()) {
MutexLock checksum_lock(&blob_file_checksums_mutex_);
if (blob_file_checksums_.find(blob_file.file_number) !=
blob_file_checksums_.end()) {
// This reader can have been opened just before finalization registered
// checksum metadata. Never let such a racing refresh repopulate the
// cache.
if (handle) {
*cached_blob_file_reader = cache_.Guard(handle);
}
blob_file_reader->reset();
return Status::OK();
}
}
if (handle) {
BlobFileReader* const cached_reader = cache_.Value(handle);
assert(cached_reader != nullptr);
+31 -6
View File
@@ -6,9 +6,12 @@
#pragma once
#include <cinttypes>
#include <string>
#include <unordered_map>
#include "cache/typed_cache.h"
#include "db/blob/blob_file_reader.h"
#include "db/blob/blob_read_request.h"
#include "rocksdb/rocksdb_namespace.h"
#include "util/mutexlock.h"
@@ -32,20 +35,30 @@ class BlobFileCache {
BlobFileCache(const BlobFileCache&) = delete;
BlobFileCache& operator=(const BlobFileCache&) = delete;
// Returns a cached reader for `blob_file_number`, opening and caching it on
// miss. If `allow_footer_skip_retry` is true, a footer-validation corruption
// retries once without requiring a footer.
// Returns a cached reader for `blob_file`, opening and caching it on miss.
// Checksum metadata is forwarded to the FileSystem through FileOptions when
// the file is opened. If `allow_footer_skip_retry` is true, a
// footer-validation corruption retries once without requiring a footer.
Status GetBlobFileReader(const ReadOptions& read_options,
uint64_t blob_file_number,
const BlobFileOpenInfo& blob_file,
CacheHandleGuard<BlobFileReader>* blob_file_reader,
bool allow_footer_skip_retry = false);
// Opens a blob file reader without inserting it into the cache.
Status OpenBlobFileReaderUncached(
const ReadOptions& read_options, uint64_t blob_file_number,
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
std::unique_ptr<BlobFileReader>* blob_file_reader,
bool allow_footer_skip_retry = false);
// Makes finalized checksum metadata available to direct-write fallback reads
// whose pinned Version predates the blob-file addition. Any cached
// footer-less reader is evicted as part of the same per-file transition.
void RegisterBlobFileChecksum(const BlobFileOpenInfo& blob_file);
// Stops retaining checksum metadata once no direct-write fallback can refer
// to the file. Manifest-backed callers continue to provide it directly.
void UnregisterBlobFileChecksum(uint64_t blob_file_number);
// Inserts a freshly opened uncached reader unless another thread already
// cached the same blob file.
Status InsertBlobFileReader(
@@ -58,7 +71,7 @@ class BlobFileCache {
// reader instead so a racing refresh cannot reintroduce an older active-file
// view after the blob file grows.
Status RefreshBlobFileReader(
uint64_t blob_file_number,
const BlobFileOpenInfo& blob_file,
std::unique_ptr<BlobFileReader>* blob_file_reader,
CacheHandleGuard<BlobFileReader>* cached_blob_file_reader);
@@ -72,6 +85,11 @@ class BlobFileCache {
}
private:
Status OpenBlobFileReader(const ReadOptions& read_options,
const BlobFileOpenInfo& blob_file,
bool allow_footer_skip_retry,
std::unique_ptr<BlobFileReader>* blob_file_reader);
using CacheInterface =
BasicTypedCacheInterface<BlobFileReader, CacheEntryRole::kMisc>;
using TypedHandle = CacheInterface::TypedHandle;
@@ -85,6 +103,13 @@ class BlobFileCache {
HistogramImpl* blob_file_read_hist_;
std::shared_ptr<IOTracer> io_tracer_;
struct BlobFileChecksum {
std::string value;
std::string function_name;
};
std::unordered_map<uint64_t, BlobFileChecksum> blob_file_checksums_;
mutable port::Mutex blob_file_checksums_mutex_;
static constexpr size_t kNumberOfMutexStripes = 1 << 7;
};
+21 -18
View File
@@ -115,13 +115,13 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader) {
BlobFileCache blob_file_cache(backing_cache.get(), &immutable_options,
&file_options, column_family_id,
blob_file_read_hist, nullptr /*IOTracer*/);
const BlobFileOpenInfo blob_file{blob_file_number, Slice(), Slice()};
// First try: reader should be opened and put in cache
CacheHandleGuard<BlobFileReader> first;
const ReadOptions read_options;
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
&first));
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file, &first));
ASSERT_NE(first.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -129,8 +129,8 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader) {
// Second try: reader should be served from cache
CacheHandleGuard<BlobFileReader> second;
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
&second));
ASSERT_OK(
blob_file_cache.GetBlobFileReader(read_options, blob_file, &second));
ASSERT_NE(second.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -163,6 +163,7 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_Race) {
BlobFileCache blob_file_cache(backing_cache.get(), &immutable_options,
&file_options, column_family_id,
blob_file_read_hist, nullptr /*IOTracer*/);
const BlobFileOpenInfo blob_file{blob_file_number, Slice(), Slice()};
CacheHandleGuard<BlobFileReader> first;
CacheHandleGuard<BlobFileReader> second;
@@ -172,16 +173,15 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_Race) {
"BlobFileCache::GetBlobFileReader:DoubleCheck", [&](void* /* arg */) {
// Disabling sync points to prevent infinite recursion
SyncPoint::GetInstance()->DisableProcessing();
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options,
blob_file_number, &second));
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file,
&second));
ASSERT_NE(second.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
&first));
ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file, &first));
ASSERT_NE(first.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -245,21 +245,22 @@ TEST_F(BlobFileCacheTest, RefreshBlobFileReaderPrefersLargestObservedFileSize) {
BlobFileCache blob_file_cache(backing_cache.get(), &immutable_options,
&file_options, column_family_id,
blob_file_read_hist, nullptr /*IOTracer*/);
const BlobFileOpenInfo blob_file{blob_file_number, Slice(), Slice()};
const ReadOptions read_options;
std::unique_ptr<BlobFileReader> stale_reader;
ASSERT_OK(blob_file_cache.OpenBlobFileReaderUncached(
read_options, blob_file_number, &stale_reader,
read_options, blob_file, &stale_reader,
/*allow_footer_skip_retry=*/true));
std::unique_ptr<BlobFileReader> initial_cached_reader;
ASSERT_OK(blob_file_cache.OpenBlobFileReaderUncached(
read_options, blob_file_number, &initial_cached_reader,
read_options, blob_file, &initial_cached_reader,
/*allow_footer_skip_retry=*/true));
CacheHandleGuard<BlobFileReader> cached_reader;
ASSERT_OK(blob_file_cache.RefreshBlobFileReader(
blob_file_number, &initial_cached_reader, &cached_reader));
blob_file, &initial_cached_reader, &cached_reader));
ASSERT_NE(cached_reader.GetValue(), nullptr);
const uint64_t initial_file_size = cached_reader.GetValue()->GetFileSize();
@@ -269,20 +270,20 @@ TEST_F(BlobFileCacheTest, RefreshBlobFileReaderPrefersLargestObservedFileSize) {
std::unique_ptr<BlobFileReader> fresh_reader;
ASSERT_OK(blob_file_cache.OpenBlobFileReaderUncached(
read_options, blob_file_number, &fresh_reader,
read_options, blob_file, &fresh_reader,
/*allow_footer_skip_retry=*/true));
ASSERT_GT(fresh_reader->GetFileSize(), initial_file_size);
CacheHandleGuard<BlobFileReader> refreshed_reader;
ASSERT_OK(blob_file_cache.RefreshBlobFileReader(
blob_file_number, &fresh_reader, &refreshed_reader));
ASSERT_OK(blob_file_cache.RefreshBlobFileReader(blob_file, &fresh_reader,
&refreshed_reader));
ASSERT_NE(refreshed_reader.GetValue(), nullptr);
ASSERT_GT(refreshed_reader.GetValue()->GetFileSize(), initial_file_size);
BlobFileReader* const largest_reader = refreshed_reader.GetValue();
CacheHandleGuard<BlobFileReader> preserved_reader;
ASSERT_OK(blob_file_cache.RefreshBlobFileReader(
blob_file_number, &stale_reader, &preserved_reader));
ASSERT_OK(blob_file_cache.RefreshBlobFileReader(blob_file, &stale_reader,
&preserved_reader));
ASSERT_NE(preserved_reader.GetValue(), nullptr);
ASSERT_EQ(preserved_reader.GetValue(), largest_reader);
ASSERT_EQ(stale_reader.get(), nullptr);
@@ -312,12 +313,13 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_IOError) {
// Note: there is no blob file with the below number
constexpr uint64_t blob_file_number = 123;
const BlobFileOpenInfo blob_file{blob_file_number, Slice(), Slice()};
CacheHandleGuard<BlobFileReader> reader;
const ReadOptions read_options;
ASSERT_TRUE(
blob_file_cache.GetBlobFileReader(read_options, blob_file_number, &reader)
blob_file_cache.GetBlobFileReader(read_options, blob_file, &reader)
.IsIOError());
ASSERT_EQ(reader.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
@@ -352,6 +354,7 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_CacheFull) {
BlobFileCache blob_file_cache(backing_cache.get(), &immutable_options,
&file_options, column_family_id,
blob_file_read_hist, nullptr /*IOTracer*/);
const BlobFileOpenInfo blob_file{blob_file_number, Slice(), Slice()};
// Insert into cache should fail since it has zero capacity and
// strict_capacity_limit is set
@@ -359,7 +362,7 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_CacheFull) {
const ReadOptions read_options;
ASSERT_TRUE(
blob_file_cache.GetBlobFileReader(read_options, blob_file_number, &reader)
blob_file_cache.GetBlobFileReader(read_options, blob_file, &reader)
.IsMemoryLimit());
ASSERT_EQ(reader.GetValue(), nullptr);
ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
+16 -5
View File
@@ -330,6 +330,11 @@ Status BlobFilePartitionManager::FinalizeBlobFile(
*addition =
BlobFileAddition(file_number, blob_count, total_blob_bytes,
std::move(checksum_method), std::move(checksum_value));
if (blob_file_cache_ != nullptr) {
const BlobFileOpenInfo blob_file{file_number, addition->GetChecksumValue(),
addition->GetChecksumMethod()};
blob_file_cache_->RegisterBlobFileChecksum(blob_file);
}
return Status::OK();
}
@@ -753,6 +758,9 @@ void BlobFilePartitionManager::UnprotectSealedBlobFileNumbers(
// manifest-visible path. Evict it here so delayed protection does not
// leave an obsolete blob reader behind until DB close.
blob_file_cache_->Evict(file_number);
if (file_to_partition_.find(file_number) == file_to_partition_.end()) {
blob_file_cache_->UnregisterBlobFileChecksum(file_number);
}
}
}
}
@@ -772,6 +780,10 @@ void BlobFilePartitionManager::RemoveFilePartitionMappings(
// this file before it was sealed. Drop it now so future manifest-visible
// reads reopen against the finalized on-disk size and footer state.
blob_file_cache_->Evict(file_number);
if (protected_blob_file_refs_.find(file_number) ==
protected_blob_file_refs_.end()) {
blob_file_cache_->UnregisterBlobFileChecksum(file_number);
}
}
}
}
@@ -809,9 +821,9 @@ Status BlobFilePartitionManager::ResolveBlobDirectWriteIndex(
}
Status s;
const BlobFileOpenInfo blob_file{blob_idx.file_number(), Slice(), Slice()};
CacheHandleGuard<BlobFileReader> reader;
s = blob_file_cache->GetBlobFileReader(read_options, blob_idx.file_number(),
&reader,
s = blob_file_cache->GetBlobFileReader(read_options, blob_file, &reader,
/*allow_footer_skip_retry=*/true);
if (!s.ok()) {
return s;
@@ -836,7 +848,7 @@ Status BlobFilePartitionManager::ResolveBlobDirectWriteIndex(
std::unique_ptr<BlobFileReader> fresh_reader;
s = blob_file_cache->OpenBlobFileReaderUncached(
read_options, blob_idx.file_number(), &fresh_reader,
read_options, blob_file, &fresh_reader,
/*allow_footer_skip_retry=*/true);
if (!s.ok()) {
return s;
@@ -850,8 +862,7 @@ Status BlobFilePartitionManager::ResolveBlobDirectWriteIndex(
if (s.ok()) {
blob_value->PinSelf(fresh_contents->data());
CacheHandleGuard<BlobFileReader> ignored;
blob_file_cache
->RefreshBlobFileReader(blob_idx.file_number(), &fresh_reader, &ignored)
blob_file_cache->RefreshBlobFileReader(blob_file, &fresh_reader, &ignored)
.PermitUncheckedError();
}
+211 -14
View File
@@ -278,25 +278,31 @@ Status BlobFileReader::ReadFromFile(const RandomAccessFileReader* file_reader,
return s;
}
FSReadRequest read_req;
read_req.offset = read_offset;
read_req.len = read_size;
if (file_reader->use_direct_io()) {
constexpr char* scratch = nullptr;
read_req.scratch = nullptr;
AlignedBufferAllocationContext direct_io_context{direct_io_buffer};
s = file_reader->Read(io_options, read_offset, read_size, slice, scratch,
&direct_io_context, &dbg);
file_reader->Read(io_options, &read_req, &direct_io_context, &dbg);
} else {
buf->reset(new char[read_size]);
s = file_reader->Read(io_options, read_offset, read_size, slice, buf->get(),
nullptr, &dbg);
read_req.scratch = buf->get();
file_reader->Read(io_options, &read_req, nullptr, &dbg);
}
s = std::move(read_req.status);
if (!s.ok()) {
return s;
}
*slice = read_req.result;
if (slice->size() != read_size) {
return Status::Corruption("Failed to read data from blob file");
return Status::Corruption("Incomplete blob read from " +
file_reader->file_name() + " at offset " +
std::to_string(read_offset) + ": expected " +
std::to_string(read_size) + " bytes, got " +
std::to_string(slice->size()));
}
return Status::OK();
@@ -330,7 +336,10 @@ Status BlobFileReader::GetBlob(
if (!IsValidBlobOffset(offset, key_size, value_size, file_size_,
has_footer_)) {
return Status::Corruption("Invalid blob offset");
return Status::Corruption(
"Invalid blob offset " + std::to_string(offset) + " for value size " +
std::to_string(value_size) + " in " + file_reader_->file_name() +
" (file size " + std::to_string(file_size_) + ")");
}
if (compression_type != compression_type_) {
@@ -438,12 +447,18 @@ Status BlobFileReader::GetBlobRange(const ReadOptions& read_options,
// performs); this guards the sub-range read below.
if (!IsValidBlobOffset(offset, key_size, value_size, file_size_,
has_footer_)) {
return Status::Corruption("Invalid blob offset");
return Status::Corruption(
"Invalid blob offset " + std::to_string(offset) + " for value size " +
std::to_string(value_size) + " in " + file_reader_->file_name() +
" (file size " + std::to_string(file_size_) + ")");
}
// The requested sub-range must lie within the value.
if (range_offset > value_size || range_length > value_size - range_offset) {
return Status::InvalidArgument("Blob range out of bounds");
return Status::InvalidArgument(
"Blob range [" + std::to_string(range_offset) + ", +" +
std::to_string(range_length) + ") out of bounds for value size " +
std::to_string(value_size) + " in " + file_reader_->file_name());
}
// Read only the requested bytes, at the value's file position plus the range
@@ -518,7 +533,10 @@ void BlobFileReader::MultiGetBlob(
if (!IsValidBlobOffset(offset, key_size, value_size, file_size_,
has_footer_)) {
*req->status = Status::Corruption("Invalid blob offset");
*req->status = Status::Corruption(
"Invalid blob offset " + std::to_string(offset) + " for value size " +
std::to_string(value_size) + " in " + file_reader_->file_name() +
" (file size " + std::to_string(file_size_) + ")");
continue;
}
if (req->compression != compression_type_) {
@@ -612,8 +630,11 @@ void BlobFileReader::MultiGetBlob(
auto& read_req = read_reqs[j++];
const auto& record_slice = read_req.result;
if (read_req.status.ok() && record_slice.size() != read_req.len) {
read_req.status =
IOStatus::Corruption("Failed to read data from blob file");
read_req.status = IOStatus::Corruption(
"Incomplete blob read from " + file_reader_->file_name() +
" at offset " + std::to_string(read_req.offset) + ": expected " +
std::to_string(read_req.len) + " bytes, got " +
std::to_string(record_slice.size()));
}
*req->status = read_req.status;
@@ -644,6 +665,182 @@ void BlobFileReader::MultiGetBlob(
}
}
void BlobFileReader::MultiGetBlobRange(
const ReadOptions& read_options,
autovector<std::pair<BlobRangeReadRequest*, std::unique_ptr<BlobContents>>>&
blob_reqs,
uint64_t* bytes_read) const {
const size_t num_blobs = blob_reqs.size();
assert(num_blobs > 0);
assert(num_blobs <= MultiGetContext::MAX_BATCH_SIZE);
// Range reads are only issued for uncompressed blob references (a strict
// sub-range of a compressed record cannot be decompressed in isolation).
// Version::MultiGetBlobLazy rejects a compressed blob index before it reaches
// here, so every request's blob index is uncompressed by contract; a
// compressed blob file therefore means the blob index disagrees with the
// file. Report that as the compression-type mismatch it is -- exactly like
// the whole-value MultiGetBlob path -- rather than as a usage error. (Unlike
// the single-read path, BlobSource has no per-request compression to check
// for a range read, so this reader-level check is the range multi path's
// compression backstop.)
if (compression_type_ != kNoCompression) {
for (auto& blob_req : blob_reqs) {
assert(blob_req.first);
assert(blob_req.first->status);
*blob_req.first->status =
Status::Corruption("Compression type mismatch when reading a blob");
}
return;
}
#ifndef NDEBUG
for (size_t i = 0; i < num_blobs - 1; ++i) {
assert(blob_reqs[i].first->offset + blob_reqs[i].first->range_offset <=
blob_reqs[i + 1].first->offset +
blob_reqs[i + 1].first->range_offset);
}
#endif // !NDEBUG
std::vector<FSReadRequest> read_reqs;
uint64_t total_len = 0;
read_reqs.reserve(num_blobs);
for (size_t i = 0; i < num_blobs; ++i) {
BlobRangeReadRequest* const req = blob_reqs[i].first;
assert(req);
assert(req->status);
const size_t key_size = req->user_key.size();
// Validate that the full value region is within the file (same check
// GetBlob performs); this guards the sub-range read below.
if (!IsValidBlobOffset(req->offset, key_size, req->value_size, file_size_,
has_footer_)) {
*req->status = Status::Corruption(
"Invalid blob offset " + std::to_string(req->offset) +
" for value size " + std::to_string(req->value_size) + " in " +
file_reader_->file_name() + " (file size " +
std::to_string(file_size_) + ")");
continue;
}
// The requested sub-range must lie within the value.
if (req->range_offset > req->value_size ||
req->range_length > req->value_size - req->range_offset) {
*req->status = Status::InvalidArgument(
"Blob range [" + std::to_string(req->range_offset) + ", +" +
std::to_string(req->range_length) +
") out of bounds for value size " + std::to_string(req->value_size) +
" in " + file_reader_->file_name());
continue;
}
// Read only the requested bytes, at the value's file position plus the
// range offset -- no record-header adjustment, no whole-record checksum.
FSReadRequest read_req;
read_req.offset = req->offset + req->range_offset;
read_req.len = req->range_length;
total_len += read_req.len;
read_reqs.emplace_back(std::move(read_req));
}
RecordTick(statistics_, BLOB_DB_BLOB_FILE_BYTES_READ, total_len);
if (read_reqs.empty()) {
if (bytes_read) {
*bytes_read = 0;
}
return;
}
Buffer buf;
AlignedBuffer direct_io_buffer;
Status s;
bool direct_io = file_reader_->use_direct_io();
if (direct_io) {
for (size_t i = 0; i < read_reqs.size(); ++i) {
read_reqs[i].scratch = nullptr;
}
} else {
buf.reset(new char[total_len]);
std::ptrdiff_t pos = 0;
for (size_t i = 0; i < read_reqs.size(); ++i) {
read_reqs[i].scratch = buf.get() + pos;
pos += read_reqs[i].len;
}
}
TEST_SYNC_POINT("BlobFileReader::MultiGetBlobRange:ReadFromFile");
PERF_COUNTER_ADD(blob_read_count, num_blobs);
PERF_COUNTER_ADD(blob_read_byte, total_len);
IOOptions opts;
IODebugContext dbg;
s = file_reader_->PrepareIOOptions(read_options, opts, &dbg);
if (s.ok()) {
AlignedBufferAllocationContext direct_io_context{&direct_io_buffer};
s = file_reader_->MultiRead(opts, read_reqs.data(), read_reqs.size(),
&direct_io_context, &dbg);
}
if (!s.ok()) {
// Batch read failed; s is written to every request's status below, so these
// per-request FS statuses carry nothing extra.
for (auto& req : read_reqs) {
req.status.PermitUncheckedError();
}
for (auto& blob_req : blob_reqs) {
BlobRangeReadRequest* const req = blob_req.first;
assert(req);
assert(req->status);
if (!req->status->IsCorruption() && !req->status->IsInvalidArgument()) {
// Avoid overwriting per-request validation errors.
*req->status = s;
}
}
return;
}
assert(s.ok());
TEST_SYNC_POINT_CALLBACK("BlobFileReader::MultiGetBlobRange:TamperWithResult",
&read_reqs);
uint64_t total_bytes = 0;
for (size_t i = 0, j = 0; i < num_blobs; ++i) {
BlobRangeReadRequest* const req = blob_reqs[i].first;
assert(req);
assert(req->status);
if (!req->status->ok()) {
continue;
}
assert(j < read_reqs.size());
auto& read_req = read_reqs[j++];
const Slice& range_slice = read_req.result;
if (read_req.status.ok() && range_slice.size() != read_req.len) {
read_req.status = IOStatus::Corruption(
"Incomplete blob range read from " + file_reader_->file_name() +
" at offset " + std::to_string(read_req.offset) + ": expected " +
std::to_string(read_req.len) + " bytes, got " +
std::to_string(range_slice.size()));
}
*req->status = read_req.status;
if (!req->status->ok()) {
continue;
}
// The blob is uncompressed, so the bytes read are exactly the requested
// value bytes; copy them into an owned BlobContents (no decompression).
BlobContentsCreator::Create(&blob_reqs[i].second, /*out_charge=*/nullptr,
range_slice, kNoCompression,
/*allocator=*/nullptr);
total_bytes += range_slice.size();
}
if (bytes_read) {
*bytes_read = total_bytes;
}
}
Status BlobFileReader::VerifyBlob(const Slice& record_slice,
const Slice& user_key, uint64_t value_size) {
PERF_TIMER_GUARD(blob_checksum_time);
+15
View File
@@ -88,6 +88,21 @@ class BlobFileReader {
blob_reqs,
uint64_t* bytes_read) const;
// Byte-range (partial) multi-read counterpart of MultiGetBlob: for each
// request reads only its sub-range [offset + range_offset, +range_length) of
// an *uncompressed* blob value, coalescing all requests into a single
// MultiRead. Unlike MultiGetBlob it reads no record header/key, verifies no
// whole-record checksum (a strict sub-range cannot cover it), and never
// decompresses -- so it is only valid when this reader's blob file is
// uncompressed (checked). Each result owns a copy of its requested bytes.
// Requests must be sorted ascending by effective read offset (offset +
// range_offset) by the caller.
void MultiGetBlobRange(
const ReadOptions& read_options,
autovector<std::pair<BlobRangeReadRequest*,
std::unique_ptr<BlobContents>>>& blob_reqs,
uint64_t* bytes_read) const;
CompressionType GetCompressionType() const { return compression_type_; }
uint64_t GetFileSize() const { return file_size_; }
+296 -21
View File
@@ -5,9 +5,11 @@
#include "db/blob/blob_gen2_format.h"
#include <algorithm>
#include <array>
#include <cstring>
#include <string>
#include <vector>
#include "file/random_access_file_reader.h"
#include "file/writable_file_writer.h"
@@ -16,9 +18,56 @@
#include "table/format.h"
#include "util/cast_util.h"
#include "util/coding.h"
#include "util/crc32c.h"
namespace ROCKSDB_NAMESPACE {
namespace {
// Verifies a SimpleGen2Blob record's builtin checksum: the stored 4-byte
// checksum (at record[payload_size + 1 ..], context-modified by the record's
// file offset) against the checksum computed over the payload plus its 1-byte
// compression marker. Returns OK, or a Corruption carrying debugging
// information -- stored (with the per-offset context modifier removed),
// computed, and checksum type, with CRC values unmasked so they can be compared
// against a reference. Mirrors VerifyBlockChecksum() for the block-based
// format, since a SimpleGen2Blob record's trailer uses the same
// builtin-checksum scheme; shared by the scalar and batched readers so both
// report the same diagnostics.
Status VerifySimpleGen2BlobChecksum(ChecksumType checksum_type,
uint32_t base_context_checksum,
const char* record, size_t payload_size,
uint64_t record_offset,
const std::string& file_name) {
uint32_t stored = DecodeFixed32(record + payload_size + 1);
const uint32_t modifier =
ChecksumModifierForContext(base_context_checksum, record_offset);
stored -= modifier;
const uint32_t computed = ComputeBuiltinChecksumWithLastByte(
checksum_type, record, payload_size, record[payload_size]);
if (stored == computed) {
return Status::OK();
}
// Unmask CRC values (as VerifyBlockChecksum does) so a reader of the error
// can compare against a reference checksum.
uint32_t stored_for_msg = stored;
uint32_t computed_for_msg = computed;
if (checksum_type == kCRC32c) {
stored_for_msg = crc32c::Unmask(stored);
computed_for_msg = crc32c::Unmask(computed);
}
return Status::Corruption("SimpleGen2 blob record checksum mismatch: stored" +
std::string(modifier ? "(context removed)" : "") +
" = " + std::to_string(stored_for_msg) +
", computed = " + std::to_string(computed_for_msg) +
", type = " + std::to_string(checksum_type) +
" in " + file_name + " offset " +
std::to_string(record_offset) + " size " +
std::to_string(payload_size));
}
} // namespace
Status ReadAndVerifySimpleGen2BlobRecord(
const ReadOptions& read_options, RandomAccessFileReader* file,
uint64_t record_offset, size_t payload_size, size_t record_size,
@@ -33,14 +82,17 @@ Status ReadAndVerifySimpleGen2BlobRecord(
IODebugContext dbg;
Status s = file->PrepareIOOptions(read_options, opts, &dbg);
if (s.ok()) {
s = file->Read(opts, record_offset, record_size, &result, buf, nullptr,
&dbg);
s = file->Read(record_offset, record_size, opts, &result, buf, &dbg);
}
if (!s.ok()) {
return s;
}
if (result.size() != record_size) {
return Status::Corruption("Could not read complete blob record");
return Status::Corruption("Incomplete blob record read from " +
file->file_name() + " at offset " +
std::to_string(record_offset) + ": expected " +
std::to_string(record_size) + " bytes, got " +
std::to_string(result.size()));
}
// With mmap reads the data lands outside `buf`; copy it in so the caller can
// rely on `buf` owning the bytes (this is the only copy on the mmap path).
@@ -54,24 +106,32 @@ Status ReadAndVerifySimpleGen2BlobRecord(
static_cast<CompressionType>(record[payload_size]);
if (compression != expected_compression) {
return Status::Corruption(
"Blob record compression does not match blob index");
"Blob record compression mismatch in " + file->file_name() +
" at offset " + std::to_string(record_offset) + ": index expects " +
std::to_string(static_cast<int>(expected_compression)) +
", record has " + std::to_string(static_cast<int>(compression)));
}
if (read_options.verify_checksums) {
uint32_t stored = DecodeFixed32(record + payload_size + 1);
stored -= ChecksumModifierForContext(base_context_checksum, record_offset);
const uint32_t computed = ComputeBuiltinChecksumWithLastByte(
checksum_type, record, payload_size, record[payload_size]);
if (stored != computed) {
return Status::Corruption("Blob record checksum mismatch in " +
file->file_name() + " offset " +
std::to_string(record_offset) + " size " +
std::to_string(payload_size));
const Status checksum_status = VerifySimpleGen2BlobChecksum(
checksum_type, base_context_checksum, record, payload_size,
record_offset, file->file_name());
if (!checksum_status.ok()) {
return checksum_status;
}
}
// The compression marker matched the blob index above, but SimpleGen2Blob
// payloads are currently written uncompressed only (see
// WriteSimpleGen2BlobRecord), so a record that actually carries a compression
// type is not readable here. This guards a corrupt or forward-incompatible
// record whose (compressed) marker happened to agree with a (compressed)
// index.
if (compression != kNoCompression) {
return Status::Corruption("Blob record compression is not supported");
return Status::Corruption("Blob record compression " +
std::to_string(static_cast<int>(compression)) +
" is not supported in " + file->file_name() +
" at offset " + std::to_string(record_offset));
}
return Status::OK();
}
@@ -88,14 +148,21 @@ Status ReadSimpleGen2BlobRange(const ReadOptions& read_options,
// A strict sub-range of a compressed payload cannot be decompressed in
// isolation; callers resolve such payloads whole and slice instead.
if (expected_compression != kNoCompression) {
return Status::Corruption("Cannot range-read a compressed blob");
return Status::Corruption(
"Cannot range-read a compressed blob (compression " +
std::to_string(static_cast<int>(expected_compression)) + ") in " +
file->file_name() + " at offset " + std::to_string(record_offset));
}
// The requested sub-range must lie within the payload (the trailer that
// follows the payload is never part of a range read).
if (range_offset > payload_size ||
range_length > payload_size - range_offset) {
return Status::InvalidArgument("Blob range out of bounds");
return Status::InvalidArgument(
"Blob range [" + std::to_string(range_offset) + ", +" +
std::to_string(range_length) + ") out of bounds for payload size " +
std::to_string(payload_size) + " in " + file->file_name() +
" at offset " + std::to_string(record_offset));
}
Slice result;
@@ -103,14 +170,18 @@ Status ReadSimpleGen2BlobRange(const ReadOptions& read_options,
IODebugContext dbg;
Status s = file->PrepareIOOptions(read_options, opts, &dbg);
if (s.ok()) {
s = file->Read(opts, record_offset + range_offset, range_length, &result,
buf, nullptr, &dbg);
s = file->Read(record_offset + range_offset, range_length, opts, &result,
buf, &dbg);
}
if (!s.ok()) {
return s;
}
if (result.size() != range_length) {
return Status::Corruption("Could not read complete blob range");
return Status::Corruption("Incomplete blob range read from " +
file->file_name() + " at offset " +
std::to_string(record_offset + range_offset) +
": expected " + std::to_string(range_length) +
" bytes, got " + std::to_string(result.size()));
}
// With mmap reads the data lands outside `buf`; copy it in so the caller can
// rely on `buf` owning the bytes (the only copy on the mmap path).
@@ -124,6 +195,210 @@ Status ReadSimpleGen2BlobRange(const ReadOptions& read_options,
return Status::OK();
}
namespace {
// One record's read description for the batched SimpleGen2Blob readers below:
// where/how much to read, where to put it, and where to report the outcome.
struct Gen2ReadSlot {
uint64_t offset = 0;
size_t len = 0;
char* buf = nullptr;
Status* status = nullptr;
};
// Issues a single MultiRead for the records described by `slots` and copies
// each result into its buf, setting *slots[i].status. Shared by the
// whole-record and range batch readers; per-record format checks (compression
// marker / checksum) are done by the whole-record caller afterwards from the
// (now populated) buf.
void MultiReadGen2(const ReadOptions& read_options,
RandomAccessFileReader* file,
std::vector<Gen2ReadSlot>& slots) {
assert(file != nullptr);
assert(!slots.empty());
// RandomAccessFileReader::MultiRead requires the requests sorted ascending by
// file offset -- it asserts this, and in direct-I/O mode it aligns and merges
// only consecutive requests, so unsorted input reads the wrong bytes. The
// callers group records by SST but not by offset, so sort here. Each slot
// carries its own output buf/status pointer, so sorting does not disturb
// result routing.
std::sort(slots.begin(), slots.end(),
[](const Gen2ReadSlot& a, const Gen2ReadSlot& b) {
return a.offset < b.offset;
});
const size_t num = slots.size();
std::vector<FSReadRequest> read_reqs;
read_reqs.reserve(num);
for (size_t i = 0; i < num; ++i) {
FSReadRequest read_req;
read_req.offset = slots[i].offset;
read_req.len = slots[i].len;
read_reqs.emplace_back(std::move(read_req));
}
AlignedBuffer direct_io_buffer;
const bool direct_io = file->use_direct_io();
if (direct_io) {
for (size_t i = 0; i < num; ++i) {
read_reqs[i].scratch = nullptr;
}
} else {
for (size_t i = 0; i < num; ++i) {
read_reqs[i].scratch = slots[i].buf;
}
}
IOOptions opts;
IODebugContext dbg;
Status s = file->PrepareIOOptions(read_options, opts, &dbg);
if (s.ok()) {
AlignedBufferAllocationContext direct_io_context{&direct_io_buffer};
s = file->MultiRead(opts, read_reqs.data(), read_reqs.size(),
&direct_io_context, &dbg);
}
if (!s.ok()) {
// Batch read failed; s is written to every slot's status below, so these
// per-request FS statuses carry nothing extra.
for (auto& read_req : read_reqs) {
read_req.status.PermitUncheckedError();
}
for (size_t i = 0; i < num; ++i) {
*slots[i].status = s;
}
return;
}
for (size_t i = 0; i < num; ++i) {
FSReadRequest& read_req = read_reqs[i];
if (read_req.status.ok() && read_req.result.size() != read_req.len) {
read_req.status = IOStatus::Corruption(
"Incomplete blob record read from " + file->file_name() +
" at offset " + std::to_string(read_req.offset) + ": expected " +
std::to_string(read_req.len) + " bytes, got " +
std::to_string(read_req.result.size()));
}
*slots[i].status = read_req.status;
if (!slots[i].status->ok()) {
continue;
}
// With mmap/direct reads the data lands outside the caller's buf; copy it
// in so the caller can rely on buf owning the bytes.
if (read_req.result.data() != slots[i].buf) {
memcpy(slots[i].buf, read_req.result.data(), read_req.len);
}
}
}
} // namespace
void ReadAndVerifySimpleGen2BlobRecords(const ReadOptions& read_options,
RandomAccessFileReader* file,
ChecksumType checksum_type,
uint32_t base_context_checksum,
size_t num_records,
SimpleGen2RecordReadRequest* reqs) {
assert(file != nullptr);
assert(num_records > 0);
std::vector<Gen2ReadSlot> slots;
slots.reserve(num_records);
for (size_t i = 0; i < num_records; ++i) {
assert(reqs[i].buf != nullptr);
assert(reqs[i].status != nullptr);
assert(reqs[i].record_size ==
reqs[i].payload_size + kSimpleGen2BlobTrailerSize);
slots.push_back({reqs[i].record_offset, reqs[i].record_size, reqs[i].buf,
reqs[i].status});
}
MultiReadGen2(read_options, file, slots);
for (size_t i = 0; i < num_records; ++i) {
if (!reqs[i].status->ok()) {
continue;
}
const char* record = reqs[i].buf;
const size_t payload_size = reqs[i].payload_size;
const CompressionType compression =
static_cast<CompressionType>(record[payload_size]);
if (compression != reqs[i].expected_compression) {
*reqs[i].status = Status::Corruption(
"Blob record compression mismatch in " + file->file_name() +
" at offset " + std::to_string(reqs[i].record_offset) +
": index expects " +
std::to_string(static_cast<int>(reqs[i].expected_compression)) +
", record has " + std::to_string(static_cast<int>(compression)));
continue;
}
if (read_options.verify_checksums) {
*reqs[i].status = VerifySimpleGen2BlobChecksum(
checksum_type, base_context_checksum, record, payload_size,
reqs[i].record_offset, file->file_name());
if (!reqs[i].status->ok()) {
continue;
}
}
// See ReadAndVerifySimpleGen2BlobRecord: the marker matched the index, but
// only uncompressed SimpleGen2Blob payloads are currently supported.
if (compression != kNoCompression) {
*reqs[i].status = Status::Corruption(
"Blob record compression " +
std::to_string(static_cast<int>(compression)) +
" is not supported in " + file->file_name() + " at offset " +
std::to_string(reqs[i].record_offset));
continue;
}
}
}
void ReadSimpleGen2BlobRanges(const ReadOptions& read_options,
RandomAccessFileReader* file, size_t num_records,
SimpleGen2RangeReadRequest* reqs) {
assert(file != nullptr);
assert(num_records > 0);
std::vector<Gen2ReadSlot> slots;
slots.reserve(num_records);
for (size_t i = 0; i < num_records; ++i) {
assert(reqs[i].buf != nullptr);
assert(reqs[i].status != nullptr);
// A strict sub-range of a compressed payload cannot be decompressed in
// isolation; callers resolve such payloads whole and slice instead.
if (reqs[i].expected_compression != kNoCompression) {
*reqs[i].status = Status::Corruption(
"Cannot range-read a compressed blob (compression " +
std::to_string(static_cast<int>(reqs[i].expected_compression)) +
") in " + file->file_name() + " at offset " +
std::to_string(reqs[i].record_offset));
continue;
}
// The requested sub-range must lie within the payload (the trailer that
// follows the payload is never part of a range read).
if (reqs[i].range_offset > reqs[i].payload_size ||
reqs[i].range_length > reqs[i].payload_size - reqs[i].range_offset) {
*reqs[i].status = Status::InvalidArgument(
"Blob range [" + std::to_string(reqs[i].range_offset) + ", +" +
std::to_string(reqs[i].range_length) +
") out of bounds for payload size " +
std::to_string(reqs[i].payload_size) + " in " + file->file_name() +
" at offset " + std::to_string(reqs[i].record_offset));
continue;
}
slots.push_back({reqs[i].record_offset + reqs[i].range_offset,
reqs[i].range_length, reqs[i].buf, reqs[i].status});
}
if (slots.empty()) {
return;
}
// slots only covers requests that passed validation above; a failed
// validation left *reqs[i].status non-OK and is skipped here.
MultiReadGen2(read_options, file, slots);
}
IOStatus WriteSimpleGen2BlobRecord(WritableFileWriter* file,
const WriteOptions& write_options,
ChecksumType checksum_type,
@@ -148,12 +423,12 @@ IOStatus WriteSimpleGen2BlobRecord(WritableFileWriter* file,
return io_s;
}
if (!payload.empty()) {
io_s = file->Append(opts, payload);
io_s = file->Append(payload, opts);
if (!io_s.ok()) {
return io_s;
}
}
return file->Append(opts, Slice(trailer.data(), trailer.size()));
return file->Append(Slice(trailer.data(), trailer.size()), opts);
}
} // namespace ROCKSDB_NAMESPACE
+51
View File
@@ -91,6 +91,57 @@ Status ReadSimpleGen2BlobRange(const ReadOptions& read_options,
uint64_t range_offset, size_t range_length,
CompressionType expected_compression, char* buf);
// One whole-record request in a batched SimpleGen2Blob read (see
// ReadAndVerifySimpleGen2BlobRecords). Mirrors the scalar
// ReadAndVerifySimpleGen2BlobRecord inputs, with the read bytes going into
// caller-provided `buf` (capacity >= record_size) and the per-record outcome in
// `*status`.
struct SimpleGen2RecordReadRequest {
uint64_t record_offset = 0;
size_t payload_size = 0;
size_t record_size = 0; // == payload_size + kSimpleGen2BlobTrailerSize
CompressionType expected_compression = kNoCompression;
char* buf = nullptr;
Status* status = nullptr;
};
// One sub-range request in a batched SimpleGen2Blob range read (see
// ReadSimpleGen2BlobRanges). Mirrors the scalar ReadSimpleGen2BlobRange inputs.
struct SimpleGen2RangeReadRequest {
uint64_t record_offset = 0;
size_t payload_size = 0;
uint64_t range_offset = 0;
size_t range_length = 0;
CompressionType expected_compression = kNoCompression;
char* buf = nullptr;
Status* status = nullptr;
};
// Batch counterpart of ReadAndVerifySimpleGen2BlobRecord: reads `num_records`
// whole records from `file` in a single MultiRead, then per record verifies the
// compression marker and (when read_options.verify_checksums) the
// context-modified checksum, copying the record into reqs[i].buf. Per-record
// outcome is written to *reqs[i].status; on success reqs[i].buf[0,
// payload_size) holds the verified payload (the trailer sits unused at the
// tail). Requests may be given in any order; they are sorted internally, as
// RandomAccessFileReader::MultiRead requires ascending offsets.
void ReadAndVerifySimpleGen2BlobRecords(const ReadOptions& read_options,
RandomAccessFileReader* file,
ChecksumType checksum_type,
uint32_t base_context_checksum,
size_t num_records,
SimpleGen2RecordReadRequest* reqs);
// Batch counterpart of ReadSimpleGen2BlobRange: reads the requested sub-range
// of `num_records` *uncompressed* payloads from `file` in a single MultiRead
// (no trailer read, no checksum). Per-record outcome is in *reqs[i].status; on
// success reqs[i].buf[0, range_length) holds the requested bytes. Requests may
// be given in any order; they are sorted internally, as
// RandomAccessFileReader::MultiRead requires ascending offsets.
void ReadSimpleGen2BlobRanges(const ReadOptions& read_options,
RandomAccessFileReader* file, size_t num_records,
SimpleGen2RangeReadRequest* reqs);
// Writes a SimpleGen2Blob record for `payload` at the current end of `file`,
// which the caller asserts is byte offset `record_offset`. Appends the payload
// bytes followed by the 5-byte trailer (compression marker + context-modified
+1 -1
View File
@@ -29,7 +29,7 @@ Status BlobLogSequentialReader::ReadSlice(uint64_t size, Slice* slice,
StopWatch read_sw(clock_, statistics_, BLOB_DB_BLOB_FILE_READ_MICROS);
// TODO: rate limit `BlobLogSequentialReader` reads (it appears unused?)
Status s = file_->Read(IOOptions(), next_byte_, static_cast<size_t>(size),
Status s = file_->Read(next_byte_, static_cast<size_t>(size), IOOptions(),
slice, buf, nullptr);
next_byte_ += size;
if (!s.ok()) {
+6 -6
View File
@@ -40,7 +40,7 @@ Status BlobLogWriter::Sync(const WriteOptions& write_options) {
IOOptions opts;
Status s = WritableFileWriter::PrepareIOOptions(write_options, opts);
if (s.ok()) {
s = dest_->Sync(opts, use_fsync_);
s = use_fsync_ ? dest_->Fsync(opts) : dest_->Sync(opts);
}
if (s.ok()) {
RecordTick(statistics_, BLOB_DB_BLOB_FILE_SYNCED);
@@ -58,7 +58,7 @@ Status BlobLogWriter::WriteHeader(const WriteOptions& write_options,
IOOptions opts;
Status s = WritableFileWriter::PrepareIOOptions(write_options, opts);
if (s.ok()) {
s = dest_->Append(opts, Slice(str));
s = dest_->Append(Slice(str), opts);
}
if (s.ok()) {
block_offset_ += str.size();
@@ -92,7 +92,7 @@ Status BlobLogWriter::AppendFooter(const WriteOptions& write_options,
IOOptions opts;
s = WritableFileWriter::PrepareIOOptions(write_options, opts);
if (s.ok()) {
s = dest_->Append(opts, Slice(str));
s = dest_->Append(Slice(str), opts);
}
if (s.ok()) {
block_offset_ += str.size();
@@ -181,13 +181,13 @@ Status BlobLogWriter::EmitPhysicalRecord(const WriteOptions& write_options,
IOOptions opts;
Status s = WritableFileWriter::PrepareIOOptions(write_options, opts);
if (s.ok()) {
s = dest_->Append(opts, Slice(headerbuf));
s = dest_->Append(Slice(headerbuf), opts);
}
if (s.ok()) {
s = dest_->Append(opts, key);
s = dest_->Append(key, opts);
}
if (s.ok()) {
s = dest_->Append(opts, val);
s = dest_->Append(val, opts);
}
if (do_flush_ && s.ok()) {
s = dest_->Flush(opts);
+56 -1
View File
@@ -14,10 +14,21 @@
namespace ROCKSDB_NAMESPACE {
// Non-owning metadata needed when opening a blob file for reading. The
// checksum slices must remain valid while this object is in use. A file without
// finalized checksum metadata, such as an active direct-write blob file, uses
// empty slices for both checksum fields.
struct BlobFileOpenInfo {
uint64_t file_number = 0;
Slice file_checksum;
Slice file_checksum_func_name;
};
// A read Blob request structure for use in BlobSource::MultiGetBlob and
// BlobFileReader::MultiGetBlob.
struct BlobReadRequest {
// User key to lookup the paired blob
// TODO: consider simplifying to copied Slice?
const Slice* user_key = nullptr;
// File offset in bytes
@@ -52,7 +63,51 @@ struct BlobReadRequest {
};
using BlobFileReadRequests =
std::tuple<uint64_t /* file_number */, uint64_t /* file_size */,
std::tuple<BlobFileOpenInfo, uint64_t /* file_size */,
autovector<BlobReadRequest>>;
// A byte-range (partial) blob read request for the lazy blob-read multi
// primitives (BlobFileReader::MultiGetBlobRange /
// BlobSource::MultiGetBlobRange). Unlike BlobReadRequest (whole value), it
// reads only the sub-range [range_offset, range_offset + range_length) of an
// *uncompressed* blob value, skipping whole-record checksum verification and
// blob-cache population -- the multi-read counterpart of
// BlobSource::GetBlobRange / BlobFileReader::GetBlobRange.
struct BlobRangeReadRequest {
// User key that pairs with the blob value (used for offset validation).
Slice user_key;
// The blob value's file offset and full (logical == on-disk, uncompressed)
// size, from the BlobIndex.
uint64_t offset = 0;
uint64_t value_size = 0;
// The requested sub-range within the value. `range_length` is the number of
// bytes to read; the caller must ensure range_offset + range_length <=
// value_size (it clamps before building the request).
uint64_t range_offset = 0;
size_t range_length = 0;
// Output parameter set to point to the requested bytes, and the read status.
PinnableSlice* result = nullptr;
Status* status = nullptr;
BlobRangeReadRequest(Slice _user_key, uint64_t _offset, uint64_t _value_size,
uint64_t _range_offset, size_t _range_length,
PinnableSlice* _result, Status* _status)
: user_key(_user_key),
offset(_offset),
value_size(_value_size),
range_offset(_range_offset),
range_length(_range_length),
result(_result),
status(_status) {}
BlobRangeReadRequest() = default;
};
using BlobFileRangeReadRequests =
std::tuple<BlobFileOpenInfo, uint64_t /* file_size */,
autovector<BlobRangeReadRequest>>;
} // namespace ROCKSDB_NAMESPACE
+364 -24
View File
@@ -7,6 +7,7 @@
#include <cassert>
#include <string>
#include <vector>
#include "cache/cache_reservation_manager.h"
#include "cache/charged_cache.h"
@@ -49,9 +50,10 @@ Status AppendBlobRefreshRetryFailure(const Status& stale_status,
template <typename ReadFn>
Status ReadBlobWithReaderRetry(BlobFileCache* blob_file_cache,
const ReadOptions& read_options,
uint64_t file_number, const ReadFn& read) {
const BlobFileOpenInfo& blob_file,
const ReadFn& read) {
CacheHandleGuard<BlobFileReader> blob_file_reader;
Status s = blob_file_cache->GetBlobFileReader(read_options, file_number,
Status s = blob_file_cache->GetBlobFileReader(read_options, blob_file,
&blob_file_reader);
if (!s.ok()) {
return s;
@@ -65,11 +67,11 @@ Status ReadBlobWithReaderRetry(BlobFileCache* blob_file_cache,
const Status stale_status = s;
blob_file_reader.Reset();
blob_file_cache->Evict(file_number);
blob_file_cache->Evict(blob_file.file_number);
std::unique_ptr<BlobFileReader> fresh_reader;
s = blob_file_cache->OpenBlobFileReaderUncached(
read_options, file_number, &fresh_reader,
read_options, blob_file, &fresh_reader,
/*allow_footer_skip_retry=*/false);
if (!s.ok()) {
return AppendBlobRefreshRetryFailure(stale_status, s);
@@ -82,7 +84,7 @@ Status ReadBlobWithReaderRetry(BlobFileCache* blob_file_cache,
CacheHandleGuard<BlobFileReader> ignored_reader;
blob_file_cache
->RefreshBlobFileReader(file_number, &fresh_reader, &ignored_reader)
->RefreshBlobFileReader(blob_file, &fresh_reader, &ignored_reader)
.PermitUncheckedError();
return s;
}
@@ -271,9 +273,9 @@ Status BlobSource::InsertEntryIntoCache(const Slice& key, BlobContents* value,
}
Status BlobSource::GetBlob(const ReadOptions& read_options,
const Slice& user_key, uint64_t file_number,
uint64_t offset, uint64_t file_size,
uint64_t value_size,
const Slice& user_key,
const BlobFileOpenInfo& blob_file, uint64_t offset,
uint64_t file_size, uint64_t value_size,
CompressionType compression_type,
FilePrefetchBuffer* prefetch_buffer,
PinnableSlice* value, uint64_t* bytes_read) {
@@ -281,7 +283,8 @@ Status BlobSource::GetBlob(const ReadOptions& read_options,
Status s;
const CacheKey cache_key = GetCacheKey(file_number, file_size, offset);
const CacheKey cache_key =
GetCacheKey(blob_file.file_number, file_size, offset);
CacheHandleGuard<BlobContents> blob_handle;
@@ -331,8 +334,7 @@ Status BlobSource::GetBlob(const ReadOptions& read_options,
uint64_t read_size = 0;
s = ReadBlobWithReaderRetry(
blob_file_cache_, read_options, file_number,
[&](BlobFileReader* reader) {
blob_file_cache_, read_options, blob_file, [&](BlobFileReader* reader) {
if (compression_type != reader->GetCompressionType()) {
return Status::Corruption(
"Compression type mismatch when reading blob");
@@ -376,7 +378,8 @@ Status BlobSource::GetBlob(const ReadOptions& read_options,
}
Status BlobSource::GetBlobRange(const ReadOptions& read_options,
const Slice& user_key, uint64_t file_number,
const Slice& user_key,
const BlobFileOpenInfo& blob_file,
uint64_t offset, uint64_t file_size,
uint64_t value_size,
CompressionType compression_type,
@@ -393,7 +396,8 @@ Status BlobSource::GetBlobRange(const ReadOptions& read_options,
Status s;
const CacheKey cache_key = GetCacheKey(file_number, file_size, offset);
const CacheKey cache_key =
GetCacheKey(blob_file.file_number, file_size, offset);
CacheHandleGuard<BlobContents> blob_handle;
@@ -416,7 +420,8 @@ Status BlobSource::GetBlobRange(const ReadOptions& read_options,
const bool no_io = read_options.read_tier == kBlockCacheTier;
if (no_io) {
return Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
s = Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
return s;
}
// Cache miss: read only the requested sub-range from the file. The result is
@@ -433,8 +438,7 @@ Status BlobSource::GetBlobRange(const ReadOptions& read_options,
// case (offset/size mismatch surfacing as Corruption), not a payload
// checksum failure.
s = ReadBlobWithReaderRetry(
blob_file_cache_, read_options, file_number,
[&](BlobFileReader* reader) {
blob_file_cache_, read_options, blob_file, [&](BlobFileReader* reader) {
if (compression_type != reader->GetCompressionType()) {
return Status::Corruption(
"Compression type mismatch when reading blob");
@@ -501,7 +505,8 @@ Status BlobSource::GetSimpleGen2Blob(
const bool no_io = read_options.read_tier == kBlockCacheTier;
if (no_io) {
return Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
s = Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
return s;
}
// Cache miss (or no cache configured). Read the record into a buffer
@@ -601,7 +606,8 @@ Status BlobSource::GetSimpleGen2BlobRange(
const bool no_io = read_options.read_tier == kBlockCacheTier;
if (no_io) {
return Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
s = Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
return s;
}
// Cache miss: read only the requested sub-range from the file. The result is
@@ -633,6 +639,188 @@ Status BlobSource::GetSimpleGen2BlobRange(
return s;
}
void BlobSource::MultiGetSimpleGen2Blob(
const ReadOptions& read_options, const OffsetableCacheKey& base_cache_key,
RandomAccessFileReader* file, ChecksumType checksum_type,
uint32_t base_context_checksum, size_t num_records,
SimpleGen2BlobReadRequest* reqs) {
assert(file);
if (num_records == 0) {
return;
}
const bool no_io = read_options.read_tier == kBlockCacheTier;
MemoryAllocator* const allocator = (blob_cache_ && read_options.fill_cache)
? blob_cache_.get()->memory_allocator()
: nullptr;
// Records that missed the cache and must be read from disk, in read order;
// each owns its buffer until we build a BlobContents from it below.
struct Pending {
size_t idx;
CacheAllocationPtr buf;
uint64_t record_size;
};
std::vector<Pending> pending;
std::vector<SimpleGen2RecordReadRequest> read_reqs;
for (size_t i = 0; i < num_records; ++i) {
SimpleGen2BlobReadRequest& req = reqs[i];
assert(req.result);
assert(req.status);
const uint64_t record_size = req.payload_size + kSimpleGen2BlobTrailerSize;
const CacheKey cache_key =
GetSimpleGen2BlobCacheKey(base_cache_key, req.record_offset);
if (blob_cache_) {
CacheHandleGuard<BlobContents> blob_handle;
const Status cs = GetBlobFromCache(cache_key.AsSlice(), &blob_handle);
if (cs.ok()) {
*req.status = cs;
PinCachedBlob(&blob_handle, req.result);
continue;
}
}
if (no_io) {
*req.status =
Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
continue;
}
CacheAllocationPtr buf =
AllocateBlock(static_cast<size_t>(record_size), allocator);
read_reqs.push_back(SimpleGen2RecordReadRequest{
req.record_offset, static_cast<size_t>(req.payload_size),
static_cast<size_t>(record_size), req.expected_compression, buf.get(),
req.status});
pending.push_back(Pending{i, std::move(buf), record_size});
}
if (read_reqs.empty()) {
return;
}
ReadAndVerifySimpleGen2BlobRecords(read_options, file, checksum_type,
base_context_checksum, read_reqs.size(),
read_reqs.data());
for (size_t k = 0; k < pending.size(); ++k) {
SimpleGen2BlobReadRequest& req = reqs[pending[k].idx];
if (!req.status->ok()) {
continue;
}
const uint64_t record_size = pending[k].record_size;
std::unique_ptr<BlobContents> blob_contents(new BlobContents(
std::move(pending[k].buf), static_cast<size_t>(req.payload_size)));
RecordTick(statistics_, BLOB_DB_BLOB_FILE_BYTES_READ, record_size);
PERF_COUNTER_ADD(blob_read_count, 1);
PERF_COUNTER_ADD(blob_read_byte, record_size);
if (read_options.io_activity == Env::IOActivity::kLazyResolve) {
RecordLazyRead(statistics_, record_size);
}
if (blob_cache_ && read_options.fill_cache) {
const CacheKey cache_key =
GetSimpleGen2BlobCacheKey(base_cache_key, req.record_offset);
CacheHandleGuard<BlobContents> blob_handle;
const Status ps =
PutBlobIntoCache(cache_key.AsSlice(), &blob_contents, &blob_handle);
if (!ps.ok()) {
*req.status = ps;
} else {
PinCachedBlob(&blob_handle, req.result);
}
} else {
PinOwnedBlob(&blob_contents, req.result);
}
}
}
void BlobSource::MultiGetSimpleGen2BlobRange(
const ReadOptions& read_options, const OffsetableCacheKey& base_cache_key,
RandomAccessFileReader* file, size_t num_records,
SimpleGen2BlobRangeReadRequest* reqs) {
assert(file);
// Range reads are only issued while resolving a lazy result (partial reads
// are lazy-only), so the lazy read stats below are recorded unconditionally.
assert(read_options.io_activity == Env::IOActivity::kLazyResolve);
if (num_records == 0) {
return;
}
const bool no_io = read_options.read_tier == kBlockCacheTier;
struct Pending {
size_t idx;
CacheAllocationPtr buf;
};
std::vector<Pending> pending;
std::vector<SimpleGen2RangeReadRequest> read_reqs;
for (size_t i = 0; i < num_records; ++i) {
SimpleGen2BlobRangeReadRequest& req = reqs[i];
assert(req.result);
assert(req.status);
const CacheKey cache_key =
GetSimpleGen2BlobCacheKey(base_cache_key, req.record_offset);
// Probe the whole-payload cache; on a hit slice the requested sub-range.
if (blob_cache_) {
CacheHandleGuard<BlobContents> blob_handle;
const Status cs = GetBlobFromCache(cache_key.AsSlice(), &blob_handle);
if (cs.ok()) {
*req.status = cs;
PinCacheHitSubRange(&blob_handle, req.range_offset, req.range_length,
req.result);
continue;
}
}
if (no_io) {
*req.status =
Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
continue;
}
// No cache-fill allocator: a partial payload is never inserted.
CacheAllocationPtr buf =
AllocateBlock(req.range_length, /*allocator=*/nullptr);
read_reqs.push_back(SimpleGen2RangeReadRequest{
req.record_offset, static_cast<size_t>(req.payload_size),
req.range_offset, req.range_length, req.expected_compression, buf.get(),
req.status});
pending.push_back(Pending{i, std::move(buf)});
}
if (read_reqs.empty()) {
return;
}
ReadSimpleGen2BlobRanges(read_options, file, read_reqs.size(),
read_reqs.data());
for (size_t k = 0; k < pending.size(); ++k) {
SimpleGen2BlobRangeReadRequest& req = reqs[pending[k].idx];
if (!req.status->ok()) {
continue;
}
std::unique_ptr<BlobContents> blob_contents(
new BlobContents(std::move(pending[k].buf), req.range_length));
RecordTick(statistics_, BLOB_DB_BLOB_FILE_BYTES_READ, req.range_length);
PERF_COUNTER_ADD(blob_read_count, 1);
PERF_COUNTER_ADD(blob_read_byte, req.range_length);
RecordLazyRead(statistics_, req.range_length);
RecordLazyPartialRead(statistics_, req.range_length, req.payload_size);
PinOwnedBlob(&blob_contents, req.result);
}
}
void BlobSource::MultiGetBlob(const ReadOptions& read_options,
autovector<BlobFileReadRequests>& blob_reqs,
uint64_t* bytes_read) {
@@ -641,7 +829,7 @@ void BlobSource::MultiGetBlob(const ReadOptions& read_options,
uint64_t total_bytes_read = 0;
uint64_t bytes_read_in_file = 0;
for (auto& [file_number, file_size, blob_reqs_in_file] : blob_reqs) {
for (auto& [blob_file, file_size, blob_reqs_in_file] : blob_reqs) {
// sort blob_reqs_in_file by file offset.
std::sort(
blob_reqs_in_file.begin(), blob_reqs_in_file.end(),
@@ -649,7 +837,7 @@ void BlobSource::MultiGetBlob(const ReadOptions& read_options,
return lhs.offset < rhs.offset;
});
MultiGetBlobFromOneFile(read_options, file_number, file_size,
MultiGetBlobFromOneFile(read_options, blob_file, file_size,
blob_reqs_in_file, &bytes_read_in_file);
total_bytes_read += bytes_read_in_file;
@@ -661,7 +849,7 @@ void BlobSource::MultiGetBlob(const ReadOptions& read_options,
}
void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
uint64_t file_number,
const BlobFileOpenInfo& blob_file,
uint64_t /*file_size*/,
autovector<BlobReadRequest>& blob_reqs,
uint64_t* bytes_read) {
@@ -679,6 +867,7 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
Mask cache_hit_mask = 0;
uint64_t total_bytes = 0;
const uint64_t file_number = blob_file.file_number;
const OffsetableCacheKey base_cache_key(db_id_, db_session_id_, file_number);
if (blob_cache_) {
@@ -750,7 +939,7 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
}
CacheHandleGuard<BlobFileReader> blob_file_reader;
Status s = blob_file_cache_->GetBlobFileReader(read_options, file_number,
Status s = blob_file_cache_->GetBlobFileReader(read_options, blob_file,
&blob_file_reader);
if (!s.ok()) {
for (size_t i = 0; i < _blob_reqs.size(); ++i) {
@@ -790,7 +979,7 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
std::unique_ptr<BlobFileReader> fresh_reader;
s = blob_file_cache_->OpenBlobFileReaderUncached(
read_options, file_number, &fresh_reader,
read_options, blob_file, &fresh_reader,
/*allow_footer_skip_retry=*/false);
if (!s.ok()) {
for (const auto& blob_req : _blob_reqs) {
@@ -851,7 +1040,7 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
if (install_fresh_reader) {
CacheHandleGuard<BlobFileReader> ignored_reader;
blob_file_cache_
->RefreshBlobFileReader(file_number, &fresh_reader, &ignored_reader)
->RefreshBlobFileReader(blob_file, &fresh_reader, &ignored_reader)
.PermitUncheckedError();
}
}
@@ -884,6 +1073,24 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
}
}
// Whole-column reads on the lazy resolve path count as lazy reads (mirrors
// BlobSource::GetBlob, which records only on a disk read, not a cache hit).
// Gated on the activity so regular MultiGet is unaffected; the misses read
// above are exactly the disk reads for these blob columns.
if (read_options.io_activity == Env::IOActivity::kLazyResolve) {
for (auto& [req, blob_contents] : _blob_reqs) {
assert(req);
if (req->status->ok()) {
const uint64_t adjustment =
read_options.verify_checksums
? BlobLogRecord::CalculateAdjustmentForRecordHeader(
req->user_key->size())
: 0;
RecordLazyRead(statistics_, req->len + adjustment);
}
}
}
total_bytes += _bytes_read;
if (bytes_read) {
*bytes_read = total_bytes;
@@ -891,6 +1098,139 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
}
}
void BlobSource::MultiGetBlobRange(
const ReadOptions& read_options,
autovector<BlobFileRangeReadRequests>& blob_reqs, uint64_t* bytes_read) {
assert(blob_reqs.size() > 0);
uint64_t total_bytes_read = 0;
uint64_t bytes_read_in_file = 0;
for (auto& [blob_file, file_size, blob_reqs_in_file] : blob_reqs) {
// Sort by effective read offset so the file system layer can coalesce
// adjacent sub-range reads within one MultiRead.
std::sort(blob_reqs_in_file.begin(), blob_reqs_in_file.end(),
[](const BlobRangeReadRequest& lhs,
const BlobRangeReadRequest& rhs) -> bool {
return lhs.offset + lhs.range_offset <
rhs.offset + rhs.range_offset;
});
MultiGetBlobRangeFromOneFile(read_options, blob_file, file_size,
blob_reqs_in_file, &bytes_read_in_file);
total_bytes_read += bytes_read_in_file;
bytes_read_in_file = 0; // reset: some early-return paths do not write it
}
if (bytes_read) {
*bytes_read = total_bytes_read;
}
}
void BlobSource::MultiGetBlobRangeFromOneFile(
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
uint64_t /*file_size*/, autovector<BlobRangeReadRequest>& blob_reqs,
uint64_t* bytes_read) {
const size_t num_blobs = blob_reqs.size();
assert(num_blobs > 0);
assert(num_blobs <= MultiGetContext::MAX_BATCH_SIZE);
// Range reads are only issued while resolving a lazy result (partial reads
// are lazy-only), so the lazy read stats below are recorded unconditionally.
assert(read_options.io_activity == Env::IOActivity::kLazyResolve);
using Mask = uint64_t;
Mask cache_hit_mask = 0;
uint64_t total_bytes = 0;
const uint64_t file_number = blob_file.file_number;
const OffsetableCacheKey base_cache_key(db_id_, db_session_id_, file_number);
// Probe the whole-value cache per request; on a hit, slice the requested
// sub-range (zero-copy, no disk read). A partial read is never cache-filled.
if (blob_cache_) {
size_t cached_blob_count = 0;
for (size_t i = 0; i < num_blobs; ++i) {
BlobRangeReadRequest& req = blob_reqs[i];
assert(req.status);
CacheHandleGuard<BlobContents> blob_handle;
const CacheKey cache_key = base_cache_key.WithOffset(req.offset);
const Status s = GetBlobFromCache(cache_key.AsSlice(), &blob_handle);
if (s.ok()) {
*req.status = s;
PinCacheHitSubRange(&blob_handle, req.range_offset, req.range_length,
req.result);
++cached_blob_count;
cache_hit_mask |= (Mask{1} << i); // hit -> no disk read
}
}
if (cached_blob_count == num_blobs) {
if (bytes_read) {
*bytes_read = 0;
}
return;
}
}
const bool no_io = read_options.read_tier == kBlockCacheTier;
if (no_io) {
for (size_t i = 0; i < num_blobs; ++i) {
if (!(cache_hit_mask & (Mask{1} << i))) {
assert(blob_reqs[i].status);
*blob_reqs[i].status =
Status::Incomplete("Cannot read blob(s): no disk I/O allowed");
}
}
return;
}
autovector<std::pair<BlobRangeReadRequest*, std::unique_ptr<BlobContents>>>
_blob_reqs;
for (size_t i = 0; i < num_blobs; ++i) {
if (!(cache_hit_mask & (Mask{1} << i))) {
_blob_reqs.emplace_back(&blob_reqs[i], std::unique_ptr<BlobContents>());
}
}
// Lazy range reads run against a pinned SuperVersion (max_open_files == -1),
// and blob file numbers are never reused, so the stale cached-reader race
// that MultiGetBlobFromOneFile guards against cannot occur here; a Corruption
// is simply surfaced per request. Hence no reader-refresh retry and no
// cache-fill (a partial value cannot represent the whole-record cache entry).
CacheHandleGuard<BlobFileReader> blob_file_reader;
Status s = blob_file_cache_->GetBlobFileReader(read_options, blob_file,
&blob_file_reader);
if (!s.ok()) {
for (auto& blob_req : _blob_reqs) {
assert(blob_req.first);
assert(blob_req.first->status);
*blob_req.first->status = s;
}
return;
}
assert(blob_file_reader.GetValue());
uint64_t _bytes_read = 0;
blob_file_reader.GetValue()->MultiGetBlobRange(read_options, _blob_reqs,
&_bytes_read);
for (auto& [req, blob_contents] : _blob_reqs) {
assert(req);
if (req->status->ok()) {
PinOwnedBlob(&blob_contents, req->result);
RecordLazyRead(statistics_, req->range_length);
RecordLazyPartialRead(statistics_, req->range_length, req->value_size);
}
}
total_bytes += _bytes_read;
if (bytes_read) {
*bytes_read = total_bytes;
}
}
bool BlobSource::TEST_BlobInCache(uint64_t file_number, uint64_t file_size,
uint64_t offset, size_t* charge) const {
const CacheKey cache_key = GetCacheKey(file_number, file_size, offset);
+82 -9
View File
@@ -28,6 +28,30 @@ class Slice;
class RandomAccessFileReader;
enum ChecksumType : char;
// One whole-record request for BlobSource::MultiGetSimpleGen2Blob (the batched
// counterpart of GetSimpleGen2Blob). The per-file context (base cache key,
// file, checksum context) is passed once to the batch call; these are the
// per-record inputs and outputs.
struct SimpleGen2BlobReadRequest {
uint64_t record_offset = 0;
uint64_t payload_size = 0;
CompressionType expected_compression = kNoCompression;
PinnableSlice* result = nullptr;
Status* status = nullptr;
};
// One sub-range request for BlobSource::MultiGetSimpleGen2BlobRange (the
// batched counterpart of GetSimpleGen2BlobRange).
struct SimpleGen2BlobRangeReadRequest {
uint64_t record_offset = 0;
uint64_t payload_size = 0;
uint64_t range_offset = 0;
size_t range_length = 0;
CompressionType expected_compression = kNoCompression;
PinnableSlice* result = nullptr;
Status* status = nullptr;
};
// BlobSource is a class that provides universal access to blobs, regardless of
// whether they are in the blob cache, secondary cache, or (remote) storage.
// Depending on user settings, it always fetch blobs from multi-tier cache and
@@ -56,8 +80,9 @@ class BlobSource {
// sets "*bytes_read" to the size of on-disk (possibly compressed) blob
// record.
Status GetBlob(const ReadOptions& read_options, const Slice& user_key,
uint64_t file_number, uint64_t offset, uint64_t file_size,
uint64_t value_size, CompressionType compression_type,
const BlobFileOpenInfo& blob_file, uint64_t offset,
uint64_t file_size, uint64_t value_size,
CompressionType compression_type,
FilePrefetchBuffer* prefetch_buffer, PinnableSlice* value,
uint64_t* bytes_read);
@@ -81,10 +106,11 @@ class BlobSource {
// *bytes_read (when non-null) is the number of bytes read from the file; on a
// hit it is 0.
Status GetBlobRange(const ReadOptions& read_options, const Slice& user_key,
uint64_t file_number, uint64_t offset, uint64_t file_size,
uint64_t value_size, CompressionType compression_type,
uint64_t range_offset, size_t range_length,
PinnableSlice* value, uint64_t* bytes_read);
const BlobFileOpenInfo& blob_file, uint64_t offset,
uint64_t file_size, uint64_t value_size,
CompressionType compression_type, uint64_t range_offset,
size_t range_length, PinnableSlice* value,
uint64_t* bytes_read);
// Reads a SimpleGen2Blob payload (see db/blob/blob_gen2_format.h) through the
// blob value cache and BLOB_DB_* statistics. This is the counterpart to
@@ -146,6 +172,33 @@ class BlobSource {
uint64_t range_offset, size_t range_length,
PinnableSlice* value, uint64_t* bytes_read);
// Batched counterpart of GetSimpleGen2Blob: resolves `num_records` whole
// embedded (SimpleGen2Blob) records from one file, coalescing all cache-miss
// reads into a single MultiRead. The per-file context (base cache key, file,
// checksum type, base context checksum) is shared by all records; per-record
// inputs/outputs are in `reqs`. Each record independently probes the blob
// cache and, on a miss, is read + verified from the file; whole records are
// cache-filled (when configured and fill_cache) exactly as GetSimpleGen2Blob.
// Per-record outcome is written to *reqs[i].status.
void MultiGetSimpleGen2Blob(const ReadOptions& read_options,
const OffsetableCacheKey& base_cache_key,
RandomAccessFileReader* file,
ChecksumType checksum_type,
uint32_t base_context_checksum,
size_t num_records,
SimpleGen2BlobReadRequest* reqs);
// Batched counterpart of GetSimpleGen2BlobRange: resolves `num_records`
// uncompressed embedded sub-ranges from one file, coalescing cache-miss reads
// into a single MultiRead. Like GetSimpleGen2BlobRange, each record probes
// the whole-payload cache (slicing on a hit) and a partial read is never
// inserted into the cache. Per-record outcome is in *reqs[i].status.
void MultiGetSimpleGen2BlobRange(const ReadOptions& read_options,
const OffsetableCacheKey& base_cache_key,
RandomAccessFileReader* file,
size_t num_records,
SimpleGen2BlobRangeReadRequest* reqs);
// Read multiple blobs from the underlying cache or blob file(s).
//
// If successful, returns ok and sets "result" in the elements of "blob_reqs"
@@ -181,14 +234,34 @@ class BlobSource {
// "*bytes_read" to the total size of on-disk (possibly compressed) blob
// records.
void MultiGetBlobFromOneFile(const ReadOptions& read_options,
uint64_t file_number, uint64_t file_size,
const BlobFileOpenInfo& blob_file,
uint64_t file_size,
autovector<BlobReadRequest>& blob_reqs,
uint64_t* bytes_read);
// Byte-range (partial) multi-read counterpart of MultiGetBlob: for each
// request reads only its sub-range of an *uncompressed* blob value, from the
// blob cache (whole-value hit, sliced) or coalesced disk reads (one MultiRead
// per file). Like GetBlobRange, a partial read never fills the blob cache.
// Groups requests by file and dispatches each file via
// MultiGetBlobRangeFromOneFile. Per-request outcome is in
// *blob_reqs[..].status.
void MultiGetBlobRange(const ReadOptions& read_options,
autovector<BlobFileRangeReadRequests>& blob_reqs,
uint64_t* bytes_read);
// Byte-range (partial) multi-read counterpart of MultiGetBlobFromOneFile for
// a single blob file. See MultiGetBlobRange.
void MultiGetBlobRangeFromOneFile(const ReadOptions& read_options,
const BlobFileOpenInfo& blob_file,
uint64_t file_size,
autovector<BlobRangeReadRequest>& blob_reqs,
uint64_t* bytes_read);
inline Status GetBlobFileReader(
const ReadOptions& read_options, uint64_t blob_file_number,
const ReadOptions& read_options, const BlobFileOpenInfo& blob_file,
CacheHandleGuard<BlobFileReader>* blob_file_reader) {
return blob_file_cache_->GetBlobFileReader(read_options, blob_file_number,
return blob_file_cache_->GetBlobFileReader(read_options, blob_file,
blob_file_reader);
}
+93 -75
View File
@@ -104,6 +104,10 @@ void WriteBlobFile(const ImmutableOptions& immutable_options,
&checksum_method, &checksum_value));
}
BlobFileOpenInfo OpenInfoWithoutChecksum(uint64_t file_number) {
return BlobFileOpenInfo{file_number, Slice(), Slice()};
}
} // anonymous namespace
class BlobSourceTest : public DBTestBase {
@@ -212,10 +216,10 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
ASSERT_FALSE(blob_source.TEST_BlobInCache(blob_file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], blob_file_number,
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i], kNoCompression,
prefetch_buffer, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i]);
ASSERT_TRUE(values[i].IsPinned());
ASSERT_EQ(bytes_read,
@@ -251,10 +255,10 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
ASSERT_FALSE(blob_source.TEST_BlobInCache(blob_file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], blob_file_number,
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i], kNoCompression,
prefetch_buffer, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i]);
ASSERT_TRUE(values[i].IsPinned());
ASSERT_EQ(bytes_read,
@@ -299,10 +303,10 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
ASSERT_TRUE(blob_source.TEST_BlobInCache(blob_file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], blob_file_number,
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i], kNoCompression,
prefetch_buffer, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i]);
ASSERT_TRUE(values[i].IsPinned());
ASSERT_EQ(bytes_read,
@@ -339,10 +343,10 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
ASSERT_TRUE(blob_source.TEST_BlobInCache(blob_file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], blob_file_number,
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i], kNoCompression,
prefetch_buffer, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i]);
ASSERT_TRUE(values[i].IsPinned());
ASSERT_EQ(bytes_read,
@@ -385,7 +389,8 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
blob_offsets[i]));
ASSERT_TRUE(blob_source
.GetBlob(read_options, keys[i], blob_file_number,
.GetBlob(read_options, keys[i],
OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read)
@@ -427,7 +432,8 @@ TEST_F(BlobSourceTest, GetBlobsFromCache) {
blob_offsets[i]));
ASSERT_TRUE(blob_source
.GetBlob(read_options, keys[i], file_number,
.GetBlob(read_options, keys[i],
OpenInfoWithoutChecksum(file_number),
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer, &values[i],
&bytes_read)
@@ -522,8 +528,8 @@ TEST_F(BlobSourceTest, GetCompressedBlobs) {
compression, blob_offsets, blob_sizes);
CacheHandleGuard<BlobFileReader> blob_file_reader;
ASSERT_OK(blob_source.GetBlobFileReader(read_options, file_number,
&blob_file_reader));
ASSERT_OK(blob_source.GetBlobFileReader(
read_options, OpenInfoWithoutChecksum(file_number), &blob_file_reader));
ASSERT_NE(blob_file_reader.GetValue(), nullptr);
const uint64_t file_size = blob_file_reader.GetValue()->GetFileSize();
@@ -541,10 +547,10 @@ TEST_F(BlobSourceTest, GetCompressedBlobs) {
for (size_t i = 0; i < num_blobs; ++i) {
ASSERT_FALSE(blob_source.TEST_BlobInCache(file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], file_number,
blob_offsets[i], file_size, blob_sizes[i],
compression, nullptr /*prefetch_buffer*/,
&values[i], &bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(file_number),
blob_offsets[i], file_size, blob_sizes[i], compression,
nullptr /*prefetch_buffer*/, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i] /*uncompressed blob*/);
ASSERT_NE(values[i].size(), blob_sizes[i] /*compressed size*/);
ASSERT_EQ(bytes_read,
@@ -564,10 +570,10 @@ TEST_F(BlobSourceTest, GetCompressedBlobs) {
blob_offsets[i]));
// Compressed blob size is passed in GetBlob
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], file_number,
blob_offsets[i], file_size, blob_sizes[i],
compression, nullptr /*prefetch_buffer*/,
&values[i], &bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(file_number),
blob_offsets[i], file_size, blob_sizes[i], compression,
nullptr /*prefetch_buffer*/, &values[i], &bytes_read));
ASSERT_EQ(values[i], blobs[i] /*uncompressed blob*/);
ASSERT_NE(values[i].size(), blob_sizes[i] /*compressed size*/);
ASSERT_EQ(bytes_read,
@@ -674,7 +680,8 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromMultiFiles) {
keys[j], blob_offsets[j], blob_sizes[j], kNoCompression,
&value_buf[i * num_blobs + j], &statuses_buf[i * num_blobs + j]);
}
blob_reqs.emplace_back(file_number, file_size, blob_reqs_in_file[i]);
blob_reqs.emplace_back(OpenInfoWithoutChecksum(file_number), file_size,
blob_reqs_in_file[i]);
}
get_perf_context()->Reset();
@@ -731,7 +738,8 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromMultiFiles) {
}
// Add a fake multi-get blob request.
blob_reqs.emplace_back(fake_file_number, file_size, fake_blob_reqs_in_file);
blob_reqs.emplace_back(OpenInfoWithoutChecksum(fake_file_number), file_size,
fake_blob_reqs_in_file);
blob_source.MultiGetBlob(read_options, blob_reqs, &bytes_read);
@@ -865,8 +873,9 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromCache) {
statistics->Reset().PermitUncheckedError();
// Get half of blobs
blob_source.MultiGetBlobFromOneFile(read_options, blob_file_number,
file_size, blob_reqs, &bytes_read);
blob_source.MultiGetBlobFromOneFile(
read_options, OpenInfoWithoutChecksum(blob_file_number), file_size,
blob_reqs, &bytes_read);
uint64_t fs_read_bytes = 0;
uint64_t ca_read_bytes = 0;
@@ -911,10 +920,10 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromCache) {
ASSERT_FALSE(blob_source.TEST_BlobInCache(blob_file_number, file_size,
blob_offsets[i]));
ASSERT_OK(blob_source.GetBlob(read_options, keys[i], blob_file_number,
blob_offsets[i], file_size, blob_sizes[i],
kNoCompression, prefetch_buffer,
&value_buf[i], &bytes_read));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[i], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[i], file_size, blob_sizes[i], kNoCompression,
prefetch_buffer, &value_buf[i], &bytes_read));
ASSERT_EQ(value_buf[i], blobs[i]);
ASSERT_TRUE(value_buf[i].IsPinned());
ASSERT_EQ(bytes_read,
@@ -935,8 +944,9 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromCache) {
kNoCompression, &value_buf[i], &statuses_buf[i]);
}
blob_source.MultiGetBlobFromOneFile(read_options, blob_file_number,
file_size, blob_reqs, &bytes_read);
blob_source.MultiGetBlobFromOneFile(
read_options, OpenInfoWithoutChecksum(blob_file_number), file_size,
blob_reqs, &bytes_read);
uint64_t blob_bytes = 0;
for (size_t i = 0; i < num_blobs; ++i) {
@@ -985,8 +995,9 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromCache) {
get_perf_context()->Reset();
statistics->Reset().PermitUncheckedError();
blob_source.MultiGetBlobFromOneFile(read_options, blob_file_number,
file_size, blob_reqs, &bytes_read);
blob_source.MultiGetBlobFromOneFile(
read_options, OpenInfoWithoutChecksum(blob_file_number), file_size,
blob_reqs, &bytes_read);
for (size_t i = 0; i < num_blobs; ++i) {
ASSERT_TRUE(statuses_buf[i].IsIncomplete());
@@ -1029,8 +1040,9 @@ TEST_F(BlobSourceTest, MultiGetBlobsFromCache) {
get_perf_context()->Reset();
statistics->Reset().PermitUncheckedError();
blob_source.MultiGetBlobFromOneFile(read_options, non_existing_file_number,
file_size, blob_reqs, &bytes_read);
blob_source.MultiGetBlobFromOneFile(
read_options, OpenInfoWithoutChecksum(non_existing_file_number),
file_size, blob_reqs, &bytes_read);
for (size_t i = 0; i < num_blobs; ++i) {
ASSERT_TRUE(statuses_buf[i].IsIOError());
@@ -1104,8 +1116,9 @@ TEST_F(BlobSourceTest, GetBlobPreservesCorruptionDetailsWhenRefreshOpenFails) {
PinnableSlice value;
uint64_t bytes_read = 0;
ASSERT_OK(blob_source.GetBlob(
read_options, keys[0], blob_file_number, blob_offsets[0], file_size,
blob_sizes[0], kNoCompression, prefetch_buffer, &value, &bytes_read));
read_options, keys[0], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[0], file_size, blob_sizes[0], kNoCompression,
prefetch_buffer, &value, &bytes_read));
ASSERT_EQ(value, blobs[0]);
const std::string blob_file_path =
@@ -1114,10 +1127,10 @@ TEST_F(BlobSourceTest, GetBlobPreservesCorruptionDetailsWhenRefreshOpenFails) {
PinnableSlice invalid_value;
bytes_read = 0;
Status s =
blob_source.GetBlob(read_options, keys[0], blob_file_number, file_size,
file_size, blob_sizes[0], kNoCompression,
prefetch_buffer, &invalid_value, &bytes_read);
Status s = blob_source.GetBlob(
read_options, keys[0], OpenInfoWithoutChecksum(blob_file_number),
file_size, file_size, blob_sizes[0], kNoCompression, prefetch_buffer,
&invalid_value, &bytes_read);
ASSERT_TRUE(s.IsCorruption());
ASSERT_EQ(bytes_read, 0);
@@ -1178,8 +1191,9 @@ TEST_F(BlobSourceTest,
PinnableSlice value;
uint64_t bytes_read = 0;
ASSERT_OK(blob_source.GetBlob(
read_options, keys[0], blob_file_number, blob_offsets[0], file_size,
blob_sizes[0], kNoCompression, prefetch_buffer, &value, &bytes_read));
read_options, keys[0], OpenInfoWithoutChecksum(blob_file_number),
blob_offsets[0], file_size, blob_sizes[0], kNoCompression,
prefetch_buffer, &value, &bytes_read));
ASSERT_EQ(value, blobs[0]);
const std::string blob_file_path =
@@ -1193,8 +1207,9 @@ TEST_F(BlobSourceTest,
&values[0], &statuses[0]);
bytes_read = 0;
blob_source.MultiGetBlobFromOneFile(read_options, blob_file_number, file_size,
blob_reqs, &bytes_read);
blob_source.MultiGetBlobFromOneFile(read_options,
OpenInfoWithoutChecksum(blob_file_number),
file_size, blob_reqs, &bytes_read);
ASSERT_TRUE(statuses[0].IsCorruption());
ASSERT_EQ(bytes_read, 0);
@@ -1303,8 +1318,8 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
CacheHandleGuard<BlobFileReader> file_reader;
ReadOptions read_options;
ASSERT_OK(
blob_source.GetBlobFileReader(read_options, file_number, &file_reader));
ASSERT_OK(blob_source.GetBlobFileReader(
read_options, OpenInfoWithoutChecksum(file_number), &file_reader));
ASSERT_NE(file_reader.GetValue(), nullptr);
const uint64_t file_size = file_reader.GetValue()->GetFileSize();
ASSERT_EQ(file_reader.GetValue()->GetCompressionType(), kNoCompression);
@@ -1322,7 +1337,8 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
get_perf_context()->Reset();
// key0 should be filled to the primary cache from the blob file.
ASSERT_OK(blob_source.GetBlob(read_options, keys[0], file_number,
ASSERT_OK(blob_source.GetBlob(read_options, keys[0],
OpenInfoWithoutChecksum(file_number),
blob_offsets[0], file_size, blob_sizes[0],
kNoCompression, nullptr /* prefetch_buffer */,
values.data(), nullptr /* bytes_read */));
@@ -1331,17 +1347,18 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
// key0 should be evicted and key0's dummy item is inserted into secondary
// cache. key1 should be filled to the primary cache from the blob file.
ASSERT_OK(blob_source.GetBlob(read_options, keys[1], file_number,
blob_offsets[1], file_size, blob_sizes[1],
kNoCompression, nullptr /* prefetch_buffer */,
&values[1], nullptr /* bytes_read */));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[1], OpenInfoWithoutChecksum(file_number),
blob_offsets[1], file_size, blob_sizes[1], kNoCompression,
nullptr /* prefetch_buffer */, &values[1], nullptr /* bytes_read */));
// Release cache handle
values[1].Reset();
// key0 should be filled to the primary cache from the blob file. key1
// should be evicted and key1's dummy item is inserted into secondary cache.
ASSERT_OK(blob_source.GetBlob(read_options, keys[0], file_number,
ASSERT_OK(blob_source.GetBlob(read_options, keys[0],
OpenInfoWithoutChecksum(file_number),
blob_offsets[0], file_size, blob_sizes[0],
kNoCompression, nullptr /* prefetch_buffer */,
values.data(), nullptr /* bytes_read */));
@@ -1354,10 +1371,10 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
// key0 should be evicted and is inserted into secondary cache.
// key1 should be filled to the primary cache from the blob file.
ASSERT_OK(blob_source.GetBlob(read_options, keys[1], file_number,
blob_offsets[1], file_size, blob_sizes[1],
kNoCompression, nullptr /* prefetch_buffer */,
&values[1], nullptr /* bytes_read */));
ASSERT_OK(blob_source.GetBlob(
read_options, keys[1], OpenInfoWithoutChecksum(file_number),
blob_offsets[1], file_size, blob_sizes[1], kNoCompression,
nullptr /* prefetch_buffer */, &values[1], nullptr /* bytes_read */));
ASSERT_EQ(values[1], blobs[1]);
ASSERT_TRUE(
blob_source.TEST_BlobInCache(file_number, file_size, blob_offsets[1]));
@@ -1422,9 +1439,10 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
// fetch key0 from the blob file to the primary cache.
// key1 is evicted and inserted into the secondary cache.
ASSERT_OK(blob_source.GetBlob(
read_options, keys[0], file_number, blob_offsets[0], file_size,
blob_sizes[0], kNoCompression, nullptr /* prefetch_buffer */,
values.data(), nullptr /* bytes_read */));
read_options, keys[0], OpenInfoWithoutChecksum(file_number),
blob_offsets[0], file_size, blob_sizes[0], kNoCompression,
nullptr /* prefetch_buffer */, values.data(),
nullptr /* bytes_read */));
ASSERT_EQ(values[0], blobs[0]);
// Release cache handle
@@ -1606,8 +1624,8 @@ TEST_F(BlobSourceCacheReservationTest, SimpleCacheReservation) {
for (size_t i = 0; i < kNumBlobs; ++i) {
ASSERT_OK(blob_source.GetBlob(
read_options, keys_[i], kBlobFileNumber, blob_offsets[i],
blob_file_size_, blob_sizes[i], kNoCompression,
read_options, keys_[i], OpenInfoWithoutChecksum(kBlobFileNumber),
blob_offsets[i], blob_file_size_, blob_sizes[i], kNoCompression,
nullptr /* prefetch_buffer */, &values[i], nullptr /* bytes_read */));
ASSERT_EQ(cache_res_mgr->GetTotalReservedCacheSize(), 0);
ASSERT_EQ(cache_res_mgr->GetTotalMemoryUsed(), 0);
@@ -1625,8 +1643,8 @@ TEST_F(BlobSourceCacheReservationTest, SimpleCacheReservation) {
uint64_t blob_bytes = 0;
for (size_t i = 0; i < kNumBlobs; ++i) {
ASSERT_OK(blob_source.GetBlob(
read_options, keys_[i], kBlobFileNumber, blob_offsets[i],
blob_file_size_, blob_sizes[i], kNoCompression,
read_options, keys_[i], OpenInfoWithoutChecksum(kBlobFileNumber),
blob_offsets[i], blob_file_size_, blob_sizes[i], kNoCompression,
nullptr /* prefetch_buffer */, &values[i], nullptr /* bytes_read */));
size_t charge = 0;
@@ -1728,8 +1746,8 @@ TEST_F(BlobSourceCacheReservationTest, IncreaseCacheReservation) {
for (size_t i = 0; i < kNumBlobs; ++i) {
ASSERT_OK(blob_source.GetBlob(
read_options, keys_[i], kBlobFileNumber, blob_offsets[i],
blob_file_size_, blob_sizes[i], kNoCompression,
read_options, keys_[i], OpenInfoWithoutChecksum(kBlobFileNumber),
blob_offsets[i], blob_file_size_, blob_sizes[i], kNoCompression,
nullptr /* prefetch_buffer */, &values[i], nullptr /* bytes_read */));
ASSERT_EQ(cache_res_mgr->GetTotalReservedCacheSize(), 0);
ASSERT_EQ(cache_res_mgr->GetTotalMemoryUsed(), 0);
@@ -1744,8 +1762,8 @@ TEST_F(BlobSourceCacheReservationTest, IncreaseCacheReservation) {
uint64_t blob_bytes = 0;
for (size_t i = 0; i < kNumBlobs; ++i) {
ASSERT_OK(blob_source.GetBlob(
read_options, keys_[i], kBlobFileNumber, blob_offsets[i],
blob_file_size_, blob_sizes[i], kNoCompression,
read_options, keys_[i], OpenInfoWithoutChecksum(kBlobFileNumber),
blob_offsets[i], blob_file_size_, blob_sizes[i], kNoCompression,
nullptr /* prefetch_buffer */, &values[i], nullptr /* bytes_read */));
// Release cache handle
+92
View File
@@ -7,6 +7,7 @@
#include <array>
#include <atomic>
#include <cstdint>
#include <mutex>
#include <set>
#include <sstream>
#include <string>
@@ -19,15 +20,19 @@
#include "db/column_family.h"
#include "db/db_test_util.h"
#include "db/db_with_timestamp_test_util.h"
#include "env/composite_env_wrapper.h"
#include "file/filename.h"
#include "file/random_access_file_reader.h"
#include "port/stack_trace.h"
#include "rocksdb/convenience.h"
#include "rocksdb/file_system.h"
#include "rocksdb/trace_reader_writer.h"
#include "rocksdb/trace_record.h"
#include "rocksdb/utilities/replayer.h"
#include "test_util/sync_point.h"
#include "util/compression.h"
#include "util/defer.h"
#include "util/file_checksum_helper.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -38,6 +43,49 @@ class DBBlobBasicTest : public DBTestBase {
: DBTestBase("db_blob_basic_test", /* env_do_fsync */ false) {}
};
class BlobFileChecksumCapturingFS : public FileSystemWrapper {
public:
explicit BlobFileChecksumCapturingFS(const std::shared_ptr<FileSystem>& base)
: FileSystemWrapper(base) {}
static const char* kClassName() { return "BlobFileChecksumCapturingFS"; }
const char* Name() const override { return kClassName(); }
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
if (fname.find(".blob") != std::string::npos) {
std::lock_guard<std::mutex> lock(mu_);
file_checksum_ = opts.file_checksum;
file_checksum_func_name_ = opts.file_checksum_func_name;
++capture_count_;
}
return target()->NewRandomAccessFile(fname, opts, result, dbg);
}
std::string GetFileChecksum() {
std::lock_guard<std::mutex> lock(mu_);
return file_checksum_;
}
std::string GetFileChecksumFuncName() {
std::lock_guard<std::mutex> lock(mu_);
return file_checksum_func_name_;
}
int GetCaptureCount() {
std::lock_guard<std::mutex> lock(mu_);
return capture_count_;
}
private:
std::mutex mu_;
std::string file_checksum_;
std::string file_checksum_func_name_;
int capture_count_ = 0;
};
TEST_F(DBBlobBasicTest, GetBlob) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -65,6 +113,41 @@ TEST_F(DBBlobBasicTest, GetBlob) {
.IsIncomplete());
}
TEST_F(DBBlobBasicTest, BlobFileChecksumInFileOptions) {
auto capturing_fs =
std::make_shared<BlobFileChecksumCapturingFS>(env_->GetFileSystem());
std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, capturing_fs));
Options options = GetDefaultOptions();
options.create_if_missing = true;
options.env = env.get();
options.enable_blob_files = true;
options.min_blob_size = 0;
options.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
Defer close_db([this]() { Close(); });
Reopen(options);
constexpr char key[] = "key";
constexpr char blob_value[] = "blob_value";
ASSERT_OK(Put(key, blob_value));
ASSERT_OK(Flush());
std::vector<ColumnFamilyMetaData> column_family_metadata;
db_->GetAllColumnFamilyMetaData(&column_family_metadata);
ASSERT_EQ(column_family_metadata.size(), 1);
ASSERT_EQ(column_family_metadata[0].blob_files.size(), 1);
const BlobMetaData& blob_metadata = column_family_metadata[0].blob_files[0];
ASSERT_FALSE(blob_metadata.checksum_value.empty());
ASSERT_EQ(blob_metadata.checksum_method, "FileChecksumCrc32c");
ASSERT_EQ(Get(key), blob_value);
ASSERT_GT(capturing_fs->GetCaptureCount(), 0);
ASSERT_EQ(capturing_fs->GetFileChecksum(), blob_metadata.checksum_value);
ASSERT_EQ(capturing_fs->GetFileChecksumFuncName(),
blob_metadata.checksum_method);
}
TEST_F(DBBlobBasicTest, BlobFileWritableFileMaxBufferSize) {
constexpr uint64_t kBlobWriterBufferSize = 128 * 1024;
@@ -754,6 +837,15 @@ TEST_F(DBBlobBasicTest, MultiGetBlobsFromCache) {
ASSERT_EQ(values[2], third_value);
}
// Load the SST blocks into the block cache. The inlined key does not read
// the blob file, so the blob cache stays empty.
{
ReadOptions fill_block_cache_options;
std::string value;
ASSERT_OK(db_->Get(fill_block_cache_options, first_key, &value));
ASSERT_EQ(value, first_value);
}
// Try again with no I/O allowed. The first (inlined) value should be
// successfully read; however, the two blob values could only be read from the
// blob file, so for those the read should return Incomplete.
+103
View File
@@ -40,6 +40,8 @@
#include "rocksdb/utilities/replayer.h"
#include "test_util/sync_point.h"
#include "util/compression.h"
#include "util/defer.h"
#include "util/file_checksum_helper.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -121,6 +123,59 @@ bool IsBlobFilePath(const std::string& fname) {
file_type == kBlobFile;
}
class BlobFileChecksumCapturingFileSystem : public FileSystemWrapper {
public:
explicit BlobFileChecksumCapturingFileSystem(
const std::shared_ptr<FileSystem>& fs)
: FileSystemWrapper(fs) {}
static const char* kClassName() {
return "BlobFileChecksumCapturingFileSystem";
}
const char* Name() const override { return kClassName(); }
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
if (IsBlobFilePath(fname)) {
std::lock_guard<std::mutex> lock(mu_);
file_checksum_ = opts.file_checksum;
file_checksum_func_name_ = opts.file_checksum_func_name;
++capture_count_;
}
return target()->NewRandomAccessFile(fname, opts, result, dbg);
}
void Reset() {
std::lock_guard<std::mutex> lock(mu_);
file_checksum_.clear();
file_checksum_func_name_.clear();
capture_count_ = 0;
}
std::string GetFileChecksum() const {
std::lock_guard<std::mutex> lock(mu_);
return file_checksum_;
}
std::string GetFileChecksumFuncName() const {
std::lock_guard<std::mutex> lock(mu_);
return file_checksum_func_name_;
}
int GetCaptureCount() const {
std::lock_guard<std::mutex> lock(mu_);
return capture_count_;
}
private:
mutable std::mutex mu_;
std::string file_checksum_;
std::string file_checksum_func_name_;
int capture_count_ = 0;
};
// Keeps track of when an active blob file stops being writer-visible so the
// test filesystem can model current-size visibility only while the file remains
// active.
@@ -1755,6 +1810,54 @@ TEST_F(
ASSERT_TRUE(iter->Valid());
ASSERT_EQ(iter->value().ToString(), values[target_idx]);
}
TEST_F(DBBlobDirectWriteTest, DirectWriteDelayedReadUsesFinalizedFileChecksum) {
auto capturing_fs = std::make_shared<BlobFileChecksumCapturingFileSystem>(
env_->GetFileSystem());
std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, capturing_fs));
Options options = GetDirectWriteOptions();
options.env = env.get();
options.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
Defer close_db([this]() { Close(); });
Reopen(options);
const std::string key = "delayed-read-key";
const std::string value(96, 'v');
ASSERT_OK(Put(key, value));
ReadOptions read_options;
read_options.allow_unprepared_value = true;
std::unique_ptr<Iterator> iter(db_->NewIterator(read_options));
iter->Seek(key);
ASSERT_TRUE(iter->Valid());
ASSERT_EQ(iter->key(), key);
ASSERT_TRUE(iter->value().empty());
ASSERT_OK(Flush());
std::vector<ColumnFamilyMetaData> column_family_metadata;
db_->GetAllColumnFamilyMetaData(&column_family_metadata);
ASSERT_EQ(column_family_metadata.size(), 1);
ASSERT_EQ(column_family_metadata[0].blob_files.size(), 1);
const BlobMetaData& blob_metadata = column_family_metadata[0].blob_files[0];
ASSERT_FALSE(blob_metadata.checksum_value.empty());
ASSERT_EQ(blob_metadata.checksum_method, "FileChecksumCrc32c");
capturing_fs->Reset();
ASSERT_TRUE(iter->PrepareValue());
ASSERT_TRUE(iter->Valid());
ASSERT_EQ(iter->value().ToString(), value);
ASSERT_GT(capturing_fs->GetCaptureCount(), 0);
ASSERT_EQ(capturing_fs->GetFileChecksum(), blob_metadata.checksum_value);
ASSERT_EQ(capturing_fs->GetFileChecksumFuncName(),
blob_metadata.checksum_method);
const int capture_count = capturing_fs->GetCaptureCount();
ASSERT_EQ(Get(key), value);
ASSERT_EQ(capturing_fs->GetCaptureCount(), capture_count);
}
TEST_F(DBBlobDirectWriteTest, OrderedTraceUsesLogicalBatchForBlobDirectWrite) {
AssertOrderedTraceStoresLogicalPut(GetDirectWriteOptions(),
/*expect_blob_files=*/true);
+47
View File
@@ -1032,6 +1032,53 @@ TEST_F(DBBlobIndexTest, EmbeddedBlobWideColumnGetEntityBlockCacheTier) {
ASSERT_TRUE(s.IsIncomplete()) << s.ToString();
}
// Companion to the test above (genuine miss -> Incomplete): with a blob cache
// configured, a warm read populates the embedded record into the blob cache, so
// a subsequent kBlockCacheTier GetEntity serves it from cache instead of
// returning Incomplete. This exercises the non-lazy Get/GetEntity embedded path
// (get_context -> EmbeddedAwareBlobFetcher -> GetSameFileBlob whole ->
// ResolveEmbeddedBlobCached), which shares the block-cache-tier behavior with
// the lazy resolver (a same-file column already in the blob cache is served).
TEST_F(DBBlobIndexTest,
EmbeddedBlobWideColumnGetEntityBlockCacheTierServesCached) {
Options options = GetTestOptions();
options.create_if_missing = true;
LRUCacheOptions co;
co.capacity = 8 << 20;
options.blob_cache = NewLRUCache(co);
DestroyAndReopen(options);
const std::string sst_path = dbname_ + "/embedded_wc_blockcache_hit.sst";
SstFileWriterEmbeddedBlobOptions embedded_blob_options;
embedded_blob_options.min_blob_size = 8;
const std::string big1(1024, 'a');
const WideColumns columns{{kDefaultWideColumnName, "d"}, {"big", big1}};
SstFileWriter writer(EnvOptions(), options);
ASSERT_OK(writer.OpenWithEmbeddedBlobs(sst_path, embedded_blob_options));
ASSERT_OK(writer.PutEntity("k", columns));
ASSERT_OK(writer.Finish());
ASSERT_OK(db_->IngestExternalFile({sst_path}, IngestExternalFileOptions()));
// Warm both the block cache (data block) and the blob cache (embedded record)
// with a full read.
{
PinnableWideColumns result;
ASSERT_OK(db_->GetEntity(ReadOptions(), db_->DefaultColumnFamily(), "k",
&result));
ASSERT_EQ(result.columns(), columns);
}
// The embedded record is now cached, so a block-cache-only read serves it.
ReadOptions read_options;
read_options.read_tier = kBlockCacheTier;
PinnableWideColumns result;
ASSERT_OK(
db_->GetEntity(read_options, db_->DefaultColumnFamily(), "k", &result));
ASSERT_EQ(result.columns(), columns);
}
// A corrupt embedded blob record backing a wide-column column must surface an
// error on GetEntity, not expose the raw same-file BlobIndex.
// A failure resolving an embedded blob record backing a wide-column column
+51 -18
View File
@@ -8,6 +8,7 @@
#include <cstddef>
#include <cstdint>
#include "db/blob/blob_constants.h"
#include "rocksdb/rocksdb_namespace.h"
#include "rocksdb/status.h"
@@ -17,6 +18,18 @@ class BlobIndex;
class PinnableSlice;
struct ReadOptions;
// One request for SameFileBlobReader::MultiGetSameFileBlob (the batched
// counterpart of GetSameFileBlob). See GetSameFileBlob for the per-field
// semantics (range_length == kWholeBlobLength selects the whole value).
struct SameFileBlobReadRequest {
const BlobIndex* blob_index = nullptr;
uint64_t range_offset = 0;
size_t range_length = kWholeBlobLength;
BlobVerifyPolicy verify_policy = BlobVerifyPolicy::kVerifyIfNoAmplification;
PinnableSlice* result = nullptr;
Status* status = nullptr;
};
// Narrow interface for resolving a same-file ("embedded") blob reference
// against the physical file that currently holds the table entry. Implemented
// by BlockBasedTable, which owns the reader context (file, footer, blob cache)
@@ -30,28 +43,48 @@ class SameFileBlobReader {
public:
virtual ~SameFileBlobReader() = default;
// Resolves a same-file blob reference (one for which blob_index.IsSameFile()
// is true) into `value`, pinning the payload zero-copy (from the blob cache
// or an owned buffer). read_tier == kBlockCacheTier without a cached record
// yields Status::Incomplete; a bad record yields Status::Corruption.
// Resolves a same-file ("embedded") blob reference (one for which
// blob_index.IsSameFile() is true) into `value`, pinning the payload
// zero-copy (from the blob cache or an owned buffer).
//
// range_length == kWholeBlobLength resolves the whole value. Any other length
// resolves only the sub-range [range_offset, range_offset + range_length),
// reading just those bytes on a cache miss -- skipping whole-record checksum
// verification (a strict sub-range cannot cover it) and never populating the
// blob cache; valid only for an uncompressed record and a verify_policy other
// than kVerifyIfPresent, with range_offset + range_length <=
// blob_index.size().
//
// verify_policy controls whole-record checksum verification (see
// BlobVerifyPolicy); a whole read under kVerifyIfPresent verifies even when
// ReadOptions::verify_checksums is off. read_tier == kBlockCacheTier without
// a cached record yields Status::Incomplete; a bad record yields
// Status::Corruption.
virtual Status GetSameFileBlob(const ReadOptions& read_options,
const BlobIndex& blob_index,
uint64_t range_offset, size_t range_length,
BlobVerifyPolicy verify_policy,
PinnableSlice* value) const = 0;
// Resolves a byte sub-range [range_offset, range_offset + range_length) of an
// *uncompressed* same-file blob reference into `value`, reading only those
// bytes on a cache miss (the embedded counterpart of a separate-file blob
// range read). Pins the payload zero-copy (a sliced cached record, or an
// owned sub-range buffer). Unlike GetSameFileBlob it skips whole-record
// checksum verification (a strict sub-range cannot cover it) and never
// populates the blob cache. The caller must ensure range_offset +
// range_length <= blob_index.size(). read_tier == kBlockCacheTier without a
// cached record yields Status::Incomplete.
virtual Status GetSameFileBlobRange(const ReadOptions& read_options,
const BlobIndex& blob_index,
uint64_t range_offset,
size_t range_length,
PinnableSlice* value) const = 0;
// Batched counterpart of GetSameFileBlob: resolves `num_reads` same-file blob
// references (whole or sub-range, mixed) against this reader, writing each
// outcome to *reqs[i].status. Implementations may coalesce the underlying
// disk reads into a single MultiRead. The default implementation simply loops
// over GetSameFileBlob (correct, but not coalesced); BlockBasedTable
// overrides it to coalesce reads of embedded records in the same SST.
virtual void MultiGetSameFileBlob(const ReadOptions& read_options,
size_t num_reads,
SameFileBlobReadRequest* reqs) const {
for (size_t i = 0; i < num_reads; ++i) {
SameFileBlobReadRequest& req = reqs[i];
const Status s =
GetSameFileBlob(read_options, *req.blob_index, req.range_offset,
req.range_length, req.verify_policy, req.result);
if (req.status) {
*req.status = s;
}
}
}
};
} // namespace ROCKSDB_NAMESPACE
+5 -3
View File
@@ -207,7 +207,8 @@ Status BuildTable(
std::move(file), fname, file_options, ioptions.clock, io_tracer,
ioptions.stats, Histograms::SST_WRITE_MICROS, ioptions.listeners,
ioptions.file_checksum_gen_factory.get(),
tmp_set.Contains(FileType::kTableFile), false));
tmp_set.Contains(FileType::kTableFile),
tmp_set.Contains(FileType::kTableFile)));
builder = NewTableBuilder(tboptions, file_writer.get());
}
@@ -395,7 +396,7 @@ Status BuildTable(
relevant_mapping,
ioptions.compaction_style == CompactionStyle::kCompactionStyleFIFO
? meta->file_creation_time
: meta->oldest_ancester_time);
: meta->oldest_ancestor_time);
s = builder->Finish();
}
if (io_status->ok()) {
@@ -451,7 +452,8 @@ Status BuildTable(
WritableFileWriter::PrepareIOOptions(tboptions.write_options, opts);
if (s.ok() && io_status->ok() && !empty) {
StopWatch sw(ioptions.clock, ioptions.stats, TABLE_SYNC_MICROS);
*io_status = file_writer->Sync(opts, ioptions.use_fsync);
*io_status = ioptions.use_fsync ? file_writer->Fsync(opts)
: file_writer->Sync(opts);
}
TEST_SYNC_POINT("BuildTable:BeforeCloseTableFile");
if (s.ok() && io_status->ok() && !empty) {
+265 -15
View File
@@ -195,6 +195,7 @@ using ROCKSDB_NAMESPACE::WALRecoveryMode;
using ROCKSDB_NAMESPACE::WritableFile;
using ROCKSDB_NAMESPACE::WriteBatch;
using ROCKSDB_NAMESPACE::WriteBatchWithIndex;
using ROCKSDB_NAMESPACE::WriteBufferFlushPolicy;
using ROCKSDB_NAMESPACE::WriteBufferManager;
using ROCKSDB_NAMESPACE::WriteOptions;
using ROCKSDB_NAMESPACE::WriteStallCondition;
@@ -2623,6 +2624,59 @@ char* rocksdb_get(rocksdb_t* db, const rocksdb_readoptions_t* options,
return result;
}
static char* rocksdb_get_with_metadata_impl(
rocksdb_t* db, const rocksdb_readoptions_t* options,
ColumnFamilyHandle* column_family, const char* key, size_t keylen,
size_t* vallen, char** timestamp, size_t* timestamp_len,
unsigned char* newer_version_present, char** errptr) {
ROCKSDB_NAMESPACE::OutputMetadata output_metadata;
if (timestamp != nullptr && timestamp_len != nullptr) {
output_metadata.WantTimestamp();
}
if (newer_version_present != nullptr) {
output_metadata.WantNewerVersionPresent();
}
PinnableSlice pinnable_val;
const Status s =
db->rep->GetWithMetadata(options->rep, column_family, Slice(key, keylen),
&pinnable_val, &output_metadata);
if (newer_version_present != nullptr) {
*newer_version_present = *output_metadata.newer_version_present ? 1 : 0;
}
char* result = nullptr;
if (s.ok()) {
*vallen = pinnable_val.size();
result = CopyString(pinnable_val);
if (output_metadata.timestamp.has_value()) {
*timestamp_len = output_metadata.timestamp->size();
*timestamp = CopyString(*output_metadata.timestamp);
}
} else {
*vallen = 0;
if (output_metadata.timestamp.has_value()) {
*timestamp = nullptr;
*timestamp_len = 0;
}
if (!s.IsNotFound()) {
SaveError(errptr, s);
}
}
return result;
}
char* rocksdb_get_with_metadata(rocksdb_t* db,
const rocksdb_readoptions_t* options,
const char* key, size_t keylen, size_t* vallen,
char** timestamp, size_t* timestamp_len,
unsigned char* newer_version_present,
char** errptr) {
return rocksdb_get_with_metadata_impl(
db, options, db->rep->DefaultColumnFamily(), key, keylen, vallen,
timestamp, timestamp_len, newer_version_present, errptr);
}
char* rocksdb_get_cf(rocksdb_t* db, const rocksdb_readoptions_t* options,
rocksdb_column_family_handle_t* column_family,
const char* key, size_t keylen, size_t* vallen,
@@ -2645,6 +2699,16 @@ char* rocksdb_get_cf(rocksdb_t* db, const rocksdb_readoptions_t* options,
return result;
}
char* rocksdb_get_cf_with_metadata(
rocksdb_t* db, const rocksdb_readoptions_t* options,
rocksdb_column_family_handle_t* column_family, const char* key,
size_t keylen, size_t* vallen, char** timestamp, size_t* timestamp_len,
unsigned char* newer_version_present, char** errptr) {
return rocksdb_get_with_metadata_impl(
db, options, column_family->rep, key, keylen, vallen, timestamp,
timestamp_len, newer_version_present, errptr);
}
char* rocksdb_get_with_ts(rocksdb_t* db, const rocksdb_readoptions_t* options,
const char* key, size_t keylen, size_t* vallen,
char** ts, size_t* tslen, char** errptr) {
@@ -2736,6 +2800,72 @@ void rocksdb_multi_get(rocksdb_t* db, const rocksdb_readoptions_t* options,
}
}
static void rocksdb_multi_get_with_metadata_impl(
rocksdb_t* db, const rocksdb_readoptions_t* options,
ColumnFamilyHandle* const* column_families, size_t num_keys,
const char* const* keys_list, const size_t* keys_list_sizes,
char** values_list, size_t* values_list_sizes, char** timestamp_list,
size_t* timestamp_list_sizes, unsigned char* newer_version_present,
char** errs) {
std::unique_ptr<Slice[]> keys(new Slice[num_keys]);
std::vector<PinnableSlice> values(num_keys);
std::vector<Status> statuses(num_keys);
for (size_t i = 0; i < num_keys; ++i) {
keys[i] = Slice(keys_list[i], keys_list_sizes[i]);
}
ROCKSDB_NAMESPACE::MultiGetOutputMetadata output_metadata;
if (timestamp_list != nullptr && timestamp_list_sizes != nullptr) {
output_metadata.WantTimestamps();
}
if (newer_version_present != nullptr) {
output_metadata.WantNewerVersionPresent();
}
db->rep->MultiGetWithMetadata(options->rep, num_keys, column_families,
keys.get(), values.data(), statuses.data(),
&output_metadata);
for (size_t i = 0; i < num_keys; ++i) {
if (newer_version_present != nullptr) {
newer_version_present[i] =
(*output_metadata.newer_version_present)[i] ? 1 : 0;
}
if (statuses[i].ok()) {
values_list[i] = CopyString(values[i]);
values_list_sizes[i] = values[i].size();
if (output_metadata.timestamps.has_value()) {
timestamp_list[i] = CopyString((*output_metadata.timestamps)[i]);
timestamp_list_sizes[i] = (*output_metadata.timestamps)[i].size();
}
errs[i] = nullptr;
} else {
values_list[i] = nullptr;
values_list_sizes[i] = 0;
if (output_metadata.timestamps.has_value()) {
timestamp_list[i] = nullptr;
timestamp_list_sizes[i] = 0;
}
errs[i] = statuses[i].IsNotFound()
? nullptr
: strdup(statuses[i].ToString().c_str());
}
}
}
void rocksdb_multi_get_with_metadata(
rocksdb_t* db, const rocksdb_readoptions_t* options, size_t num_keys,
const char* const* keys_list, const size_t* keys_list_sizes,
char** values_list, size_t* values_list_sizes, char** timestamp_list,
size_t* timestamp_list_sizes, unsigned char* newer_version_present,
char** errs) {
std::vector<ColumnFamilyHandle*> column_families(
num_keys, db->rep->DefaultColumnFamily());
rocksdb_multi_get_with_metadata_impl(
db, options, column_families.data(), num_keys, keys_list, keys_list_sizes,
values_list, values_list_sizes, timestamp_list, timestamp_list_sizes,
newer_version_present, errs);
}
void rocksdb_multi_get_with_ts(rocksdb_t* db,
const rocksdb_readoptions_t* options,
size_t num_keys, const char* const* keys_list,
@@ -2811,6 +2941,24 @@ void rocksdb_multi_get_cf(
}
}
void rocksdb_multi_get_cf_with_metadata(
rocksdb_t* db, const rocksdb_readoptions_t* options,
const rocksdb_column_family_handle_t* const* column_families,
size_t num_keys, const char* const* keys_list,
const size_t* keys_list_sizes, char** values_list,
size_t* values_list_sizes, char** timestamp_list,
size_t* timestamp_list_sizes, unsigned char* newer_version_present,
char** errs) {
std::vector<ColumnFamilyHandle*> cfs(num_keys);
for (size_t i = 0; i < num_keys; ++i) {
cfs[i] = column_families[i]->rep;
}
rocksdb_multi_get_with_metadata_impl(
db, options, cfs.data(), num_keys, keys_list, keys_list_sizes,
values_list, values_list_sizes, timestamp_list, timestamp_list_sizes,
newer_version_present, errs);
}
void rocksdb_multi_get_cf_with_ts(
rocksdb_t* db, const rocksdb_readoptions_t* options,
const rocksdb_column_family_handle_t* const* column_families,
@@ -5050,6 +5198,11 @@ uint64_t rocksdb_table_properties_column_family_id(
return props->rep.column_family_id;
}
uint64_t rocksdb_table_properties_oldest_ancestor_time(
const rocksdb_table_properties_t* props) {
return props->rep.oldest_ancestor_time;
}
uint64_t rocksdb_table_properties_creation_time(
const rocksdb_table_properties_t* props) {
return props->rep.creation_time;
@@ -5080,11 +5233,6 @@ uint64_t rocksdb_table_properties_fast_compression_estimated_data_size(
return props->rep.fast_compression_estimated_data_size;
}
uint64_t rocksdb_table_properties_external_sst_file_global_seqno_offset(
const rocksdb_table_properties_t* props) {
return props->rep.external_sst_file_global_seqno_offset;
}
uint64_t rocksdb_table_properties_tail_start_offset(
const rocksdb_table_properties_t* props) {
return props->rep.tail_start_offset;
@@ -5120,6 +5268,11 @@ uint64_t rocksdb_table_properties_separate_key_value_in_data_block(
return props->rep.separate_key_value_in_data_block;
}
uint64_t rocksdb_table_properties_lsm_info_at_creation(
const rocksdb_table_properties_t* props) {
return props->rep.lsm_info_at_creation;
}
const char* rocksdb_table_properties_db_id(
const rocksdb_table_properties_t* props, size_t* size) {
*size = props->rep.db_id.size();
@@ -7609,6 +7762,35 @@ rocksdb_write_buffer_manager_t* rocksdb_write_buffer_manager_create(
return wbm;
}
static bool rocksdb_write_buffer_manager_parse_flush_policy(
int value, WriteBufferFlushPolicy* policy) {
switch (value) {
case rocksdb_write_buffer_manager_flush_policy_oldest:
*policy = WriteBufferFlushPolicy::kFlushOldest;
return true;
case rocksdb_write_buffer_manager_flush_policy_largest:
*policy = WriteBufferFlushPolicy::kFlushLargest;
return true;
case rocksdb_write_buffer_manager_flush_policy_largest_across_dbs:
*policy = WriteBufferFlushPolicy::kFlushLargestAcrossDBs;
return true;
default:
return false;
}
}
rocksdb_write_buffer_manager_t* rocksdb_write_buffer_manager_create_with_policy(
size_t buffer_size, bool allow_stall, int flush_policy) {
WriteBufferFlushPolicy policy;
if (!rocksdb_write_buffer_manager_parse_flush_policy(flush_policy, &policy)) {
return nullptr;
}
rocksdb_write_buffer_manager_t* wbm = new rocksdb_write_buffer_manager_t;
wbm->rep = std::make_shared<WriteBufferManager>(
buffer_size, std::shared_ptr<Cache>{}, allow_stall, policy);
return wbm;
}
rocksdb_write_buffer_manager_t* rocksdb_write_buffer_manager_create_with_cache(
size_t buffer_size, const rocksdb_cache_t* cache, bool allow_stall) {
rocksdb_write_buffer_manager_t* wbm = new rocksdb_write_buffer_manager_t;
@@ -7617,6 +7799,20 @@ rocksdb_write_buffer_manager_t* rocksdb_write_buffer_manager_create_with_cache(
return wbm;
}
rocksdb_write_buffer_manager_t*
rocksdb_write_buffer_manager_create_with_cache_and_policy(
size_t buffer_size, const rocksdb_cache_t* cache, bool allow_stall,
int flush_policy) {
WriteBufferFlushPolicy policy;
if (!rocksdb_write_buffer_manager_parse_flush_policy(flush_policy, &policy)) {
return nullptr;
}
rocksdb_write_buffer_manager_t* wbm = new rocksdb_write_buffer_manager_t;
wbm->rep = std::make_shared<WriteBufferManager>(buffer_size, cache->rep,
allow_stall, policy);
return wbm;
}
void rocksdb_write_buffer_manager_destroy(rocksdb_write_buffer_manager_t* wbm) {
delete wbm;
}
@@ -7657,6 +7853,19 @@ ROCKSDB_LIBRARY_API void rocksdb_write_buffer_manager_set_allow_stall(
wbm->rep->SetAllowStall(new_allow_stall);
}
int rocksdb_write_buffer_manager_flush_policy(
rocksdb_write_buffer_manager_t* wbm) {
return static_cast<int>(wbm->rep->flush_policy());
}
void rocksdb_write_buffer_manager_set_flush_policy(
rocksdb_write_buffer_manager_t* wbm, int flush_policy) {
WriteBufferFlushPolicy policy;
if (rocksdb_write_buffer_manager_parse_flush_policy(flush_policy, &policy)) {
wbm->rep->SetFlushPolicy(policy);
}
}
rocksdb_sst_file_manager_t* rocksdb_sst_file_manager_create(
rocksdb_env_t* env) {
rocksdb_sst_file_manager_t* sfm = new rocksdb_sst_file_manager_t;
@@ -9883,6 +10092,26 @@ const char* rocksdb_options_get_db_host_id(rocksdb_options_t* opt,
return opt->rep.db_host_id.data();
}
void rocksdb_options_set_remote_compaction_manifest_floor(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.remote_compaction_manifest_floor = v;
}
unsigned char rocksdb_options_get_remote_compaction_manifest_floor(
rocksdb_options_t* opt) {
return opt->rep.remote_compaction_manifest_floor;
}
void rocksdb_options_set_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.use_session_tmp_dir_for_remote_compaction = v;
}
unsigned char rocksdb_options_get_use_session_tmp_dir_for_remote_compaction(
rocksdb_options_t* opt) {
return opt->rep.use_session_tmp_dir_for_remote_compaction;
}
void rocksdb_options_set_lowest_used_cache_tier(rocksdb_options_t* opt, int v) {
opt->rep.lowest_used_cache_tier =
static_cast<decltype(opt->rep.lowest_used_cache_tier)>(v);
@@ -9923,6 +10152,16 @@ uint64_t rocksdb_options_get_max_compaction_trigger_wakeup_seconds(
return opt->rep.max_compaction_trigger_wakeup_seconds;
}
void rocksdb_options_set_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt, int v) {
opt->rep.periodic_compaction_phase_recovery_percent = v;
}
int rocksdb_options_get_periodic_compaction_phase_recovery_percent(
rocksdb_options_t* opt) {
return opt->rep.periodic_compaction_phase_recovery_percent;
}
void rocksdb_options_set_follower_refresh_catchup_period_ms(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.follower_refresh_catchup_period_ms = v;
@@ -11094,16 +11333,6 @@ unsigned char rocksdb_ingestexternalfileoptions_get_ingest_behind(
return opt->rep.ingest_behind;
}
void rocksdb_ingestexternalfileoptions_set_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
opt->rep.write_global_seqno = v;
}
unsigned char rocksdb_ingestexternalfileoptions_get_write_global_seqno(
rocksdb_ingestexternalfileoptions_t* opt) {
return opt->rep.write_global_seqno;
}
void rocksdb_ingestexternalfileoptions_set_verify_checksums_before_ingest(
rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
opt->rep.verify_checksums_before_ingest = v;
@@ -11184,6 +11413,16 @@ int rocksdb_ingestexternalfileoptions_get_file_opening_threads(
/* OpenAndCompactOptions */
void rocksdb_open_and_compact_options_set_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt, uint32_t v) {
opt->rep.max_secondary_open_retries = v;
}
uint32_t rocksdb_open_and_compact_options_get_max_secondary_open_retries(
rocksdb_open_and_compact_options_t* opt) {
return opt->rep.max_secondary_open_retries;
}
unsigned char rocksdb_open_and_compact_options_get_allow_resumption(
rocksdb_open_and_compact_options_t* opt) {
return opt->rep.allow_resumption;
@@ -11581,6 +11820,17 @@ double rocksdb_block_based_options_get_uniform_cv_threshold(
return opt->rep.uniform_cv_threshold;
}
void rocksdb_block_based_options_set_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt, int v) {
opt->rep.optimize_key_common_prefix =
static_cast<decltype(opt->rep.optimize_key_common_prefix)>(v);
}
int rocksdb_block_based_options_get_optimize_key_common_prefix(
rocksdb_block_based_table_options_t* opt) {
return static_cast<int>(opt->rep.optimize_key_common_prefix);
}
void rocksdb_block_based_options_set_enable_index_compression(
rocksdb_block_based_table_options_t* opt, unsigned char v) {
opt->rep.enable_index_compression = v;
+59
View File
@@ -2474,6 +2474,35 @@ int main(int argc, char** argv) {
CheckNoError(err);
rocksdb_readoptions_set_snapshot(roptions, snap);
CheckGet(db, roptions, "foo", "hello");
size_t metadata_value_len = 0;
unsigned char newer_version_present = 0;
char* metadata_value =
rocksdb_get_with_metadata(db, roptions, "foo", 3, &metadata_value_len,
NULL, NULL, &newer_version_present, &err);
// NOLINTNEXTLINE(clang-analyzer-security.insecureAPI.DeprecatedOrUnsafeBufferHandling)
CheckNoError(err);
CheckEqual("hello", metadata_value, metadata_value_len);
// NOLINTNEXTLINE(clang-analyzer-security.insecureAPI.DeprecatedOrUnsafeBufferHandling)
CheckCondition(newer_version_present == 1);
Free(&metadata_value);
const char* metadata_keys[] = {"foo"};
const size_t metadata_key_sizes[] = {3};
char* metadata_values[1];
size_t metadata_value_sizes[1];
unsigned char newer_versions[1];
char* metadata_errors[1];
rocksdb_multi_get_with_metadata(
db, roptions, 1, metadata_keys, metadata_key_sizes, metadata_values,
metadata_value_sizes, NULL, NULL, newer_versions, metadata_errors);
// NOLINTNEXTLINE(clang-analyzer-security.insecureAPI.DeprecatedOrUnsafeBufferHandling)
CheckCondition(metadata_errors[0] == NULL);
CheckEqual("hello", metadata_values[0], metadata_value_sizes[0]);
// NOLINTNEXTLINE(clang-analyzer-security.insecureAPI.DeprecatedOrUnsafeBufferHandling)
CheckCondition(newer_versions[0] == 1);
Free(&metadata_values[0]);
rocksdb_readoptions_set_snapshot(roptions, NULL);
CheckGet(db, roptions, "foo", NULL);
rocksdb_release_snapshot(db, snap);
@@ -6687,6 +6716,36 @@ int main(int argc, char** argv) {
300 == rocksdb_write_buffer_manager_buffer_size(write_buffer_manager));
rocksdb_write_buffer_manager_destroy(write_buffer_manager);
write_buffer_manager = rocksdb_write_buffer_manager_create_with_policy(
200, true,
rocksdb_write_buffer_manager_flush_policy_largest_across_dbs);
CheckCondition(write_buffer_manager != NULL);
CheckCondition(
rocksdb_write_buffer_manager_flush_policy_largest_across_dbs ==
rocksdb_write_buffer_manager_flush_policy(write_buffer_manager));
rocksdb_write_buffer_manager_set_flush_policy(
write_buffer_manager,
rocksdb_write_buffer_manager_flush_policy_largest);
CheckCondition(
rocksdb_write_buffer_manager_flush_policy_largest ==
rocksdb_write_buffer_manager_flush_policy(write_buffer_manager));
rocksdb_write_buffer_manager_set_flush_policy(write_buffer_manager, 999);
CheckCondition(
rocksdb_write_buffer_manager_flush_policy_largest ==
rocksdb_write_buffer_manager_flush_policy(write_buffer_manager));
rocksdb_write_buffer_manager_destroy(write_buffer_manager);
write_buffer_manager =
rocksdb_write_buffer_manager_create_with_cache_and_policy(
200, lru, true,
rocksdb_write_buffer_manager_flush_policy_largest_across_dbs);
CheckCondition(write_buffer_manager != NULL);
rocksdb_write_buffer_manager_destroy(write_buffer_manager);
write_buffer_manager =
rocksdb_write_buffer_manager_create_with_policy(200, true, 999);
CheckCondition(write_buffer_manager == NULL);
rocksdb_cache_destroy(lru);
}
+127 -10
View File
@@ -33,6 +33,7 @@
#include "db/write_controller.h"
#include "file/sst_file_manager_impl.h"
#include "logging/logging.h"
#include "memtable/flush_initiator.h"
#include "monitoring/thread_status_util.h"
#include "options/options_helper.h"
#include "port/port.h"
@@ -672,7 +673,7 @@ ColumnFamilyData::ColumnFamilyData(
new InternalStats(ioptions_.num_levels, ioptions_.clock, this));
table_cache_.reset(new TableCache(ioptions_, file_options, _table_cache,
block_cache_tracer, io_tracer,
db_session_id, fast_sst_open));
db_session_id, fast_sst_open, name_));
blob_file_cache_.reset(
new BlobFileCache(_table_cache, &ioptions(), soptions(), id_,
internal_stats_->GetBlobFileReadHist(), io_tracer));
@@ -834,6 +835,11 @@ void ColumnFamilyData::SetDropped() {
// can't drop default CF
assert(id_ != 0);
dropped_ = true;
// A tracker can still contain accounting published while a different WBM
// policy was active. Cleanup therefore cannot depend on the current policy.
if (mem_ != nullptr) {
mem_->StopWBMTracking();
}
write_controller_token_.reset();
// remove from column_family_set
@@ -1246,7 +1252,10 @@ uint64_t ColumnFamilyData::GetLiveSstFilesSize() const {
MemTable* ColumnFamilyData::ConstructNewMemtable(
const MutableCFOptions& mutable_cf_options, SequenceNumber earliest_seq) {
return new MemTable(internal_comparator_, ioptions_, mutable_cf_options,
write_buffer_manager_, earliest_seq, id_);
write_buffer_manager_, earliest_seq, id_,
column_family_set_ != nullptr
? column_family_set_->flush_initiator()
: nullptr);
}
void ColumnFamilyData::CreateNewMemtable(SequenceNumber earliest_seq) {
@@ -1281,15 +1290,15 @@ Compaction* ColumnFamilyData::PickCompaction(
}
bool ColumnFamilyData::RangeOverlapWithCompaction(
const Slice& smallest_user_key, const Slice& largest_user_key,
int level) const {
const Slice& smallest_user_key, const Slice& largest_user_key, int level,
bool range_limit_exclusive) const {
return compaction_picker_->RangeOverlapWithCompaction(
smallest_user_key, largest_user_key, level);
smallest_user_key, largest_user_key, level, range_limit_exclusive);
}
Status ColumnFamilyData::RangesOverlapWithMemtables(
const autovector<UserKeyRange>& ranges, SuperVersion* super_version,
bool allow_data_in_errors, bool* overlap) {
bool allow_data_in_errors, bool* overlap, bool range_limit_exclusive) {
assert(overlap != nullptr);
*overlap = false;
// Create an InternalIterator over all unflushed memtables
@@ -1334,12 +1343,15 @@ Status ColumnFamilyData::RangesOverlapWithMemtables(
}
if (status.ok()) {
const int limit_cmp = memtable_iter->Valid()
? ucmp->CompareWithoutTimestamp(
seek_result.user_key, ranges[i].limit)
: 1;
if (memtable_iter->Valid() &&
ucmp->CompareWithoutTimestamp(seek_result.user_key,
ranges[i].limit) <= 0) {
(range_limit_exclusive ? limit_cmp < 0 : limit_cmp <= 0)) {
*overlap = true;
} else if (range_del_agg.IsRangeOverlapped(ranges[i].start,
ranges[i].limit)) {
} else if (range_del_agg.IsRangeOverlapped(
ranges[i].start, ranges[i].limit, range_limit_exclusive)) {
*overlap = true;
}
}
@@ -1896,6 +1908,102 @@ size_t ColumnFamilySet::NumberOfColumnFamilies() const {
return column_families_.size();
}
void ColumnFamilySet::RefreshFlushableMemAccounting() {
if (flush_initiator_ == nullptr || write_buffer_manager_ == nullptr ||
!write_buffer_manager_->ShouldTrackFlushInitiator()) {
return;
}
if (bulk_flushable_mem_accounting_depth_ > 0) {
bulk_flushable_mem_accounting_dirty_ = true;
flush_initiator_->InvalidateLargestFlushableCFMem();
return;
}
// Keep candidate readers from consuming a mixture of the previous snapshot
// and the values published by this rebuild.
flush_initiator_->InvalidateLargestFlushableCFMem();
constexpr int kMaxRefreshAttempts = 3;
for (int attempt = 0; attempt < kMaxRefreshAttempts; ++attempt) {
const uint64_t update_seq =
flush_initiator_->GetFlushableMemUpdateSequence();
bool has_mutable_mem = false;
bool has_ineligible_cf = false;
size_t largest = 0;
size_t waiting_immutable_mem = 0;
for (auto cfd : *this) {
if (cfd->IsDropped() || !cfd->initialized()) {
continue;
}
const bool mutable_not_empty = !cfd->mem()->IsEmpty();
if (mutable_not_empty) {
cfd->mem()->RefreshWBMTracking();
}
const bool has_mutable =
mutable_not_empty && cfd->mem()->IsWBMTrackingActive();
has_mutable_mem = has_mutable_mem || has_mutable;
if (flush_initiator_->UsesTotalMutableMem()) {
waiting_immutable_mem +=
cfd->imm()->WBMTrackedUnstartedMemTablesMemoryUsage();
continue;
}
if (cfd->imm()->IsFlushPendingOrRunning()) {
has_ineligible_cf = true;
continue;
}
const size_t mutable_mem =
has_mutable ? cfd->mem()->WBMTrackedMemoryUsage() : 0;
const size_t immutable_mem =
cfd->imm()->WBMTrackedUnflushedMemTablesMemoryUsage();
waiting_immutable_mem += immutable_mem;
largest = std::max(largest, mutable_mem + immutable_mem);
}
if (flush_initiator_->UsesTotalMutableMem()) {
flush_initiator_->SetHasIneligibleCF(false);
flush_initiator_->SetHasFlushableCF(has_mutable_mem ||
waiting_immutable_mem > 0);
if (flush_initiator_->TrySetLargestFlushableCFMem(
0, waiting_immutable_mem, update_seq)) {
write_buffer_manager_->NotifyFlushInitiatorChanged();
return;
}
continue;
}
flush_initiator_->SetHasIneligibleCF(has_ineligible_cf);
flush_initiator_->SetHasFlushableCF(largest > 0);
if (flush_initiator_->TrySetLargestFlushableCFMem(
largest, waiting_immutable_mem, update_seq)) {
write_buffer_manager_->NotifyFlushInitiatorChanged();
return;
}
}
flush_initiator_->InvalidateLargestFlushableCFMem();
}
void ColumnFamilySet::BeginBulkFlushableMemAccountingUpdate() {
++bulk_flushable_mem_accounting_depth_;
if (bulk_flushable_mem_accounting_depth_ == 1) {
bulk_flushable_mem_accounting_dirty_ = true;
if (flush_initiator_ != nullptr && write_buffer_manager_ != nullptr &&
write_buffer_manager_->ShouldTrackFlushInitiator()) {
flush_initiator_->InvalidateLargestFlushableCFMem();
write_buffer_manager_->NotifyFlushInitiatorChanged();
}
}
}
void ColumnFamilySet::EndBulkFlushableMemAccountingUpdate() {
assert(bulk_flushable_mem_accounting_depth_ > 0);
--bulk_flushable_mem_accounting_depth_;
if (bulk_flushable_mem_accounting_depth_ == 0 &&
bulk_flushable_mem_accounting_dirty_) {
bulk_flushable_mem_accounting_dirty_ = false;
TEST_SYNC_POINT(
"ColumnFamilySet::EndBulkFlushableMemAccountingUpdate:Refresh");
RefreshFlushableMemAccounting();
}
}
// under a DB mutex AND write thread
ColumnFamilyData* ColumnFamilySet::CreateColumnFamily(
const std::string& name, uint32_t id, Version* dummy_versions,
@@ -1936,6 +2044,15 @@ void ColumnFamilySet::RemoveColumnFamily(ColumnFamilyData* cfd) {
column_families_.erase(cfd->GetName());
running_ts_sz_.erase(cf_id);
ts_sz_for_record_.erase(cf_id);
if (bulk_flushable_mem_accounting_depth_ > 0) {
bulk_flushable_mem_accounting_dirty_ = true;
if (flush_initiator_ != nullptr) {
flush_initiator_->InvalidateLargestFlushableCFMem();
}
return;
}
TEST_SYNC_POINT("ColumnFamilySet::RemoveColumnFamily:Refresh");
RefreshFlushableMemAccounting();
}
// under a DB mutex OR from a write thread
+22 -3
View File
@@ -52,6 +52,7 @@ struct SuperVersionContext;
class BlobFileCache;
class BlobFilePartitionManager;
class BlobSource;
class FlushInitiator;
extern const double kIncSlowdownRatio;
// This file contains a list of data structures for managing column family
@@ -445,8 +446,8 @@ class ColumnFamilyData {
// Check if the passed range overlap with any running compactions.
// REQUIRES: DB mutex held
bool RangeOverlapWithCompaction(const Slice& smallest_user_key,
const Slice& largest_user_key,
int level) const;
const Slice& largest_user_key, int level,
bool range_limit_exclusive) const;
// Check if the passed ranges overlap with any unflushed memtables
// (immutable or mutable).
@@ -457,7 +458,8 @@ class ColumnFamilyData {
// Thread-safe
Status RangesOverlapWithMemtables(const autovector<UserKeyRange>& ranges,
SuperVersion* super_version,
bool allow_data_in_errors, bool* overlap);
bool allow_data_in_errors, bool* overlap,
bool range_limit_exclusive);
// A flag to tell a manual compaction is to compact all levels together
// instead of a specific level.
@@ -870,6 +872,19 @@ class ColumnFamilySet {
WriteBufferManager* write_buffer_manager() { return write_buffer_manager_; }
void SetFlushInitiator(FlushInitiator* flush_initiator) {
flush_initiator_ = flush_initiator;
}
FlushInitiator* flush_initiator() const { return flush_initiator_; }
void RefreshFlushableMemAccounting();
// Coalesces accounting rebuilds while a caller removes multiple column
// families. Both methods require the DB mutex.
void BeginBulkFlushableMemAccountingUpdate();
void EndBulkFlushableMemAccountingUpdate();
WriteController* write_controller() { return write_controller_; }
private:
@@ -897,6 +912,9 @@ class ColumnFamilySet {
// column families with non-zero user-defined timestamp size.
UnorderedMap<uint32_t, size_t> ts_sz_for_record_;
int bulk_flushable_mem_accounting_depth_ = 0;
bool bulk_flushable_mem_accounting_dirty_ = false;
uint32_t max_column_family_;
const FileOptions file_options_;
@@ -911,6 +929,7 @@ class ColumnFamilySet {
const ImmutableDBOptions* const db_options_;
Cache* table_cache_;
WriteBufferManager* write_buffer_manager_;
FlushInitiator* flush_initiator_ = nullptr;
WriteController* write_controller_;
BlockCacheTracer* const block_cache_tracer_;
std::shared_ptr<IOTracer> io_tracer_;
+46
View File
@@ -774,6 +774,8 @@ TEST_P(ColumnFamilyTest, BulkAddDrop) {
constexpr int kNumCF = 1000;
ColumnFamilyOptions cf_options;
WriteOptions write_options;
db_options_.write_buffer_manager = std::make_shared<WriteBufferManager>(
1 << 30, nullptr, false, WriteBufferFlushPolicy::kFlushLargestAcrossDBs);
Open();
std::vector<std::string> cf_names;
std::vector<ColumnFamilyHandle*> cf_handles;
@@ -784,7 +786,22 @@ TEST_P(ColumnFamilyTest, BulkAddDrop) {
for (int i = 1; i <= kNumCF; i++) {
ASSERT_OK(db_->Put(write_options, cf_handles[i - 1], "foo", "bar"));
}
std::atomic<int> per_removal_refreshes{0};
std::atomic<int> bulk_refreshes{0};
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ColumnFamilySet::RemoveColumnFamily:Refresh",
[&](void*) { ++per_removal_refreshes; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ColumnFamilySet::EndBulkFlushableMemAccountingUpdate:Refresh",
[&](void*) { ++bulk_refreshes; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Defer cleanup_sync_points([] {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
});
ASSERT_OK(db_->DropColumnFamilies(cf_handles));
EXPECT_EQ(0, per_removal_refreshes.load());
EXPECT_EQ(1, bulk_refreshes.load());
std::vector<ColumnFamilyDescriptor> cf_descriptors;
for (auto* handle : cf_handles) {
delete handle;
@@ -798,7 +815,10 @@ TEST_P(ColumnFamilyTest, BulkAddDrop) {
for (int i = 1; i <= kNumCF; i++) {
ASSERT_OK(db_->Put(write_options, cf_handles[i - 1], "foo", "bar"));
}
bulk_refreshes.store(0);
ASSERT_OK(db_->DropColumnFamilies(cf_handles));
EXPECT_EQ(0, per_removal_refreshes.load());
EXPECT_EQ(1, bulk_refreshes.load());
for (auto* handle : cf_handles) {
delete handle;
}
@@ -3167,6 +3187,32 @@ TEST_P(ColumnFamilyTest, CreateDropAndDestroy) {
ASSERT_OK(db_->DestroyColumnFamilyHandle(cfh));
}
TEST_P(ColumnFamilyTest, DropWaitsForUnorderedMemtableWrite) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WriteImpl:UnorderedWriteAfterWriteWAL",
"ColumnFamilyTest::DropWaitsForUnorderedMemtableWrite:AfterWAL"},
{"DBImpl::WaitForPendingWrites:BeforeBlock",
"DBImpl::WriteImpl:BeforeUnorderedWriteMemtable"}});
db_options_.unordered_write = true;
Open();
ColumnFamilyHandle* cfh;
ASSERT_OK(db_->CreateColumnFamily(ColumnFamilyOptions(), "yoyo", &cfh));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
port::Thread writer(
[&] { ASSERT_OK(db_->Put(WriteOptions(), cfh, "foo", "bar")); });
TEST_SYNC_POINT(
"ColumnFamilyTest::DropWaitsForUnorderedMemtableWrite:AfterWAL");
ASSERT_OK(db_->DropColumnFamily(cfh));
writer.join();
ASSERT_OK(db_->DestroyColumnFamilyHandle(cfh));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(ColumnFamilyTest, CreateDropAndDestroyWithoutFileDeletion) {
ColumnFamilyHandle* cfh;
Open();
+12 -10
View File
@@ -491,7 +491,8 @@ int Compaction::GetProximalLevel() const { return proximal_level_; }
// smallest_key and largest_key include timestamps if user-defined timestamp is
// enabled.
bool Compaction::OverlapProximalLevelOutputRange(
const Slice& smallest_key, const Slice& largest_key) const {
const Slice& smallest_key, const Slice& largest_key,
bool range_limit_exclusive) const {
if (!SupportsPerKeyPlacement()) {
return false;
}
@@ -506,10 +507,11 @@ bool Compaction::OverlapProximalLevelOutputRange(
const Comparator* ucmp =
input_vstorage_->InternalComparator()->user_comparator();
const int largest_cmp = ucmp->CompareWithoutTimestamp(
largest_key, proximal_level_smallest_.user_key());
return ucmp->CompareWithoutTimestamp(
smallest_key, proximal_level_largest_.user_key()) <= 0 &&
ucmp->CompareWithoutTimestamp(
largest_key, proximal_level_smallest_.user_key()) >= 0;
(range_limit_exclusive ? largest_cmp > 0 : largest_cmp >= 0);
}
// key includes timestamp if user-defined timestamp is enabled.
@@ -984,9 +986,9 @@ uint64_t Compaction::MaxInputFileNewestKeyTime(const InternalKey* start,
return newest_key_time;
}
uint64_t Compaction::MinInputFileOldestAncesterTime(
uint64_t Compaction::MinInputFileOldestAncestorTime(
const InternalKey* start, const InternalKey* end) const {
uint64_t min_oldest_ancester_time = std::numeric_limits<uint64_t>::max();
uint64_t min_oldest_ancestor_time = std::numeric_limits<uint64_t>::max();
const InternalKeyComparator& icmp =
column_family_data()->internal_comparator();
for (const auto& level_files : inputs_) {
@@ -997,14 +999,14 @@ uint64_t Compaction::MinInputFileOldestAncesterTime(
if (end != nullptr && icmp.Compare(file->smallest, *end) > 0) {
continue;
}
uint64_t oldest_ancester_time = file->TryGetOldestAncesterTime();
if (oldest_ancester_time != 0) {
min_oldest_ancester_time =
std::min(min_oldest_ancester_time, oldest_ancester_time);
uint64_t oldest_ancestor_time = file->TryGetOldestAncestorTime();
if (oldest_ancestor_time != 0) {
min_oldest_ancestor_time =
std::min(min_oldest_ancestor_time, oldest_ancestor_time);
}
}
}
return min_oldest_ancester_time;
return min_oldest_ancestor_time;
}
uint64_t Compaction::MinInputFileEpochNumber() const {
+4 -3
View File
@@ -397,7 +397,8 @@ class Compaction {
// Both smallest_key and largest_key include timestamps if user-defined
// timestamp is enabled.
bool OverlapProximalLevelOutputRange(const Slice& smallest_key,
const Slice& largest_key) const;
const Slice& largest_key,
bool range_limit_exclusive) const;
// For testing purposes, check that a key is within proximal level
// output range for per_key_placement feature, which is safe to place the key
@@ -437,8 +438,8 @@ class Compaction {
const InternalKey* end) const;
// start and end are sub compact range. Null if no boundary.
// This is used to filter out some input files' ancester's time range.
uint64_t MinInputFileOldestAncesterTime(const InternalKey* start,
// This is used to filter out some input files' ancestor's time range.
uint64_t MinInputFileOldestAncestorTime(const InternalKey* start,
const InternalKey* end) const;
// Return the minimum epoch number among
// input files' associated with this compaction
+24 -15
View File
@@ -1909,8 +1909,7 @@ void CompactionIterator::PrepareOutput() {
// KeyNotExistsBeyondOutputLevel() return true?
if (Valid() && bottommost_level_ &&
DefinitelyInSnapshot(ikey_.sequence, earliest_snapshot_) &&
ikey_.type != kTypeMerge && current_key_committed_ &&
ikey_.sequence <= preserve_seqno_after_ && !is_range_del_) {
ikey_.type != kTypeMerge && current_key_committed_ && !is_range_del_) {
assert(compaction_ != nullptr && !compaction_->allow_ingest_behind());
if (ikey_.type == kTypeDeletion ||
(ikey_.type == kTypeSingleDeletion && timestamp_size_ == 0)) {
@@ -1932,19 +1931,29 @@ void CompactionIterator::PrepareOutput() {
}
bool zeroed_seqno = false;
if (!timestamp_size_) {
current_key_.UpdateInternalKey(0, ikey_.type);
zeroed_seqno = true;
} else if (full_history_ts_low_ && cmp_with_history_ts_low_ < 0) {
// For UDT, the seqno and timestamp could only be zeroed out after the
// key is below history_ts_low_.
// For the same user key (excluding timestamp), the timestamp-based
// history can be collapsed to save some space if the timestamp is
// older than *full_history_ts_low_.
const std::string kTsMin(timestamp_size_, static_cast<char>(0));
const Slice ts_slice = kTsMin;
ikey_.SetTimestamp(ts_slice);
current_key_.UpdateInternalKey(0, ikey_.type, &ts_slice);
// Whether the sequence number can actually be zeroed out here is decided
// by the shared BottommostSeqnoCanBeZeroed() predicate, which the
// bottommost-file marking logic
// (VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction) also
// consults. Keeping both on one predicate ensures a file is only marked
// for a compaction that can make progress; otherwise the file would be
// re-marked forever (infinite compaction loop).
if (BottommostSeqnoCanBeZeroed(
ikey_.sequence, preserve_seqno_after_, timestamp_size_,
full_history_ts_low_ != nullptr, cmp_with_history_ts_low_ < 0)) {
if (!timestamp_size_) {
current_key_.UpdateInternalKey(0, ikey_.type);
} else {
// For UDT, the seqno and timestamp could only be zeroed out after the
// key is below history_ts_low_.
// For the same user key (excluding timestamp), the timestamp-based
// history can be collapsed to save some space if the timestamp is
// older than *full_history_ts_low_.
const std::string kTsMin(timestamp_size_, static_cast<char>(0));
const Slice ts_slice = kTsMin;
ikey_.SetTimestamp(ts_slice);
current_key_.UpdateInternalKey(0, ikey_.type, &ts_slice);
}
zeroed_seqno = true;
}
+47 -16
View File
@@ -395,6 +395,24 @@ void CompactionJob::Prepare(
std::min(preclude_last_level_min_seqno, preserve_time_min_seqno);
#endif
// For a bottommost-file compaction, which exists only because the marker
// (VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction) decided the
// file's largest sequence number could be zeroed, honor that same
// version-carried preserve boundary here. The marker reads DBImpl's live,
// DB-wide seqno->time mapping, whereas the value computed above is derived
// only from the selected file's persisted mapping, which can be sparse or
// empty (e.g. a file written before preserve/preclude was enabled). Taking
// the max ensures this compaction can zero at least the sequence numbers the
// marker considered zeroable, so a marked file makes progress instead of
// being re-marked forever (infinite compaction loop). Other compaction
// reasons keep the per-input-file boundary so their seqno-zeroing behavior is
// unchanged. When preserve/preclude is inactive the version value is
// kMaxSequenceNumber, making this a no-op.
if (c->compaction_reason() == CompactionReason::kBottommostFiles) {
preserve_time_min_seqno = std::max(preserve_time_min_seqno,
storage_info->GetPreserveTimeMinSeqno());
}
// Preserve sequence numbers for preserved write times and snapshots, though
// the specific sequence number of the earliest snapshot can be zeroed.
preserve_seqno_after_ =
@@ -423,6 +441,17 @@ void CompactionJob::Prepare(
c->GetKeepInLastLevelThroughSeqno());
options_file_number_ = versions_->options_file_number();
// For remote compaction hardening: capture the current MANIFEST position as a
// floor for the worker's recovery. manifest_file_size() is a physical byte
// offset that VersionSet::ProcessManifestWrites records under the DB mutex
// only after the whole version-edit batch is written and sync'd, so it is a
// durable, complete-batch offset -- never mid-record or mid-atomic-group --
// at or past the compaction's (already committed) input version.
if (mutable_db_options_copy_.remote_compaction_manifest_floor) {
min_manifest_file_number_ = versions_->manifest_file_number();
min_manifest_file_size_ = versions_->manifest_file_size();
}
}
void CompactionJob::MaybeAssignCompactionProgressAndWriter(
@@ -2131,20 +2160,20 @@ Status CompactionJob::FinishCompactionOutputFile(
if (s.ok()) {
// With accurate smallest and largest key, we can get a slightly more
// accurate oldest ancester time.
// This makes oldest ancester time in manifest more accurate than in
// accurate oldest ancestor time.
// This makes oldest ancestor time in manifest more accurate than in
// table properties. Not sure how to resolve it.
if (meta->smallest.size() > 0 && meta->largest.size() > 0) {
uint64_t refined_oldest_ancester_time;
uint64_t refined_oldest_ancestor_time;
Slice new_smallest = meta->smallest.user_key();
Slice new_largest = meta->largest.user_key();
if (!new_largest.empty() && !new_smallest.empty()) {
refined_oldest_ancester_time =
sub_compact->compaction->MinInputFileOldestAncesterTime(
refined_oldest_ancestor_time =
sub_compact->compaction->MinInputFileOldestAncestorTime(
&(meta->smallest), &(meta->largest));
if (refined_oldest_ancester_time !=
if (refined_oldest_ancestor_time !=
std::numeric_limits<uint64_t>::max()) {
meta->oldest_ancester_time = refined_oldest_ancester_time;
meta->oldest_ancestor_time = refined_oldest_ancestor_time;
}
}
}
@@ -2526,7 +2555,7 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
return s;
}
// Try to figure out the output file's oldest ancester time.
// Try to figure out the output file's oldest ancestor time.
int64_t temp_current_time = 0;
auto get_time_status = db_options_.clock->GetCurrentTime(&temp_current_time);
// Safe to proceed even if GetCurrentTime fails. So, log and proceed.
@@ -2543,14 +2572,14 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
if (sub_compact->end.has_value()) {
tmp_end.SetMinPossibleForUserKey(*(sub_compact->end));
}
uint64_t oldest_ancester_time =
sub_compact->compaction->MinInputFileOldestAncesterTime(
uint64_t oldest_ancestor_time =
sub_compact->compaction->MinInputFileOldestAncestorTime(
sub_compact->start.has_value() ? &tmp_start : nullptr,
sub_compact->end.has_value() ? &tmp_end : nullptr);
if (oldest_ancester_time == std::numeric_limits<uint64_t>::max()) {
if (oldest_ancestor_time == std::numeric_limits<uint64_t>::max()) {
// TODO: fix DBSSTTest.GetTotalSstFilesSize and use
// kUnknownOldestAncesterTime
oldest_ancester_time = current_time;
// kUnknownOldestAncestorTime
oldest_ancestor_time = current_time;
}
uint64_t newest_key_time = sub_compact->compaction->MaxInputFileNewestKeyTime(
@@ -2563,7 +2592,7 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
FileMetaData meta;
meta.fd = FileDescriptor(file_number,
sub_compact->compaction->output_path_id(), 0);
meta.oldest_ancester_time = oldest_ancester_time;
meta.oldest_ancestor_time = oldest_ancestor_time;
meta.file_creation_time = current_time;
meta.epoch_number = epoch_number;
meta.temperature = temperature;
@@ -2605,7 +2634,8 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
std::move(writable_file), fname, fo_copy, db_options_.clock, io_tracer_,
db_options_.stats, Histograms::SST_WRITE_MICROS, listeners,
db_options_.file_checksum_gen_factory.get(),
tmp_set.Contains(FileType::kTableFile), false));
tmp_set.Contains(FileType::kTableFile),
tmp_set.Contains(FileType::kTableFile)));
// TODO(hx235): pass in the correct `oldest_key_time` instead of `0`
const ReadOptions read_options(Env::IOActivity::kCompaction);
@@ -3238,7 +3268,8 @@ Status CompactionJob::PersistSubcompactionProgress(
s = WritableFileWriter::PrepareIOOptions(write_options, opts);
}
if (s.ok()) {
s = compaction_progress_writer_->file()->Sync(opts, db_options_.use_fsync);
s = db_options_.use_fsync ? compaction_progress_writer_->file()->Fsync(opts)
: compaction_progress_writer_->file()->Sync(opts);
}
if (!s.ok()) {
+24 -4
View File
@@ -508,7 +508,7 @@ class CompactionJob {
std::atomic<int>* num_running_remote_compactions_;
// Stores the sequence number to time mapping gathered from all input files
// it also collects the smallest_seqno -> oldest_ancester_time from the SST.
// it also collects the smallest_seqno -> oldest_ancestor_time from the SST.
SeqnoToTimeMapping seqno_to_time_mapping_;
// Max seqno that can be zeroed out in last level, including for preserving
@@ -524,6 +524,13 @@ class CompactionJob {
// Setting this requires DBMutex.
uint64_t options_file_number_ = 0;
// MANIFEST position (file number and size, in bytes) captured in Prepare()
// (mutex held) to send to the remote worker as a floor. Zero when
// DBOptions::remote_compaction_manifest_floor is off (or this is not a remote
// compaction), meaning no floor is sent. Setting these requires DBMutex.
uint64_t min_manifest_file_number_ = 0;
uint64_t min_manifest_file_size_ = 0;
// Writer for persisting compaction progress during compaction
log::Writer* compaction_progress_writer_ = nullptr;
@@ -584,6 +591,7 @@ class CompactionJob {
struct CompactionServiceInput {
std::string cf_name;
// Snapshot sequence numbers in strictly increasing order.
std::vector<SequenceNumber> snapshots;
// SST files for compaction, it should already be expended to include all the
@@ -603,6 +611,18 @@ struct CompactionServiceInput {
uint64_t options_file_number = 0;
// MANIFEST position (file number and size, in bytes) the primary scheduled
// this compaction from, used as a floor by the remote worker: it refuses to
// reconstruct the compaction against an older MANIFEST view (an older file
// number, or the same file number read to a smaller size) and falls back to
// local compaction. A non-zero min_manifest_file_number also signals the
// worker to use "trust the MANIFEST" recovery. Zero means the primary did not
// provide a floor (kill switch off, or an older primary); the worker then
// uses the original recovery with no floor check.
// See DBOptions::remote_compaction_manifest_floor.
uint64_t min_manifest_file_number = 0;
uint64_t min_manifest_file_size = 0;
// serialization interface to read and write the object
static Status Read(const std::string& data_str, CompactionServiceInput* obj);
Status Write(std::string* output);
@@ -621,7 +641,7 @@ struct CompactionServiceOutputFile {
SequenceNumber largest_seqno{};
std::string smallest_internal_key;
std::string largest_internal_key;
uint64_t oldest_ancester_time = kUnknownOldestAncesterTime;
uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
uint64_t file_creation_time = kUnknownFileCreationTime;
uint64_t epoch_number = kUnknownEpochNumber;
std::string file_checksum = kUnknownFileChecksum;
@@ -637,7 +657,7 @@ struct CompactionServiceOutputFile {
CompactionServiceOutputFile(
const std::string& name, uint64_t size, SequenceNumber smallest,
SequenceNumber largest, std::string _smallest_internal_key,
std::string _largest_internal_key, uint64_t _oldest_ancester_time,
std::string _largest_internal_key, uint64_t _oldest_ancestor_time,
uint64_t _file_creation_time, uint64_t _epoch_number,
const std::string& _file_checksum,
const std::string& _file_checksum_func_name, uint64_t _paranoid_hash,
@@ -650,7 +670,7 @@ struct CompactionServiceOutputFile {
largest_seqno(largest),
smallest_internal_key(std::move(_smallest_internal_key)),
largest_internal_key(std::move(_largest_internal_key)),
oldest_ancester_time(_oldest_ancester_time),
oldest_ancestor_time(_oldest_ancestor_time),
file_creation_time(_file_creation_time),
epoch_number(_epoch_number),
file_checksum(_file_checksum),
+25 -2
View File
@@ -387,7 +387,7 @@ class CompactionJobTestBase : public testing::Test {
edit.AddFile(
level, file_number, 0, file_size, smallest_key, largest_key,
smallest_seqno, largest_seqno, false, Temperature::kUnknown,
oldest_blob_file_number, kUnknownOldestAncesterTime,
oldest_blob_file_number, kUnknownOldestAncestorTime,
kUnknownFileCreationTime,
versions_->GetColumnFamilySet()->GetDefault()->NewEpochNumber(),
kUnknownFileChecksum, kUnknownFileChecksumFuncName, kNullUniqueId64x2,
@@ -1572,6 +1572,10 @@ TEST_F(CompactionJobTest, InputSerialization) {
while (!rnd.OneIn(10)) {
input.snapshots.emplace_back(rnd64.Uniform(UINT64_MAX));
}
std::sort(input.snapshots.begin(), input.snapshots.end());
input.snapshots.erase(
std::unique(input.snapshots.begin(), input.snapshots.end()),
input.snapshots.end());
while (!rnd.OneIn(10)) {
input.input_files.emplace_back(rnd.RandomString(
rnd.Uniform(kStrMaxLen - 1) +
@@ -1641,6 +1645,25 @@ TEST_F(CompactionJobTest, InputSerialization) {
ASSERT_TRUE(s.IsNotSupported());
}
TEST_F(CompactionJobTest, OpenAndCompactRejectsUnorderedSnapshots) {
for (const std::vector<SequenceNumber>& snapshots :
{std::vector<SequenceNumber>{2, 1}, std::vector<SequenceNumber>{1, 1}}) {
CompactionServiceInput input;
input.snapshots = snapshots;
std::string serialized_input;
ASSERT_OK(input.Write(&serialized_input));
std::string serialized_result;
Status s =
DB::OpenAndCompact("", "output", serialized_input, &serialized_result,
CompactionServiceOptionsOverride());
ASSERT_TRUE(s.IsInvalidArgument());
ASSERT_NE(s.ToString().find("snapshots must be strictly increasing"),
std::string::npos);
}
}
TEST_F(CompactionJobTest, ResultSerialization) {
// Setup a random CompactionServiceResult
CompactionServiceResult result;
@@ -1679,7 +1702,7 @@ TEST_F(CompactionJobTest, ResultSerialization) {
rnd.Uniform(kStrMaxLen)) /* smallest_internal_key */,
rnd.RandomBinaryString(
rnd.Uniform(kStrMaxLen)) /* largest_internal_key */,
rnd64.Uniform(UINT64_MAX) /* oldest_ancester_time */,
rnd64.Uniform(UINT64_MAX) /* oldest_ancestor_time */,
rnd64.Uniform(UINT64_MAX) /* file_creation_time */,
rnd64.Uniform(UINT64_MAX) /* epoch_number */,
file_checksum /* file_checksum */,
+38 -11
View File
@@ -32,7 +32,7 @@ Status CompactionOutputs::Finish(
meta->fd.largest_seqno);
relevant_mapping.SetCapacity(kMaxSeqnoTimePairsPerSST);
builder_->SetSeqnoTimeTableProperties(relevant_mapping,
meta->oldest_ancester_time);
meta->oldest_ancestor_time);
s = builder_->Finish();
} else {
@@ -73,7 +73,7 @@ IOStatus CompactionOutputs::WriterSyncClose(const Status& input_status,
WriteOptions(Env::IOActivity::kCompaction), opts);
if (input_status.ok() && io_s.ok()) {
StopWatch sw(clock, statistics, COMPACTION_OUTFILE_SYNC_MICROS);
io_s = file_writer_->Sync(opts, use_fsync);
io_s = use_fsync ? file_writer_->Fsync(opts) : file_writer_->Sync(opts);
}
if (input_status.ok() && io_s.ok()) {
io_s = file_writer_->Close(opts);
@@ -267,11 +267,22 @@ bool CompactionOutputs::ShouldStopBefore(const CompactionIterator& c_iter) {
return true;
}
// If there's user defined partitioner, check that first
if (partitioner_ && partitioner_->ShouldPartition(PartitionerRequest(
last_key_for_partitioner_, c_iter.user_key(),
current_output_file_size_)) == kRequired) {
return true;
if (partitioner_) {
if (partitioner_prefix_valid_) {
// Fast path: no per-key copy or virtual call. A partition is requested at
// the first key that no longer shares the established prefix. An empty
// prefix means no further partitions (every key "starts with" it).
if (UNLIKELY(!c_iter.user_key().starts_with(partitioner_prefix_))) {
return true;
}
} else {
// Slow path: per-key callback (and copy elsewhere)
if (UNLIKELY(partitioner_->ShouldPartition(PartitionerRequest(
last_key_for_partitioner_, c_iter.user_key(),
current_output_file_size_)) == kRequired)) {
return true;
}
}
}
// files output to Level 0 won't be split
@@ -401,11 +412,27 @@ Status CompactionOutputs::AddToOutput(
if (!s.ok()) {
return s;
}
// A new output file starts a fresh partition-tracking window. Establish the
// partitioner fast path from this file's first key: if the partitioner can
// express its next boundary as "first key not sharing this prefix", cache
// that prefix; otherwise fall back to per-key ShouldPartition(). This is
// the only place the prefix needs to be (re)derived, since a partition
// boundary always forces a file cut, and re-deriving on non-partition cuts
// just recomputes the same still-valid prefix.
if (partitioner_) {
OptSlice prefix =
partitioner_->ShouldPartitionByPrefix(c_iter.user_key());
partitioner_prefix_valid_ = prefix.has_value();
if (partitioner_prefix_valid_) {
partitioner_prefix_.assign(prefix->data(), prefix->size());
}
}
}
// c_iter may emit range deletion keys, so update `last_key_for_partitioner_`
// here before returning below when `is_range_del` is true
if (partitioner_) {
// here before returning below when `is_range_del` is true. Only needed in the
// fallback mode; the prefix fast path does not use it.
if (partitioner_ && !partitioner_prefix_valid_) {
last_key_for_partitioner_.assign(c_iter.user_key().data_,
c_iter.user_key().size_);
}
@@ -790,10 +817,10 @@ void CompactionOutputs::FillFilesToCutForTtl() {
*(compaction_->inputs(compaction_->num_input_levels() - 1));
for (FileMetaData* file : olevel) {
// Worth filtering out by start and end?
uint64_t oldest_ancester_time = file->TryGetOldestAncesterTime();
uint64_t oldest_ancestor_time = file->TryGetOldestAncestorTime();
// We put old files if they are not too small to prevent a flood
// of small files.
if (oldest_ancester_time < old_age_thres &&
if (oldest_ancestor_time < old_age_thres &&
file->fd.GetFileSize() >
compaction_->mutable_cf_options().target_file_size_base / 2) {
files_to_cut_for_ttl_.push_back(file);
+13 -2
View File
@@ -349,8 +349,19 @@ class CompactionOutputs {
const bool is_proximal_level_;
// partitioner information
std::string last_key_for_partitioner_;
std::unique_ptr<SstPartitioner> partitioner_;
// Previous user key, only maintained (copied per key) in the fallback mode
// (see below).
std::string last_key_for_partitioner_;
// Fast path for partitioners that support ShouldPartitionByPrefix(): when
// partitioner_prefix_valid_ is true, all keys in the current output file
// share partitioner_prefix_, and a partition boundary is requested at the
// first future key that does not start with that prefix. This is (re)derived
// once per output file from its first key, avoiding a per-key copy and
// virtual ShouldPartition() call. When false, we fall back to comparing
// against last_key_for_partitioner_ via ShouldPartition() on every key.
std::string partitioner_prefix_;
bool partitioner_prefix_valid_ = false;
// A flag determines if this subcompaction has been split by the cursor
// for RoundRobin compaction
@@ -360,7 +371,7 @@ class CompactionOutputs {
// repetitive comparison in ShouldStopBefore()
const InternalKey* local_output_split_key_ = nullptr;
// Some identified files with old oldest ancester time and the range should be
// Some identified files with old oldest ancestor time and the range should be
// isolated out so that the output file(s) in that range can be merged down
// for TTL and clear the timestamps for the range.
std::vector<FileMetaData*> files_to_cut_for_ttl_;
+13 -9
View File
@@ -319,21 +319,22 @@ bool CompactionPicker::ExpandInputsToCleanCut(const std::string& /*cf_name*/,
}
bool CompactionPicker::RangeOverlapWithCompaction(
const Slice& smallest_user_key, const Slice& largest_user_key,
int level) const {
const Slice& smallest_user_key, const Slice& largest_user_key, int level,
bool range_limit_exclusive) const {
const Comparator* ucmp = icmp_->user_comparator();
for (Compaction* c : compactions_in_progress_) {
const int largest_cmp = ucmp->CompareWithoutTimestamp(
largest_user_key, c->GetSmallestUserKey());
if (c->output_level() == level &&
ucmp->CompareWithoutTimestamp(smallest_user_key,
c->GetLargestUserKey()) <= 0 &&
ucmp->CompareWithoutTimestamp(largest_user_key,
c->GetSmallestUserKey()) >= 0) {
(range_limit_exclusive ? largest_cmp > 0 : largest_cmp >= 0)) {
// Overlap
return true;
}
if (c->SupportsPerKeyPlacement()) {
if (c->OverlapProximalLevelOutputRange(smallest_user_key,
largest_user_key)) {
if (c->OverlapProximalLevelOutputRange(
smallest_user_key, largest_user_key, range_limit_exclusive)) {
return true;
}
}
@@ -364,7 +365,8 @@ bool CompactionPicker::FilesRangeOverlapWithCompaction(
if (proximal_level != Compaction::kInvalidLevel) {
if (ioptions_.compaction_style == kCompactionStyleUniversal) {
if (RangeOverlapWithCompaction(smallest.user_key(), largest.user_key(),
proximal_level)) {
proximal_level,
/*range_limit_exclusive=*/false)) {
return true;
}
} else {
@@ -372,14 +374,16 @@ bool CompactionPicker::FilesRangeOverlapWithCompaction(
GetRange(inputs, &proximal_smallest, &proximal_largest, level);
if (RangeOverlapWithCompaction(proximal_smallest.user_key(),
proximal_largest.user_key(),
proximal_level)) {
proximal_level,
/*range_limit_exclusive=*/false)) {
return true;
}
}
}
return RangeOverlapWithCompaction(smallest.user_key(), largest.user_key(),
level);
level,
/*range_limit_exclusive=*/false);
}
// Returns true if any one of specified files are being compacted
+2 -2
View File
@@ -154,8 +154,8 @@ class CompactionPicker {
// Return true if the passed key range overlap with a compaction output
// that is currently running.
bool RangeOverlapWithCompaction(const Slice& smallest_user_key,
const Slice& largest_user_key,
int level) const;
const Slice& largest_user_key, int level,
bool range_limit_exclusive) const;
// Stores the minimal range that covers all entries in inputs in
// *smallest, *largest.
+6 -5
View File
@@ -105,9 +105,10 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
TableReader* reader = f->fd.pinned_reader.Get();
if (reader != nullptr && reader->GetTableProperties() != nullptr) {
uint64_t newest_key_time = f->TryGetNewestKeyTime();
uint64_t creation_time = reader->GetTableProperties()->creation_time;
uint64_t oldest_ancestor_time =
reader->GetTableProperties()->oldest_ancestor_time;
uint64_t est_newest_key_time = newest_key_time == kUnknownNewestKeyTime
? creation_time
? oldest_ancestor_time
: newest_key_time;
if (est_newest_key_time == kUnknownNewestKeyTime ||
est_newest_key_time >= (current_time - mutable_cf_options.ttl)) {
@@ -165,13 +166,13 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
for (const auto& f : inputs[0].files) {
assert(f);
uint64_t newest_key_time = f->TryGetNewestKeyTime();
uint64_t creation_time = 0;
uint64_t oldest_ancestor_time = 0;
TableReader* reader = f->fd.pinned_reader.Get();
if (reader != nullptr && reader->GetTableProperties() != nullptr) {
creation_time = reader->GetTableProperties()->creation_time;
oldest_ancestor_time = reader->GetTableProperties()->oldest_ancestor_time;
}
uint64_t est_newest_key_time = newest_key_time == kUnknownNewestKeyTime
? creation_time
? oldest_ancestor_time
: newest_key_time;
ROCKS_LOG_BUFFER(log_buffer,
"[%s] FIFO compaction: picking file %" PRIu64
+213 -78
View File
@@ -90,7 +90,8 @@ class CompactionPickerTestBase : public testing::Test {
&icmp_, ucmp_, options_.num_levels, style, nullptr, false,
EpochNumberRequirement::kMustPresent, ioptions_.clock,
options_.bottommost_file_compaction_delay,
OffpeakTimeOption(mutable_db_options_.daily_offpeak_time_utc)));
OffpeakTimeOption(mutable_db_options_.daily_offpeak_time_utc),
PeriodicCompactionPhaseParams{}));
vstorage_->PrepareForVersionAppend(ioptions_, mutable_cf_options_);
}
@@ -101,7 +102,8 @@ class CompactionPickerTestBase : public testing::Test {
&icmp_, ucmp_, options_.num_levels, ioptions_.compaction_style,
vstorage_.get(), false, EpochNumberRequirement::kMustPresent,
ioptions_.clock, options_.bottommost_file_compaction_delay,
OffpeakTimeOption(mutable_db_options_.daily_offpeak_time_utc)));
OffpeakTimeOption(mutable_db_options_.daily_offpeak_time_utc),
PeriodicCompactionPhaseParams{}));
}
void DeleteVersionStorage() {
@@ -118,7 +120,7 @@ class CompactionPickerTestBase : public testing::Test {
SequenceNumber smallest_seq = 100, SequenceNumber largest_seq = 100,
size_t compensated_file_size = 0, bool marked_for_compact = false,
Temperature temperature = Temperature::kUnknown,
uint64_t oldest_ancestor_time = kUnknownOldestAncesterTime,
uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime,
uint64_t newest_key_time = kUnknownNewestKeyTime,
Slice ts_of_smallest = Slice(), Slice ts_of_largest = Slice(),
uint64_t epoch_number = kUnknownEpochNumber) {
@@ -160,15 +162,15 @@ class CompactionPickerTestBase : public testing::Test {
FileMetaData* f = new FileMetaData(
file_number, path_id, file_size, smallest_ikey, largest_ikey,
smallest_seq, largest_seq, marked_for_compact, temperature,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kInvalidBlobFileNumber, kUnknownOldestAncestorTime,
kUnknownFileCreationTime, epoch_number, kUnknownFileChecksum,
kUnknownFileChecksumFuncName, kNullUniqueId64x2, 0, 0,
true /* user_defined_timestamps_persisted */, "" /* min timestamp */,
"" /* max timestamp */);
f->compensated_file_size =
(compensated_file_size != 0) ? compensated_file_size : file_size;
// oldest_ancester_time is only used if newest_key_time is not available
f->oldest_ancester_time = oldest_ancestor_time;
// oldest_ancestor_time is only used if newest_key_time is not available
f->oldest_ancestor_time = oldest_ancestor_time;
// Set min/max timestamps for UDT support
if (!ts_of_smallest.empty()) {
f->min_timestamp = ts_of_smallest.ToString();
@@ -323,7 +325,7 @@ class CompactionPickerU64TsTest : public CompactionPickerTestBase {
/*smallest_seq=*/10, /*largest_seq=*/40,
/*compensated_file_size=*/1000,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, ts_small, ts_large);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts_small, ts_large);
std::string full_history_ts_low = MakeU64Timestamp(full_history_ts_low_val);
@@ -348,12 +350,12 @@ class CompactionPickerU64TsTest : public CompactionPickerTestBase {
/*smallest_seq=*/100, /*largest_seq=*/100,
/*compensated_file_size=*/file_size,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, ts1, ts2);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts1, ts2);
Add(0, 2U, "150", "250", file_size, /*path_id=*/0,
/*smallest_seq=*/200, /*largest_seq=*/200,
/*compensated_file_size=*/file_size,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, ts1, ts2);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts1, ts2);
}
};
@@ -1450,14 +1452,14 @@ TEST_F(CompactionPickerTest, FIFOToCold1) {
static_cast<uint64_t>(current_time) - kColdThreshold;
Add(0 /* level */, 4U /* file_number */, "260", "300", 1 * kFileSize, 0,
2500, 2600, 0, true, Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: threshold_time - 2000 /* oldest_ancestor_time */,
newestKeyTimeKnown ? threshold_time - 2000
: kUnknownNewestKeyTime /* newest_key_time */);
// Qualifies for compaction to kCold.
Add(0, 3U, "200", "300", 4 * kFileSize, 0, 2300, 2400, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 3000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 3000,
newestKeyTimeKnown ? threshold_time - 3000
: kUnknownNewestKeyTime /* newest_key_time */);
UpdateVersionStorageInfo();
@@ -1502,32 +1504,32 @@ TEST_F(CompactionPickerTest, FIFOToColdMaxCompactionSize) {
static_cast<uint64_t>(current_time) - kColdThreshold;
Add(0, 6U, "240", "290", 2 * kFileSize, 0, 2900, 3000, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: static_cast<uint64_t>(current_time) - 100,
newestKeyTimeKnown ? static_cast<uint64_t>(current_time) - 100
: kUnknownNewestKeyTime);
Add(0, 5U, "240", "290", 2 * kFileSize, 0, 2700, 2800, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time + 100,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time + 100,
newestKeyTimeKnown ? threshold_time + 100 : kUnknownNewestKeyTime);
Add(0, 4U, "260", "300", 1 * kFileSize, 0, 2500, 2600, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 2000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 2000,
newestKeyTimeKnown ? threshold_time - 2000 : kUnknownNewestKeyTime);
// The following two files qualify for compaction to kCold.
// But only the last two should be included to respect
// `max_compaction_bytes`.
Add(0, 3U, "200", "300", 4 * kFileSize, 0, 2300, 2400, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 3000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 3000,
newestKeyTimeKnown ? threshold_time - 3000 : kUnknownNewestKeyTime);
Add(0, 2U, "200", "300", 4 * kFileSize, 0, 2100, 2200, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 4000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 4000,
newestKeyTimeKnown ? threshold_time - 4000 : kUnknownNewestKeyTime);
Add(0, 1U, "200", "300", 4 * kFileSize, 0, 2000, 2100, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 5000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 5000,
newestKeyTimeKnown ? threshold_time - 5000 : kUnknownNewestKeyTime);
UpdateVersionStorageInfo();
@@ -1572,30 +1574,30 @@ TEST_F(CompactionPickerTest, FIFOToColdWithExistingCold) {
static_cast<uint64_t>(current_time) - kColdThreshold;
Add(0, 6U, "240", "290", 2 * kFileSize, 0, 2900, 3000, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: static_cast<uint64_t>(current_time) - 100,
newestKeyTimeKnown ? static_cast<uint64_t>(current_time) - 100
: kUnknownNewestKeyTime);
Add(0, 5U, "240", "290", 2 * kFileSize, 0, 2700, 2800, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time + 100,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time + 100,
newestKeyTimeKnown ? threshold_time + 100 : kUnknownNewestKeyTime);
Add(0, 4U, "260", "300", 1 * kFileSize, 0, 2500, 2600, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 2000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 2000,
newestKeyTimeKnown ? threshold_time - 2000 : kUnknownNewestKeyTime);
// The following two files qualify for compaction to kCold.
Add(0, 3U, "200", "300", 4 * kFileSize, 0, 2300, 2400, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 3000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 3000,
newestKeyTimeKnown ? threshold_time - 3000 : kUnknownNewestKeyTime);
Add(0, 2U, "200", "300", 4 * kFileSize, 0, 2100, 2200, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 4000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 4000,
newestKeyTimeKnown ? threshold_time - 4000 : kUnknownNewestKeyTime);
Add(0, 1U, "200", "300", 4 * kFileSize, 0, 2000, 2100, 0, true,
Temperature::kCold,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 5000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 5000,
newestKeyTimeKnown ? threshold_time - 5000 : kUnknownNewestKeyTime);
UpdateVersionStorageInfo();
@@ -1640,30 +1642,30 @@ TEST_F(CompactionPickerTest, FIFOToColdWithHotBetweenCold) {
static_cast<uint64_t>(current_time) - kColdThreshold;
Add(0, 6U, "240", "290", 2 * kFileSize, 0, 2900, 3000, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: static_cast<uint64_t>(current_time) - 100,
newestKeyTimeKnown ? static_cast<uint64_t>(current_time) - 100
: kUnknownNewestKeyTime);
Add(0, 5U, "240", "290", 2 * kFileSize, 0, 2700, 2800, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time + 100,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time + 100,
newestKeyTimeKnown ? threshold_time + 100 : kUnknownNewestKeyTime);
Add(0, 4U, "260", "300", 1 * kFileSize, 0, 2500, 2600, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 2000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 2000,
newestKeyTimeKnown ? threshold_time - 2000 : kUnknownNewestKeyTime);
Add(0, 3U, "200", "300", 4 * kFileSize, 0, 2300, 2400, 0, true,
Temperature::kCold,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 3000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 3000,
newestKeyTimeKnown ? threshold_time - 3000 : kUnknownNewestKeyTime);
// Qualifies for compaction to kCold.
Add(0, 2U, "200", "300", 4 * kFileSize, 0, 2100, 2200, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 4000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 4000,
newestKeyTimeKnown ? threshold_time - 4000 : kUnknownNewestKeyTime);
Add(0, 1U, "200", "300", 4 * kFileSize, 0, 2000, 2100, 0, true,
Temperature::kCold,
newestKeyTimeKnown ? kUnknownOldestAncesterTime : threshold_time - 5000,
newestKeyTimeKnown ? kUnknownOldestAncestorTime : threshold_time - 5000,
newestKeyTimeKnown ? threshold_time - 5000 : kUnknownNewestKeyTime);
UpdateVersionStorageInfo();
@@ -1712,36 +1714,36 @@ TEST_F(CompactionPickerTest, FIFOToHotAndWarm) {
static_cast<uint64_t>(current_time) - kWarmThreshold;
Add(0, 6U, "240", "290", 2 * kFileSize, 0, 2900, 3000, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: static_cast<uint64_t>(current_time) - 100,
newestKeyTimeKnown ? static_cast<uint64_t>(current_time) - 100
: kUnknownNewestKeyTime);
Add(0, 5U, "240", "290", 2 * kFileSize, 0, 2700, 2800, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: hot_threshold_time + 100,
newestKeyTimeKnown ? hot_threshold_time + 100 : kUnknownNewestKeyTime);
Add(0, 4U, "260", "300", 1 * kFileSize, 0, 2500, 2600, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: hot_threshold_time - 200,
newestKeyTimeKnown ? hot_threshold_time - 200 : kUnknownNewestKeyTime);
// Qualifies for Hot
Add(0, 3U, "200", "300", 4 * kFileSize, 0, 2300, 2400, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: warm_threshold_time - 100,
newestKeyTimeKnown ? warm_threshold_time - 100 : kUnknownNewestKeyTime);
// Qualifies for Warm
Add(0, 2U, "200", "300", 4 * kFileSize, 0, 2100, 2200, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: warm_threshold_time - 4000,
newestKeyTimeKnown ? warm_threshold_time - 4000
: kUnknownNewestKeyTime);
Add(0, 1U, "200", "300", 4 * kFileSize, 0, 2000, 2100, 0, true,
Temperature::kUnknown,
newestKeyTimeKnown ? kUnknownOldestAncesterTime
newestKeyTimeKnown ? kUnknownOldestAncestorTime
: warm_threshold_time - 5000,
newestKeyTimeKnown ? warm_threshold_time - 5000
: kUnknownNewestKeyTime);
@@ -2488,7 +2490,7 @@ TEST_F(CompactionPickerTest, FileTtlBoosterLargeNumLevels) {
FileTtlBooster booster(kCurrentTime, /*ttl=*/2048,
/*num_non_empty_levels=*/100, /*level=*/1);
FileMetaData meta;
meta.oldest_ancester_time = kCurrentTime - 1023;
meta.oldest_ancestor_time = kCurrentTime - 1023;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
}
@@ -2507,47 +2509,47 @@ TEST_F(CompactionPickerTest, FileTtlBooster) {
FileTtlBooster booster(kCurrentTime, 2048, 7, 3);
// Not triggering if the file is younger than ttl/2
meta.oldest_ancester_time = kCurrentTime - 1023;
meta.oldest_ancestor_time = kCurrentTime - 1023;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime - 1024;
meta.oldest_ancestor_time = kCurrentTime - 1024;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime + 10;
meta.oldest_ancestor_time = kCurrentTime + 10;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
// Within one boosting step
meta.oldest_ancester_time = kCurrentTime - (1024 + 120 + 6);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 120 + 6);
ASSERT_EQ(1, booster.GetBoostScore(&meta));
// One boosting step
meta.oldest_ancester_time = kCurrentTime - (1024 + 120 + 7);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 120 + 7);
ASSERT_EQ(2, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime - (1024 + 120 + 8);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 120 + 8);
ASSERT_EQ(2, booster.GetBoostScore(&meta));
// Multiple boosting steps
meta.oldest_ancester_time = kCurrentTime - (1024 + 120 + 30);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 120 + 30);
ASSERT_EQ(5, booster.GetBoostScore(&meta));
// Very high boosting steps
meta.oldest_ancester_time = kCurrentTime - (1024 + 120 + 700);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 120 + 700);
ASSERT_EQ(101, booster.GetBoostScore(&meta));
}
{
// Test second last level
FileTtlBooster booster(kCurrentTime, 2048, 7, 5);
meta.oldest_ancester_time = kCurrentTime - (1024 + 480);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 480);
ASSERT_EQ(1, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime - (1024 + 480 + 60);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 480 + 60);
ASSERT_EQ(3, booster.GetBoostScore(&meta));
}
{
// Test last level
FileTtlBooster booster(kCurrentTime, 2048, 7, 6);
meta.oldest_ancester_time = kCurrentTime - (1024 + 480);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 480);
ASSERT_EQ(1, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime - (1024 + 480 + 60);
meta.oldest_ancestor_time = kCurrentTime - (1024 + 480 + 60);
ASSERT_EQ(1, booster.GetBoostScore(&meta));
meta.oldest_ancester_time = kCurrentTime - 3000;
meta.oldest_ancestor_time = kCurrentTime - 3000;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
}
}
@@ -3575,21 +3577,21 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
Add(0, 1U, "150", "200", kFileSize, 0, 500, 550, /*compensated_file_size*/ 0,
/*marked_for_compact*/ false, /* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 3);
Add(0, 2U, "201", "250", 2 * kFileSize, 0, 401, 450,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 2);
Add(0, 4U, "260", "300", 4 * kFileSize, 0, 260, 300,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 1);
@@ -3617,7 +3619,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
// Simulate a flush and mark the file for compaction
Add(0, 7U, "150", "200", kFileSize, 0, 551, 600, 0, true,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 4);
@@ -3645,7 +3647,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap2) {
// Mark file number 4 for compaction
Add(0, 4U, "260", "300", 4 * kFileSize, 0, 260, 300, 0, true,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 1);
@@ -3672,14 +3674,14 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap2) {
AddVersionStorage();
Add(0, 1U, "150", "200", kFileSize, 0, 500, 550, /*compensated_file_size*/ 0,
/*marked_for_compact*/ false, /* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 3);
Add(0, 2U, "201", "250", 2 * kFileSize, 0, 401, 450,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 2);
@@ -3857,27 +3859,27 @@ TEST_F(CompactionPickerTest, UniversalMarkedL0Overlap2) {
Add(0, 4U, "260", "300", 1 * kFileSize, 0, 260, 300,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 4);
Add(0, 5U, "240", "290", 2 * kFileSize, 0, 201, 250, 0, true,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 3);
Add(0, 3U, "301", "350", 4 * kFileSize, 0, 101, 150,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 2);
Add(0, 6U, "501", "750", 8 * kFileSize, 0, 50, 100,
/*compensated_file_size*/ 0, /*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 1);
@@ -3904,14 +3906,14 @@ TEST_F(CompactionPickerTest, UniversalMarkedL0Overlap2) {
Add(0, 1U, "150", "200", kFileSize, 0, 500, 550, /*compensated_file_size*/ 0,
/*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 6);
Add(0, 2U, "201", "250", kFileSize, 0, 401, 450, /*compensated_file_size*/ 0,
/*marked_for_compact*/ false,
/* temperature*/ Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime,
/*ts_of_smallest*/ Slice(), /*ts_of_largest*/ Slice(),
/*epoch_number*/ 5);
@@ -4147,7 +4149,7 @@ TEST_F(CompactionPickerU64TsTest, Overlap) {
/*file_size=*/1U, /*path_id=*/0,
/*smallest_seq=*/100, /*largest_seq=*/100, /*compensated_file_size=*/0,
/*marked_for_compact=*/false, /*temperature=*/Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime, ts1, ts2);
UpdateVersionStorageInfo();
}
@@ -4162,6 +4164,18 @@ TEST_F(CompactionPickerU64TsTest, Overlap) {
CompactionOptions(), input_files, level, vstorage_.get(),
mutable_cf_options_, mutable_db_options_, /*output_path_id=*/0));
{
std::string range_start("400");
PutFixed64(&range_start, ts_of_smallest);
std::string range_limit(smallest);
PutFixed64(&range_limit, ts_of_smallest);
ASSERT_FALSE(level_compaction_picker.RangeOverlapWithCompaction(
range_start, range_limit, level, /*range_limit_exclusive=*/true));
ASSERT_TRUE(level_compaction_picker.RangeOverlapWithCompaction(
range_start, range_limit, level,
/*range_limit_exclusive=*/false));
}
{
// [600, ts=50000] to [600, ts=50000] is the range to check.
// ucmp->Compare(smallest_user_key, c->GetLargestUserKey()) > 0, but
@@ -4173,7 +4187,8 @@ TEST_F(CompactionPickerU64TsTest, Overlap) {
std::string user_key_with_ts2(largest);
PutFixed64(&user_key_with_ts2, ts_of_largest - 1);
ASSERT_TRUE(level_compaction_picker.RangeOverlapWithCompaction(
user_key_with_ts1, user_key_with_ts2, level));
user_key_with_ts1, user_key_with_ts2, level,
/*range_limit_exclusive=*/false));
}
{
// [500, ts=60000] to [500, ts=60000] is the range to check.
@@ -4186,7 +4201,8 @@ TEST_F(CompactionPickerU64TsTest, Overlap) {
std::string user_key_with_ts2(smallest);
PutFixed64(&user_key_with_ts2, ts_of_smallest + 1);
ASSERT_TRUE(level_compaction_picker.RangeOverlapWithCompaction(
user_key_with_ts1, user_key_with_ts2, level));
user_key_with_ts1, user_key_with_ts2, level,
/*range_limit_exclusive=*/false));
}
}
@@ -4215,12 +4231,12 @@ TEST_F(CompactionPickerU64TsTest, CannotTrivialMoveUniversal) {
Add(1, 1U, "150", "150", kFileSize, /*path_id=*/0, /*smallest_seq=*/100,
/*largest_seq=*/100, /*compensated_file_size=*/kFileSize,
/*marked_for_compact=*/false, Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime, ts1, ts2);
Add(2, 2U, "150", "150", kFileSize, /*path_id=*/0, /*smallest_seq=*/100,
/*largest_seq=*/100, /*compensated_file_size=*/kFileSize,
/*marked_for_compact=*/false, Temperature::kUnknown,
/*oldest_ancestor_time*/ kUnknownOldestAncesterTime,
/*oldest_ancestor_time*/ kUnknownOldestAncestorTime,
/*newest_key_time*/ kUnknownNewestKeyTime, ts3, ts4);
UpdateVersionStorageInfo();
@@ -5017,12 +5033,12 @@ TEST_F(CompactionPickerTest, StandaloneRangeDeletionOnlyPicksOlderFiles) {
// Create L0 files with overlapping ranges
// File 1: newest regular file (epoch 5), keys [100, 200]
Add(0, 1U, "100", "200", 1U, 0, 100, 100, 0, false, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, Slice(), Slice(), 5);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, Slice(), Slice(), 5);
// File 2: standalone range deletion (epoch 4), keys [150, 250]
// This file should be marked as having only range deletions
Add(0, 2U, "150", "250", 1U, 0, 200, 200, 0, true, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, Slice(), Slice(), 4);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, Slice(), Slice(), 4);
// Manually set file 2 as standalone range deletion
FileMetaData* range_del_file = file_map_[2U].first;
@@ -5058,6 +5074,41 @@ TEST_F(CompactionPickerTest, StandaloneRangeDeletionOnlyPicksOlderFiles) {
ASSERT_EQ(10U, compaction->input(1, 0)->fd.GetNumber());
}
// Tests for the seqno->time preserve window controlling bottommost file
// marking. A bottommost file whose largest seqno is still within the preserve
// window cannot have its seqno zeroed, so it must not be marked (otherwise the
// rewrite makes no progress and the file is re-marked forever -- an infinite
// loop).
TEST_F(CompactionPickerTest, BottommostFileNotMarkedWithinPreserveWindow) {
NewVersionStorage(6, kCompactionStyleLevel);
Add(5, 1U, "100", "200", /*file_size=*/1000, /*path_id=*/0,
/*smallest_seq=*/90, /*largest_seq=*/100);
// preserve_time_min_seqno = 100 => preserve_seqno_after = 99; largest_seqno
// 100 is within the window (> 99) and cannot be zeroed.
vstorage_->SetPreserveTimeMinSeqno(100);
UpdateVersionStorageInfo();
vstorage_->UpdateOldestSnapshot(/*oldest_snapshot_seqnum=*/200,
/*allow_ingest_behind=*/false, /*ucmp=*/ucmp_,
/*full_history_ts_low=*/"");
ASSERT_EQ(0U, vstorage_->BottommostFilesMarkedForCompaction().size());
}
TEST_F(CompactionPickerTest, BottommostFileMarkedBelowPreserveWindow) {
NewVersionStorage(6, kCompactionStyleLevel);
Add(5, 1U, "100", "200", /*file_size=*/1000, /*path_id=*/0,
/*smallest_seq=*/90, /*largest_seq=*/100);
// preserve_time_min_seqno = 150 => preserve_seqno_after = 149; largest_seqno
// 100 <= 149 so it can be zeroed and the file is marked as usual.
vstorage_->SetPreserveTimeMinSeqno(150);
UpdateVersionStorageInfo();
vstorage_->UpdateOldestSnapshot(/*oldest_snapshot_seqnum=*/200,
/*allow_ingest_behind=*/false, /*ucmp=*/ucmp_,
/*full_history_ts_low=*/"");
ASSERT_EQ(1U, vstorage_->BottommostFilesMarkedForCompaction().size());
ASSERT_EQ(1U, vstorage_->BottommostFilesMarkedForCompaction()[0]
.second->fd.GetNumber());
}
// Tests for full_history_ts_low parameter in compaction picker.
// The full_history_ts_low parameter is used to control bottommost file marking
// for compaction when user-defined timestamps (UDT) are enabled.
@@ -5118,7 +5169,7 @@ TEST_F(CompactionPickerU64TsTest,
/*smallest_seq=*/10, /*largest_seq=*/40,
/*compensated_file_size=*/1000,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncesterTime, kUnknownNewestKeyTime, ts_small, ts_large);
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts_small, ts_large);
// Update version storage with empty full_history_ts_low
UpdateVersionStorageInfo();
@@ -5137,6 +5188,90 @@ TEST_F(CompactionPickerU64TsTest,
ASSERT_EQ(0U, vstorage_->BottommostFilesMarkedForCompaction().size());
}
TEST_F(CompactionPickerU64TsTest,
BottommostNotMarkedWithEmptyFullHistoryTsLowAndUnknownMaxTs) {
// Regression test for an infinite bottommost compaction loop. A bottommost
// UDT file whose max_timestamp was not recorded in its FileMetaData (e.g. a
// file written by an older version, or one whose timestamp property is
// otherwise absent) must NOT be marked for compaction when
// full_history_ts_low is unset. Under UDT the only progress a bottommost
// compaction can make here is zeroing sequence numbers, which
// CompactionIterator does only when full_history_ts_low is set. With it unset
// the compaction cannot advance the file, so the rewritten output
// re-qualifies forever.
NewVersionStorage(6, kCompactionStyleLevel);
std::string ts_small = MakeU64Timestamp(50);
std::string ts_large = MakeU64Timestamp(100);
// Add a bottommost file with seqno < oldest_snapshot.
Add(5, 1U, "100", "200", /*file_size=*/1000, /*path_id=*/0,
/*smallest_seq=*/10, /*largest_seq=*/40,
/*compensated_file_size=*/1000,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts_small, ts_large);
// Simulate a file whose min/max timestamp was never recorded.
ASSERT_EQ(1U, vstorage_->LevelFiles(5).size());
FileMetaData* f = vstorage_->LevelFiles(5)[0];
f->min_timestamp.clear();
f->max_timestamp.clear();
UpdateVersionStorageInfo();
vstorage_->UpdateOldestSnapshot(
/*oldest_snapshot_seqnum=*/50,
/*allow_ingest_behind=*/false,
/*ucmp=*/ucmp_,
/*full_history_ts_low=*/"");
// No bottommost file can make progress with empty full_history_ts_low, so
// nothing is marked even though max_timestamp is unknown. Before the fix this
// file was marked, producing an infinite compaction loop.
ASSERT_EQ(0U, vstorage_->BottommostFilesMarkedForCompaction().size());
}
TEST_F(CompactionPickerU64TsTest,
BottommostMarkedWithSetFullHistoryTsLowAndUnknownMaxTs) {
// A bottommost UDT file whose max_timestamp was not recorded (e.g. written by
// an older version) must still be marked for a one-time backfill compaction
// when full_history_ts_low IS set: that compaction can collapse timestamp
// history / zero seqnos and record max_timestamp, so it makes progress
// (bounded, not a loop). This complements
// BottommostNotMarkedWithEmptyFullHistoryTsLowAndUnknownMaxTs, which covers
// the unset case where no progress is possible.
NewVersionStorage(6, kCompactionStyleLevel);
std::string ts_small = MakeU64Timestamp(50);
std::string ts_large = MakeU64Timestamp(100);
Add(5, 1U, "100", "200", /*file_size=*/1000, /*path_id=*/0,
/*smallest_seq=*/10, /*largest_seq=*/40,
/*compensated_file_size=*/1000,
/*marked_for_compact=*/false, Temperature::kUnknown,
kUnknownOldestAncestorTime, kUnknownNewestKeyTime, ts_small, ts_large);
// Simulate a file whose min/max timestamp was never recorded.
ASSERT_EQ(1U, vstorage_->LevelFiles(5).size());
FileMetaData* f = vstorage_->LevelFiles(5)[0];
f->min_timestamp.clear();
f->max_timestamp.clear();
UpdateVersionStorageInfo();
std::string full_history_ts_low = MakeU64Timestamp(500);
vstorage_->UpdateOldestSnapshot(
/*oldest_snapshot_seqnum=*/50,
/*allow_ingest_behind=*/false,
/*ucmp=*/ucmp_, full_history_ts_low);
// With full_history_ts_low set, the unknown-max_timestamp file is marked so a
// bounded compaction can backfill its metadata and reclaim obsolete data.
ASSERT_EQ(1U, vstorage_->BottommostFilesMarkedForCompaction().size());
ASSERT_EQ(1U, vstorage_->BottommostFilesMarkedForCompaction()[0]
.second->fd.GetNumber());
}
TEST_F(CompactionPickerU64TsTest, LevelPickCompactionWithFullHistoryTsLow) {
// Test that level compaction correctly passes full_history_ts_low
// and picks compaction appropriately
@@ -5580,13 +5715,13 @@ TEST_F(CompactionPickerTest, FIFOTTLBlobEstimationSingleLevel) {
uint64_t recent_time = static_cast<uint64_t>(time(nullptr));
Add(0, 1U, "100", "200", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, recent_time);
Temperature::kUnknown, kUnknownOldestAncestorTime, recent_time);
Add(0, 2U, "200", "300", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, recent_time);
Temperature::kUnknown, kUnknownOldestAncestorTime, recent_time);
Add(0, 3U, "300", "400", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, 1);
Temperature::kUnknown, kUnknownOldestAncestorTime, 1);
Add(0, 4U, "400", "500", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, 1);
Temperature::kUnknown, kUnknownOldestAncestorTime, 1);
if (blob_total > 0) {
AddBlobFile(100, blob_total / 2);
AddBlobFile(101, blob_total / 2);
@@ -5659,13 +5794,13 @@ TEST_F(CompactionPickerTest, FIFOTTLBlobEstimationMultiLevel) {
uint64_t recent_time = static_cast<uint64_t>(time(nullptr));
// L0 files: 2 recent, 2 expired
Add(0, 1U, "100", "200", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, recent_time);
Temperature::kUnknown, kUnknownOldestAncestorTime, recent_time);
Add(0, 2U, "200", "300", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, recent_time);
Temperature::kUnknown, kUnknownOldestAncestorTime, recent_time);
Add(0, 3U, "300", "400", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, 1);
Temperature::kUnknown, kUnknownOldestAncestorTime, 1);
Add(0, 4U, "400", "500", 50 * 1024, 0, 100, 100, 0, false,
Temperature::kUnknown, kUnknownOldestAncesterTime, 1);
Temperature::kUnknown, kUnknownOldestAncestorTime, 1);
// L2 legacy migration file
Add(2, 10U, "100", "600", 200 * 1024);
// Blob files (associated with ALL levels)
+30 -23
View File
@@ -50,6 +50,8 @@ CompactionJob::ProcessKeyValueCompactionWithCompactionService(
compaction_input.end =
compaction_input.has_end ? sub_compact->end->ToString() : "";
compaction_input.options_file_number = options_file_number_;
compaction_input.min_manifest_file_number = min_manifest_file_number_;
compaction_input.min_manifest_file_size = min_manifest_file_size_;
TEST_SYNC_POINT_CALLBACK(
"CompactionServiceJob::ProcessKeyValueCompactionWithCompactionService",
@@ -82,7 +84,6 @@ CompactionJob::ProcessKeyValueCompactionWithCompactionService(
compaction->is_full_compaction(), compaction->is_manual_compaction(),
compaction->bottommost_level(), compaction->start_level(),
compaction->output_level());
CompactionServiceScheduleResponse response =
db_options_.compaction_service->Schedule(info, compaction_input_binary);
switch (response.status) {
@@ -270,7 +271,7 @@ CompactionJob::ProcessKeyValueCompactionWithCompactionService(
file.smallest_seqno, file.largest_seqno);
meta.smallest.DecodeFrom(file.smallest_internal_key);
meta.largest.DecodeFrom(file.largest_internal_key);
meta.oldest_ancester_time = file.oldest_ancester_time;
meta.oldest_ancestor_time = file.oldest_ancestor_time;
meta.file_creation_time = file.file_creation_time;
meta.epoch_number = file.epoch_number;
meta.file_checksum = file.file_checksum;
@@ -468,7 +469,7 @@ Status CompactionServiceCompactionJob::Run() {
MakeTableFileName(meta.fd.GetNumber()), meta.fd.GetFileSize(),
meta.fd.smallest_seqno, meta.fd.largest_seqno,
meta.smallest.Encode().ToString(), meta.largest.Encode().ToString(),
meta.oldest_ancester_time, meta.file_creation_time, meta.epoch_number,
meta.oldest_ancestor_time, meta.file_creation_time, meta.epoch_number,
meta.file_checksum, meta.file_checksum_func_name,
output_file.validator.GetHash(), meta.marked_for_compaction,
meta.unique_id, *output_file.table_properties,
@@ -564,6 +565,14 @@ static std::unordered_map<std::string, OptionTypeInfo> cs_input_type_info = {
{offsetof(struct CompactionServiceInput, options_file_number),
OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"min_manifest_file_number",
{offsetof(struct CompactionServiceInput, min_manifest_file_number),
OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"min_manifest_file_size",
{offsetof(struct CompactionServiceInput, min_manifest_file_size),
OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
};
static std::unordered_map<std::string, OptionTypeInfo>
@@ -592,8 +601,9 @@ static std::unordered_map<std::string, OptionTypeInfo>
{offsetof(struct CompactionServiceOutputFile, largest_internal_key),
OptionType::kEncodedString, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
// Misspelled key kept for compatibility with other versions
{"oldest_ancester_time",
{offsetof(struct CompactionServiceOutputFile, oldest_ancester_time),
{offsetof(struct CompactionServiceOutputFile, oldest_ancestor_time),
OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"file_creation_time",
@@ -911,24 +921,11 @@ static std::unordered_map<std::string, OptionTypeInfo>
{offsetof(struct InternalStats::CompactionStats, count),
OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
// The remote worker may serialize a different number of
// CompactionReason counts than this primary expects (RocksDB versions
// can disagree on kNumOfReasons). Deserialization tolerates the
// mismatch: extra trailing elements are dropped, missing ones are
// zero-filled.
//
// The serialized width is intentionally kept at kNumOfReasons - 1 (the
// reduced width) so this change does NOT alter what is written -- only
// the read path becomes tolerant. Widening the serialized width to the
// full kNumOfReasons is a separate, later change that must not land
// until this tolerant read has been deployed to every primary,
// including rollback targets.
{"counts",
SizeTolerantDeserializeArray<
int, static_cast<int>(CompactionReason::kNumOfReasons) - 1>(
offsetof(struct InternalStats::CompactionStats, counts),
OptionVerificationType::kNormal, OptionTypeFlags::kNone,
{0, OptionType::kInt})},
{"counts", SizeTolerantDeserializeArray<
int, static_cast<int>(CompactionReason::kNumOfReasons)>(
offsetof(struct InternalStats::CompactionStats, counts),
OptionVerificationType::kNormal, OptionTypeFlags::kNone,
{0, OptionType::kInt})},
};
static std::unordered_map<std::string, OptionTypeInfo>
@@ -1075,9 +1072,19 @@ Status CompactionServiceInput::Read(const std::string& data_str,
ConfigOptions cf;
cf.invoke_prepare_options = false;
cf.ignore_unknown_options = true;
return OptionTypeInfo::ParseType(
Status s = OptionTypeInfo::ParseType(
cf, data_str.substr(sizeof(BinaryFormatVersion)), cs_input_type_info,
obj);
if (!s.ok()) {
return s;
}
for (size_t i = 1; i < obj->snapshots.size(); ++i) {
if (obj->snapshots[i - 1] >= obj->snapshots[i]) {
return Status::InvalidArgument(
"CompactionServiceInput snapshots must be strictly increasing");
}
}
return Status::OK();
} else {
return Status::NotSupported(
"Compaction Service Input data version not supported: " +
+805 -29
View File
@@ -5,9 +5,12 @@
#include <atomic>
#include <memory>
#include <string>
#include <vector>
#include "db/db_test_util.h"
#include "file/file_util.h"
#include "file/filename.h"
#include "port/stack_trace.h"
#include "rocksdb/advanced_compression.h"
#include "rocksdb/filter_policy.h"
@@ -81,6 +84,82 @@ size_t CountFirstCountsFieldElements(const std::string& s) {
return count;
}
// Simulates a filesystem without POSIX read-after-unlink semantics (a remote
// or object store): once the primary renames a freshly written CURRENT over
// the old one, a reader that already opened the previous CURRENT object can no
// longer read it and the filesystem reports the path as gone rather than
// serving the unlinked object. Fails the first `fail_times` reads of a CURRENT
// file that way; later reads succeed, as they do once the reader picks up the
// replacement.
class VanishingCurrentFileFS : public FileSystemWrapper {
public:
explicit VanishingCurrentFileFS(const std::shared_ptr<FileSystem>& base)
: FileSystemWrapper(base) {}
static const char* kClassName() { return "VanishingCurrentFileFS"; }
const char* Name() const override { return kClassName(); }
IOStatus NewSequentialFile(const std::string& fname, const FileOptions& opts,
std::unique_ptr<FSSequentialFile>* result,
IODebugContext* dbg) override {
IOStatus io_s = target()->NewSequentialFile(fname, opts, result, dbg);
if (io_s.ok() && IsCurrentFileName(fname)) {
*result = std::make_unique<MaybeVanishedFile>(std::move(*result), this);
}
return io_s;
}
void SetFailuresToInject(int failures) {
failures_to_inject_.store(failures);
}
int injected_failures() const { return injected_failures_.load(); }
private:
static bool IsCurrentFileName(const std::string& fname) {
const std::string suffix = "/" + kCurrentFileName;
return fname.size() > suffix.size() &&
fname.compare(fname.size() - suffix.size(), suffix.size(), suffix) ==
0;
}
// Returns true (consuming one injection) while failures remain.
bool ConsumeFailure() {
int remaining = failures_to_inject_.load();
while (remaining > 0) {
if (failures_to_inject_.compare_exchange_weak(remaining, remaining - 1)) {
injected_failures_.fetch_add(1);
return true;
}
}
return false;
}
class MaybeVanishedFile : public FSSequentialFileOwnerWrapper {
public:
MaybeVanishedFile(std::unique_ptr<FSSequentialFile>&& target,
VanishingCurrentFileFS* fs)
: FSSequentialFileOwnerWrapper(std::move(target)), fs_(fs) {}
IOStatus Read(size_t n, const IOOptions& options, Slice* result,
char* scratch, IODebugContext* dbg) override {
if (fs_->ConsumeFailure()) {
return IOStatus::PathNotFound(
"Failed to read CURRENT: the object was replaced and is no longer "
"readable");
}
return FSSequentialFileOwnerWrapper::Read(n, options, result, scratch,
dbg);
}
private:
VanishingCurrentFileFS* fs_;
};
std::atomic<int> failures_to_inject_{0};
std::atomic<int> injected_failures_{0};
};
} // namespace
class MyTestCompactionService : public CompactionService {
@@ -113,6 +192,7 @@ class MyTestCompactionService : public CompactionService {
start_info_ = info;
assert(info.db_name == db_path_);
std::string unique_id = Env::Default()->GenerateUniqueId();
last_output_directory_name_ = unique_id;
jobs_.emplace(unique_id, compaction_service_input);
infos_.emplace(unique_id, info);
CompactionServiceScheduleResponse response(
@@ -149,10 +229,10 @@ class MyTestCompactionService : public CompactionService {
OpenAndCompactOptions options;
options.canceled = &canceled_;
options.max_secondary_open_retries = max_secondary_open_retries_;
Status s =
DB::OpenAndCompact(options, db_path_, GetOutputPath(scheduled_job_id),
compaction_input, result, options_override);
Status s = DB::OpenAndCompact(options, db_path_, scheduled_job_id,
compaction_input, result, options_override);
{
InstrumentedMutexLock l(&mutex_);
if (is_override_wait_result_) {
@@ -163,6 +243,14 @@ class MyTestCompactionService : public CompactionService {
compaction_num_.fetch_add(1);
if (s.ok()) {
return CompactionServiceJobStatus::kSuccess;
} else if (s.IsIncomplete() && !s.IsManualCompactionPaused()) {
// The remote worker declined the job (e.g. the MANIFEST floor check
// rejected an older/stale filesystem view). Mapping that decline to
// kUseLocal (retry locally) vs. kFailure (surface it) is the integrator's
// choice; this example picks kUseLocal to exercise the fallback path,
// while DbStressCompactionService deliberately picks kFailure. Real
// cancellation (kManualCompactionPaused) is handled as a failure.
return CompactionServiceJobStatus::kUseLocal;
} else {
return CompactionServiceJobStatus::kFailure;
}
@@ -204,6 +292,11 @@ class MyTestCompactionService : public CompactionService {
int GetCompactionNum() { return compaction_num_.load(); }
std::string GetLastOutputDirectoryName() {
InstrumentedMutexLock l(&mutex_);
return last_output_directory_name_;
}
CompactionServiceJobInfo GetCompactionInfoForStart() { return start_info_; }
CompactionServiceJobInfo GetCompactionInfoForWait() { return wait_info_; }
@@ -231,6 +324,10 @@ class MyTestCompactionService : public CompactionService {
void SetCanceled(bool canceled) { canceled_ = canceled; }
bool GetCanceled() { return canceled_; }
void SetMaxSecondaryOpenRetries(uint32_t retries) {
max_secondary_open_retries_ = retries;
}
void GetResult(CompactionServiceResult* deserialized) {
CompactionServiceResult::Read(result_, deserialized).PermitUncheckedError();
}
@@ -248,9 +345,12 @@ class MyTestCompactionService : public CompactionService {
std::map<std::string, std::string> jobs_;
std::map<std::string, CompactionServiceJobInfo> infos_;
std::string result_;
std::string last_output_directory_name_;
std::string GetOutputPath(const std::string& scheduled_job_id) {
return db_path_ + "/" + scheduled_job_id;
return RemoteCompactionJobDir(
db_path_, options_.use_session_tmp_dir_for_remote_compaction,
scheduled_job_id);
}
private:
@@ -272,6 +372,8 @@ class MyTestCompactionService : public CompactionService {
std::atomic_int installation_callback_count_{0};
std::atomic<CompactionServiceJobStatus> final_updated_status_{
CompactionServiceJobStatus::kUseLocal};
uint32_t max_secondary_open_retries_ =
OpenAndCompactOptions().max_secondary_open_retries;
protected:
std::atomic_bool canceled_{false};
@@ -399,8 +501,12 @@ TEST_F(CompactionServiceTest, BasicCompactions) {
ASSERT_EQ(primary_statistics->getTickerCount(REMOTE_COMPACT_WRITE_BYTES),
compactor_statistics->getTickerCount(COMPACT_WRITE_BYTES));
ASSERT_GE(primary_statistics->getTickerCount(REMOTE_COMPACT_READ_BYTES), 1);
ASSERT_GT(primary_statistics->getTickerCount(COMPACT_READ_BYTES),
primary_statistics->getTickerCount(REMOTE_COMPACT_READ_BYTES));
// The remote worker reads the full compaction inputs on demand, while the
// primary only re-reads the (smaller) output files for verify_table(), so
// remote read bytes dominate. (The worker no longer eagerly opens every file
// during secondary recovery, so its input reads are attributed accurately.)
ASSERT_GT(primary_statistics->getTickerCount(REMOTE_COMPACT_READ_BYTES),
primary_statistics->getTickerCount(COMPACT_READ_BYTES));
// compactor is already the remote side, which doesn't have remote
ASSERT_EQ(compactor_statistics->getTickerCount(REMOTE_COMPACT_READ_BYTES), 0);
ASSERT_EQ(compactor_statistics->getTickerCount(REMOTE_COMPACT_WRITE_BYTES),
@@ -438,6 +544,27 @@ TEST_F(CompactionServiceTest, BasicCompactions) {
}
ASSERT_TRUE(s.IsAborted());
CompactionServiceResult result;
my_cs->GetResult(&result);
ASSERT_NOK(result.status);
// The aborted job's output files were never installed, so they are still in
// the job's staging directory. Size them before reopening: the reopen starts
// a new DB session, and DB::Open() deletes staging directories belonging to
// any other session.
uint64_t total_size = 0;
Env* const env = options.env;
ASSERT_TRUE(options.env != nullptr);
for (const auto& output_file : result.output_files) {
std::string file_name = result.output_path + "/" + output_file.file_name;
uint64_t file_size = 0;
ASSERT_OK(env->GetFileSize(file_name, &file_size));
ASSERT_GT(file_size, 0);
total_size += file_size;
}
ASSERT_EQ(total_size, result.internal_stats.TotalBytesWritten());
// Test re-open and successful unique id verification
std::atomic_int verify_passed{0};
SyncPoint::GetInstance()->SetCallBack(
@@ -452,13 +579,6 @@ TEST_F(CompactionServiceTest, BasicCompactions) {
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "cf_1", "cf_2", "cf_3"},
options);
ASSERT_GT(verify_passed, 0);
CompactionServiceResult result;
my_cs->GetResult(&result);
if (s.IsAborted()) {
ASSERT_NOK(result.status);
} else {
ASSERT_OK(result.status);
}
ASSERT_GE(result.internal_stats.output_level_stats.micros, 1);
ASSERT_GE(result.internal_stats.output_level_stats.cpu_micros, 1);
@@ -466,17 +586,6 @@ TEST_F(CompactionServiceTest, BasicCompactions) {
ASSERT_EQ(result.output_files.size(),
result.internal_stats.output_level_stats.num_output_files);
uint64_t total_size = 0;
for (auto output_file : result.output_files) {
std::string file_name = result.output_path + "/" + output_file.file_name;
uint64_t file_size = 0;
ASSERT_OK(options.env->GetFileSize(file_name, &file_size));
ASSERT_GT(file_size, 0);
total_size += file_size;
}
ASSERT_EQ(total_size, result.internal_stats.TotalBytesWritten());
ASSERT_TRUE(result.stats.is_remote_compaction);
ASSERT_TRUE(result.stats.is_manual_compaction);
ASSERT_FALSE(result.stats.is_full_compaction);
@@ -1791,7 +1900,7 @@ TEST_F(CompactionServiceTest, InvalidResultFallsBackToLocal) {
TEST_F(CompactionServiceTest, CompatCheckCountsWidthTolerance) {
// Number of per-CompactionReason counts this binary serializes and expects.
constexpr size_t kExpected =
static_cast<size_t>(CompactionReason::kNumOfReasons) - 1;
static_cast<size_t>(CompactionReason::kNumOfReasons);
constexpr size_t kLastIndex = kExpected - 1;
constexpr int kFirstCount = 3;
constexpr int kLastCount = 5;
@@ -2169,6 +2278,498 @@ TEST_F(CompactionServiceTest, ConcurrentCompaction) {
ASSERT_EQ(FilesPerLevel(), "0,0,10");
}
TEST_F(CompactionServiceTest, RemoteCompactionMissingLowerFileTrustsManifest) {
// A remote compaction reconstructs its view of the LSM (which files sit below
// the output level) from the secondary's recovered MANIFEST version. That
// view drives correctness-critical decisions like whether a deletion
// tombstone can be dropped (Compaction::KeyNotExistsBeyondOutputLevel /
// IsBottommostLevel, which read input_version_'s VersionStorageInfo).
//
// If a non-input file that is still referenced by the MANIFEST is momentarily
// missing (as on a live primary), the OLD point-in-time recovery rolls back
// to an earlier version. Here that rollback drops the (later, whole-DB)
// manifest record entirely, so the remote worker can no longer find the
// compaction input and the job fails ("cannot find matched SST files") --
// i.e. it fails safe rather than mis-compacting. The fix makes recovery TRUST
// the manifest: it keeps the current version, the job completes, and the
// deleted key correctly stays deleted (the below-output tombstone is
// retained).
Options options = CurrentOptions();
options.num_levels = 7;
// -1 keeps the primary's descriptor for the (unlinked) lower file valid, so
// the final read below is served from the primary regardless of the deletion.
options.max_open_files = -1;
options.level0_file_num_compaction_trigger = 100; // no auto compaction
ReopenWithCompactionService(&options);
// Old value for "k", pushed down to L2 (below the compaction output level).
ASSERT_OK(Put("k", "OLD"));
ASSERT_OK(Flush());
MoveFilesToLevel(2); // L2 file contains k=OLD
// Newer tombstone for "k" at L0 (the compaction input).
ASSERT_OK(Delete("k"));
ASSERT_OK(Flush()); // L0 file contains Delete(k)
// Reopen so the MANIFEST is rewritten as current state before we perturb it.
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "cf_1", "cf_2", "cf_3"},
options);
// Closing on reopen cancels pending remote jobs; re-arm the service.
GetCompactionService()->SetCanceled(false);
ColumnFamilyMetaData meta;
db_->GetColumnFamilyMetaData(&meta); // default CF
ASSERT_EQ(meta.levels[0].files.size(), 1);
ASSERT_EQ(meta.levels[2].files.size(), 1);
const auto& f_del = meta.levels[0].files[0];
const auto& f_old = meta.levels[2].files[0];
// Make the lower (L2) file momentarily unavailable to a fresh open, as a live
// primary would if it had compacted it away. It is NOT a compaction input.
ASSERT_OK(env_->DeleteFile(f_old.db_path + "/" + f_old.name));
// Remote-compact the L0 tombstone into L1. L2 (with k=OLD) is below the
// output level, so a correct compaction must retain the tombstone. Without
// the fix, recovery rolls back and this job fails; with the fix it completes.
std::string fname = f_del.db_path + "/" + f_del.name;
ASSERT_OK(db_->CompactFiles(CompactionOptions(), {fname}, 1));
// Deleted key stays deleted (tombstone retained above the still-present L2
// value, which the primary can still read via its open descriptor).
ASSERT_EQ("NOT_FOUND", Get("k"));
}
TEST_F(CompactionServiceTest, RemoteCompactionManifestFloorSendsPosition) {
// By default (remote_compaction_manifest_floor=true), the primary attaches a
// non-zero MANIFEST floor to every remote compaction request, and the worker
// runs its floor check. Here the worker recovers the same-or-later MANIFEST
// position the primary scheduled from, so the check accepts (the
// later-manifest positive case).
Options options = CurrentOptions();
options.disable_auto_compactions = true;
ReopenWithCompactionService(&options);
ASSERT_TRUE(db_->GetDBOptions().remote_compaction_manifest_floor);
GenerateTestData();
uint64_t floor_number = 0;
uint64_t floor_size = 0;
int floor_check_count = 0;
bool floor_check_rejected = false;
SyncPoint::GetInstance()->SetCallBack(
"CompactionServiceJob::ProcessKeyValueCompactionWithCompactionService",
[&](void* arg) {
auto* input = static_cast<CompactionServiceInput*>(arg);
floor_number = input->min_manifest_file_number;
floor_size = input->min_manifest_file_size;
});
SyncPoint::GetInstance()->SetCallBack(
"DBImplSecondary::OpenAndCompact::ManifestFloorCheck", [&](void* arg) {
floor_check_count++;
if (*static_cast<bool*>(arg)) {
floor_check_rejected = true;
}
});
SyncPoint::GetInstance()->EnableProcessing();
std::string start_str = Key(15);
std::string end_str = Key(45);
Slice start(start_str);
Slice end(end_str);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), &start, &end));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
// A floor was populated and the worker accepted its (later-or-equal) view.
ASSERT_GT(floor_number, 0U);
ASSERT_GT(floor_size, 0U);
ASSERT_GE(floor_check_count, 1);
ASSERT_FALSE(floor_check_rejected);
VerifyTestData();
}
TEST_F(CompactionServiceTest, RemoteCompactionManifestFloorRejectsStaleView) {
// If the worker recovers an OLDER MANIFEST view than the primary scheduled
// from (simulated here by inflating the floor above the real position), the
// floor check rejects the job and the primary falls back to a correct local
// compaction rather than compacting against a stale/older LSM shape.
Options options = CurrentOptions();
options.num_levels = 7;
options.level0_file_num_compaction_trigger = 100; // no auto compaction
ReopenWithCompactionService(&options);
// Old value for "k", pushed down to L2 (below the compaction output level).
ASSERT_OK(Put("k", "OLD"));
ASSERT_OK(Flush());
MoveFilesToLevel(2); // L2: k=OLD
// Newer tombstone for "k" at L0 (the compaction input).
ASSERT_OK(Delete("k"));
ASSERT_OK(Flush()); // L0: Delete(k)
int floor_check_rejected = 0;
SyncPoint::GetInstance()->SetCallBack(
"CompactionServiceJob::ProcessKeyValueCompactionWithCompactionService",
[&](void* arg) {
auto* input = static_cast<CompactionServiceInput*>(arg);
// Simulate the worker seeing an older MANIFEST than the primary
// scheduled from by raising the floor above the real position.
input->min_manifest_file_number += 1000000;
});
SyncPoint::GetInstance()->SetCallBack(
"DBImplSecondary::OpenAndCompact::ManifestFloorCheck", [&](void* arg) {
if (*static_cast<bool*>(arg)) {
floor_check_rejected++;
}
});
SyncPoint::GetInstance()->EnableProcessing();
ColumnFamilyMetaData meta;
db_->GetColumnFamilyMetaData(&meta); // default CF
ASSERT_EQ(meta.levels[0].files.size(), 1);
const auto& f_del = meta.levels[0].files[0];
std::string fname = f_del.db_path + "/" + f_del.name;
// The worker rejects the stale view -> local fallback -> the compaction still
// completes correctly (CompactFiles returns OK).
ASSERT_OK(db_->CompactFiles(CompactionOptions(), {fname}, 1));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_GE(floor_check_rejected, 1);
// Deleted key stays deleted: the local fallback retained the below-output
// tombstone. A silent mis-compaction against the older view would resurrect
// "OLD".
ASSERT_EQ("NOT_FOUND", Get("k"));
}
TEST_F(CompactionServiceTest, RemoteCompactionManifestFloorKillSwitchOff) {
// With the kill switch off, the primary sends no floor, so the worker uses
// the original point-in-time recovery (no trust-manifest, no floor check). A
// transiently-missing non-input file then rolls recovery back and fails the
// job -- confirming the escape hatch reverts to the pre-fix behavior.
Options options = CurrentOptions();
options.num_levels = 7;
options.max_open_files = -1;
options.open_files_async = true;
options.skip_stats_update_on_db_open = true;
options.level0_file_num_compaction_trigger = 100; // no auto compaction
options.remote_compaction_manifest_floor = false; // kill switch OFF
ReopenWithCompactionService(&options);
ASSERT_OK(Put("k", "OLD"));
ASSERT_OK(Flush());
MoveFilesToLevel(2); // L2: k=OLD
ASSERT_OK(Delete("k"));
ASSERT_OK(Flush()); // L0: Delete(k)
// Reopen so the MANIFEST is rewritten as current state before we perturb it.
SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::BGWorkAsyncFileOpen:Done",
"CompactionServiceTest::ManifestFloorKillSwitchOff:OpenDone"}});
SyncPoint::GetInstance()->EnableProcessing();
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "cf_1", "cf_2", "cf_3"},
options);
TEST_SYNC_POINT("CompactionServiceTest::ManifestFloorKillSwitchOff:OpenDone");
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
GetCompactionService()->SetCanceled(false);
bool floor_provided_seen = true;
int floor_check_count = 0;
SyncPoint::GetInstance()->SetCallBack(
"CompactionServiceJob::ProcessKeyValueCompactionWithCompactionService",
[&](void* arg) {
auto* input = static_cast<CompactionServiceInput*>(arg);
floor_provided_seen = (input->min_manifest_file_number != 0);
});
SyncPoint::GetInstance()->SetCallBack(
"DBImplSecondary::OpenAndCompact::ManifestFloorCheck",
[&](void* /*arg*/) { floor_check_count++; });
SyncPoint::GetInstance()->EnableProcessing();
ColumnFamilyMetaData meta;
db_->GetColumnFamilyMetaData(&meta);
ASSERT_EQ(meta.levels[0].files.size(), 1);
ASSERT_EQ(meta.levels[2].files.size(), 1);
const auto& f_del = meta.levels[0].files[0];
const auto& f_old = meta.levels[2].files[0];
// Make the lower (L2) non-input file momentarily unavailable, as a live
// primary would if it had compacted it away.
ASSERT_OK(env_->DeleteFile(f_old.db_path + "/" + f_old.name));
std::string fname = f_del.db_path + "/" + f_del.name;
Status s = db_->CompactFiles(CompactionOptions(), {fname}, 1);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
// No floor was sent and the worker skipped the floor check; the original
// recovery rolled back and failed the job (reverted behavior).
ASSERT_FALSE(floor_provided_seen);
ASSERT_EQ(0, floor_check_count);
ASSERT_NOK(s);
}
TEST_F(CompactionServiceTest, RemoteCompactionManifestFloorMutable) {
// remote_compaction_manifest_floor is a mutable DB option: flipping it via
// SetDBOptions changes whether the primary attaches a floor to subsequent
// remote compaction requests (and thus whether the worker runs the floor
// check).
Options options = CurrentOptions();
options.disable_auto_compactions = true;
ReopenWithCompactionService(&options);
GenerateTestData();
bool floor_provided = false;
int floor_check_count = 0;
SyncPoint::GetInstance()->SetCallBack(
"CompactionServiceJob::ProcessKeyValueCompactionWithCompactionService",
[&](void* arg) {
auto* input = static_cast<CompactionServiceInput*>(arg);
floor_provided = (input->min_manifest_file_number != 0);
});
SyncPoint::GetInstance()->SetCallBack(
"DBImplSecondary::OpenAndCompact::ManifestFloorCheck",
[&](void* /*arg*/) { floor_check_count++; });
SyncPoint::GetInstance()->EnableProcessing();
std::string start_str = Key(15);
std::string end_str = Key(45);
Slice start(start_str);
Slice end(end_str);
// Default on: floor attached, floor check runs.
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), &start, &end));
ASSERT_TRUE(floor_provided);
ASSERT_GE(floor_check_count, 1);
// Turn the kill switch off dynamically.
ASSERT_OK(db_->SetDBOptions({{"remote_compaction_manifest_floor", "false"}}));
ASSERT_FALSE(db_->GetDBOptions().remote_compaction_manifest_floor);
floor_provided = true;
floor_check_count = 0;
// Create a fresh compaction target overlapping the range and re-compact.
ASSERT_OK(Put(Key(20), "v"));
ASSERT_OK(Flush());
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), &start, &end));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
// Now no floor is attached and the worker skips the floor check.
ASSERT_FALSE(floor_provided);
ASSERT_EQ(0, floor_check_count);
}
namespace {
// A remote compaction service whose output directory already contains an SST
// left behind by a previously interrupted (canceled) fresh attempt, then runs
// OpenAndCompact with allow_resumption=true. Used to verify that when
// resumption is internally disabled, the output directory is cleaned before the
// fresh compaction starts.
class LeftoverOutputCompactionService : public MyTestCompactionService {
public:
using MyTestCompactionService::MyTestCompactionService;
CompactionServiceJobStatus Wait(const std::string& scheduled_job_id,
std::string* result) override {
std::string compaction_input;
{
InstrumentedMutexLock l(&mutex_);
auto it = jobs_.find(scheduled_job_id);
if (it == jobs_.end()) {
return CompactionServiceJobStatus::kFailure;
}
compaction_input = it->second;
jobs_.erase(it);
}
CompactionServiceOptionsOverride options_override = GetOptionsOverride();
Env* env = options_override.env;
const std::string output_dir = GetOutputPath(scheduled_job_id);
EXPECT_OK(env->CreateDirIfMissing(output_dir));
// An output SST left behind by a previously interrupted fresh attempt. Uses
// a high file number the fresh compaction will not reuse, so whether it is
// cleaned up is directly observable.
leftover_file_ = MakeTableFileName(output_dir, 999999);
{
std::unique_ptr<WritableFile> f;
EXPECT_OK(env->NewWritableFile(leftover_file_, &f, EnvOptions()));
EXPECT_OK(f->Append("leftover"));
EXPECT_OK(f->Close());
}
OpenAndCompactOptions open_options;
open_options.allow_resumption = true;
Status s = DB::OpenAndCompact(open_options, db_path_, scheduled_job_id,
compaction_input, result, options_override);
{
InstrumentedMutexLock l(&mutex_);
result_ = *result;
}
if (!s.ok()) {
return CompactionServiceJobStatus::kFailure;
}
leftover_cleaned_ = env->FileExists(leftover_file_).IsNotFound();
return CompactionServiceJobStatus::kSuccess;
}
bool leftover_cleaned() const { return leftover_cleaned_; }
private:
std::string leftover_file_;
bool leftover_cleaned_ = false;
};
} // namespace
TEST_F(CompactionServiceTest, CleansOutputDirWhenResumptionDisabled) {
// Regression test: allow_resumption=true is requested, but resumption is
// internally disabled because paranoid_file_checks=true is incompatible with
// the resumption path's output-hash verification. OpenAndCompact then starts
// a fresh compaction, which must clean the output directory first (honoring
// the documented OpenAndCompactOptions::allow_resumption fallback contract),
// so an output file left by a previously interrupted attempt does not collide
// with the file numbers the fresh compaction reuses. Before the fix the
// leftover file was kept, which under a no-reopen filesystem (as in the crash
// test) fails the fresh compaction's NewWritableFile with a
// "no-reopen-for-write contract" violation.
Options options = CurrentOptions();
options.disable_auto_compactions = true;
options.paranoid_file_checks = true; // disables resumption on the worker
options.use_session_tmp_dir_for_remote_compaction = true;
options.env = env_;
auto statistics = CreateDBStatistics();
auto cs = std::make_shared<LeftoverOutputCompactionService>(
dbname_, options, statistics,
std::vector<std::shared_ptr<EventListener>>{},
std::vector<std::shared_ptr<TablePropertiesCollectorFactory>>{});
options.compaction_service = cs;
DestroyAndReopen(options);
// Two overlapping flushed files so the manual compaction has real work.
ASSERT_OK(Put(Key(1), "v1"));
ASSERT_OK(Put(Key(3), "v3"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(2), "v2"));
ASSERT_OK(Put(Key(3), "v3new"));
ASSERT_OK(Flush());
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
// The fresh compaction cleaned the leftover output before starting.
ASSERT_TRUE(cs->leftover_cleaned());
ASSERT_EQ(Get(Key(1)), "v1");
ASSERT_EQ(Get(Key(2)), "v2");
ASSERT_EQ(Get(Key(3)), "v3new");
}
TEST_F(CompactionServiceTest, RetriesSecondaryOpenWhenCurrentFileVanishes) {
// The remote compaction worker opens the live primary's DB directory
// read-only inside OpenAndCompact. The primary replaces CURRENT on every
// MANIFEST rotation, and on a filesystem without POSIX read-after-unlink
// semantics the CURRENT object the worker already opened becomes unreadable.
// That is a transient race, so the secondary open must be retried; otherwise
// a CompactionService that does not fall back to a local compaction
// (db_stress with remote_compaction_failure_fall_back_to_local=false) turns
// it into a primary background error that fails every later flush.
auto fs = std::make_shared<VanishingCurrentFileFS>(env_->GetFileSystem());
std::unique_ptr<Env> vanishing_env(NewCompositeEnv(fs));
Options options = CurrentOptions();
options.disable_auto_compactions = true;
options.env = vanishing_env.get();
auto statistics = CreateDBStatistics();
auto cs = std::make_shared<MyTestCompactionService>(
dbname_, options, statistics,
std::vector<std::shared_ptr<EventListener>>{},
std::vector<std::shared_ptr<TablePropertiesCollectorFactory>>{});
cs->SetMaxSecondaryOpenRetries(3);
options.compaction_service = cs;
DestroyAndReopen(options);
// Two overlapping flushed files so the manual compaction has real work.
ASSERT_OK(Put(Key(1), "v1"));
ASSERT_OK(Put(Key(3), "v3"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(2), "v2"));
ASSERT_OK(Put(Key(3), "v3new"));
ASSERT_OK(Flush());
// Arm the injection only now: the primary is done opening, so the failed
// CURRENT reads below happen in the worker's secondary open. Three failures
// exercise a non-default retry limit and require the fourth attempt.
fs->SetFailuresToInject(3);
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
const Status compact_s = db_->CompactRange(cro, nullptr, nullptr);
// Collect everything before closing: the DB must be closed while the
// injecting Env is still alive, so no assertion may return early above it.
const int injected_failures = fs->injected_failures();
const int compactions = cs->GetCompactionNum();
const std::string v1 = Get(Key(1));
const std::string v2 = Get(Key(2));
const std::string v3 = Get(Key(3));
Close();
ASSERT_OK(compact_s);
ASSERT_EQ(3, injected_failures);
ASSERT_GE(compactions, 1);
ASSERT_EQ(v1, "v1");
ASSERT_EQ(v2, "v2");
ASSERT_EQ(v3, "v3new");
}
TEST_F(CompactionServiceTest, CanDisableSecondaryOpenRetry) {
auto fs = std::make_shared<VanishingCurrentFileFS>(env_->GetFileSystem());
std::unique_ptr<Env> vanishing_env(NewCompositeEnv(fs));
Options options = CurrentOptions();
options.disable_auto_compactions = true;
options.env = vanishing_env.get();
auto statistics = CreateDBStatistics();
auto cs = std::make_shared<MyTestCompactionService>(
dbname_, options, statistics,
std::vector<std::shared_ptr<EventListener>>{},
std::vector<std::shared_ptr<TablePropertiesCollectorFactory>>{});
cs->SetMaxSecondaryOpenRetries(0);
options.compaction_service = cs;
DestroyAndReopen(options);
ASSERT_OK(Put(Key(1), "v1"));
ASSERT_OK(Put(Key(3), "v3"));
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(2), "v2"));
ASSERT_OK(Put(Key(3), "v3new"));
ASSERT_OK(Flush());
fs->SetFailuresToInject(1);
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
const Status compact_s = db_->CompactRange(cro, nullptr, nullptr);
const int injected_failures = fs->injected_failures();
const int compactions = cs->GetCompactionNum();
Close();
ASSERT_TRUE(compact_s.IsIncomplete());
ASSERT_EQ(1, injected_failures);
ASSERT_EQ(1, compactions);
}
TEST_F(CompactionServiceTest, CompactionInfo) {
Options options = CurrentOptions();
ReopenWithCompactionService(&options);
@@ -2854,7 +3455,7 @@ class ResumableCompactionService : public MyTestCompactionService {
EXPECT_OK(statistics_->Reset());
Status s =
DB::OpenAndCompact(options, db_path_, GetOutputPath(scheduled_job_id),
DB::OpenAndCompact(options, db_path_, scheduled_job_id,
compaction_input, &temp_result, override_options);
EXPECT_TRUE(s.IsManualCompactionPaused());
@@ -2871,9 +3472,8 @@ class ResumableCompactionService : public MyTestCompactionService {
std::string* result) {
EXPECT_OK(statistics_->Reset());
Status s =
DB::OpenAndCompact(options, db_path_, GetOutputPath(scheduled_job_id),
compaction_input, result, override_options);
Status s = DB::OpenAndCompact(options, db_path_, scheduled_job_id,
compaction_input, result, override_options);
EXPECT_TRUE(s.ok());
@@ -3378,6 +3978,182 @@ TEST_F(ResumableCompactionKeyTypeTest, CancelAndResumeWithTimedPut) {
VerifyResumeBytes();
}
// Covers the DB session temporary directory lifecycle when remote compaction
// opts in to using it.
class SessionTmpDirTest : public DBTestBase {
public:
SessionTmpDirTest()
: DBTestBase("session_tmp_dir_test",
/*env_do_fsync=*/true) {}
protected:
// Opens with a CompactionService opted in to the session temporary directory.
Options OptionsWithCompactionService() {
Options options = CurrentOptions();
options.env = env_;
options.use_session_tmp_dir_for_remote_compaction = true;
worker_statistics_ = CreateDBStatistics();
compaction_service_ = std::make_shared<MyTestCompactionService>(
dbname_, options, worker_statistics_,
std::vector<std::shared_ptr<EventListener>>{},
std::vector<std::shared_ptr<TablePropertiesCollectorFactory>>{});
options.compaction_service = compaction_service_;
return options;
}
std::string SessionTmpPath() { return SessionTmpDir(dbname_); }
// Creates and closes a DB session before planting temporary output for the
// next session to delete.
void CreateDbAndQuiesce(const Options& options) {
DestroyAndReopen(options);
ASSERT_TRUE(Exists(SessionTmpPath()));
Close();
}
// Creates `parent/name/` holding one file, mimicking a leaked job directory.
void PlantDir(const std::string& parent, const std::string& name) {
SpecialEnv* const env = env_;
if (env == nullptr) {
ADD_FAILURE() << "Test environment is not initialized";
return;
}
ASSERT_OK(env->CreateDirIfMissing(parent));
const std::string dir = parent + "/" + name;
ASSERT_OK(env->CreateDirIfMissing(dir));
ASSERT_OK(WriteStringToFile(env, "leftover", dir + "/000123.sst"));
}
bool Exists(const std::string& path) {
return env_ != nullptr && env_->FileExists(path).ok();
}
// Reopens after planting stale output. Cleanup completes before DB::Open()
// schedules compactions or returns.
void ReopenAndDeleteStaleOutput(const Options& options) { Reopen(options); }
std::shared_ptr<Statistics> worker_statistics_;
std::shared_ptr<CompactionService> compaction_service_;
};
TEST_F(SessionTmpDirTest, ResolvesJobDirectory) {
const std::string job_dir_name = "SESSION01_4294967296";
const std::string legacy_output_path = dbname_ + "/" + job_dir_name;
ASSERT_EQ(RemoteCompactionJobDir(dbname_, false, job_dir_name),
legacy_output_path);
ASSERT_EQ(RemoteCompactionJobDir(dbname_, false, legacy_output_path),
legacy_output_path);
ASSERT_EQ(RemoteCompactionJobDir(dbname_, true, job_dir_name),
SessionTmpPath() + "/" + job_dir_name);
}
TEST_F(SessionTmpDirTest, DeletesDirsFromPreviousSessions) {
Options options = OptionsWithCompactionService();
CreateDbAndQuiesce(options);
// Two leaked job directories, one of them with a nested subdirectory to prove
// the removal is recursive.
PlantDir(SessionTmpPath(), "SESSIONFROMDEADHOST01_4294967296");
PlantDir(SessionTmpPath() + "/SESSIONFROMDEADHOST01_4294967296", "nested");
PlantDir(SessionTmpPath(), "ANOTHERDEADSESSION02_8589934592");
ReopenAndDeleteStaleOutput(options);
ASSERT_TRUE(Exists(SessionTmpPath()));
ASSERT_FALSE(Exists(SessionTmpPath() + "/SESSIONFROMDEADHOST01_4294967296"));
ASSERT_FALSE(Exists(SessionTmpPath() + "/ANOTHERDEADSESSION02_8589934592"));
}
TEST_F(SessionTmpDirTest, IgnoresDirsOutsideSessionTmpDir) {
Options options = OptionsWithCompactionService();
CreateDbAndQuiesce(options);
// A CompactionService that stages output outside the configured root owns
// cleaning it up; the DB must not touch it.
const std::string elsewhere = "service_owned_staging_dir";
PlantDir(dbname_, elsewhere);
PlantDir(SessionTmpPath(), "SESSIONFROMDEADHOST01_4294967296");
ReopenAndDeleteStaleOutput(options);
ASSERT_TRUE(Exists(dbname_ + "/" + elsewhere));
ASSERT_FALSE(Exists(SessionTmpPath() + "/SESSIONFROMDEADHOST01_4294967296"));
// Nothing in RocksDB owns this directory, so nothing else will remove it --
// including the DestroyDB that the next test's setup runs against the shared
// per-thread DB path.
ASSERT_OK(DestroyDir(env_, dbname_ + "/" + elsewhere));
}
TEST_F(SessionTmpDirTest, DisabledOptionSkipsCleanup) {
if (Exists(SessionTmpPath())) {
ASSERT_OK(DestroyDir(env_, SessionTmpPath()));
}
Options options = CurrentOptions();
options.env = env_;
DestroyAndReopen(options);
ASSERT_FALSE(Exists(SessionTmpPath()));
PlantDir(SessionTmpPath(), "SESSIONFROMDEADHOST01_4294967296");
Reopen(options);
Close();
ASSERT_TRUE(Exists(SessionTmpPath() + "/SESSIONFROMDEADHOST01_4294967296"));
ASSERT_OK(DestroyDir(env_, SessionTmpPath()));
}
TEST_F(SessionTmpDirTest, EnabledSessionTmpDirIsCreated) {
Options options = CurrentOptions();
options.use_session_tmp_dir_for_remote_compaction = true;
DestroyAndReopen(options);
ASSERT_TRUE(Exists(SessionTmpPath()));
}
TEST_F(SessionTmpDirTest, DestroyDBRemovesSessionTmpDir) {
Options options = OptionsWithCompactionService();
CreateDbAndQuiesce(options);
PlantDir(SessionTmpPath(), "SESSIONFROMDEADHOST01_4294967296");
// DestroyDB's final DeleteDir(dbname) is non-recursive, so it can only remove
// the DB directory if the session temporary directory is gone first.
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_FALSE(Exists(dbname_));
}
TEST_F(SessionTmpDirTest, CompactionUsesSessionTmpDir) {
Options options = OptionsWithCompactionService();
options.disable_auto_compactions = true;
DestroyAndReopen(options);
for (int i = 0; i < 20; i++) {
ASSERT_OK(Put(Key(i), "value" + std::to_string(i)));
}
ASSERT_OK(Flush());
for (int i = 0; i < 20; i += 2) {
ASSERT_OK(Put(Key(i), "value_new" + std::to_string(i)));
}
ASSERT_OK(Flush());
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
auto* my_cs = static_cast_with_check<MyTestCompactionService>(
compaction_service_.get());
ASSERT_GE(my_cs->GetCompactionNum(), 1);
CompactionServiceResult result;
my_cs->GetResult(&result);
ASSERT_OK(result.status);
ASSERT_EQ(result.output_path,
RemoteCompactionJobDir(dbname_, true,
my_cs->GetLastOutputDirectoryName()));
for (int i = 0; i < 20; i++) {
ASSERT_EQ(Get(Key(i)), (i % 2 ? "value" : "value_new") + std::to_string(i));
}
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+3 -3
View File
@@ -68,11 +68,11 @@ class FileTtlBooster {
if (!enabled_) {
return 1;
}
uint64_t oldest_ancester_time = f->TryGetOldestAncesterTime();
if (oldest_ancester_time >= current_time_) {
uint64_t oldest_ancestor_time = f->TryGetOldestAncestorTime();
if (oldest_ancestor_time >= current_time_) {
return 1;
}
uint64_t age = current_time_ - oldest_ancester_time;
uint64_t age = current_time_ - oldest_ancestor_time;
if (age > boost_age_start_) {
// Use integer just for convenience.
// We could make all file_to_order double if we want.
+12
View File
@@ -25,6 +25,18 @@ SstPartitionerFixedPrefixFactory::SstPartitionerFixedPrefixFactory(size_t len)
RegisterOptions("Length", &len_, &sst_fixed_prefix_type_info);
}
OptSlice SstPartitionerFixedPrefix::ShouldPartitionByPrefix(
const Slice& user_key) {
if (user_key.size() >= len_) {
return {user_key.data(), len_};
} else {
// ShouldPartition() cuts a partition for EVERY UNIQUE KEY outside the
// prefix length domain, for better or worse. We can't directly represent
// that so we defer to ShouldPartition().
return {};
}
}
PartitionerResult SstPartitionerFixedPrefix::ShouldPartition(
const PartitionerRequest& request) {
Slice last_key_fixed(*request.prev_user_key);
+189 -3
View File
@@ -1339,9 +1339,13 @@ TEST_F(TieredCompactionTest, SequenceBasedTieredStorageLevel) {
// This test was essentially for a hacked-up version on future functionality.
// It can be resurrected if/when a form of range-based tiering is properly
// implemented.
// FIXME: aside from that, this test reproduces a near-endless compaction
// cycle that needs to be reproduced independently and fixed before
// leveled compaction can be used with the preclude feature in production.
// The near-endless compaction cycle this used to reproduce (a bottommost file
// whose seqno could not be zeroed within the preserve window being re-marked
// forever) is now prevented by sharing the seqno-zeroability decision between
// the compaction output path and bottommost-file marking (see
// BottommostSeqnoCanBeZeroed and CompactionPickerTest.BottommostFile*
// PreserveWindow). This test remains disabled only because range-based tiering
// is not implemented.
TEST_F(TieredCompactionTest, DISABLED_RangeBasedTieredStorageLevel) {
const int kNumTrigger = 4;
const int kNumLevels = 7;
@@ -1558,6 +1562,188 @@ class PrecludeLastLevelTest : public PrecludeLastLevelTestBase,
}
};
// Regression test for an infinite kBottommostFiles compaction loop on a
// leveled column family that has the seqno->time preserve feature enabled. A
// bottommost file whose largest sequence number is still within the preserve
// window cannot have its seqno zeroed by compaction, so it must not be marked
// for bottommost compaction; otherwise each rewrite makes no progress and the
// file is re-marked forever. This exercises the DBImpl plumbing that feeds
// VersionStorageInfo::preserve_time_min_seqno_ (DBImpl::
// MaybeUpdatePreserveTimeMinSeqno) end to end, complementing the isolated
// marking-predicate coverage in compaction_picker_test.cc.
class PrecludeBottommostLoopTest : public PrecludeLastLevelTestBase {};
TEST_F(PrecludeBottommostLoopTest, LeveledPreserveTimeNoBottommostLoop) {
const int kNumLevels = 7;
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.preserve_internal_time_seconds = 10000;
options.env = mock_env_.get();
options.num_levels = kNumLevels;
// Prevent the extra L0 file below from auto-triggering an L0 compaction.
options.level0_file_num_compaction_trigger = 4;
SyncPoint::GetInstance()->EnableProcessing();
DestroyAndReopen(options);
// Populate the seqno->time mapping with recent samples so the preserve window
// maps the file's (recent) largest seqno as non-zeroable. Total elapsed mock
// time stays far below preserve_internal_time_seconds, so nothing ages out.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put("infinite", "v" + std::to_string(i)));
dbfull()->TEST_WaitForPeriodicTaskRun(
[&] { mock_clock_->MockSleepForSeconds(10); });
}
ASSERT_OK(Flush());
// Push the file with the hot key down to the bottommost level.
MoveFilesToLevel(kNumLevels - 1);
ASSERT_EQ(1, NumTableFilesAtLevel(kNumLevels - 1));
// Confirm the bottommost file's largest seqno was not zeroed (it is within
// the preserve window), so it is a genuine marking candidate that only the
// preserve-window guard should protect -- this keeps the test from passing
// vacuously.
{
std::vector<LiveFileMetaData> files;
db_->GetLiveFilesMetaData(&files);
bool found_bottommost = false;
for (const auto& f : files) {
if (f.level == kNumLevels - 1) {
ASSERT_GT(f.largest_seqno, 0U);
found_bottommost = true;
}
}
ASSERT_TRUE(found_bottommost);
}
// Bump the sequence number above the bottommost file so that releasing a
// snapshot advances oldest_snapshot_seqnum_ past the file's largest seqno,
// which makes the file a candidate for bottommost marking (the snapshot gate
// passes). Only the preserve-window guard should keep it from being marked.
ASSERT_OK(Put("bumpseqnum", ""));
ASSERT_OK(Flush());
std::atomic<int> bottommost_compactions{0};
SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
auto* c = static_cast<Compaction*>(arg);
if (c != nullptr &&
c->compaction_reason() == CompactionReason::kBottommostFiles) {
bottommost_compactions.fetch_add(1);
}
});
const Snapshot* snapshot = db_->GetSnapshot();
db_->ReleaseSnapshot(snapshot);
// The bottommost file's largest seqno is within the preserve window, so it is
// not marked and the DB quiesces. Before the fix the file was marked and the
// rewrite re-qualified it forever (this wait would hang).
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, bottommost_compactions.load());
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
// Close explicitly because mock_env_ is destroyed before the DB otherwise.
Close();
}
// Regression test for the empty-mapping variant of the same loop: a bottommost
// file that predates a preserve/preclude rollout has an EMPTY persisted
// seqno->time mapping. Once the live, DB-wide mapping ages the file out of the
// preserve window, the marker (which reads the live mapping) marks it. The
// compaction that rewrites it must use the same version-carried preserve
// boundary rather than the file's own empty mapping; otherwise it computes
// preserve_seqno_after_ = 0, zeroes nothing, and re-marks the file forever.
// With the fix the file's seqno is zeroed in one bounded compaction.
TEST_F(PrecludeBottommostLoopTest, EmptyMappingAgedOutNoBottommostLoop) {
const int kNumLevels = 7;
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.env = mock_env_.get();
options.num_levels = kNumLevels;
options.level0_file_num_compaction_trigger = 4;
// Start with preserve/preclude disabled so the bottommost file below is
// written with an EMPTY persisted seqno->time mapping, mimicking a file that
// predates a preserve/preclude rollout.
options.preserve_internal_time_seconds = 0;
options.preclude_last_level_data_seconds = 0;
SyncPoint::GetInstance()->EnableProcessing();
DestroyAndReopen(options);
// Hold a snapshot below the "aged" key so moving it to the bottommost level
// does not zero its seqno (there is no preserve window yet to protect it).
ASSERT_OK(Put("keeplow", "v"));
const Snapshot* protect = db_->GetSnapshot();
ASSERT_OK(Put("aged", "v"));
ASSERT_OK(Flush());
MoveFilesToLevel(kNumLevels - 1);
ASSERT_EQ(1, NumTableFilesAtLevel(kNumLevels - 1));
SequenceNumber aged_seqno = 0;
{
std::vector<LiveFileMetaData> files;
db_->GetLiveFilesMetaData(&files);
for (const auto& f : files) {
if (f.level == kNumLevels - 1) {
aged_seqno = f.largest_seqno;
}
}
}
ASSERT_GT(aged_seqno, 0U);
// Roll out the preserve feature dynamically and let the live, DB-wide
// seqno->time mapping accumulate samples while mock time advances well past
// preserve_internal_time_seconds, so the pre-existing file's seqno ages out
// of the preserve window per the live mapping (even though the file's own
// persisted mapping is empty). The protecting snapshot keeps the file from
// being marked during this loop (oldest_snapshot_seqnum_ stays below it).
ASSERT_OK(db_->SetOptions({{"preserve_internal_time_seconds", "10000"}}));
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put("hot", "v" + std::to_string(i)));
dbfull()->TEST_WaitForPeriodicTaskRun(
[&] { mock_clock_->MockSleepForSeconds(200); });
}
std::atomic<int> bottommost_compactions{0};
SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
auto* c = static_cast<Compaction*>(arg);
if (c != nullptr &&
c->compaction_reason() == CompactionReason::kBottommostFiles) {
bottommost_compactions.fetch_add(1);
}
});
// Release the protecting snapshot so oldest_snapshot_seqnum_ advances past
// the file's largest seqno, making it a bottommost marking candidate. The
// marker (using the live mapping) marks it; the compaction that rewrites it
// must use the same version-carried preserve boundary (not the file's empty
// mapping) to actually zero the seqno, otherwise it makes no progress and
// loops.
db_->ReleaseSnapshot(protect);
// With the fix the file's seqno is zeroed in a bounded number of bottommost
// compactions and the DB quiesces. Before the fix this wait would hang.
ASSERT_OK(dbfull()->TEST_WaitForCompact());
{
std::vector<LiveFileMetaData> files;
db_->GetLiveFilesMetaData(&files);
for (const auto& f : files) {
if (f.level == kNumLevels - 1) {
ASSERT_EQ(0U, f.largest_seqno);
}
}
}
// The file was genuinely marked and rewritten (not vacuously skipped), and
// the rewrite made progress instead of looping.
ASSERT_GE(bottommost_compactions.load(), 1);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
Close();
}
TEST_P(PrecludeLastLevelTest, MigrationFromPreserveTimeManualCompaction) {
const int kNumTrigger = 4;
const int kNumLevels = 7;
+212 -6
View File
@@ -10,6 +10,7 @@
#include <algorithm>
#include <array>
#include <cassert>
#include <utility>
#include <vector>
#include "folly/Executor.h"
@@ -18,7 +19,10 @@
#include "folly/io/async/EventBase.h"
#include "rocksdb/coro_db.h"
#include "rocksdb/file_system.h"
#include "rocksdb/sst_file_reader.h"
#include "table/multiget_context.h"
#include "table/sst_file_reader_impl.h"
#include "util/coro_stats_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -33,14 +37,28 @@ folly::coro::Task<Status> CoroDB::CoGet(DB* db, const ReadOptions& options,
file_system == nullptr ? nullptr : file_system->GetReadExecutor();
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
Status status;
if (coro_db == nullptr || read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
db->GetEnv());
status = db->Get(options, column_family, key, value, timestamp);
} else {
auto task = [](CoroutineStatsConfig task_stats_config, Env* task_env,
CoroDB* task_db, const ReadOptions& task_options,
ColumnFamilyHandle* task_column_family, Slice task_key,
PinnableSlice* task_value,
std::string* task_timestamp) -> folly::coro::Task<Status> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_return co_await folly::coro::co_nothrow(
task_db->GetCoroutine(task_options, task_column_family, task_key,
task_value, task_timestamp));
}(std::move(stats_config), db->GetEnv(), coro_db, options, column_family,
key, value, timestamp);
status = co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base),
coro_db->GetCoroutine(options, column_family, key, value, timestamp)));
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
co_return status;
}
@@ -73,6 +91,131 @@ folly::coro::Task<Status> CoroDB::CoGet(DB* db, const ReadOptions& options,
return CoGet(db, options, db->DefaultColumnFamily(), key, value, timestamp);
}
folly::coro::Task<Status> CoroDB::CoGet(SstFileReader* reader,
const ReadOptions& options,
const Slice& key,
PinnableSlice* value) {
assert(reader != nullptr);
SstFileReader::Rep* rep = reader->rep_.get();
auto* read_executor = rep->read_executor;
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
Status status;
if (read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
rep->options.env);
status = reader->Get(options, key, value);
} else {
auto task = [](CoroutineStatsConfig task_stats_config, Env* task_env,
SstFileReader* task_reader, const ReadOptions& task_options,
Slice task_key,
PinnableSlice* task_value) -> folly::coro::Task<Status> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_return co_await folly::coro::co_nothrow(
task_reader->rep_->GetCoroutine(task_options, task_key, task_value));
}(std::move(stats_config), rep->options.env, reader, options, key, value);
status = co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
co_return status;
}
folly::coro::Task<Status> CoroDB::CoGet(SstFileReader* reader,
const ReadOptions& options,
const Slice& key, std::string* value) {
assert(value != nullptr);
PinnableSlice pinnable_value(value);
assert(!pinnable_value.IsPinned());
Status status = co_await folly::coro::co_nothrow(
CoGet(reader, options, key, &pinnable_value));
if (status.ok() && pinnable_value.IsPinned()) {
value->assign(pinnable_value.data(), pinnable_value.size());
}
co_return status;
}
folly::coro::Task<Status> CoroDB::CoGetEntity(DB* db,
const ReadOptions& options,
ColumnFamilyHandle* column_family,
const Slice& key,
PinnableWideColumns* columns) {
assert(db != nullptr);
CoroDB* coro_db = db->GetCoroDB();
auto* file_system = db->GetFileSystem();
auto* read_executor =
file_system == nullptr ? nullptr : file_system->GetReadExecutor();
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
Status status;
if (coro_db == nullptr || read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
db->GetEnv());
status = db->GetEntity(options, column_family, key, columns);
} else {
auto task =
[](CoroutineStatsConfig task_stats_config, Env* task_env,
CoroDB* task_db, const ReadOptions& task_options,
ColumnFamilyHandle* task_column_family, Slice task_key,
PinnableWideColumns* task_columns) -> folly::coro::Task<Status> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_return co_await folly::coro::co_nothrow(task_db->GetEntityCoroutine(
task_options, task_column_family, task_key, task_columns));
}(std::move(stats_config), db->GetEnv(), coro_db, options, column_family,
key, columns);
status = co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
co_return status;
}
folly::coro::Task<Status> CoroDB::CoGetEntity(DB* db,
const ReadOptions& options,
const Slice& key,
PinnableWideColumns* columns) {
assert(db != nullptr);
return CoGetEntity(db, options, db->DefaultColumnFamily(), key, columns);
}
folly::coro::Task<Status> CoroDB::CoGetEntity(DB* db,
const ReadOptions& options,
const Slice& key,
PinnableAttributeGroups* result) {
assert(db != nullptr);
CoroDB* coro_db = db->GetCoroDB();
auto* file_system = db->GetFileSystem();
auto* read_executor =
file_system == nullptr ? nullptr : file_system->GetReadExecutor();
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
Status status;
if (coro_db == nullptr || read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
db->GetEnv());
status = db->GetEntity(options, key, result);
} else {
auto task =
[](CoroutineStatsConfig task_stats_config, Env* task_env,
CoroDB* task_db, const ReadOptions& task_options, Slice task_key,
PinnableAttributeGroups* task_result) -> folly::coro::Task<Status> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_return co_await folly::coro::co_nothrow(
task_db->GetEntityCoroutine(task_options, task_key, task_result));
}(std::move(stats_config), db->GetEnv(), coro_db, options, key, result);
status = co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
co_return status;
}
folly::coro::Task<std::vector<Status>> CoroDB::CoMultiGet(
DB* db, const ReadOptions& options,
const std::vector<ColumnFamilyHandle*>& column_families,
@@ -110,6 +253,54 @@ folly::coro::Task<std::vector<Status>> CoroDB::CoMultiGet(
CoMultiGet(db, options, column_families, keys, values, timestamps));
}
folly::coro::Task<std::vector<Status>> CoroDB::CoMultiGet(
SstFileReader* reader, const ReadOptions& options,
const std::vector<Slice>& keys, std::vector<PinnableSlice>* values) {
assert(reader != nullptr);
SstFileReader::Rep* rep = reader->rep_.get();
auto* read_executor = rep->read_executor;
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
std::vector<Status> statuses;
if (read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
rep->options.env);
statuses = reader->MultiGet(options, keys, values);
} else {
auto task = [](CoroutineStatsConfig task_stats_config, Env* task_env,
SstFileReader* task_reader, const ReadOptions& task_options,
const std::vector<Slice>& task_keys,
std::vector<PinnableSlice>* task_values)
-> folly::coro::Task<std::vector<Status>> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_return co_await folly::coro::co_nothrow(
task_reader->rep_->MultiGetCoroutine(task_options, task_keys,
task_values));
}(std::move(stats_config), rep->options.env, reader, options, keys, values);
statuses = co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
co_return statuses;
}
folly::coro::Task<std::vector<Status>> CoroDB::CoMultiGet(
SstFileReader* reader, const ReadOptions& options,
const std::vector<Slice>& keys, std::vector<std::string>* values) {
std::vector<PinnableSlice> pinnable_values;
std::vector<Status> statuses = co_await folly::coro::co_nothrow(
CoMultiGet(reader, options, keys, &pinnable_values));
values->resize(keys.size());
for (size_t i = 0; i < keys.size(); ++i) {
if (statuses[i].ok()) {
(*values)[i].assign(pinnable_values[i].data(), pinnable_values[i].size());
}
}
co_return statuses;
}
folly::coro::Task<void> CoroDB::CoMultiGet(
DB* db, const ReadOptions& options, size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
@@ -122,16 +313,31 @@ folly::coro::Task<void> CoroDB::CoMultiGet(
file_system == nullptr ? nullptr : file_system->GetReadExecutor();
auto* read_event_base =
read_executor == nullptr ? nullptr : read_executor->getEventBase();
CoroutineStatsConfig stats_config = CaptureAndDisableCoroutineStatsConfig();
if (coro_db == nullptr || read_event_base == nullptr) {
CoroutineStatsContextScope stats_scope(std::move(stats_config),
db->GetEnv());
db->MultiGet(options, num_keys, column_families, keys, values, timestamps,
statuses, sorted_input);
} else {
auto task = [](CoroutineStatsConfig task_stats_config, Env* task_env,
CoroDB* task_db, const ReadOptions& task_options,
size_t task_num_keys,
ColumnFamilyHandle** task_column_families,
const Slice* task_keys, PinnableSlice* task_values,
std::string* task_timestamps, Status* task_statuses,
bool task_sorted_input) -> folly::coro::Task<void> {
CoroutineStatsContextScope stats_scope(std::move(task_stats_config),
task_env);
co_await folly::coro::co_nothrow(task_db->MultiGetCoroutine(
task_options, task_num_keys, task_column_families, task_keys,
task_values, task_timestamps, task_statuses, task_sorted_input));
}(std::move(stats_config), db->GetEnv(), coro_db, options, num_keys,
column_families, keys, values,
timestamps, statuses, sorted_input);
co_await folly::coro::co_nothrow(folly::coro::co_withExecutor(
folly::Executor::getKeepAliveToken(read_event_base),
coro_db->MultiGetCoroutine(options, num_keys, column_families, keys,
values, timestamps, statuses,
sorted_input)));
folly::Executor::getKeepAliveToken(read_event_base), std::move(task)));
}
}
+764 -32
View File
@@ -11,6 +11,7 @@
#include "db/db_test_util.h"
#include "db/log_writer.h"
#include "db/manifest_ops.h"
#include "db/version_edit.h"
#include "file/writable_file_writer.h"
#include "options/options_helper.h"
@@ -20,10 +21,14 @@
#include "rocksdb/merge_operator.h"
#include "rocksdb/perf_context.h"
#include "rocksdb/table.h"
#include "rocksdb/utilities/checkpoint.h"
#include "rocksdb/utilities/debug.h"
#include "rocksdb/write_buffer_manager.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#include "table/format.h"
#include "test_util/sync_point.h"
#include "util/defer.h"
#include "util/file_checksum_helper.h"
#include "util/random.h"
#include "utilities/counted_fs.h"
@@ -34,6 +39,19 @@
namespace ROCKSDB_NAMESPACE {
namespace {
OutputMetadata NewerVersionOutputMetadata() {
OutputMetadata output_metadata;
output_metadata.WantNewerVersionPresent();
return output_metadata;
}
MultiGetOutputMetadata NewerVersionMultiGetOutputMetadata() {
MultiGetOutputMetadata output_metadata;
output_metadata.WantNewerVersionPresent();
return output_metadata;
}
class MyFlushBlockPolicy : public FlushBlockPolicy {
public:
explicit MyFlushBlockPolicy(const int num_keys_in_block,
@@ -141,6 +159,7 @@ struct RecoveryOptimizationCounters {
sp->ClearAllCallBacks();
}
};
} // namespace
// optimize_manifest_for_recovery=true: a clean reopen of a flushed DB must
@@ -351,6 +370,82 @@ TEST_F(DBBasicTest,
synthetic_number);
}
TEST_F(DBBasicTest,
OptimizeManifestForRecoveryKeepsCurrentOptionsFileWithStaleOptions) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.optimize_manifest_for_recovery = true;
options.avoid_unnecessary_blocking_io = false;
DestroyAndReopen(options);
ASSERT_OK(Put("k", "v"));
ASSERT_OK(Flush());
const uint64_t next_before_close =
dbfull()->GetVersionSet()->current_next_file_number();
Close();
const uint64_t stale_options_number = next_before_close + 100;
ASSERT_OK(WriteStringToFile(
env_, "" /*data*/, dbname_ + "/" + OptionsFileName(stale_options_number),
/*should_sync=*/true));
ASSERT_OK(WriteStringToFile(
env_, "" /*data*/,
dbname_ + "/" + OptionsFileName(stale_options_number + 1),
/*should_sync=*/true));
RecoveryOptimizationCounters counters;
counters.Install();
Reopen(options);
counters.Uninstall();
const uint64_t current_options_number =
dbfull()->GetVersionSet()->options_file_number();
ASSERT_EQ(1, counters.next_file_number.load());
ASSERT_GT(dbfull()->GetVersionSet()->current_next_file_number(),
stale_options_number + 1);
ASSERT_GT(current_options_number, 0u);
ASSERT_OK(env_->FileExists(OptionsFileName(dbname_, current_options_number)));
const std::string checkpoint_dir = dbname_ + "_checkpoint";
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
Checkpoint* checkpoint = nullptr;
ASSERT_OK(Checkpoint::Create(db_.get(), &checkpoint));
std::unique_ptr<Checkpoint> checkpoint_guard(checkpoint);
ASSERT_OK(checkpoint_guard->CreateCheckpoint(checkpoint_dir));
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
ASSERT_EQ("v", Get("k"));
}
TEST_F(DBBasicTest,
OptimizeManifestForRecoveryReservesTempOptionsFileNumbersInMemory) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.optimize_manifest_for_recovery = true;
DestroyAndReopen(options);
ASSERT_OK(Put("k", "v"));
ASSERT_OK(Flush());
const uint64_t next_before_close =
dbfull()->GetVersionSet()->current_next_file_number();
Close();
const uint64_t stale_temp_options_number = next_before_close + 100;
ASSERT_OK(
WriteStringToFile(env_, "" /*data*/,
TempOptionsFileName(dbname_, stale_temp_options_number),
/*should_sync=*/true));
RecoveryOptimizationCounters counters;
counters.Install();
Reopen(options);
counters.Uninstall();
ASSERT_EQ(1, counters.next_file_number.load());
ASSERT_GT(dbfull()->GetVersionSet()->current_next_file_number(),
stale_temp_options_number);
ASSERT_EQ("v", Get("k"));
}
// optimize_manifest_for_recovery=true: after a clean Put + Flush + Close, the
// next Open's min_log_number_to_keep equals (max_wal_before_close + 1),
// proving the close-time MANIFEST write took effect.
@@ -978,27 +1073,19 @@ TEST_F(DBBasicTest, WritableFileWriterInitialFileSizeAdoptsExistingSize) {
TEST_F(DBBasicTest, ReuseManifestOnOpenSkipsOnTailCorruption) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.optimize_manifest_for_recovery = true;
options.reuse_manifest_on_open = true;
options.write_dbid_to_manifest = false;
DestroyAndReopen(options);
ASSERT_OK(Put("k", "v"));
ASSERT_OK(Flush());
Close();
// Find the MANIFEST file and append garbage to it.
std::string manifest_path;
{
std::vector<std::string> files;
ASSERT_OK(env_->GetChildren(dbname_, &files));
for (const auto& f : files) {
uint64_t number;
FileType type;
if (ParseFileName(f, &number, &type) && type == kDescriptorFile) {
manifest_path = dbname_ + "/" + f;
break;
}
}
}
ASSERT_FALSE(manifest_path.empty());
uint64_t manifest_file_number = 0;
ASSERT_OK(GetCurrentManifestPath(dbname_, env_->GetFileSystem().get(),
/*is_retry=*/false, &manifest_path,
&manifest_file_number));
{
std::string contents;
ASSERT_OK(ReadFileToString(env_, manifest_path, &contents));
@@ -1007,9 +1094,13 @@ TEST_F(DBBasicTest, ReuseManifestOnOpenSkipsOnTailCorruption) {
}
std::atomic<int> reopened{0};
std::atomic<int> new_manifest{0};
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionSet::ReopenManifestForAppend:Reopened",
[&](void* /*arg*/) { reopened.fetch_add(1); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionSet::ProcessManifestWrites:BeforeNewManifest",
[&](void* /*arg*/) { new_manifest.fetch_add(1); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
@@ -1018,6 +1109,12 @@ TEST_F(DBBasicTest, ReuseManifestOnOpenSkipsOnTailCorruption) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_EQ(0, reopened.load());
ASSERT_EQ(1, new_manifest.load());
std::string manifest_path_after;
ASSERT_OK(GetCurrentManifestPath(dbname_, env_->GetFileSystem().get(),
/*is_retry=*/false, &manifest_path_after,
&manifest_file_number));
ASSERT_NE(manifest_path, manifest_path_after);
ASSERT_EQ("v", Get("k"));
}
@@ -1030,27 +1127,19 @@ TEST_F(DBBasicTest, ReuseManifestOnOpenSkipsOnTailCorruption) {
TEST_F(DBBasicTest, ReuseManifestOnOpenIncompleteAtomicGroupAtTail) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.optimize_manifest_for_recovery = true;
options.reuse_manifest_on_open = true;
options.write_dbid_to_manifest = false;
DestroyAndReopen(options);
ASSERT_OK(Put("k", "v"));
ASSERT_OK(Flush());
Close();
// Find the MANIFEST file.
std::string manifest_path;
{
std::vector<std::string> files;
ASSERT_OK(env_->GetChildren(dbname_, &files));
for (const auto& f : files) {
uint64_t number;
FileType type;
if (ParseFileName(f, &number, &type) && type == kDescriptorFile) {
manifest_path = dbname_ + "/" + f;
break;
}
}
}
ASSERT_FALSE(manifest_path.empty());
uint64_t manifest_file_number = 0;
ASSERT_OK(GetCurrentManifestPath(dbname_, env_->GetFileSystem().get(),
/*is_retry=*/false, &manifest_path,
&manifest_file_number));
// Append an incomplete atomic group (2 records of a 3-member group) to the
// MANIFEST. These are valid log records with correct CRCs but the atomic
@@ -1088,11 +1177,29 @@ TEST_F(DBBasicTest, ReuseManifestOnOpenIncompleteAtomicGroupAtTail) {
// Deliberately NOT writing the third record (remaining_entries=0)
}
// Reopen: recovery should succeed (incomplete trailing group is ignored).
// With the fix, ReopenManifestForAppend will NOT reuse the MANIFEST because
// manifest_last_valid_record_end_ < physical_size (the incomplete group
// records are excluded from the valid end).
std::atomic<int> reopened{0};
std::atomic<int> new_manifest{0};
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionSet::ReopenManifestForAppend:Reopened",
[&](void* /*arg*/) { reopened.fetch_add(1); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionSet::ProcessManifestWrites:BeforeNewManifest",
[&](void* /*arg*/) { new_manifest.fetch_add(1); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Reopen: recovery should succeed, but ReopenManifestForAppend must not
// reuse a MANIFEST ending with an incomplete atomic group.
Reopen(options);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_EQ(0, reopened.load());
ASSERT_EQ(1, new_manifest.load());
std::string manifest_path_after;
ASSERT_OK(GetCurrentManifestPath(dbname_, env_->GetFileSystem().get(),
/*is_retry=*/false, &manifest_path_after,
&manifest_file_number));
ASSERT_NE(manifest_path, manifest_path_after);
ASSERT_EQ("v", Get("k"));
// Write new data to create new MANIFEST records.
@@ -1228,6 +1335,24 @@ TEST_F(DBBasicTest, ReadOnlyDB) {
ASSERT_OK(EnforcedReadOnlyReopen(options));
ASSERT_EQ("v3", Get("foo"));
ASSERT_EQ("v2", Get("bar"));
std::string metadata_value;
ReadOptions metadata_read_options;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(metadata_read_options, "foo", &metadata_value,
&output_metadata));
ASSERT_EQ("v3", metadata_value);
ASSERT_FALSE(*output_metadata.newer_version_present);
std::vector<Slice> metadata_keys{Slice("foo")};
std::vector<std::string> metadata_values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> metadata_statuses =
db_->MultiGetWithMetadata(metadata_read_options, metadata_keys,
&metadata_values, &multiget_output_metadata);
ASSERT_EQ(1, metadata_statuses.size());
ASSERT_OK(metadata_statuses[0]);
ASSERT_EQ("v3", metadata_values[0]);
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[0]);
verify_all_iters();
ASSERT_EQ(db_->SyncWAL().code(), Status::Code::kNotSupported);
@@ -1409,6 +1534,35 @@ TEST_F(DBBasicTest, CompactedDB) {
ASSERT_EQ(DummyString(kFileSize / 2, 'j'), Get("jjj"));
ASSERT_EQ("NOT_FOUND", Get("kkk"));
std::string metadata_value;
ReadOptions metadata_read_options;
const Snapshot* metadata_snapshot = db_->GetSnapshot();
metadata_read_options.snapshot = metadata_snapshot;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(metadata_read_options, "aaa", &metadata_value,
&output_metadata));
ASSERT_EQ(DummyString(kFileSize / 2, 'a'), metadata_value);
ASSERT_FALSE(*output_metadata.newer_version_present);
std::vector<Slice> metadata_keys{Slice("aaa"), Slice("ccc")};
std::vector<std::string> metadata_values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> metadata_statuses =
db_->MultiGetWithMetadata(metadata_read_options, metadata_keys,
&metadata_values, &multiget_output_metadata);
ASSERT_EQ(metadata_keys.size(), metadata_statuses.size());
ASSERT_OK(metadata_statuses[0]);
ASSERT_EQ(DummyString(kFileSize / 2, 'a'), metadata_values[0]);
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[0]);
ASSERT_TRUE(metadata_statuses[1].IsNotFound());
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[1]);
db_->ReleaseSnapshot(metadata_snapshot);
ASSERT_TRUE(db_->GetWithMetadata(ReadOptions(), db_->DefaultColumnFamily(),
"aaa", static_cast<PinnableSlice*>(nullptr),
&output_metadata)
.IsInvalidArgument());
// TODO: validate that other write ops return NotImplemented
// (CompactedDB is missing some overrides)
@@ -1596,6 +1750,430 @@ TEST_P(DBBasicGetWithParam, GetSnapshot) {
} while (ChangeOptions());
}
TEST_F(DBBasicTest, GetNewerVersionPresent) {
// Snapshot reads report later point and range-delete writes without changing
// the snapshot-visible value.
Options options = CurrentOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "v1"));
ASSERT_OK(Put(1, "merge_key", "base"));
ASSERT_OK(Put(1, "stable", "s1"));
ASSERT_OK(Put(1, "range_key", "r1"));
const Snapshot* snapshot = db_->GetSnapshot();
ReadOptions snapshot_read;
snapshot_read.snapshot = snapshot;
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("v1", value);
ASSERT_FALSE(*output_metadata.newer_version_present);
ASSERT_OK(Put(1, "other", "v2"));
ASSERT_OK(Put(1, "created_after_put", "new"));
ASSERT_OK(Merge(1, "created_after_merge", "new"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), handles_[1], "range_key", "range_key~"));
ASSERT_OK(Merge(1, "merge_key", "new"));
value.clear();
*output_metadata.newer_version_present = true;
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "stable", &value,
&output_metadata));
ASSERT_EQ("s1", value);
ASSERT_FALSE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = true;
ASSERT_TRUE(db_->GetWithMetadata(snapshot_read, handles_[1],
"created_after_put", &value,
&output_metadata)
.IsNotFound());
ASSERT_TRUE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_TRUE(db_->GetWithMetadata(snapshot_read, handles_[1],
"created_after_merge", &value,
&output_metadata)
.IsNotFound());
ASSERT_TRUE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "range_key",
&value, &output_metadata));
ASSERT_EQ("r1", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "merge_key",
&value, &output_metadata));
ASSERT_EQ("base", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
ASSERT_OK(Put(1, "key", "v2"));
value.clear();
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("v1", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
ASSERT_OK(Flush(1));
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("v1", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(snapshot_read, handles_[1], "range_key",
&value, &output_metadata));
ASSERT_EQ("r1", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
ReadOptions latest_read;
value.clear();
*output_metadata.newer_version_present = true;
ASSERT_OK(db_->GetWithMetadata(latest_read, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("v2", value);
ASSERT_FALSE(*output_metadata.newer_version_present);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, NewerVersionPresentRequestedByOutputMetadata) {
// Newer-version tracking is opt-in through OutputMetadata and
// MultiGetOutputMetadata.
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "old"));
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(Put(1, "key", "new"));
ReadOptions read_options;
read_options.snapshot = snapshot;
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("old", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
std::vector<Slice> keys{"key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_EQ(1, multiget_output_metadata.newer_version_present->size());
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[0]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, NewerVersionPresentIgnoreRangeDeletions) {
// ignore_range_deletions affects the returned value, not newer-version
// metadata about covering range tombstones.
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "range_key", "old"));
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(
db_->DeleteRange(WriteOptions(), handles_[1], "range_key", "range_key~"));
ReadOptions read_options;
read_options.snapshot = snapshot;
read_options.ignore_range_deletions = true;
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "range_key", &value,
&output_metadata));
ASSERT_EQ("old", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
std::vector<Slice> keys{"range_key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[0]);
ASSERT_OK(Flush(1));
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "range_key", &value,
&output_metadata));
ASSERT_EQ("old", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
values.clear();
multiget_output_metadata.newer_version_present->clear();
statuses = db_->MultiGetWithMetadata(read_options, cfs, keys, &values,
&multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[0]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, NewerVersionPresentFromImmutableMemTable) {
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "point_key", "old_point"));
ASSERT_OK(Put(1, "range_key", "old_range"));
ManagedSnapshot snapshot(db_.get());
ASSERT_OK(Put(1, "point_key", "new_point"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), handles_[1], "range_key", "range_key~"));
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(handles_[1])->cfd();
Status resume_status;
{
ASSERT_OK(db_->PauseBackgroundWork());
Defer resume_background(
[&] { resume_status = db_->ContinueBackgroundWork(); });
ASSERT_OK(dbfull()->TEST_SwitchMemtable(cfd));
ReadOptions read_options;
read_options.snapshot = snapshot.snapshot();
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "point_key",
&value, &output_metadata));
ASSERT_EQ("old_point", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
value.clear();
*output_metadata.newer_version_present = false;
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "range_key",
&value, &output_metadata));
ASSERT_EQ("old_range", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
std::vector<Slice> keys{"point_key", "range_key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
const std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(2, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_OK(statuses[1]);
ASSERT_EQ(std::vector<std::string>({"old_point", "old_range"}), values);
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[0]);
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[1]);
}
ASSERT_OK(resume_status);
ASSERT_OK(Flush(1));
}
TEST_F(DBBasicTest, NewerVersionPresentBypassesRowCacheWhileTracking) {
// Prime row-cache entries for the snapshot values, then verify metadata
// reads still inspect SST sequence numbers instead of replaying those
// entries.
option_config_ = kRowCache;
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "get_key", "old_get"));
ASSERT_OK(Put(1, "multiget_key", "old_multiget"));
ASSERT_OK(Flush(1));
ManagedSnapshot snapshot(db_.get());
ASSERT_OK(Put(1, "get_key", "new_get"));
ASSERT_OK(Put(1, "multiget_key", "new_multiget"));
ASSERT_OK(Flush(1));
ReadOptions read_options;
read_options.snapshot = snapshot.snapshot();
std::string primed_value;
ASSERT_OK(db_->Get(read_options, handles_[1], "get_key", &primed_value));
ASSERT_EQ("old_get", primed_value);
std::vector<Slice> primed_keys{"multiget_key"};
std::vector<ColumnFamilyHandle*> primed_cfs(primed_keys.size(), handles_[1]);
std::vector<std::string> primed_values;
const std::vector<Status> primed_statuses =
db_->MultiGet(read_options, primed_cfs, primed_keys, &primed_values);
ASSERT_OK(primed_statuses[0]);
ASSERT_EQ("old_multiget", primed_values[0]);
for (int i = 0; i < 2; ++i) {
SCOPED_TRACE(i);
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "get_key", &value,
&output_metadata));
ASSERT_EQ("old_get", value);
ASSERT_TRUE(*output_metadata.newer_version_present);
}
std::vector<Slice> keys{"multiget_key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
for (int i = 0; i < 2; ++i) {
SCOPED_TRACE(i);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("old_multiget", values[0]);
ASSERT_TRUE((*multiget_output_metadata.newer_version_present)[0]);
}
}
TEST_F(DBBasicTest, NewerVersionPresentUsesRowCacheAfterObservation) {
option_config_ = kRowCache;
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "old"));
ASSERT_OK(Flush(1));
ManagedSnapshot snapshot(db_.get());
ReadOptions read_options;
read_options.snapshot = snapshot.snapshot();
std::vector<Slice> keys{"key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
std::vector<Status> statuses =
db_->MultiGet(read_options, cfs, keys, &values);
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_OK(Put(1, "key", "new"));
const uint64_t row_cache_hits = TestGetTickerCount(options, ROW_CACHE_HIT);
values.clear();
MultiGetOutputMetadata output_metadata = NewerVersionMultiGetOutputMetadata();
statuses = db_->MultiGetWithMetadata(read_options, cfs, keys, &values,
&output_metadata);
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_TRUE((*output_metadata.newer_version_present)[0]);
ASSERT_EQ(row_cache_hits + 1, TestGetTickerCount(options, ROW_CACHE_HIT));
}
TEST_F(DBBasicTest, NewerVersionPresentPersistedTierNotSupported) {
// kPersistedTier is incompatible with snapshot newer-version tracking but not
// with untracked latest-version reads.
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "old"));
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(Put(1, "key", "new"));
ReadOptions read_options;
read_options.snapshot = snapshot;
read_options.read_tier = kPersistedTier;
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_TRUE(db_->GetWithMetadata(read_options, handles_[1], "key", &value,
&output_metadata)
.IsNotSupported());
std::vector<Slice> keys{"key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_TRUE(statuses[0].IsNotSupported());
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[0]);
read_options.snapshot = nullptr;
value.clear();
*output_metadata.newer_version_present = true;
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("new", value);
ASSERT_FALSE(*output_metadata.newer_version_present);
values.clear();
multiget_output_metadata.newer_version_present->clear();
statuses = db_->MultiGetWithMetadata(read_options, cfs, keys, &values,
&multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("new", values[0]);
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[0]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, ReadOnlyNewerVersionPresentReturnsFalse) {
// Read-only DBs do not widen snapshot reads, so requested metadata remains
// false.
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "key", "value"));
ASSERT_OK(Flush(1));
ASSERT_OK(
TryReopenReadOnlyWithColumnFamilies({"default", "pikachu"}, options));
const Snapshot* snapshot = db_->GetSnapshot();
ReadOptions read_options;
read_options.snapshot = snapshot;
std::string value;
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_OK(db_->GetWithMetadata(read_options, handles_[1], "key", &value,
&output_metadata));
ASSERT_EQ("value", value);
ASSERT_FALSE(*output_metadata.newer_version_present);
std::vector<Slice> keys{"key"};
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata multiget_output_metadata =
NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &multiget_output_metadata);
ASSERT_EQ(1, statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("value", values[0]);
ASSERT_EQ(1, multiget_output_metadata.newer_version_present->size());
ASSERT_FALSE((*multiget_output_metadata.newer_version_present)[0]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, CheckLock) {
do {
std::unique_ptr<DB> localdb;
@@ -2188,6 +2766,151 @@ TEST_F(DBBasicTest, MultiGetSimple) {
} while (ChangeCompactOptions());
}
TEST_F(DBBasicTest, MultiGetNewerVersionPresent) {
// A mixed MultiGet batch reports newer writes independently for each key in
// input order.
Options options = CurrentOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_OK(Put(1, "updated", "old"));
ASSERT_OK(Put(1, "unchanged", "same"));
ASSERT_OK(Put(1, "deleted", "old"));
ASSERT_OK(Put(1, "merged", "old"));
ASSERT_OK(Put(1, "range_deleted", "old"));
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(Put(1, "updated", "new"));
ASSERT_OK(Put(1, "created_after_put", "new"));
ASSERT_OK(Merge(1, "created_after_merge", "new"));
ASSERT_OK(Delete(1, "deleted"));
ASSERT_OK(Merge(1, "merged", "new"));
ASSERT_OK(db_->DeleteRange(WriteOptions(), handles_[1], "range_deleted",
"range_deleted~"));
ReadOptions read_options;
read_options.snapshot = snapshot;
std::vector<Slice> keys({"updated", "unchanged", "deleted", "merged",
"range_deleted", "created_after_put",
"created_after_merge", "missing"});
std::vector<ColumnFamilyHandle*> cfs(keys.size(), handles_[1]);
std::vector<std::string> values;
MultiGetOutputMetadata output_metadata = NewerVersionMultiGetOutputMetadata();
std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, cfs, keys, &values, &output_metadata);
ASSERT_EQ(keys.size(), statuses.size());
ASSERT_OK(statuses[0]);
ASSERT_EQ("old", values[0]);
ASSERT_TRUE((*output_metadata.newer_version_present)[0]);
ASSERT_OK(statuses[1]);
ASSERT_EQ("same", values[1]);
ASSERT_FALSE((*output_metadata.newer_version_present)[1]);
ASSERT_OK(statuses[2]);
ASSERT_EQ("old", values[2]);
ASSERT_TRUE((*output_metadata.newer_version_present)[2]);
ASSERT_OK(statuses[3]);
ASSERT_EQ("old", values[3]);
ASSERT_TRUE((*output_metadata.newer_version_present)[3]);
ASSERT_OK(statuses[4]);
ASSERT_EQ("old", values[4]);
ASSERT_TRUE((*output_metadata.newer_version_present)[4]);
ASSERT_TRUE(statuses[5].IsNotFound());
ASSERT_TRUE((*output_metadata.newer_version_present)[5]);
ASSERT_TRUE(statuses[6].IsNotFound());
ASSERT_TRUE((*output_metadata.newer_version_present)[6]);
ASSERT_TRUE(statuses[7].IsNotFound());
ASSERT_FALSE((*output_metadata.newer_version_present)[7]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, MultiGetNewerVersionPresentAcrossColumnFamilies) {
// Unsorted mixed-CF reads keep metadata aligned with input order after each
// CF is processed independently.
Options options = CurrentOptions();
CreateAndReopenWithCF({"one", "two"}, options);
ASSERT_OK(Put(1, "updated_one", "old_one"));
ASSERT_OK(Put(2, "updated_two", "old_two"));
ASSERT_OK(Put(2, "stable_two", "stable"));
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(Put(1, "updated_one", "new_one"));
ASSERT_OK(Put(2, "updated_two", "new_two"));
ASSERT_OK(Flush(1));
ASSERT_OK(Flush(2));
ReadOptions read_options;
read_options.snapshot = snapshot;
std::vector<Slice> keys{"updated_two", "updated_one", "stable_two"};
std::vector<ColumnFamilyHandle*> column_families{handles_[2], handles_[1],
handles_[2]};
std::vector<std::string> values;
MultiGetOutputMetadata output_metadata = NewerVersionMultiGetOutputMetadata();
const std::vector<Status> statuses = db_->MultiGetWithMetadata(
read_options, column_families, keys, &values, &output_metadata);
for (const Status& status : statuses) {
ASSERT_OK(status);
}
ASSERT_EQ(std::vector<std::string>({"old_two", "old_one", "stable"}), values);
ASSERT_TRUE((*output_metadata.newer_version_present)[0]);
ASSERT_TRUE((*output_metadata.newer_version_present)[1]);
ASSERT_FALSE((*output_metadata.newer_version_present)[2]);
db_->ReleaseSnapshot(snapshot);
}
TEST_F(DBBasicTest, MultiGetWithMetadataColumnFamiliesKeysSizeMismatch) {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
std::vector<Slice> keys({"k1", "k2"});
std::vector<ColumnFamilyHandle*> cfs({handles_[1]});
std::vector<std::string> values({"stale"});
MultiGetOutputMetadata output_metadata;
output_metadata.timestamps.emplace();
output_metadata.newer_version_present = std::vector<uint8_t>({1});
std::vector<Status> statuses = db_->MultiGetWithMetadata(
ReadOptions(), cfs, keys, &values, &output_metadata);
ASSERT_EQ(keys.size(), statuses.size());
ASSERT_EQ(keys.size(), values.size());
for (const Status& status : statuses) {
ASSERT_TRUE(status.IsInvalidArgument());
}
ASSERT_EQ(keys.size(), output_metadata.timestamps->size());
ASSERT_EQ(keys.size(), output_metadata.newer_version_present->size());
ASSERT_FALSE((*output_metadata.newer_version_present)[0]);
ASSERT_FALSE((*output_metadata.newer_version_present)[1]);
}
TEST_F(DBBasicTest, GetWithMetadataNullValueInvalidArgument) {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
OutputMetadata output_metadata = NewerVersionOutputMetadata();
ASSERT_TRUE(db_->GetWithMetadata(ReadOptions(), handles_[1], "key",
static_cast<PinnableSlice*>(nullptr),
&output_metadata)
.IsInvalidArgument());
ASSERT_FALSE(*output_metadata.newer_version_present);
*output_metadata.newer_version_present = true;
ASSERT_TRUE(db_->GetWithMetadata(ReadOptions(), "key",
static_cast<std::string*>(nullptr),
&output_metadata)
.IsInvalidArgument());
ASSERT_FALSE(*output_metadata.newer_version_present);
}
TEST_F(DBBasicTest, MultiGetEmpty) {
do {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
@@ -2227,6 +2950,9 @@ INSTANTIATE_TEST_CASE_P(FormatVersions, DBBlockChecksumTest,
TEST_P(DBBlockChecksumTest, BlockChecksumTest) {
BlockBasedTableOptions table_options;
table_options.format_version = GetParam();
// kFooterFormatVersionsToTest includes the unpublished draft format_version
// 8; writing it requires this opt-in.
SaveAndRestore<bool> allow_draft(&TEST_AllowUnsupportedFormatVersion(), true);
Options options = CurrentOptions();
const int kNumPerFile = 2;
@@ -5042,6 +5768,9 @@ INSTANTIATE_TEST_CASE_P(BestEffortsRecoverIncompleteVersionTest,
TEST_F(DBBasicTest, BestEffortsRecoveryTryMultipleManifests) {
Options options = CurrentOptions();
options.env = env_;
options.write_buffer_manager = std::make_shared<WriteBufferManager>(
64 << 20, nullptr /* cache */, false /* allow_stall */,
WriteBufferFlushPolicy::kFlushLargestAcrossDBs);
DestroyAndReopen(options);
ASSERT_OK(Put("foo", "value0"));
ASSERT_OK(Flush());
@@ -5063,6 +5792,9 @@ TEST_F(DBBasicTest, BestEffortsRecoveryTryMultipleManifests) {
Reopen(options);
ASSERT_OK(Put("bar", "value"));
options.write_buffer_manager->TEST_RefreshFlushInitiatorCandidate();
ASSERT_TRUE(
options.write_buffer_manager->TEST_ScheduleFlushOnLargestDB(nullptr));
}
TEST_F(DBBasicTest, RecoverWithNoCurrentFile) {
+6
View File
@@ -2542,9 +2542,13 @@ TEST_P(DBBloomFilterTestVaryPrefixAndFormatVer, PartitionedMultiGet) {
values[i] = PinnableSlice();
}
get_perf_context()->Reset();
const PerfLevel previous_perf_level = GetPerfLevel();
SetPerfLevel(PerfLevel::kEnableTime);
db_->MultiGet(ropts, Q, column_families.data(), key_slices.data(),
values.data(),
/*timestamps=*/nullptr, statuses.data(), true);
SetPerfLevel(previous_perf_level);
// Confirm correct status results
uint32_t number_not_found = 0;
@@ -2574,6 +2578,8 @@ TEST_P(DBBloomFilterTestVaryPrefixAndFormatVer, PartitionedMultiGet) {
// Confirm no duplicate loading same filter partition
uint64_t filter_accesses = PopTicker(options, BLOCK_CACHE_FILTER_HIT) +
PopTicker(options, BLOCK_CACHE_FILTER_MISS);
// Partition reads must contribute to the filter-block timing metric.
EXPECT_GT(get_perf_context()->read_filter_block_nanos, 0);
if (stride == 1) {
EXPECT_EQ(filter_accesses, 1);
} else {
+81
View File
@@ -3,8 +3,11 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include <atomic>
#include "db/db_test_util.h"
#include "port/port.h"
#include "test_util/sync_point.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -14,6 +17,62 @@ class DBClipTest : public DBTestBase {
DBClipTest() : DBTestBase("db_clip_test", /*env_do_fsync=*/true) {}
};
// ClipColumnFamily reads the CF's boundary keys under the DB mutex, then drops
// it and issues several writes with those keys. Those keys must not point into
// Version-owned FileMetaData, which a background job can replace and free in
// that window. Here a full compaction runs inside the window and obsoletes the
// Version the keys were read from.
TEST_F(DBClipTest, BoundaryKeysSurviveConcurrentVersionChange) {
Options options = CurrentOptions();
options.disable_auto_compactions = true;
options.num_levels = 3;
DestroyAndReopen(options);
Random rnd(301);
std::map<int32_t, std::string> values;
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 100; j++) {
int k = i * 100 + j;
values[k] = rnd.RandomString(1024);
ASSERT_OK(Put(Key(k), values[k]));
}
ASSERT_OK(Flush());
}
std::atomic<bool> fired{false};
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::ClipColumnFamily:PostBoundaryKeys", [&](void*) {
if (fired.exchange(true)) {
return;
}
// Replace and free the Version the boundary keys were read from, on
// another thread so this does not re-enter Clip's own call stack.
port::Thread compactor([&] {
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForPurge());
});
compactor.join();
});
SyncPoint::GetInstance()->EnableProcessing();
const auto begin_key = Key(251);
const auto end_key = Key(751);
ASSERT_OK(
db_->ClipColumnFamily(db_->DefaultColumnFamily(), begin_key, end_key));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_TRUE(fired.load());
for (int i = 251; i < 751; i++) {
ASSERT_EQ(Get(Key(i)), values[i]);
}
for (int i : {0, 250, 751, 999}) {
std::string result;
ASSERT_TRUE(db_->Get(ReadOptions(), Key(i), &result).IsNotFound());
}
}
TEST_F(DBClipTest, TestClipRange) {
Options options = CurrentOptions();
options.write_buffer_size = 10 * 1024 * 1024;
@@ -133,6 +192,28 @@ TEST_F(DBClipTest, TestClipRange) {
ASSERT_TRUE(in_range);
}
}
TEST_F(DBClipTest, KeepFileEndingAtBeginKey) {
Options options = CurrentOptions();
options.disable_auto_compactions = true;
DestroyAndReopen(options);
ASSERT_OK(Put("a", "va"));
ASSERT_OK(Put("b", "vb"));
ASSERT_OK(Flush());
MoveFilesToLevel(1);
ASSERT_OK(Put("c", "vc"));
ASSERT_OK(Put("d", "vd"));
ASSERT_OK(Flush());
MoveFilesToLevel(1);
ASSERT_EQ("0,2", FilesPerLevel(0));
ASSERT_OK(db_->ClipColumnFamily(db_->DefaultColumnFamily(), "b", "d"));
ASSERT_EQ("NOT_FOUND", Get("a"));
ASSERT_EQ("vb", Get("b"));
ASSERT_EQ("vc", Get("c"));
ASSERT_EQ("NOT_FOUND", Get("d"));
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+294 -10
View File
@@ -14,6 +14,7 @@
#include "db/blob/blob_index.h"
#include "db/db_test_util.h"
#include "db/dbformat.h"
#include "db/periodic_compaction_phaser.h"
#include "db/table_cache.h"
#include "env/mock_env.h"
#include "file/filename.h"
@@ -29,6 +30,8 @@
#include "test_util/sync_point.h"
#include "test_util/testutil.h"
#include "util/concurrent_task_limiter_impl.h"
#include "util/fastrange.h"
#include "util/hash.h"
#include "util/random.h"
#include "utilities/fault_injection_env.h"
#include "utilities/fault_injection_fs.h"
@@ -1334,6 +1337,118 @@ TEST_F(DBCompactionTest, CompactionSstPartitionerNonTrivial) {
ASSERT_EQ("B", Get("bbbb1"));
}
TEST_F(DBCompactionTest, CompactionSstPartitionerPrefixFastPath) {
// Exercises the ShouldPartitionByPrefix() fast path in CompactionOutputs:
// multiple keys sharing a fixed prefix must stay together in one output file
// (no spurious cuts), with boundaries landing exactly at prefix changes.
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 3;
options.sst_partitioner_factory = NewSstPartitionerFixedPrefixFactory(4);
DestroyAndReopen(options);
// Three prefixes with multiple keys each, spread across two flushed files so
// that a non-trivial compaction merges them.
ASSERT_OK(Put("aaaa1", "v"));
ASSERT_OK(Put("aaaa3", "v"));
ASSERT_OK(Put("bbbb1", "v"));
ASSERT_OK(Put("cccc2", "v"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(Put("aaaa2", "v"));
ASSERT_OK(Put("bbbb2", "v"));
ASSERT_OK(Put("cccc1", "v"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
// Exactly one output file per distinct 4-byte prefix.
std::vector<LiveFileMetaData> files;
dbfull()->GetLiveFilesMetaData(&files);
ASSERT_EQ(3, files.size());
// Each file holds a whole prefix group (multiple keys => the fast path
// correctly did NOT cut within a shared prefix).
for (const auto& f : files) {
ASSERT_EQ(f.smallestkey.substr(0, 4), f.largestkey.substr(0, 4));
}
for (const std::string k :
{"aaaa1", "aaaa2", "aaaa3", "bbbb1", "bbbb2", "cccc1", "cccc2"}) {
ASSERT_EQ("v", Get(k));
}
}
TEST_F(DBCompactionTest, CompactionSstPartitionerPrefixShortKeyFallback) {
// Keys shorter than the fixed prefix length cannot be represented as a
// prefix, so ShouldPartitionByPrefix() returns no value and CompactionOutputs
// falls back to per-key ShouldPartition(). This also exercises short->long
// and long->short mode transitions across output file boundaries.
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 3;
options.sst_partitioner_factory = NewSstPartitionerFixedPrefixFactory(4);
DestroyAndReopen(options);
// "a" and "bb" are shorter than len 4 (fallback, each in its own file);
// "cccc1"/"cccc2" share prefix "cccc" (fast path, one file); "dd" is short
// again (fallback, own file).
ASSERT_OK(Put("a", "v"));
ASSERT_OK(Put("bb", "v"));
ASSERT_OK(Put("cccc1", "v"));
ASSERT_OK(Put("dd", "v"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
// Second file overlaps to force a non-trivial (rewriting) compaction.
ASSERT_OK(Put("cccc2", "v"));
ASSERT_OK(Put("a", "v2"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
// "a" | "bb" | {"cccc1","cccc2"} | "dd" => 4 files.
std::vector<LiveFileMetaData> files;
dbfull()->GetLiveFilesMetaData(&files);
ASSERT_EQ(4, files.size());
ASSERT_EQ("v2", Get("a"));
for (const std::string k : {"bb", "cccc1", "cccc2", "dd"}) {
ASSERT_EQ("v", Get(k));
}
}
TEST_F(DBCompactionTest, CompactionSstPartitionerZeroLenNoSplit) {
// A zero-length fixed prefix yields an empty (but present) prefix from
// ShouldPartitionByPrefix(), which means "no further partitions": every key
// "starts with" the empty prefix, so the fast path never cuts.
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 3;
options.sst_partitioner_factory = NewSstPartitionerFixedPrefixFactory(0);
DestroyAndReopen(options);
ASSERT_OK(Put("aaaa1", "v1"));
ASSERT_OK(Put("bbbb1", "v1"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
// Overlapping key forces a non-trivial compaction (not a trivial move).
ASSERT_OK(Put("aaaa1", "v2"));
ASSERT_OK(Put("cccc1", "v2"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
std::vector<LiveFileMetaData> files;
dbfull()->GetLiveFilesMetaData(&files);
ASSERT_EQ(1, files.size());
ASSERT_EQ("v2", Get("aaaa1"));
ASSERT_EQ("v1", Get("bbbb1"));
ASSERT_EQ("v2", Get("cccc1"));
}
TEST_F(DBCompactionTest, ZeroSeqIdCompaction) {
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
@@ -5235,10 +5350,10 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
*max_open_files = 2;
});
// In the case where all files are opened and doing DB restart
// forcing the oldest ancester time in manifest file to be 0 to
// forcing the oldest ancestor time in manifest file to be 0 to
// simulate the case of reading from an old version.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionEdit::EncodeTo:VarintOldestAncesterTime", [&](void* arg) {
"VersionEdit::EncodeTo:VarintOldestAncestorTime", [&](void* arg) {
if (if_restart && if_open_all_files) {
std::string* encoded_field = static_cast<std::string*>(arg);
*encoded_field = "";
@@ -5275,7 +5390,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
std::vector<std::vector<FileMetaData>> level_to_files;
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
&level_to_files);
uint64_t oldest_time = level_to_files[0][0].oldest_ancester_time;
uint64_t oldest_time = level_to_files[0][0].oldest_ancestor_time;
// Add 1 hour and do another flush.
env_->MockSleepForSeconds(1 * 60 * 60);
for (int i = 101; i <= 200; ++i) {
@@ -5343,7 +5458,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
&level_to_files);
ASSERT_EQ(oldest_time, level_to_files[6][0].oldest_ancester_time);
ASSERT_EQ(oldest_time, level_to_files[6][0].oldest_ancestor_time);
env_->MockSleepForSeconds(25 * 60 * 60);
ASSERT_OK(Put(Key(2), "1"));
@@ -5371,6 +5486,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
for (bool if_open_all_files : {false, true}) {
Options options = CurrentOptions();
options.periodic_compaction_seconds = 48 * 60 * 60; // 2 days
options.periodic_compaction_phase_recovery_percent = 0;
if (if_open_all_files) {
options.max_open_files = -1; // needed for ttl compaction
} else {
@@ -5466,6 +5582,169 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
}
}
TEST_F(DBCompactionTest, PeriodicCompactionPhaseTriggerTime) {
// Unit-tests PeriodicCompactionPhaser::TriggerTime, the core of
// preferred-phase periodic compaction (see
// DBOptions::periodic_compaction_phase_recovery_percent).
using Params = PeriodicCompactionPhaseParams;
const uint64_t kN = 1000; // periodic_compaction_seconds
// The trigger-time helper treats seed_hash as an opaque value; use a fixed
// constant and derive the resulting preferred-phase target the same way
// production does (FastRange64 of the seed over the interval).
const uint64_t kSeed = 0x9e3779b97f4a7c15ULL;
const uint64_t target = FastRange64(kSeed, kN);
ASSERT_LT(target, kN);
// recovery_percent == 0 => classic behavior: trigger at file_mod + N.
{
Params p;
p.seed_hash = kSeed;
p.recovery_percent = 0;
for (uint64_t fm : {uint64_t{1}, uint64_t{7}, uint64_t{12345}}) {
ASSERT_EQ(PeriodicCompactionPhaser::TriggerTime(fm, kN, p), fm + kN);
}
}
// Steady state (anchor in the distant past): trigger is pulled earlier by
// recovery_percent% of the phase offset, and never later than the deadline.
{
Params p;
p.seed_hash = kSeed;
p.recovery_percent = 50;
p.anchor_time = 0;
const uint64_t fm = 500;
const uint64_t natural = fm + kN;
const uint64_t offset = (fm % kN + kN - target) % kN;
const uint64_t expected = natural - offset * 50 / 100;
ASSERT_EQ(PeriodicCompactionPhaser::TriggerTime(fm, kN, p), expected);
ASSERT_LE(expected, natural);
}
// Geometric convergence: repeatedly re-stamp the file at its trigger time and
// confirm the phase error is non-increasing and converges to (near) zero.
for (int recovery_percent : {50, 100}) {
Params p;
p.seed_hash = kSeed;
p.recovery_percent = recovery_percent;
p.anchor_time = 0;
uint64_t fm = 123;
uint64_t prev_error = kN;
for (int cycle = 0; cycle < 64; ++cycle) {
const uint64_t trig = PeriodicCompactionPhaser::TriggerTime(fm, kN, p);
ASSERT_LE(trig, fm + kN); // never later than the deadline
const uint64_t error = (trig % kN + kN - target) % kN;
ASSERT_LE(error, prev_error);
prev_error = error;
fm = trig;
}
// rp=100 pulls the whole gap so it reaches the phase exactly; smaller rates
// decay geometrically and leave a tiny integer-rounding residue.
if (recovery_percent == 100) {
ASSERT_EQ(prev_error, uint64_t{0});
} else {
ASSERT_LE(prev_error, uint64_t{1});
}
}
// Already past the hard deadline when phasing took effect (e.g. the DB was
// down, or periodic_compaction_seconds was turned down): instead of firing
// the whole cohort immediately, spread it over the first quarter-period after
// the anchor on an absolute (epoch-aligned) grid of period N/4.
{
Params p;
p.seed_hash = kSeed;
p.recovery_percent = 50;
p.anchor_time = 200;
const uint64_t n = 100;
const uint64_t grid = n / 4; // 25
const uint64_t phase = FastRange64(kSeed, grid);
// natural deadline (10 + 100 = 110) is before the anchor (200).
const uint64_t expected = 200 + (phase + grid - 200 % grid) % grid;
const uint64_t trig = PeriodicCompactionPhaser::TriggerTime(10, n, p);
ASSERT_EQ(trig, expected);
ASSERT_GE(trig, uint64_t{200}); // never before the anchor
ASSERT_LT(trig, 200 + grid); // within N/4 of the anchor
}
// Past due against the recovery target but not yet at the deadline => spread
// across [anchor, deadline] using the phase; never later than the deadline.
{
const uint64_t n = 100;
const uint64_t small_target = FastRange64(kSeed, n);
const uint64_t fm = small_target + 50 + 3 * n; // phase offset == 50
const uint64_t natural = fm + n;
const uint64_t anchor =
natural - 25; // recovery target (natural-50) < anchor
Params p;
p.seed_hash = kSeed;
p.recovery_percent = 100;
p.anchor_time = anchor;
ASSERT_EQ((fm % n + n - small_target) % n, uint64_t{50});
const uint64_t expected = anchor + FastRange64(kSeed, natural - anchor);
const uint64_t trig = PeriodicCompactionPhaser::TriggerTime(fm, n, p);
ASSERT_EQ(trig, expected);
ASSERT_GE(trig, anchor);
ASSERT_LT(trig, natural);
}
}
TEST_F(DBCompactionTest, PeriodicCompactionPhaseCfSpread) {
// Unit-tests PeriodicCompactionPhaser::CfPhaseSeedHash: a DB's column
// families are spread quasi-uniformly around the DB base phase via the
// golden-ratio recurrence, for any base and any number of CFs.
using Phaser = PeriodicCompactionPhaser;
// cf_id 0 leaves the base phase unchanged; distinct cf_ids -> distinct
// phases.
for (uint64_t base : {uint64_t{0}, uint64_t{0x9e3779b97f4a7c15ULL},
uint64_t{12345678901234567ULL}, ~uint64_t{0}}) {
ASSERT_EQ(Phaser::CfPhaseSeedHash(base, 0), base);
std::vector<uint64_t> hs;
for (uint32_t id = 0; id < 64; ++id) {
hs.push_back(Phaser::CfPhaseSeedHash(base, id));
}
std::sort(hs.begin(), hs.end());
for (size_t i = 1; i < hs.size(); ++i) {
ASSERT_NE(hs[i], hs[i - 1]) << "duplicate CF phase seed at base=" << base;
}
}
// Low-discrepancy: for N CFs the phases over an interval are well spread --
// no gap larger than ~2x the average (a hash would allow chance clustering).
for (uint64_t base : {uint64_t{0}, uint64_t{0xdeadbeefcafef00dULL}}) {
for (uint32_t N : {uint32_t{3}, uint32_t{8}, uint32_t{16}, uint32_t{40}}) {
const uint64_t n = 100000;
std::vector<uint64_t> phases;
for (uint32_t id = 0; id < N; ++id) {
phases.push_back(FastRange64(Phaser::CfPhaseSeedHash(base, id), n));
}
std::sort(phases.begin(), phases.end());
uint64_t max_gap = n - phases.back() + phases.front(); // wrap-around gap
for (size_t i = 1; i < phases.size(); ++i) {
max_gap = std::max(max_gap, phases[i] - phases[i - 1]);
}
ASSERT_LE(max_gap, 2 * (n / N))
<< "base=" << base << " N=" << N << " max_gap=" << max_gap;
}
}
}
TEST_F(DBCompactionTest, PeriodicCompactionPhaseOptions) {
// Verifies periodic_compaction_phase_recovery_percent is plumbed and
// dynamically settable.
Options options = CurrentOptions();
options.periodic_compaction_phase_recovery_percent = 25;
DestroyAndReopen(options);
ASSERT_EQ(dbfull()->GetDBOptions().periodic_compaction_phase_recovery_percent,
25);
ASSERT_OK(dbfull()->SetDBOptions(
{{"periodic_compaction_phase_recovery_percent", "0"}}));
ASSERT_EQ(dbfull()->GetDBOptions().periodic_compaction_phase_recovery_percent,
0);
}
TEST_F(DBCompactionTest, LevelPeriodicCompactionOffpeak) {
// This test simply checks if offpeak adjustment works in Leveled
// Compactions. For testing offpeak periodic compactions in various
@@ -5483,6 +5762,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionOffpeak) {
Options options = CurrentOptions();
options.ttl = 0;
options.periodic_compaction_seconds = 5 * kSecondsPerDay; // 5 days
options.periodic_compaction_phase_recovery_percent = 0;
// In the case where all files are opened and doing DB restart
// forcing the file creation time in manifest file to be 0 to
// simulate the case of reading from an old version.
@@ -5601,7 +5881,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
props->file_creation_time = 0;
}
if (set_creation_time_to_zero) {
props->creation_time = 0;
props->oldest_ancestor_time = 0;
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -5630,6 +5910,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
// Forward the clock by 2 days.
env_->MockSleepForSeconds(2 * 24 * 60 * 60);
options.periodic_compaction_seconds = 1 * 24 * 60 * 60; // 1 day
options.periodic_compaction_phase_recovery_percent = 0;
Reopen(options);
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -5648,6 +5929,7 @@ TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
Options options = CurrentOptions();
options.ttl = 10 * 60 * 60; // 10 hours
options.periodic_compaction_seconds = 48 * 60 * 60; // 2 days
options.periodic_compaction_phase_recovery_percent = 0;
options.max_open_files = -1; // needed for both periodic and ttl compactions
env_->SetMockSleep();
options.env = env_;
@@ -5730,6 +6012,7 @@ TEST_F(DBCompactionTest, LevelTtlBooster) {
Options options = CurrentOptions();
options.ttl = 10 * 60 * 60; // 10 hours
options.periodic_compaction_seconds = 480 * 60 * 60; // very long
options.periodic_compaction_phase_recovery_percent = 0;
options.level0_file_num_compaction_trigger = 2;
options.max_bytes_for_level_base = 5 * uint64_t{kNumKeysPerFile * kValueSize};
options.max_open_files = -1; // needed for both periodic and ttl compactions
@@ -12134,7 +12417,7 @@ TEST_F(DBCompactionTest, RecordNewestKeyTimeForTtlCompaction) {
ASSERT_EQ(newest_key_time, file_metadatas[idx]
->fd.pinned_reader.Get()
->GetTableProperties()
->creation_time);
->oldest_ancestor_time);
}
// The delta between the first and last newest_key_times is 15s
uint64_t last_newest_key_time = prev_newest_key_time;
@@ -12153,14 +12436,14 @@ TEST_F(DBCompactionTest, RecordNewestKeyTimeForTtlCompaction) {
->GetTableProperties()
->newest_key_time,
first_newest_key_time);
// Contrast newest_key_time with creation_time, which records the oldest
// ancestor time (15s older than newest_key_time)
// Contrast newest_key_time with oldest_ancestor_time (15s older than
// newest_key_time)
ASSERT_EQ(file_metadatas[0]
->fd.pinned_reader.Get()
->GetTableProperties()
->creation_time,
->oldest_ancestor_time,
last_newest_key_time);
ASSERT_EQ(file_metadatas[0]->oldest_ancester_time, last_newest_key_time);
ASSERT_EQ(file_metadatas[0]->oldest_ancestor_time, last_newest_key_time);
// Make sure TTL of 5s causes compaction
env_->MockSleepForSeconds(6);
@@ -12279,6 +12562,7 @@ TEST_F(DBCompactionTest, PeriodicTask) {
options.statistics = CreateDBStatistics();
int kPeriodicCompactionSeconds = 7 * 24 * 60 * 60; // 1 week
options.periodic_compaction_seconds = kPeriodicCompactionSeconds;
options.periodic_compaction_phase_recovery_percent = 0;
options.num_levels = 50;
auto listener = std::make_shared<PeriodicCompactionListener>();
options.listeners.push_back(listener);
+183
View File
@@ -4271,6 +4271,147 @@ TEST_F(DBTest2, TraceWithLimit) {
ASSERT_OK(DestroyDB(dbname2, options));
}
namespace {
Status ReplayTraceFile(Env* env, const std::string& trace_filename, DB* db,
const std::vector<ColumnFamilyHandle*>& handles,
uint32_t num_threads,
TraceExecutionResultHandler* res_handler = nullptr) {
std::unique_ptr<TraceReader> trace_reader;
Status s =
NewFileTraceReader(env, EnvOptions(), trace_filename, &trace_reader);
if (!s.ok()) {
return s;
}
std::unique_ptr<Replayer> replayer;
s = db->NewDefaultReplayer(handles, std::move(trace_reader), &replayer);
if (!s.ok()) {
return s;
}
s = replayer->Prepare();
if (!s.ok()) {
return s;
}
// Count the replayed records by type in `res_handler`, if given.
std::function<void(Status, std::unique_ptr<TraceRecordResult>&&)> res_cb;
if (res_handler != nullptr) {
res_cb = [res_handler](const Status& exec_s,
std::unique_ptr<TraceRecordResult>&& res) {
exec_s.PermitUncheckedError();
if (res != nullptr) {
EXPECT_OK(res->Accept(res_handler));
}
};
}
return replayer->Replay(ReplayOptions(num_threads, 1.0), res_cb);
}
} // anonymous namespace
TEST_F(DBTest2, TraceReplayDefaultColumnFamilyWithoutHandles) {
Options options = CurrentOptions();
std::string trace_filename = dbname_ + "/rocksdb.trace";
std::unique_ptr<TraceWriter> trace_writer;
ASSERT_OK(
NewFileTraceWriter(env_, EnvOptions(), trace_filename, &trace_writer));
ASSERT_OK(db_->StartTrace(TraceOptions(), std::move(trace_writer)));
ASSERT_OK(Put("a", "1"));
ASSERT_OK(Put("b", "2"));
ASSERT_EQ("1", Get("a"));
std::vector<std::string> values;
for (const Status& s :
db_->MultiGet(ReadOptions(), {Slice("a"), Slice("b")}, &values)) {
ASSERT_OK(s);
}
std::unique_ptr<Iterator> iter(db_->NewIterator(ReadOptions()));
iter->Seek("a");
iter->SeekForPrev("b");
ASSERT_OK(iter->status());
iter.reset();
ASSERT_OK(db_->EndTrace());
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay2");
options.create_if_missing = true;
for (uint32_t num_threads : {1U, 2U}) {
SCOPED_TRACE("num_threads=" + std::to_string(num_threads));
ASSERT_OK(DestroyDB(dbname2, options));
std::unique_ptr<DB> db2;
ASSERT_OK(DB::Open(options, dbname2, &db2));
// No handles: every read record on the default column family must still
// replay.
TraceExecutionResultHandler res_handler;
ASSERT_OK(ReplayTraceFile(env_, trace_filename, db2.get(),
/*handles=*/{}, num_threads, &res_handler));
// Each record replayed and produced a result.
ASSERT_EQ(2, res_handler.GetNumWrites());
ASSERT_EQ(1, res_handler.GetNumGets());
ASSERT_EQ(1, res_handler.GetNumMultiGets());
ASSERT_EQ(2, res_handler.GetNumIterSeeks());
std::string value;
ASSERT_OK(db2->Get(ReadOptions(), "a", &value));
ASSERT_EQ("1", value);
ASSERT_OK(db2->Get(ReadOptions(), "b", &value));
ASSERT_EQ("2", value);
}
ASSERT_OK(DestroyDB(dbname2, options));
}
TEST_F(DBTest2, TraceReplayReturnsExecutionError) {
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
std::string trace_filename = dbname_ + "/rocksdb.trace";
std::unique_ptr<TraceWriter> trace_writer;
ASSERT_OK(
NewFileTraceWriter(env_, EnvOptions(), trace_filename, &trace_writer));
ASSERT_OK(db_->StartTrace(TraceOptions(), std::move(trace_writer)));
ASSERT_EQ("NOT_FOUND", Get(1, "a"));
ASSERT_EQ("NOT_FOUND", Get(1, "b"));
ASSERT_OK(db_->EndTrace());
// The replay DB has no "pikachu" column family, so both Get records fail.
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay2");
options.create_if_missing = true;
ASSERT_OK(DestroyDB(dbname2, options));
std::unique_ptr<DB> db2;
ASSERT_OK(DB::Open(options, dbname2, &db2));
for (uint32_t num_threads : {1U, 2U}) {
SCOPED_TRACE("num_threads=" + std::to_string(num_threads));
Status s = ReplayTraceFile(env_, trace_filename, db2.get(),
{db2->DefaultColumnFamily()}, num_threads);
ASSERT_TRUE(s.IsCorruption()) << s.ToString();
}
db2.reset();
ASSERT_OK(DestroyDB(dbname2, options));
}
TEST_F(DBTest2, TraceReplayReturnsNotSupportedExecutionError) {
Options options = CurrentOptions();
std::string trace_filename = dbname_ + "/rocksdb.trace";
std::unique_ptr<TraceWriter> trace_writer;
ASSERT_OK(
NewFileTraceWriter(env_, EnvOptions(), trace_filename, &trace_writer));
ASSERT_OK(db_->StartTrace(TraceOptions(), std::move(trace_writer)));
ASSERT_OK(Put("a", "1"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
ASSERT_OK(db_->EndTrace());
// DeleteRange is not compatible with row_cache, so replaying the DeleteRange
// write fails with NotSupported. Unlike an unsupported trace type, this must
// not be skipped.
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay2");
options.create_if_missing = true;
options.row_cache = NewLRUCache(1 << 20);
for (uint32_t num_threads : {1U, 2U}) {
SCOPED_TRACE("num_threads=" + std::to_string(num_threads));
ASSERT_OK(DestroyDB(dbname2, options));
std::unique_ptr<DB> db2;
ASSERT_OK(DB::Open(options, dbname2, &db2));
Status s = ReplayTraceFile(env_, trace_filename, db2.get(),
/*handles=*/{}, num_threads);
ASSERT_TRUE(s.IsNotSupported()) << s.ToString();
}
ASSERT_OK(DestroyDB(dbname2, options));
}
TEST_F(DBTest2, TraceWithSampling) {
Options options = CurrentOptions();
ReadOptions ro;
@@ -7651,6 +7792,48 @@ TEST_F(DBTest2, TableCacheMissDuringReadFromBlockCacheTier) {
ASSERT_EQ(orig_num_file_opens, TestGetTickerCount(options, NO_FILE_OPENS));
}
TEST_F(DBTest2, MultiGetBlockCacheTierSkipsColdSstData) {
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
// Use buffered reads to reach the non-mmap RetrieveMultipleBlocks path.
options.allow_mmap_reads = false;
BlockBasedTableOptions table_options;
// Data blocks use the block cache. Index and filter blocks stay in the table
// reader (cache_index_and_filter_blocks is false by default), so a
// kBlockCacheTier read gets to the data block without metadata I/O.
table_options.block_cache = NewLRUCache(1 << 20);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
DestroyAndReopen(options);
ASSERT_OK(Put("k1", "v1"));
ASSERT_OK(Put("k2", "v2"));
ASSERT_OK(Flush());
// Open the table reader so that kBlockCacheTier does not fail at table
// open. Do not add the data block to the block cache.
ReadOptions warm_ro;
warm_ro.fill_cache = false;
std::string value;
ASSERT_OK(db_->Get(warm_ro, "k1", &value));
ASSERT_EQ(value, "v1");
// Make sure the data block is not in the block cache.
table_options.block_cache->EraseUnRefEntries();
ReadOptions no_io_ro;
no_io_ro.read_tier = kBlockCacheTier;
ASSERT_TRUE(db_->Get(no_io_ro, "k1", &value).IsIncomplete());
ASSERT_TRUE(db_->Get(no_io_ro, "k2", &value).IsIncomplete());
Slice keys[2] = {Slice("k1"), Slice("k2")};
PinnableSlice values[2];
Status statuses[2];
db_->MultiGet(no_io_ro, db_->DefaultColumnFamily(), 2, keys, values,
statuses);
ASSERT_TRUE(statuses[0].IsIncomplete());
ASSERT_TRUE(statuses[1].IsIncomplete());
}
TEST_F(DBTest2, GetFileChecksumsFromCurrentManifest_CRC32) {
Options opts = CurrentOptions();
opts.create_if_missing = true;
+8 -1
View File
@@ -415,7 +415,14 @@ Status DBImpl::GetLiveFilesStorageInfoImpl(
// Capture some final info before releasing mutex
const uint64_t manifest_number = versions_->manifest_file_number();
const uint64_t manifest_size = versions_->manifest_file_size();
uint64_t manifest_size = versions_->manifest_file_size();
if (cf_subset && !excluded_cf_ids->empty()) {
Status manifest_s = versions_->GetManifestAppendBoundary(&manifest_size);
if (!manifest_s.ok()) {
mutex_.Unlock();
return manifest_s;
}
}
const uint64_t options_number = versions_->options_file_number();
const uint64_t options_size = versions_->options_file_size_;
const uint64_t min_log_num = MinLogNumberToKeep();
+238
View File
@@ -20,9 +20,11 @@
#include "port/stack_trace.h"
#include "rocksdb/utilities/transaction_db.h"
#include "table/block_based/block_based_table_builder.h"
#include "table/format.h"
#include "test_util/sync_point.h"
#include "test_util/testutil.h"
#include "util/cast_util.h"
#include "util/defer.h"
#include "util/mutexlock.h"
#include "utilities/fault_injection_env.h"
#include "utilities/fault_injection_fs.h"
@@ -2980,6 +2982,34 @@ TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
}
}
// Two handles naming the same column family make the selected-cfd list longer
// than the per-CF memtable-id list, which is keyed by CF. The wait loop must
// not read the id list past its own length.
TEST_P(DBAtomicFlushTest, ManualAtomicFlushDuplicateColumnFamilyHandles) {
if (!GetParam()) {
// Only the atomic path reaches the shared wait list; without atomic_flush
// each handle is flushed independently.
return;
}
Options options = CurrentOptions();
options.create_if_missing = true;
options.atomic_flush = true;
options.write_buffer_size = (static_cast<size_t>(64) << 20);
DestroyAndReopen(options);
WriteOptions wopts;
wopts.disableWAL = true;
ASSERT_OK(Put("key", "value", wopts));
FlushOptions flush_options; // wait == true
ASSERT_OK(db_->Flush(
flush_options, {db_->DefaultColumnFamily(), db_->DefaultColumnFamily()}));
auto cfh = static_cast<ColumnFamilyHandleImpl*>(db_->DefaultColumnFamily());
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
}
TEST_P(DBAtomicFlushTest, PrecomputeMinLogNumberToKeepNon2PC) {
Options options = CurrentOptions();
options.create_if_missing = true;
@@ -3723,6 +3753,108 @@ TEST_F(DBFlushTest, NonAtomicNormalFlushAbortWhenBGError) {
SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBFlushTest, NonRecoveryFlushDoesNotStarveErrorRecovery) {
constexpr uint64_t kRecoveryTimeoutMicros = 15 * 1000 * 1000;
constexpr int kLateWriteCount = 32;
Options opts = CurrentOptions();
opts.atomic_flush = true;
opts.memtable_factory.reset(test::NewSpecialSkipListFactory(1));
opts.max_write_buffer_number = 64;
opts.max_background_flushes = 1;
DestroyAndReopen(opts);
env_->SetBackgroundThreads(1, Env::HIGH);
std::atomic_int flush_write_table_count{0};
port::Mutex wait_mutex;
port::CondVar wait_cv(&wait_mutex);
bool recovery_wait_started = false;
bool recovery_succeeded = false;
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->SetCallBack(
"FlushJob::WriteLevel0Table:s", [&](void* s_ptr) {
if (flush_write_table_count.fetch_add(1) == 0) {
Status* s = static_cast<Status*>(s_ptr);
IOStatus io_error = IOStatus::IOError("injected foobar");
io_error.SetRetryable(true);
*s = io_error;
TEST_SYNC_POINT(
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"FirstFlushFailed");
TEST_SYNC_POINT(
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"ContinueFirstFlush");
}
});
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::ResumeImpl:BeforeWaitForBackgroundWork", [&](void*) {
MutexLock l(&wait_mutex);
recovery_wait_started = true;
wait_cv.SignalAll();
});
SyncPoint::GetInstance()->SetCallBack(
"RecoverFromRetryableBGIOError:RecoverSuccess", [&](void*) {
MutexLock l(&wait_mutex);
recovery_succeeded = true;
wait_cv.SignalAll();
});
SyncPoint::GetInstance()->LoadDependency(
{{"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"FirstFlushFailed",
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"FailureObserved"},
{"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"ReleaseFirstFlush",
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"ContinueFirstFlush"}});
SyncPoint::GetInstance()->EnableProcessing();
auto wait_until = [&](const auto& predicate) {
const uint64_t abs_time =
SystemClock::Default()->NowMicros() + kRecoveryTimeoutMicros;
MutexLock l(&wait_mutex);
while (!predicate()) {
if (wait_cv.TimedWait(abs_time)) {
return false;
}
}
return true;
};
ASSERT_OK(Put(Key(1), "val1"));
ASSERT_OK(Put(Key(2), "val2"));
TEST_SYNC_POINT(
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"FailureObserved");
TEST_SYNC_POINT(
"DBFlushTest::NonRecoveryFlushDoesNotStarveErrorRecovery:"
"ReleaseFirstFlush");
const bool recovery_wait_reached =
wait_until([&] { return recovery_wait_started; });
Status late_write_status;
if (recovery_wait_reached) {
for (int key = 3; key < 3 + kLateWriteCount && late_write_status.ok();
++key) {
late_write_status = Put(Key(key), "val");
}
}
const bool recovered = wait_until([&] { return recovery_succeeded; });
if (!recovered) {
IOStatus cleanup_error = IOStatus::IOError("stop stuck error recovery");
dbfull()->TEST_SetBGError(cleanup_error, BackgroundErrorReason::kFlush);
dbfull()->TEST_SignalAllBgCv();
}
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
ASSERT_TRUE(recovery_wait_reached);
ASSERT_OK(late_write_status);
ASSERT_TRUE(recovered);
}
TEST_F(DBFlushTest, DBStuckAfterAtomicFlushError) {
// Test for a bug with atomic flush where DB can become stuck
// after a flush error. A repro timeline:
@@ -4007,6 +4139,112 @@ INSTANTIATE_TEST_CASE_P(
// larger than the max allowed padding size
testing::Values(4, kLowSpaceOverheadRatio)));
// super_block_alignment pads data blocks, giving the padded block's index entry
// a non-contiguous handle. At format_version <= 7 the writer forces that index
// entry to shared==0 (full key AND full value). format_version 8 instead uses
// the index value-delta escape (see IndexValue::EncodeTo), keeping the key
// delta-encoded and the restart cadence intact. This verifies that fv8 returns
// byte-identical read results to fv7 for the same keys (also with
// separate_key_value_in_data_block, the intersection that motivated the
// escape), that padding -- and hence the escape -- is actually exercised, and
// that fv8 index values are delta encoded.
TEST_F(DBFlushTest, SuperBlockAlignmentValueDeltaEscape) {
// Writing the unpublished draft format_version 8 requires this opt-in.
SaveAndRestore<bool> allow_draft(&TEST_AllowUnsupportedFormatVersion(), true);
constexpr int kKeyCount = 5000;
auto format_key = [](int i) {
char buf[16];
snprintf(buf, sizeof(buf), "%010d", i);
return std::string(buf);
};
// Deterministic values so fv7 and fv8 see identical input.
Random rnd(test::RandomSeed());
std::vector<std::string> values;
values.reserve(kKeyCount);
for (int i = 0; i < kKeyCount; ++i) {
values.push_back(rnd.RandomString(static_cast<int>(rnd.Uniform(1000))));
}
auto build_and_read = [&](uint32_t format_version, bool separate_kv,
std::vector<std::string>* got_get,
std::vector<std::string>* got_iter, int* pad_count,
bool* index_delta_encoded) {
Options options = CurrentOptions();
options.compression = kNoCompression;
BlockBasedTableOptions bbto;
bbto.format_version = format_version;
bbto.index_block_restart_interval = 3; // > 1 so value delta encoding is on
bbto.super_block_alignment_size = 16 * 1024;
bbto.super_block_alignment_space_overhead_ratio = 8;
bbto.separate_key_value_in_data_block = separate_kv;
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
DestroyAndReopen(options);
int local_pad = 0;
SyncPoint::GetInstance()->SetCallBack(
"BlockBasedTableBuilder::WriteMaybeCompressedBlock:SuperBlockAlignment",
[&local_pad](void*) { local_pad++; });
SyncPoint::GetInstance()->EnableProcessing();
for (int i = 0; i < kKeyCount; ++i) {
ASSERT_OK(Put(format_key(i), values[i]));
}
ASSERT_OK(Flush());
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
*pad_count = local_pad;
got_get->clear();
for (int i = 0; i < kKeyCount; ++i) {
PinnableSlice v;
ASSERT_OK(Get(format_key(i), &v));
got_get->push_back(v.ToString());
}
got_iter->clear();
{
std::unique_ptr<Iterator> it(db_->NewIterator(ReadOptions()));
for (it->SeekToFirst(); it->Valid(); it->Next()) {
ASSERT_OK(it->status());
got_iter->push_back(it->value().ToString());
}
ASSERT_OK(it->status());
}
// Confirm the index value encoding of the produced SST(s).
TablePropertiesCollection props;
ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
ASSERT_FALSE(props.empty());
*index_delta_encoded = true;
for (const auto& kv : props) {
if (kv.second->index_value_is_delta_encoded == 0) {
*index_delta_encoded = false;
}
}
};
for (bool separate_kv : {false, true}) {
SCOPED_TRACE("separate_kv=" + std::to_string(separate_kv));
std::vector<std::string> get7, iter7, get8, iter8;
int pad7 = 0, pad8 = 0;
bool delta7 = false, delta8 = false;
build_and_read(7, separate_kv, &get7, &iter7, &pad7, &delta7);
build_and_read(/*format_version=*/8, separate_kv, &get8, &iter8, &pad8,
&delta8);
// Reads must be identical across versions.
ASSERT_EQ(get7, get8);
ASSERT_EQ(iter7, iter8);
ASSERT_EQ(static_cast<int>(iter8.size()), kKeyCount);
// Padding (the non-contiguity that drives the escape) actually happened.
EXPECT_GT(pad8, 0);
// fv8 index values are delta encoded, so the escape path was exercised.
EXPECT_TRUE(delta8);
}
}
// Test that when the table builder's io_status becomes bad during flush
// (simulating write fault injection), BuildTable properly propagates the
// builder's IO error instead of producing a misleading Corruption from the
+39
View File
@@ -204,6 +204,19 @@ class DBFollowerTest : public DBTestBase {
return result;
}
void VerifyFollowerBatchValue(const std::string& key,
const std::string& expected) {
std::vector<Iterator*> iterators;
ASSERT_OK(follower()->NewIterators(
ReadOptions(), {follower()->DefaultColumnFamily()}, &iterators));
ASSERT_EQ(1U, iterators.size());
std::unique_ptr<Iterator> iterator(iterators[0]);
iterator->Seek(key);
ASSERT_OK(iterator->status());
ASSERT_TRUE(iterator->Valid());
ASSERT_EQ(expected, iterator->value());
}
DB* follower() { return follower_.get(); }
DBFollowerTestFS* follower_fs() {
return static_cast<DBFollowerTestFS*>(follower_env_->GetFileSystem().get());
@@ -253,6 +266,32 @@ TEST_F(DBFollowerTest, Basic) {
CheckDirs();
}
TEST_F(DBFollowerTest, NewIteratorsRefreshesAfterCatchUp) {
ASSERT_OK(Put("key", "v1"));
ASSERT_OK(Flush());
SyncPoint::GetInstance()->LoadDependency({
{"DBImplFollower::TryCatchupWithLeader:Begin1", "Leader::Start"},
{"Leader::Done", "DBImplFollower::TryCatchupWithLeader:Begin2"},
{"DBImplFollower::TryCatchupWithLeader:End", "Follower::WaitForCatchup"},
});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(OpenAsFollower());
VerifyFollowerBatchValue("key", "v1");
TEST_SYNC_POINT("Leader::Start");
ASSERT_OK(Put("key", "v2"));
ASSERT_OK(Flush());
TEST_SYNC_POINT("Leader::Done");
TEST_SYNC_POINT("Follower::WaitForCatchup");
VerifyFollowerBatchValue("key", "v2");
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearTrace();
}
TEST_F(DBFollowerTest, Flush) {
SyncPoint::GetInstance()->LoadDependency({
{"DBImplFollower::TryCatchupWithLeader:Begin1", "Leader::Start"},
+41
View File
@@ -7,6 +7,7 @@
#include "db/blob/blob_fetcher.h"
#include "db/db_impl/db_impl.h"
#include "db/db_impl/db_impl_metadata.h"
#include "db/version_set.h"
#include "logging/logging.h"
#include "table/get_context.h"
@@ -34,6 +35,46 @@ size_t CompactedDBImpl::FindFile(const Slice& key) {
files_.files);
}
Status CompactedDBImpl::GetWithMetadata(const ReadOptions& options,
ColumnFamilyHandle* column_family,
const Slice& key, PinnableSlice* value,
OutputMetadata* output_metadata) {
std::string* timestamp = GetOutputTimestamp(output_metadata);
bool* newer_version_present = GetOutputNewerVersionPresent(output_metadata);
if (newer_version_present != nullptr) {
*newer_version_present = false;
}
if (value == nullptr) {
return Status::InvalidArgument(
"Cannot call GetWithMetadata with a null value");
}
return Get(options, column_family, key, value, timestamp);
}
void CompactedDBImpl::MultiGetWithMetadata(
const ReadOptions& options, size_t num_keys,
ColumnFamilyHandle* const* column_families, const Slice* keys,
PinnableSlice* values, Status* statuses,
MultiGetOutputMetadata* output_metadata, const bool sorted_input) {
std::vector<std::string>* timestamps = GetOutputTimestamps(output_metadata);
if (timestamps != nullptr) {
timestamps->resize(num_keys);
}
std::vector<uint8_t>* newer_version_present =
GetOutputNewerVersionPresent(output_metadata);
if (newer_version_present != nullptr) {
newer_version_present->assign(num_keys, false);
}
autovector<ColumnFamilyHandle*, MultiGetContext::MAX_BATCH_SIZE>
stack_column_families;
std::vector<ColumnFamilyHandle*> heap_column_families;
ColumnFamilyHandle** mutable_column_families = MakeMutableCfHandles(
column_families, num_keys, &stack_column_families, &heap_column_families);
MultiGet(options, num_keys, mutable_column_families, keys, values,
timestamps != nullptr ? timestamps->data() : nullptr, statuses,
sorted_input);
}
Status CompactedDBImpl::Init(const Options& options) {
SuperVersionContext sv_context(/* create_superversion */ true);
mutex_.Lock();
+14
View File
@@ -31,6 +31,12 @@ class CompactedDBImpl : public DBImpl {
const Slice& key, PinnableSlice* value,
std::string* timestamp);
using DB::GetWithMetadata;
Status GetWithMetadata(const ReadOptions& options,
ColumnFamilyHandle* column_family, const Slice& key,
PinnableSlice* value,
OutputMetadata* output_metadata) override;
using DB::MultiGet;
// Note that CompactedDBImpl::MultiGet is not the optimized version of
// MultiGet to use.
@@ -42,6 +48,14 @@ class CompactedDBImpl : public DBImpl {
std::string* timestamps, Status* statuses,
const bool sorted_input);
using DB::MultiGetWithMetadata;
void MultiGetWithMetadata(const ReadOptions& options, size_t num_keys,
ColumnFamilyHandle* const* column_families,
const Slice* keys, PinnableSlice* values,
Status* statuses,
MultiGetOutputMetadata* output_metadata,
const bool sorted_input) override;
using DBImpl::Put;
Status Put(const WriteOptions& /*options*/,
ColumnFamilyHandle* /*column_family*/, const Slice& /*key*/,
@@ -6,10 +6,6 @@
#include "util/coro_utils.h"
#if defined(USE_COROUTINES) && defined(WITH_COROUTINES)
#include "util/coro_stats_util.h"
#endif // USE_COROUTINES && WITH_COROUTINES
#if defined(WITHOUT_COROUTINES) || \
(defined(USE_COROUTINES) && defined(WITH_COROUTINES))
@@ -18,10 +14,6 @@ namespace ROCKSDB_NAMESPACE {
DEFINE_SYNC_AND_ASYNC(Status, CompactedDBImpl::Get)
(const ReadOptions& _read_options, ColumnFamilyHandle*, const Slice& key,
PinnableSlice* value, std::string* timestamp) {
#ifdef WITH_COROUTINES
INSTALL_COROUTINE_STATS_CONTEXT_SCOPE(
immutable_db_options_.fs->GetReadExecutor(), immutable_db_options_.env);
#endif
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
_read_options.io_activity != Env::IOActivity::kGet) {
CO_RETURN Status::InvalidArgument(
@@ -110,10 +102,6 @@ DEFINE_SYNC_AND_ASYNC(void, CompactedDBImpl::MultiGet)
ColumnFamilyHandle** /*column_families*/, const Slice* keys,
PinnableSlice* values, std::string* timestamps, Status* statuses,
const bool /*sorted_input*/) {
#ifdef WITH_COROUTINES
INSTALL_COROUTINE_STATS_CONTEXT_SCOPE(
immutable_db_options_.fs->GetReadExecutor(), immutable_db_options_.env);
#endif
assert(user_comparator_);
Status s;
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
+1075 -347
View File
File diff suppressed because it is too large. Load diff
+276 -18
View File
@@ -9,6 +9,7 @@
#pragma once
#include <atomic>
#include <chrono>
#include <cstdint>
#include <deque>
#include <functional>
@@ -51,6 +52,7 @@
#include "db/write_controller.h"
#include "db/write_thread.h"
#include "logging/event_logger.h"
#include "memtable/flush_initiator.h"
#include "memtable/wbwi_memtable.h"
#include "monitoring/instrumented_mutex.h"
#include "options/db_options.h"
@@ -290,12 +292,22 @@ class DBImpl : public DB
const Slice& key, PinnableSlice* value,
std::string* timestamp);
using DB::GetWithMetadata;
Status GetWithMetadata(const ReadOptions& _read_options,
ColumnFamilyHandle* column_family, const Slice& key,
PinnableSlice* value,
OutputMetadata* output_metadata) override;
using DB::GetEntity;
Status GetEntity(const ReadOptions& options,
ColumnFamilyHandle* column_family, const Slice& key,
PinnableWideColumns* columns) override;
Status GetEntity(const ReadOptions& options, const Slice& key,
PinnableAttributeGroups* result) override;
DECLARE_SYNC_AND_ASYNC_OVERRIDE(Status, GetEntity,
const ReadOptions& _read_options,
ColumnFamilyHandle* column_family,
const Slice& key,
PinnableWideColumns* columns);
DECLARE_SYNC_AND_ASYNC_OVERRIDE(Status, GetEntity,
const ReadOptions& _read_options,
const Slice& key,
PinnableAttributeGroups* result);
Status GetEntityLazy(const ReadOptions& options,
ColumnFamilyHandle* column_family, const Slice& key,
@@ -332,6 +344,15 @@ class DBImpl : public DB
std::string* timestamps, Status* statuses,
const bool sorted_input = false);
using DB::MultiGetWithMetadata;
void MultiGetWithMetadata(const ReadOptions& _read_options,
const size_t num_keys,
ColumnFamilyHandle* const* column_families,
const Slice* keys, PinnableSlice* values,
Status* statuses,
MultiGetOutputMetadata* output_metadata,
const bool sorted_input = false) override;
void MultiGetWithCallback(
const ReadOptions& _read_options, ColumnFamilyHandle* column_family,
ReadCallback* callback,
@@ -419,6 +440,9 @@ class DBImpl : public DB
Status NewIterators(const ReadOptions& _read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
Status NewIterators(const std::vector<ReadOptions>& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
using DB::NewMultiScan;
std::unique_ptr<MultiScan> NewMultiScan(
@@ -765,6 +789,7 @@ class DBImpl : public DB
PinnableSlice* value = nullptr;
PinnableWideColumns* columns = nullptr;
std::string* timestamp = nullptr;
bool* newer_version_present = nullptr;
bool* value_found = nullptr;
ReadCallback* callback = nullptr;
bool* is_blob_index = nullptr;
@@ -795,6 +820,18 @@ class DBImpl : public DB
Cleanable* lazy_columns_pin = nullptr;
const Version** lazy_columns_version = nullptr;
const SameFileBlobReader** lazy_columns_same_file_reader = nullptr;
// Batched lazy read (MultiGetEntityLazy). When `lazy_columns_shared_sv` is
// non-null, GetImpl reads against this externally-owned SuperVersion at
// `lazy_columns_snapshot_seq` (the consistent per-batch sequence number)
// instead of acquiring its own SuperVersion and deriving its own sequence
// number, and it does NOT transfer a per-key pin -- the batch holds one
// shared SuperVersion pin per column family. `lazy_columns_pin` must then
// be null; `lazy_columns_version` / `lazy_columns_same_file_reader` are
// still filled. (Single-key GetEntityLazy leaves this null and uses
// `lazy_columns_pin`.)
SuperVersion* lazy_columns_shared_sv = nullptr;
SequenceNumber lazy_columns_snapshot_seq = kMaxSequenceNumber;
};
DECLARE_SYNC_AND_ASYNC(Status, GetImpl, const ReadOptions& read_options,
@@ -805,6 +842,11 @@ class DBImpl : public DB
ColumnFamilyHandle* column_family, const Slice& key,
PinnableSlice* value, std::string* timestamp);
Status GetImpl(const ReadOptions& read_options,
ColumnFamilyHandle* column_family, const Slice& key,
PinnableSlice* value, std::string* timestamp,
bool* newer_version_present);
// Function that Get and KeyMayExist call with no_io true or false
// Note: 'value_found' from KeyMayExist propagates here
// This function is also called by GetMergeOperands
@@ -827,6 +869,18 @@ class DBImpl : public DB
ColumnFamilyHandle* column_family, const Slice& key,
LazyWideColumns* result);
// Batched-read counterpart of GetEntityLazyImpl for one key of
// MultiGetEntityLazy: performs the lazy point lookup against the batch's
// shared `super_version` at the batch's consistent `snapshot_seq` (rather
// than acquiring its own), and binds the entity's resolver to that Version
// without taking a per-key pin (the batch owns one shared pin per column
// family).
Status GetEntityLazyForBatch(const ReadOptions& read_options,
ColumnFamilyHandle* column_family,
SuperVersion* super_version,
SequenceNumber snapshot_seq, const Slice& key,
LazyWideColumns* result);
// If `snapshot` == kMaxSequenceNumber, set a recent one inside the file.
ArenaWrappedDBIter* NewIteratorImpl(const ReadOptions& options,
ColumnFamilyHandleImpl* cfh,
@@ -1156,10 +1210,12 @@ class DBImpl : public DB
}
};
using RecoveredTransactionMap =
std::unordered_map<std::string, RecoveredTransaction*>;
bool allow_2pc() const { return immutable_db_options_.allow_2pc; }
std::unordered_map<std::string, RecoveredTransaction*>
recovered_transactions() {
RecoveredTransactionMap recovered_transactions() {
return recovered_transactions_;
}
@@ -1190,16 +1246,24 @@ class DBImpl : public DB
logs_with_prep_tracker_.MarkLogAsContainingPrepSection(log);
}
void DeleteRecoveredTransaction(const std::string& name) {
auto it = recovered_transactions_.find(name);
// Deletes the recovered transaction `it` points to and returns the iterator
// following it, like std::unordered_map::erase().
RecoveredTransactionMap::iterator DeleteRecoveredTransaction(
RecoveredTransactionMap::iterator it) {
assert(it != recovered_transactions_.end());
auto* trx = it->second;
recovered_transactions_.erase(it);
RecoveredTransactionMap::iterator next = recovered_transactions_.erase(it);
for (const auto& info : trx->batches_) {
logs_with_prep_tracker_.MarkLogAsHavingPrepSectionFlushed(
info.second.log_number_);
}
delete trx;
return next;
}
void DeleteRecoveredTransaction(const std::string& name) {
RecoveredTransactionMap::iterator it = recovered_transactions_.find(name);
DeleteRecoveredTransaction(it);
}
void DeleteAllRecoveredTransactions() {
@@ -1280,6 +1344,9 @@ class DBImpl : public DB
Status TEST_SwitchMemtable(ColumnFamilyData* cfd = nullptr);
size_t TEST_GetWBMFlushableMemUsage();
bool TEST_IsWBMFlushInitiatorFlushable() const;
// Force current memtable contents to be flushed.
Status TEST_FlushMemTable(bool wait = true, bool allow_write_stall = false,
ColumnFamilyHandle* cfh = nullptr);
@@ -1300,7 +1367,10 @@ class DBImpl : public DB
// finishes. For example in CompactRange.
Status TEST_AtomicFlushMemTables(
const autovector<ColumnFamilyData*>& provided_candidate_cfds,
const FlushOptions& flush_opts);
const FlushOptions& flush_opts, bool non_blocking_write_thread = false);
void TEST_BeginWriteStall();
void TEST_EndWriteStall();
// Wait for background threads to complete scheduled work.
Status TEST_WaitForBackgroundWork();
@@ -1318,6 +1388,8 @@ class DBImpl : public DB
// Get the background error status
Status TEST_GetBGError();
void TEST_SetBGError(const IOStatus& error, BackgroundErrorReason reason);
bool TEST_IsRecoveryInProgress();
Status TEST_ResumeImpl(DBRecoverContext context);
@@ -1441,6 +1513,20 @@ class DBImpl : public DB
// REQUIRES: DB mutex held or during open
void EnsureSeqnoToTimeMapping(const MinAndMaxPreserveSeconds& preserve_secs);
// Computes the seqno->time preserve-window lower bound from
// seqno_to_time_mapping_ and stores it on cfd's current version, so
// bottommost file marking does not mark files whose largest seqno cannot be
// zeroed yet (which would loop). No-op for column families without
// preserve/preclude enabled. Returns true if the stored value changed, so
// callers can recompute bottommost marking when the boundary moves.
// Note: right after opening an existing DB, seqno_to_time_mapping_ may not be
// fully reconstructed, so this bound can be imprecise until the first
// periodic RecordSeqnoToTimeMapping. That is safe: CompactionJob folds this
// same bound into kBottommostFiles compactions, so any marked file still
// makes progress (never loops), and the periodic task self-corrects the
// bound. REQUIRES: DB mutex held
bool MaybeUpdatePreserveTimeMinSeqno(ColumnFamilyData* cfd);
// Only called during open
void PrepopulateSeqnoToTimeMapping(
const MinAndMaxPreserveSeconds& preserve_secs);
@@ -1477,6 +1563,21 @@ class DBImpl : public DB
}
}
// Returns false when the wait times out while the DB is still blocked.
bool BlockFor(uint64_t timeout_micros) {
MutexLock lock(&state_mutex_);
const std::chrono::microseconds deadline(
SystemClock::Default()->NowMicros() + timeout_micros);
while (state_ == State::BLOCKED) {
TEST_SYNC_POINT("WBMStallInterface::BlockDB");
if (SystemClock::Default()->TimedWait(&state_cv_, deadline) &&
state_ == State::BLOCKED) {
return false;
}
}
return true;
}
// Called from WriteBufferManager. This function changes the state_
// to State::RUNNING indicating the stall is cleared and DB can proceed.
void Signal() override {
@@ -1497,6 +1598,24 @@ class DBImpl : public DB
State state_;
};
// Adapts DBImpl to the shared WriteBufferManager flush registry.
class WBMFlushInitiator : public FlushInitiator {
public:
WBMFlushInitiator(DBImpl* db, bool atomic_flush)
: FlushInitiator(atomic_flush), db_(db) {}
bool ScheduleFlush() override {
return db_->ScheduleWriteBufferManagerFlush();
}
bool TryRefreshMemoryAccounting() override {
return db_->TryRefreshFlushableMemAccounting();
}
private:
DBImpl* const db_;
};
static void TEST_ResetDbSessionIdGen();
static std::string GenerateDbSessionId(Env* env);
@@ -1514,6 +1633,9 @@ class DBImpl : public DB
// db_session_id_ is an identifier that gets reset
// every time the DB is opened
std::string db_session_id_;
// Declared before versions_ so memtables cannot outlive their allocation
// accounting target during destruction.
std::unique_ptr<FlushInitiator> wbm_flush_initiator_;
std::unique_ptr<VersionSet> versions_;
// Flag to check whether we allocated and own the info log file
bool own_info_log_;
@@ -1525,8 +1647,7 @@ class DBImpl : public DB
FileSystemPtr fs_;
MutableDBOptions mutable_db_options_;
Statistics* stats_;
std::unordered_map<std::string, RecoveredTransaction*>
recovered_transactions_;
RecoveredTransactionMap recovered_transactions_;
std::unique_ptr<Tracer> tracer_;
InstrumentedMutex trace_mutex_;
BlockCacheTracer block_cache_tracer_;
@@ -1627,6 +1748,15 @@ class DBImpl : public DB
edit_lists_[i].emplace_back(new VersionEdit(edit));
}
bool HasVersionEdits() const {
for (const auto& edit_list : edit_lists_) {
if (!edit_list.empty()) {
return true;
}
}
return false;
}
std::unordered_map<uint32_t, uint32_t> map_; // cf_id to index;
autovector<ColumnFamilyData*> cfds_;
autovector<autovector<VersionEdit*>> edit_lists_;
@@ -1996,7 +2126,7 @@ class DBImpl : public DB
// LogAndApplyForRecovery should be called only once during recovery and it
// should be called when RocksDB writes to a first new MANIFEST since this
// recovery.
Status LogAndApplyForRecovery(const RecoveryContext& recovery_ctx);
Status LogAndApplyForRecovery(RecoveryContext& recovery_ctx);
// Schedule background work to open and validate SST files asynchronously.
// Called when open_files_async is enabled.
@@ -2545,7 +2675,7 @@ class DBImpl : public DB
// Begin stalling of writes when memory usage increases beyond a certain
// threshold.
void WriteBufferManagerStallWrites();
Status WriteBufferManagerStallWrites(WriteContext* write_context);
Status ThrottleLowPriWritesIfNeeded(const WriteOptions& write_options,
WriteBatch* my_batch);
@@ -2573,6 +2703,16 @@ class DBImpl : public DB
ReadOnlyMemTable* new_imm = nullptr,
SequenceNumber last_seqno = 0);
// `refresh_wbm_accounting` can be false when a caller switches a batch of
// memtables and refreshes the DB-wide accounting once after the batch.
Status SwitchMemtableImpl(ColumnFamilyData* cfd, WriteContext* context,
ReadOnlyMemTable* new_imm,
SequenceNumber last_seqno,
bool refresh_wbm_accounting);
// REQUIRES: mutex_ is held
void UpdateWBMAccountingAfterMemtableSwitch(bool refresh_now);
// Select and output column families qualified for atomic flush in
// `selected_cfds`. If `provided_candidate_cfds` is non-empty, it will be used
// as candidate CFs to select qualified ones from. Otherwise, all column
@@ -2597,6 +2737,18 @@ class DBImpl : public DB
const autovector<ColumnFamilyData*>& provided_candidate_cfds = {},
bool entered_write_thread = false);
enum class WriteThreadJoinMode { kBlocking, kNonBlocking };
Status FlushMemTableImpl(ColumnFamilyData* cfd, const FlushOptions& options,
FlushReason flush_reason, bool entered_write_thread,
WriteThreadJoinMode join_mode, bool* made_progress);
Status AtomicFlushMemTablesImpl(
const FlushOptions& options, FlushReason flush_reason,
const autovector<ColumnFamilyData*>& provided_candidate_cfds,
bool entered_write_thread, WriteThreadJoinMode join_mode,
bool* made_progress);
// REQUIRES: mutex locked and write queues drained up to the recovery flush
// fence that is about to switch memtables.
void MaybeSyncLastSequenceWithAllocatedForRecovery(FlushReason flush_reason);
@@ -2683,6 +2835,18 @@ class DBImpl : public DB
return static_cast<uint8_t*>(static_cast<void*>(this)) + type;
}
// Checks DB-local write queues; manager-wide stall state would reject healthy
// DBs. REQUIRES: mutex_ held.
bool WouldBlockJoiningWriteThread() {
mutex_.AssertHeld();
return write_thread_.GetBegunCountOfOutstandingStall() != 0 ||
(two_write_queues_ &&
nonmem_write_thread_.GetBegunCountOfOutstandingStall() != 0);
}
Status EnterWriteThreadForNonBlockingFlush(WriteThread::Writer* w,
WriteThread::Writer* nonmem_w);
// REQUIRES: mutex locked and in write thread.
void AssignAtomicFlushSeq(const autovector<ColumnFamilyData*>& cfds);
@@ -2692,6 +2856,31 @@ class DBImpl : public DB
// REQUIRES: mutex locked and in write thread.
Status HandleWriteBufferManagerFlush(WriteContext* write_context);
// Column families and memory eligible for a WriteBufferManager flush.
struct FlushableCFs {
ColumnFamilyData* largest = nullptr; // most reclaimable memtable memory
ColumnFamilyData* oldest = nullptr; // smallest memtable creation seq
size_t largest_mem = 0;
size_t total_mem = 0;
};
// Shared selection for bidding and flushing. REQUIRES: mutex_ held.
FlushableCFs CollectFlushableCFs(bool include_waiting_immutable);
// Registers an opened read-write DB without holding mutex_.
void MaybeRegisterFlushInitiator();
// Refreshes the published mutable-memory counters after memtables change.
// REQUIRES: mutex_ held.
void RefreshFlushableMemAccounting();
bool TryRefreshFlushableMemAccounting();
// Queues one WBM flush; false lets the coordinator try another DB.
bool ScheduleWriteBufferManagerFlush();
void BackgroundCallWBMFlush();
// REQUIRES: mutex locked
Status PreprocessWrite(const WriteOptions& write_options,
WalContext* log_context, WriteContext* write_context);
@@ -2730,7 +2919,13 @@ class DBImpl : public DB
// Used by WriteImpl to update bg_error_ when IO error happens, e.g., write
// WAL, sync WAL fails, if paranoid check is enabled.
void WALIOStatusCheck(const IOStatus& status);
void WALIOStatusCheck(const IOStatus& status, uint64_t failed_wal_number = 0,
SequenceNumber failed_wal_sequence = 0,
SequenceNumber failed_wal_sequence_start = 0);
// Extends the inclusive WAL-number cutoff that must not receive more I/O.
// This is lock-free because a WAL error can race with public SyncWAL().
void ExtendWALRecoveryCutoff(uint64_t failed_wal_number);
// Used by WriteImpl to update bg_error_ in case of memtable insert error.
void HandleMemTableInsertFailure(const Status& nonok_memtable_insert_status);
@@ -2774,7 +2969,11 @@ class DBImpl : public DB
// equal to this per-column-family specified value, this flush request is
// considered to have completed its work of flushing this column family.
// After completing the work for all column families in this request, this
// flush is considered complete.
// flush is considered complete. EnqueuePendingFlush() acquires one
// reference for each CFD when it successfully queues this request.
// PopFirstFromFlushQueue() transfers responsibility for those references
// to its caller, which must release each one with UnrefAndTryDelete() after
// processing or discarding the request.
std::unordered_map<ColumnFamilyData*, uint64_t>
cfd_to_max_mem_id_to_persist;
@@ -2827,6 +3026,9 @@ class DBImpl : public DB
Env::Priority thread_pri);
void BackgroundCallFlush(Env::Priority thread_pri);
void BackgroundCallPurge();
// Recursively removes all children of the DB session temporary directory.
// Best effort: errors are logged, not returned. REQUIRES: mutex_ not held.
void CleanupSessionTmpDir();
Status BackgroundCompaction(bool* madeProgress, JobContext* job_context,
LogBuffer* log_buffer,
PrepickedCompaction* prepicked_compaction,
@@ -3092,7 +3294,9 @@ class DBImpl : public DB
ColumnFamilyHandle** column_families,
const Slice* keys, PinnableSlice* values,
PinnableWideColumns* columns, std::string* timestamps,
Status* statuses, bool sorted_input);
Status* statuses,
std::vector<uint8_t>* newer_version_present,
bool sorted_input);
// A structure to hold the information required to process MultiGet of keys
// belonging to one column family. For a multi column family MultiGet, there
@@ -3307,6 +3511,16 @@ class DBImpl : public DB
// * whenever a column family is dropped.
InstrumentedCondVar bg_cv_;
// Set only while ResumeImpl performs a file-scoped WAL recovery. Protected
// by mutex_; SwitchMemtable uses it to avoid touching the failed writer.
bool recovering_from_wal_write_error_ = false;
uint64_t recover_wal_through_number_ = 0;
// Once a file-scoped WAL error is reported, all writers through this
// inclusive cutoff are quarantined until recovery detaches them. A
// concurrent SyncWAL() reads this without taking the DB mutex.
std::atomic<uint64_t> wal_recovery_cutoff_{0};
ColumnFamilyHandleImpl* persist_stats_cf_handle_ = nullptr;
bool persistent_stats_cfd_exists_ = true;
@@ -3570,6 +3784,12 @@ class DBImpl : public DB
// number of background memtable flush jobs, submitted to the HIGH pool
int bg_flush_scheduled_ = 0;
// LOW avoids consuming a flush worker while joining the target write thread.
static constexpr Env::Priority kWBMFlushPriority = Env::Priority::LOW;
// Outstanding cross-DB WBM flush jobs, drained during shutdown.
int bg_wbm_flush_scheduled_ = 0;
// stores the number of flushes are currently running
int num_running_flushes_ = 0;
@@ -3776,9 +3996,47 @@ class GetWithTimestampReadCallback : public ReadCallback {
public:
explicit GetWithTimestampReadCallback(SequenceNumber seq)
: ReadCallback(seq) {}
// A null result is used by MultiGet, whose per-key results live in
// KeyContext/GetContext.
void EnableNewerVersionTracking(SequenceNumber read_snapshot_seq,
SequenceNumber upper_bound_seq,
bool* single_key_result) {
metadata_read_bounds_.emplace(
MetadataReadBounds{read_snapshot_seq, upper_bound_seq});
newer_version_present_ = single_key_result;
}
const MetadataReadBounds* GetMetadataReadBounds() const override {
return metadata_read_bounds_.has_value() ? &*metadata_read_bounds_
: nullptr;
}
bool NeedToTrackNewerVersions(
const bool* per_key_result = nullptr) const override {
const bool* result =
per_key_result != nullptr ? per_key_result : newer_version_present_;
return metadata_read_bounds_.has_value() && result != nullptr && !*result;
}
bool IsVisibleFullCheck(SequenceNumber seq) override {
return seq <= max_visible_seq_;
}
bool IsNewerVisibleForMetadataRead(SequenceNumber seq) override {
const MetadataReadBounds* bounds = GetMetadataReadBounds();
return bounds != nullptr && bounds->read_snapshot_seq < seq &&
seq <= bounds->newer_version_upper_bound_seq;
}
void MaybeRecordNewerVersion(SequenceNumber seq, ValueType type,
bool* per_key_result = nullptr) override {
bool* result =
per_key_result != nullptr ? per_key_result : newer_version_present_;
if (result != nullptr && !*result &&
(IsValueType(type) || type == kTypeRangeDeletion) &&
IsNewerVisibleForMetadataRead(seq)) {
*result = true;
}
}
private:
std::optional<MetadataReadBounds> metadata_read_bounds_;
bool* newer_version_present_ = nullptr;
};
Options SanitizeOptions(const std::string& db, const Options& src,
+462 -72
View File
@@ -9,6 +9,7 @@
#include <algorithm>
#include <cinttypes>
#include <deque>
#include <memory>
#include <unordered_map>
#include "db/blob/blob_file_partition_manager.h"
@@ -1339,7 +1340,7 @@ Status DBImpl::CompactRangeInternal(const CompactRangeOptions& options,
SuperVersion* super_version = cfd->GetReferencedSuperVersion(this);
s = cfd->RangesOverlapWithMemtables(
{range}, super_version, immutable_db_options_.allow_data_in_errors,
&flush_needed);
&flush_needed, /*range_limit_exclusive=*/false);
CleanupSuperVersion(super_version);
}
@@ -1712,7 +1713,7 @@ void DBImpl::PrepareTrivialMoveEdit(Compaction& c, LogBuffer* log_buffer,
c.output_level(), f->fd.GetNumber(), f->fd.GetPathId(),
f->fd.GetFileSize(), f->smallest, f->largest, f->fd.smallest_seqno,
f->fd.largest_seqno, f->marked_for_compaction, f->temperature,
f->oldest_blob_file_number, f->oldest_ancester_time,
f->oldest_blob_file_number, f->oldest_ancestor_time,
f->file_creation_time, f->epoch_number, f->file_checksum,
f->file_checksum_func_name, f->unique_id,
f->compensated_range_deletion_size, f->tail_size,
@@ -2033,7 +2034,7 @@ Status DBImpl::PauseBackgroundWork() {
InstrumentedMutexLock guard_lock(&mutex_);
bg_compaction_paused_++;
while (bg_bottom_compaction_scheduled_ > 0 || bg_compaction_scheduled_ > 0 ||
bg_flush_scheduled_ > 0) {
bg_flush_scheduled_ > 0 || bg_wbm_flush_scheduled_ > 0) {
bg_cv_.Wait();
}
bg_work_paused_++;
@@ -2224,8 +2225,8 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
"Levels between source and target are not empty for a move.");
}
if (cfd->RangeOverlapWithCompaction(refit_level_smallest.user_key(),
refit_level_largest.user_key(),
l)) {
refit_level_largest.user_key(), l,
/*range_limit_exclusive=*/false)) {
refitting_level_ = false;
return Status::NotSupported(
"Levels between source and target "
@@ -2248,8 +2249,8 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
"Levels between source and target are not empty for a move.");
}
if (cfd->RangeOverlapWithCompaction(refit_level_smallest.user_key(),
refit_level_largest.user_key(),
l)) {
refit_level_largest.user_key(), l,
/*range_limit_exclusive=*/false)) {
refitting_level_ = false;
return Status::NotSupported(
"Levels between source and target "
@@ -2289,7 +2290,7 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
to_level, f->fd.GetNumber(), f->fd.GetPathId(), f->fd.GetFileSize(),
f->smallest, f->largest, f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->temperature, f->oldest_blob_file_number,
f->oldest_ancester_time, f->file_creation_time, f->epoch_number,
f->oldest_ancestor_time, f->file_creation_time, f->epoch_number,
f->file_checksum, f->file_checksum_func_name, f->unique_id,
f->compensated_range_deletion_size, f->tail_size,
f->user_defined_timestamps_persisted, f->min_timestamp,
@@ -2724,12 +2725,54 @@ void DBImpl::MaybeSyncLastSequenceWithAllocatedForRecovery(
}
}
Status DBImpl::EnterWriteThreadForNonBlockingFlush(
WriteThread::Writer* w, WriteThread::Writer* nonmem_w) {
mutex_.AssertHeld();
if (!write_thread_.EnterUnbatchedNonBlocking(w, &mutex_)) {
return Status::Incomplete(
"Write stall in progress, unable to join the write thread to switch "
"memtables");
}
if (two_write_queues_ &&
!nonmem_write_thread_.EnterUnbatchedNonBlocking(nonmem_w, &mutex_)) {
write_thread_.ExitUnbatched(w);
return Status::Incomplete(
"Write stall in progress, unable to join the write thread to switch "
"memtables");
}
if (!error_handler_.IsBGWorkStopped() && !write_controller_.IsStopped() &&
!WouldBlockJoiningWriteThread()) {
return Status::OK();
}
write_thread_.ExitUnbatched(w);
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(nonmem_w);
}
return Status::Incomplete(
"Write stop started while joining the write thread to switch memtables");
}
Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
const FlushOptions& flush_options,
FlushReason flush_reason,
bool entered_write_thread) {
return FlushMemTableImpl(cfd, flush_options, flush_reason,
entered_write_thread, WriteThreadJoinMode::kBlocking,
nullptr /* made_progress */);
}
Status DBImpl::FlushMemTableImpl(ColumnFamilyData* cfd,
const FlushOptions& flush_options,
FlushReason flush_reason,
bool entered_write_thread,
WriteThreadJoinMode join_mode,
bool* made_progress) {
if (made_progress != nullptr) {
*made_progress = false;
}
// This method should not be called if atomic_flush is true.
assert(!immutable_db_options_.atomic_flush);
assert(!IsRecoveryFlush(flush_reason));
if (!flush_options.wait && write_controller_.IsStopped()) {
std::ostringstream oss;
oss << "Writes have been stopped, thus unable to perform manual flush. "
@@ -2763,13 +2806,45 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
WriteThread::Writer w;
WriteThread::Writer nonmem_w;
if (needs_to_join_write_thread) {
write_thread_.EnterUnbatched(&w, &mutex_);
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
if (join_mode == WriteThreadJoinMode::kNonBlocking) {
s = EnterWriteThreadForNonBlockingFlush(&w, &nonmem_w);
if (!s.ok()) {
return s;
}
} else {
write_thread_.EnterUnbatched(&w, &mutex_);
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
}
}
if (join_mode == WriteThreadJoinMode::kNonBlocking && cfd->IsDropped()) {
TEST_SYNC_POINT("DBImpl::FlushMemTableImpl:Dropped");
if (needs_to_join_write_thread) {
write_thread_.ExitUnbatched(&w);
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
}
s.PermitUncheckedOk();
return Status::ColumnFamilyDropped();
}
WaitForPendingWrites();
if (join_mode == WriteThreadJoinMode::kNonBlocking &&
flush_reason == FlushReason::kWriteBufferManager &&
cfd->imm()->IsFlushPendingOrRunning()) {
TEST_SYNC_POINT("DBImpl::FlushMemTableImpl:AlreadyFlushing");
if (needs_to_join_write_thread) {
write_thread_.ExitUnbatched(&w);
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
}
return Status::Incomplete(
"Column family started flushing before the WBM handoff completed");
}
// Recovery may have released `mutex_` after the earlier `ResumeImpl()`
// sync. Refresh sequence state at the actual memtable-switch fence, after
// both write queues are drained and before `SwitchMemtable()` consumes
@@ -2779,6 +2854,9 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
if (!cfd->mem()->IsEmpty() || !cached_recoverable_state_empty_.load() ||
IsRecoveryFlush(flush_reason)) {
s = SwitchMemtable(cfd, &context);
if (s.ok() && made_progress != nullptr) {
*made_progress = true;
}
}
const uint64_t flush_memtable_id = std::numeric_limits<uint64_t>::max();
if (s.ok()) {
@@ -2851,7 +2929,11 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
req.cfd_to_max_mem_id_to_persist.begin()->first;
bool already_queued_for_flush = loop_cfd->queued_for_flush();
bool flush_req_enqueued = EnqueuePendingFlush(req);
if (already_queued_for_flush || flush_req_enqueued) {
if (flush_req_enqueued && made_progress != nullptr) {
*made_progress = true;
}
if (flush_reason != FlushReason::kWriteBufferManager &&
(already_queued_for_flush || flush_req_enqueued)) {
loop_cfd->SetFlushSkipReschedule();
}
}
@@ -2866,7 +2948,9 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
}
}
NotifyOnManualFlushScheduled({cfd}, flush_reason);
if (flush_reason != FlushReason::kWriteBufferManager) {
NotifyOnManualFlushScheduled({cfd}, flush_reason);
}
TEST_SYNC_POINT("DBImpl::FlushMemTable:AfterScheduleFlush");
TEST_SYNC_POINT("DBImpl::FlushMemTable:BeforeWaitForBgFlush");
if (s.ok() && flush_options.wait) {
@@ -2895,6 +2979,20 @@ Status DBImpl::AtomicFlushMemTables(
const FlushOptions& flush_options, FlushReason flush_reason,
const autovector<ColumnFamilyData*>& provided_candidate_cfds,
bool entered_write_thread) {
return AtomicFlushMemTablesImpl(flush_options, flush_reason,
provided_candidate_cfds, entered_write_thread,
WriteThreadJoinMode::kBlocking,
nullptr /* made_progress */);
}
Status DBImpl::AtomicFlushMemTablesImpl(
const FlushOptions& flush_options, FlushReason flush_reason,
const autovector<ColumnFamilyData*>& provided_candidate_cfds,
bool entered_write_thread, WriteThreadJoinMode join_mode,
bool* made_progress) {
if (made_progress != nullptr) {
*made_progress = false;
}
if (!flush_options.wait && write_controller_.IsStopped()) {
std::ostringstream oss;
oss << "Writes have been stopped, thus unable to perform manual flush. "
@@ -2925,32 +3023,30 @@ Status DBImpl::AtomicFlushMemTables(
candidate_cfds = provided_candidate_cfds;
}
const auto unref_generated_candidates = [&]() {
if (!provided_candidate_cfds.empty()) {
return;
}
for (auto candidate_cfd : candidate_cfds) {
candidate_cfd->UnrefAndTryDelete();
}
candidate_cfds.clear();
};
if (!flush_options.allow_write_stall) {
int num_cfs_to_flush = 0;
for (auto cfd : candidate_cfds) {
bool flush_needed = true;
s = WaitUntilFlushWouldNotStallWrites(cfd, &flush_needed);
if (!s.ok()) {
// Unref the newly generated candidate cfds (when not provided) in
// `candidate_cfds`
if (provided_candidate_cfds.empty()) {
for (auto candidate_cfd : candidate_cfds) {
candidate_cfd->UnrefAndTryDelete();
}
}
unref_generated_candidates();
return s;
} else if (flush_needed) {
++num_cfs_to_flush;
}
}
if (0 == num_cfs_to_flush) {
// Unref the newly generated candidate cfds (when not provided) in
// `candidate_cfds`
if (provided_candidate_cfds.empty()) {
for (auto candidate_cfd : candidate_cfds) {
candidate_cfd->UnrefAndTryDelete();
}
}
unref_generated_candidates();
return s;
}
}
@@ -2965,39 +3061,54 @@ Status DBImpl::AtomicFlushMemTables(
WriteThread::Writer w;
WriteThread::Writer nonmem_w;
if (needs_to_join_write_thread) {
write_thread_.EnterUnbatched(&w, &mutex_);
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
if (join_mode == WriteThreadJoinMode::kNonBlocking) {
s = EnterWriteThreadForNonBlockingFlush(&w, &nonmem_w);
if (!s.ok()) {
unref_generated_candidates();
return s;
}
} else {
write_thread_.EnterUnbatched(&w, &mutex_);
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
}
}
WaitForPendingWrites();
// Keep atomic recovery flushes consistent with the single-CF path: the
// sequence sync must happen after both write queues are drained and before
// any recovery memtable switch reads `LastSequence()`.
// Recovery may have released `mutex_` after the earlier `ResumeImpl()`
// sync. Refresh sequence state at the atomic memtable-switch fence, after
// both write queues are drained and before `SwitchMemtable()` consumes
// `LastSequence()`.
MaybeSyncLastSequenceWithAllocatedForRecovery(flush_reason);
SelectColumnFamiliesForAtomicFlush(&cfds, candidate_cfds, flush_reason);
// Unref the newly generated candidate cfds (when not provided) in
// `candidate_cfds`
if (provided_candidate_cfds.empty()) {
for (auto candidate_cfd : candidate_cfds) {
candidate_cfd->UnrefAndTryDelete();
}
}
unref_generated_candidates();
const bool defer_wbm_accounting_refresh = cfds.size() > 1;
bool switch_attempted = false;
for (auto cfd : cfds) {
if (cfd->mem()->IsEmpty() && cached_recoverable_state_empty_.load() &&
!IsRecoveryFlush(flush_reason)) {
continue;
}
cfd->Ref();
s = SwitchMemtable(cfd, &context);
switch_attempted = true;
s = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfd, &context, nullptr, 0,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfd, &context);
cfd->UnrefAndTryDelete();
if (!s.ok()) {
break;
}
if (made_progress != nullptr) {
*made_progress = true;
}
}
if (defer_wbm_accounting_refresh && switch_attempted) {
UpdateWBMAccountingAfterMemtableSwitch(true /* refresh_now */);
}
if (s.ok()) {
AssignAtomicFlushSeq(cfds);
@@ -3014,7 +3125,9 @@ Status DBImpl::AtomicFlushMemTables(
}
}
GenerateFlushRequest(cfds, flush_reason, &flush_req);
EnqueuePendingFlush(flush_req);
if (EnqueuePendingFlush(flush_req) && made_progress != nullptr) {
*made_progress = true;
}
MaybeScheduleFlushOrCompaction();
}
@@ -3025,14 +3138,39 @@ Status DBImpl::AtomicFlushMemTables(
}
}
}
NotifyOnManualFlushScheduled(cfds, flush_reason);
if (flush_reason != FlushReason::kWriteBufferManager) {
NotifyOnManualFlushScheduled(cfds, flush_reason);
}
TEST_SYNC_POINT("DBImpl::AtomicFlushMemTables:AfterScheduleFlush");
TEST_SYNC_POINT("DBImpl::AtomicFlushMemTables:BeforeWaitForBgFlush");
if (s.ok() && flush_options.wait) {
// Walk cfds and look each one up, so the ids line up with cfds by
// construction. Iterating the map instead pairs ids with the wrong CF
// (its order is unrelated to cfds), and a CF that dropped out of the map
// (an idle CF de-selected during the switch loop, or a duplicate handle
// collapsed by the map) would make the two vectors different lengths.
autovector<const uint64_t*> flush_memtable_ids;
for (auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
flush_memtable_ids.push_back(&(iter.second));
flush_memtable_ids.reserve(cfds.size());
for (auto* cfd : cfds) {
auto iter = flush_req.cfd_to_max_mem_id_to_persist.find(cfd);
flush_memtable_ids.push_back(
iter == flush_req.cfd_to_max_mem_id_to_persist.end()
? nullptr
: &(iter->second));
}
#ifndef NDEBUG
// WaitForFlushMemTables pairs flush_memtable_ids[i] with cfds[i], so each
// entry must point at this cfd's own map id (or be null when absent). Fires
// if the ids are ever rebuilt in the request map's order.
assert(flush_memtable_ids.size() == cfds.size());
for (size_t i = 0; i < cfds.size(); ++i) {
auto iter = flush_req.cfd_to_max_mem_id_to_persist.find(cfds[i]);
assert(flush_memtable_ids[i] ==
(iter == flush_req.cfd_to_max_mem_id_to_persist.end()
? nullptr
: &(iter->second)));
}
#endif // NDEBUG
s = WaitForFlushMemTables(
cfds, flush_memtable_ids,
flush_reason == FlushReason::kErrorRecovery /* resuming_from_bg_err */,
@@ -3072,9 +3210,29 @@ Status DBImpl::RetryFlushesForErrorRecovery(FlushReason flush_reason,
flush_req.atomic_flush = true;
GenerateFlushRequest(cfds, flush_reason, &flush_req);
EnqueuePendingFlush(flush_req);
for (auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
flush_memtable_ids.push_back(iter.second);
// Keep flush_memtable_ids aligned with cfds by walking cfds, since the
// request map's order is unrelated to cfds. A CF not in the map waits on
// its whole imm list, matching the max-bound the non-atomic branch uses.
for (auto* cfd : cfds) {
auto iter = flush_req.cfd_to_max_mem_id_to_persist.find(cfd);
flush_memtable_ids.push_back(
iter == flush_req.cfd_to_max_mem_id_to_persist.end()
? std::numeric_limits<uint64_t>::max()
: iter->second);
}
#ifndef NDEBUG
// WaitForFlushMemTables pairs flush_memtable_ids[i] with cfds[i], so each
// entry must be this cfd's own max id. Fires if the ids are ever rebuilt in
// the request map's order, which is unrelated to cfds order.
assert(flush_memtable_ids.size() == cfds.size());
for (size_t i = 0; i < cfds.size(); ++i) {
auto iter = flush_req.cfd_to_max_mem_id_to_persist.find(cfds[i]);
assert(flush_memtable_ids[i] ==
(iter == flush_req.cfd_to_max_mem_id_to_persist.end()
? std::numeric_limits<uint64_t>::max()
: iter->second));
}
#endif // NDEBUG
} else {
for (auto cfd : cfds) {
flush_memtable_ids.push_back(
@@ -3512,14 +3670,9 @@ void DBImpl::MaybeScheduleFlushOrCompaction() {
return;
} else if (error_handler_.IsBGWorkStopped() &&
!error_handler_.IsRecoveryInProgress()) {
// There has been a hard error and this call is not part of the recovery
// sequence. Bail out here so we don't get into an endless loop of
// scheduling BG work which will again call this function
//
// Note that a non-recovery flush can still be scheduled if
// error_handler_.IsRecoveryInProgress() returns true. We rely on
// BackgroundCallFlush() to check flush reason and drop non-recovery
// flushes.
// Recovery must be able to schedule its flushes while background work is
// stopped. New non-recovery requests are rejected by EnqueuePendingFlush,
// and BackgroundFlush drops any request queued before the error.
return;
} else if (shutting_down_.load(std::memory_order_acquire)) {
// DB is being deleted; no more background compactions
@@ -3817,6 +3970,10 @@ bool DBImpl::EnqueuePendingFlush(const FlushRequest& flush_req) {
if (reject_new_background_jobs_) {
return enqueued;
}
if (error_handler_.IsBGWorkStopped() &&
!IsRecoveryFlush(flush_req.flush_reason)) {
return enqueued;
}
if (flush_req.cfd_to_max_mem_id_to_persist.empty()) {
return enqueued;
}
@@ -3836,11 +3993,34 @@ bool DBImpl::EnqueuePendingFlush(const FlushRequest& flush_req) {
enqueued = true;
}
} else {
// Atomic flush requests bypass the queued_for_flush() deduplication guard.
// This means an atomic request and a non-atomic request for the same CF
// can coexist in the flush queue. This is safe because PickMemtablesToFlush
// uses flush_in_progress_ to prevent double-picking of memtables, and
// BackgroundFlush filters out CFs where !IsFlushPending().
if (flush_req.flush_reason == FlushReason::kWriteBufferManager &&
write_buffer_manager_->ShouldTrackFlushInitiator()) {
for (const FlushRequest& queued : flush_queue_) {
if (!queued.atomic_flush) {
continue;
}
bool covered = true;
for (const auto& requested : flush_req.cfd_to_max_mem_id_to_persist) {
const auto queued_entry =
queued.cfd_to_max_mem_id_to_persist.find(requested.first);
if (queued_entry == queued.cfd_to_max_mem_id_to_persist.end() ||
queued_entry->second < requested.second) {
covered = false;
break;
}
}
if (covered) {
TEST_SYNC_POINT(
"DBImpl::EnqueuePendingFlush:AtomicWBMRequestDeduplicated");
return false;
}
}
}
// Atomic flush requests normally bypass the per-CF queued_for_flush()
// guard. Cross-DB WBM requests are deduplicated above because its
// autonomous retry loop must not repeatedly enqueue the same atomic cut
// while HIGH workers are unavailable. Other atomic requests retain their
// existing behavior.
for (auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
ColumnFamilyData* cfd = iter.first;
cfd->Ref();
@@ -3965,6 +4145,178 @@ void DBImpl::UnscheduleFlushCallback(void* arg) {
TEST_SYNC_POINT("DBImpl::UnscheduleFlushCallback");
}
void DBImpl::RefreshFlushableMemAccounting() {
mutex_.AssertHeld();
versions_->GetColumnFamilySet()->RefreshFlushableMemAccounting();
}
bool DBImpl::TryRefreshFlushableMemAccounting() {
if (!mutex_.TryLock()) {
return false;
}
Defer unlock_mutex([this] { mutex_.Unlock(); });
RefreshFlushableMemAccounting();
return wbm_flush_initiator_ != nullptr &&
wbm_flush_initiator_->HasAccurateFlushableMemUsage();
}
DBImpl::FlushableCFs DBImpl::CollectFlushableCFs(
bool include_waiting_immutable) {
mutex_.AssertHeld();
FlushableCFs result;
SequenceNumber oldest_seq = kMaxSequenceNumber;
for (auto cfd : *versions_->GetColumnFamilySet()) {
if (cfd->IsDropped() || !cfd->initialized()) {
continue;
}
const size_t mutable_mem =
cfd->mem()->IsEmpty() ? 0 : cfd->mem()->WBMTrackedMemoryUsage();
size_t waiting_immutable_mem = 0;
if (include_waiting_immutable) {
if (immutable_db_options_.atomic_flush) {
waiting_immutable_mem =
cfd->imm()->WBMTrackedUnstartedMemTablesMemoryUsage();
} else if (!cfd->imm()->IsFlushPendingOrRunning()) {
waiting_immutable_mem =
cfd->imm()->WBMTrackedUnflushedMemTablesMemoryUsage();
}
}
result.total_mem += mutable_mem + waiting_immutable_mem;
// Non-atomic flushes skip CFs already flushing.
if (cfd->imm()->IsFlushPendingOrRunning()) {
continue;
}
size_t mem = mutable_mem;
if (include_waiting_immutable) {
mem += waiting_immutable_mem;
}
if (mem == 0) {
continue;
}
if (result.largest == nullptr || mem > result.largest_mem) {
result.largest = cfd;
result.largest_mem = mem;
}
const SequenceNumber seq = cfd->mem()->GetCreationSeq();
if (result.oldest == nullptr || seq < oldest_seq) {
result.oldest = cfd;
oldest_seq = seq;
}
}
return result;
}
bool DBImpl::ScheduleWriteBufferManagerFlush() {
{
if (!mutex_.TryLock()) {
return false;
}
Defer unlock_mutex([this] { mutex_.Unlock(); });
// Recheck bid eligibility because DB state may have changed.
if (!write_buffer_manager_->ShouldTrackFlushInitiator() ||
shutdown_initiated_.load(std::memory_order_acquire) ||
shutting_down_.load(std::memory_order_acquire) ||
reject_new_background_jobs_ || !opened_successfully_ || read_only_ ||
error_handler_.IsBGWorkStopped() || write_controller_.IsStopped() ||
// Reject both the pause drain and fully paused states.
bg_work_paused_ > 0 || bg_compaction_paused_ > 0 ||
// Never queue work that would wait behind this DB's stall.
WouldBlockJoiningWriteThread()) {
return false;
}
// A queued or running job does not count as new progress. Returning false
// lets the coordinator try another DB instead of queuing more work behind
// a busy LOW pool.
if (bg_wbm_flush_scheduled_ > 0) {
return false;
}
const FlushableCFs flushable =
CollectFlushableCFs(true /* include_waiting_immutable */);
const bool has_flushable_cf = immutable_db_options_.atomic_flush
? flushable.total_mem > 0
: flushable.largest != nullptr;
if (!has_flushable_cf) {
wbm_flush_initiator_->SetHasFlushableCF(false);
write_buffer_manager_->NotifyFlushInitiatorChanged();
return false;
}
wbm_flush_initiator_->SetHasFlushableCF(false);
write_buffer_manager_->NotifyFlushInitiatorChanged();
++bg_wbm_flush_scheduled_;
}
// The WBM-owned executor provides bounded capacity independent of Env::LOW.
// Run outside mutex_ so DB-local work cannot block other DB mutex users.
BackgroundCallWBMFlush();
return true;
}
void DBImpl::BackgroundCallWBMFlush() {
const bool atomic_flush = immutable_db_options_.atomic_flush;
ColumnFamilyData* cfd_to_flush = nullptr;
bool has_flushable_cf = false;
{
InstrumentedMutexLock l(&mutex_);
if (!shutdown_initiated_.load(std::memory_order_acquire) &&
!shutting_down_.load(std::memory_order_acquire) &&
!reject_new_background_jobs_ && opened_successfully_ && !read_only_ &&
!error_handler_.IsBGWorkStopped() && !write_controller_.IsStopped() &&
bg_work_paused_ == 0 && bg_compaction_paused_ == 0 &&
!WouldBlockJoiningWriteThread()) {
const FlushableCFs flushable =
CollectFlushableCFs(true /* include_waiting_immutable */);
has_flushable_cf =
atomic_flush ? flushable.total_mem > 0 : flushable.largest != nullptr;
if (!atomic_flush) {
cfd_to_flush = flushable.largest;
if (cfd_to_flush != nullptr) {
cfd_to_flush->Ref();
}
}
}
}
TEST_SYNC_POINT("DBImpl::BackgroundCallWBMFlush:AfterPick");
FlushOptions flush_options;
flush_options.wait = false;
flush_options.allow_write_stall = true;
bool made_progress = false;
if (has_flushable_cf) {
Status s;
if (atomic_flush) {
s = AtomicFlushMemTablesImpl(
flush_options, FlushReason::kWriteBufferManager,
{} /* provided_candidate_cfds */, false /* entered_write_thread */,
WriteThreadJoinMode::kNonBlocking, &made_progress);
} else {
s = FlushMemTableImpl(cfd_to_flush, flush_options,
FlushReason::kWriteBufferManager,
false /* entered_write_thread */,
WriteThreadJoinMode::kNonBlocking, &made_progress);
}
if (!s.ok()) {
ROCKS_LOG_WARN(
immutable_db_options_.info_log,
"Flush on behalf of a shared WriteBufferManager failed: %s",
s.ToString().c_str());
}
s.PermitUncheckedError();
}
TEST_SYNC_POINT_CALLBACK("DBImpl::BackgroundCallWBMFlush:MadeProgress",
&made_progress);
InstrumentedMutexLock l(&mutex_);
if (cfd_to_flush != nullptr) {
cfd_to_flush->UnrefAndTryDelete();
}
RefreshFlushableMemAccounting();
--bg_wbm_flush_scheduled_;
write_buffer_manager_->NotifyFlushInitiatorFlushCompleted(
wbm_flush_initiator_.get(), made_progress);
TEST_SYNC_POINT_CALLBACK("DBImpl::BGWorkWBMFlush:done", this);
bg_cv_.SignalAll();
}
Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
LogBuffer* log_buffer, FlushReason* reason,
bool* flush_rescheduled_to_retain_udt,
@@ -3973,8 +4325,8 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
Status status;
*reason = FlushReason::kOthers;
// If BG work is stopped due to an error, but a recovery is in progress,
// that means this flush is part of the recovery. So allow it to go through
// During recovery, allow flush workers to inspect the queue. Non-recovery
// requests are dropped below.
if (!error_handler_.IsBGWorkStopped()) {
if (shutting_down_.load(std::memory_order_acquire)) {
status = Status::ShutdownInProgress();
@@ -3996,12 +4348,9 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
FlushRequest flush_req = PopFirstFromFlushQueue();
FlushReason flush_reason = flush_req.flush_reason;
if (!error_handler_.GetBGError().ok() && error_handler_.IsBGWorkStopped() &&
flush_reason != FlushReason::kErrorRecovery &&
flush_reason != FlushReason::kErrorRecoveryRetryFlush) {
// Stop non-recovery flush when bg work is stopped
// Note that we drop the flush request here.
// Recovery thread should schedule further flushes after bg error
// is cleared.
!IsRecoveryFlush(flush_reason)) {
// A request queued before the error can reach a worker before recovery
// clears the queue. Drop it; recovery rebuilds flush work for all CFs.
status = error_handler_.GetBGError();
assert(!status.ok());
ROCKS_LOG_BUFFER(log_buffer,
@@ -4198,6 +4547,9 @@ void DBImpl::BackgroundCallFlush(Env::Priority thread_pri) {
assert(num_running_flushes_ > 0);
num_running_flushes_--;
bg_flush_scheduled_--;
if (write_buffer_manager_->ShouldTrackFlushInitiator()) {
RefreshFlushableMemAccounting();
}
// See if there's more work to be done
MaybeScheduleFlushOrCompaction();
@@ -4764,7 +5116,8 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
immutable_db_options_.stats, Histograms::SST_WRITE_MICROS,
c->immutable_options().listeners,
immutable_db_options_.file_checksum_gen_factory.get(),
tmp_set.Contains(FileType::kTableFile), false));
tmp_set.Contains(FileType::kTableFile),
tmp_set.Contains(FileType::kTableFile)));
}
ROCKS_LOG_BUFFER(
@@ -4855,7 +5208,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
false /* marked_for_compact */,
c->GetOutputTemperature() /* temperature */,
in_file->oldest_blob_file_number,
in_file->oldest_ancester_time,
in_file->oldest_ancestor_time,
out_file_creation_time,
c->MinInputFileEpochNumber(),
dest_writer->GetFileChecksum(),
@@ -5525,6 +5878,11 @@ void DBImpl::InstallSuperVersionAndScheduleWork(
cfd->InstallSuperVersion(sv_context, &mutex_,
std::move(new_seqno_to_time_mapping));
// Refresh the seqno->time preserve-window bound used to gate bottommost file
// marking, so a file whose largest seqno is still within the preserve window
// is not marked for a compaction that cannot zero it out (infinite loop).
MaybeUpdatePreserveTimeMinSeqno(cfd);
// There may be a small data race here. The snapshot tricking bottommost
// compaction may already be released here. But assuming there will always be
// newer snapshot created and released frequently, the compaction will be
@@ -5656,15 +6014,47 @@ Status DBImpl::WaitForCompact(
const auto deadline = immutable_db_options_.clock->NowMicros() +
wait_for_compact_options.timeout.count();
for (;;) {
// The coordinator reserves this DB before invoking its non-blocking
// scheduling callback. Drain that DB-specific reservation without holding
// mutex_; successful callbacks publish bg_wbm_flush_scheduled_ before
// releasing the reservation.
if (write_buffer_manager_ != nullptr &&
write_buffer_manager_->ShouldTrackFlushInitiator() &&
wbm_flush_initiator_ != nullptr) {
// Fence all coordinator requests visible before this local quiescence
// check. Candidate retrieval either observes this generation and skips
// the DB, or atomically publishes a handoff reservation first.
write_buffer_manager_->FenceFlushInitiator(wbm_flush_initiator_.get());
}
if (wbm_flush_initiator_ != nullptr &&
wbm_flush_initiator_->HasPendingFlushHandoff()) {
std::chrono::microseconds handoff_timeout{0};
if (wait_for_compact_options.timeout.count() != 0) {
const uint64_t now = immutable_db_options_.clock->NowMicros();
if (now >= deadline) {
return Status::TimedOut();
}
handoff_timeout = std::chrono::microseconds(deadline - now);
}
mutex_.Unlock();
const bool handoff_finished =
wbm_flush_initiator_->WaitForPendingFlushHandoff(handoff_timeout);
mutex_.Lock();
if (!handoff_finished) {
return Status::TimedOut();
}
continue;
}
if (shutting_down_.load(std::memory_order_acquire)) {
return Status::ShutdownInProgress();
}
if (bg_work_paused_ && wait_for_compact_options.abort_on_pause) {
return Status::Aborted();
}
// A WBM job can enqueue a regular flush, so wait for both to quiesce.
if ((bg_bottom_compaction_scheduled_ || bg_compaction_scheduled_ ||
bg_flush_scheduled_ || unscheduled_compactions_ ||
!parked_compaction_cfds_.empty() ||
bg_flush_scheduled_ || bg_wbm_flush_scheduled_ ||
unscheduled_compactions_ || !parked_compaction_cfds_.empty() ||
(wait_for_compact_options.wait_for_purge && bg_purge_scheduled_) ||
unscheduled_flushes_ || error_handler_.IsRecoveryInProgress()) &&
(error_handler_.GetBGError().ok())) {
+43 -3
View File
@@ -144,6 +144,18 @@ Status DBImpl::TEST_SwitchMemtable(ColumnFamilyData* cfd) {
return s;
}
size_t DBImpl::TEST_GetWBMFlushableMemUsage() {
InstrumentedMutexLock l(&mutex_);
assert(wbm_flush_initiator_ != nullptr);
RefreshFlushableMemAccounting();
return wbm_flush_initiator_->GetFlushableMemUsage();
}
bool DBImpl::TEST_IsWBMFlushInitiatorFlushable() const {
assert(wbm_flush_initiator_ != nullptr);
return wbm_flush_initiator_->IsFlushable();
}
Status DBImpl::TEST_FlushMemTable(bool wait, bool allow_write_stall,
ColumnFamilyHandle* cfh) {
FlushOptions fo;
@@ -182,9 +194,31 @@ Status DBImpl::TEST_FlushMemTableWithListenerWait(bool allow_write_stall,
Status DBImpl::TEST_AtomicFlushMemTables(
const autovector<ColumnFamilyData*>& provided_candidate_cfds,
const FlushOptions& flush_opts) {
return AtomicFlushMemTables(flush_opts, FlushReason::kTest,
provided_candidate_cfds);
const FlushOptions& flush_opts, bool non_blocking_write_thread) {
const auto join_mode = non_blocking_write_thread
? WriteThreadJoinMode::kNonBlocking
: WriteThreadJoinMode::kBlocking;
return AtomicFlushMemTablesImpl(
flush_opts, FlushReason::kTest, provided_candidate_cfds,
false /* entered_write_thread */, join_mode, nullptr /* made_progress */);
}
void DBImpl::TEST_BeginWriteStall() {
InstrumentedMutexLock l(&mutex_);
write_thread_.BeginWriteStall();
if (wbm_flush_initiator_ != nullptr &&
write_buffer_manager_->ShouldTrackFlushInitiator()) {
wbm_flush_initiator_->SetFlushable(false);
}
}
void DBImpl::TEST_EndWriteStall() {
InstrumentedMutexLock l(&mutex_);
write_thread_.EndWriteStall();
if (wbm_flush_initiator_ != nullptr &&
write_buffer_manager_->ShouldTrackFlushInitiator()) {
wbm_flush_initiator_->SetFlushable(true);
}
}
Status DBImpl::TEST_WaitForBackgroundWork() {
@@ -225,6 +259,12 @@ Status DBImpl::TEST_GetBGError() {
return error_handler_.GetBGError();
}
void DBImpl::TEST_SetBGError(const IOStatus& error,
BackgroundErrorReason reason) {
InstrumentedMutexLock l(&mutex_);
error_handler_.SetBGError(error, reason);
}
bool DBImpl::TEST_IsRecoveryInProgress() {
InstrumentedMutexLock l(&mutex_);
return error_handler_.IsRecoveryInProgress();
+1 -1
View File
@@ -140,7 +140,7 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->temperature,
f->oldest_blob_file_number, f->oldest_ancester_time,
f->oldest_blob_file_number, f->oldest_ancestor_time,
f->file_creation_time, f->epoch_number, f->file_checksum,
f->file_checksum_func_name, f->unique_id,
f->compensated_range_deletion_size, f->tail_size,
+20 -10
View File
@@ -489,31 +489,33 @@ void DBImpl::DeleteObsoleteFileImpl(int job_id, const std::string& fname,
&file_deletion_status);
if (file_deletion_status.ok()) {
ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[JOB %d] Delete %s type=%d #%" PRIu64 " -- %s\n", job_id,
fname.c_str(), type, number,
"[JOB %d] [db_session_id=%s] Delete %s type=%d #%" PRIu64
" -- %s\n",
job_id, db_session_id_.c_str(), fname.c_str(), type, number,
file_deletion_status.ToString().c_str());
} else if (env_->FileExists(fname).IsNotFound()) {
ROCKS_LOG_INFO(
immutable_db_options_.info_log,
"[JOB %d] Tried to delete a non-existing file %s type=%d #%" PRIu64
" -- %s\n",
job_id, fname.c_str(), type, number,
"[JOB %d] [db_session_id=%s] Tried to delete a non-existing file %s"
" type=%d #%" PRIu64 " -- %s\n",
job_id, db_session_id_.c_str(), fname.c_str(), type, number,
file_deletion_status.ToString().c_str());
} else {
ROCKS_LOG_ERROR(immutable_db_options_.info_log,
"[JOB %d] Failed to delete %s type=%d #%" PRIu64 " -- %s\n",
job_id, fname.c_str(), type, number,
"[JOB %d] [db_session_id=%s] Failed to delete %s type=%d"
" #%" PRIu64 " -- %s\n",
job_id, db_session_id_.c_str(), fname.c_str(), type, number,
file_deletion_status.ToString().c_str());
}
if (type == kTableFile) {
EventHelpers::LogAndNotifyTableFileDeletion(
&event_logger_, job_id, number, fname, file_deletion_status, GetName(),
immutable_db_options_.listeners);
db_session_id_, immutable_db_options_.listeners);
}
if (type == kBlobFile) {
EventHelpers::LogAndNotifyBlobFileDeletion(
&event_logger_, immutable_db_options_.listeners, job_id, number, fname,
file_deletion_status, GetName());
file_deletion_status, GetName(), db_session_id_);
}
}
@@ -1192,6 +1194,7 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
const uint64_t prev_next_file_number = versions_->current_next_file_number();
uint64_t largest_file_number = prev_next_file_number;
bool on_disk_file_advanced_counter = false;
bool manifest_relevant_file_advanced_counter = false;
Status s;
for (const auto& path : CollectAllDBPaths()) {
std::vector<std::string> files;
@@ -1213,6 +1216,9 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
// NewFileNumber() would collide with it.
if (number >= prev_next_file_number) {
on_disk_file_advanced_counter = true;
if (type != kOptionsFile && type != kTempFile) {
manifest_relevant_file_advanced_counter = true;
}
}
largest_file_number = std::max(largest_file_number, number);
if ((type == kTableFile || type == kBlobFile)) {
@@ -1234,7 +1240,8 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
mutable_db_options_.optimize_manifest_for_recovery &&
!immutable_db_options_.best_efforts_recovery;
if (!optimize_manifest_for_recovery || on_disk_file_advanced_counter) {
if (!optimize_manifest_for_recovery ||
manifest_relevant_file_advanced_counter) {
if (largest_file_number >= prev_next_file_number) {
versions_->next_file_number_.store(largest_file_number + 1);
}
@@ -1247,6 +1254,9 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
assert(default_cfd);
recovery_ctx->UpdateVersionEdits(default_cfd, edit);
} else {
if (on_disk_file_advanced_counter) {
versions_->next_file_number_.store(largest_file_number + 1);
}
TEST_SYNC_POINT("DBImpl::Recovery:SkippedNoopEdit:NextFileNumber");
}
return s;
+19 -14
View File
@@ -131,21 +131,21 @@ Status DBImplFollower::TryCatchUpWithLeader() {
for (auto cfd : cfds_changed) {
if (cfd->mem()->GetEarliestSequenceNumber() <
versions_->LastSequence()) {
// Construct a new memtable with earliest sequence number set to the
// last sequence number in the VersionSet. This matters when
// DBImpl::MultiCFSnapshot tries to get consistent references
// to super versions in a lock free manner, it checks the earliest
// sequence number to detect if there was a change in version in
// the meantime.
MemTable* new_mem = cfd->ConstructNewMemtable(
cfd->GetLatestMutableCFOptions(), versions_->LastSequence());
cfd->mem()->SetNextLogNumber(cfd->GetLogNumber());
cfd->mem()->ConstructFragmentedRangeTombstones();
cfd->imm()->Add(cfd->mem(), &job_context.memtables_to_free);
new_mem->Ref();
cfd->SetMemtable(new_mem);
// Seal the active memtable so that its replacement gets its earliest
// sequence number set to the last sequence number in the VersionSet.
// This matters when DBImpl::MultiCFSnapshot tries to get consistent
// references to super versions in a lock free manner, it checks the
// earliest sequence number to detect if there was a change in
// version in the meantime.
SealActiveMemtable(cfd, cfd->GetLogNumber(),
versions_->LastSequence(), &job_context);
}
// `cfd->GetLogNumber()` is a safe watermark here, unlike in
// DBImplSecondary::TryCatchUpWithPrimary(): this catch-up path never
// replays WALs, so the memtables it collects hold nothing that only a
// not-yet-installed Version could make readable.
//
// This will check if the old memtable is still referenced
cfd->imm()->RemoveOldMemTables(cfd->GetLogNumber(),
&job_context.memtables_to_free);
@@ -153,6 +153,7 @@ Status DBImplFollower::TryCatchUpWithLeader() {
cfd->InstallSuperVersion(&sv_context, &mutex_);
sv_context.NewSuperVersion();
}
PublishSecondaryReadView();
}
for (auto& file : files_to_delete) {
@@ -164,6 +165,7 @@ Status DBImplFollower::TryCatchUpWithLeader() {
}
}
}
CleanupRetiredSecondaryReadViews();
job_context.Clean();
// Cleanup unused, obsolete files.
@@ -296,7 +298,8 @@ Status DB::OpenAsFollower(
dbname, &impl->immutable_db_options_, impl->mutable_db_options_,
impl->file_options_, impl->table_cache_.get(),
impl->write_buffer_manager_, &impl->write_controller_, impl->io_tracer_,
impl->db_id_, impl->db_session_id_));
impl->db_id_, impl->db_session_id_,
/*defer_sst_file_opening=*/false));
impl->column_family_memtables_.reset(
new ColumnFamilyMemTablesImpl(impl->versions_->GetColumnFamilySet()));
impl->wal_in_db_path_ = impl->immutable_db_options_.IsWalDirSameAsDBPath();
@@ -322,9 +325,11 @@ Status DB::OpenAsFollower(
sv_context.NewSuperVersion();
cfd->InstallSuperVersion(&sv_context, &impl->mutex_);
}
impl->PublishSecondaryReadView();
impl->MarkAsyncFileOpenNotNeeded();
}
impl->mutex_.Unlock();
impl->CleanupRetiredSecondaryReadViews();
sv_context.Clean();
if (s.ok()) {
dbptr->reset(impl);
+67
View File
@@ -0,0 +1,67 @@
// Copyright (c) Meta Platforms, Inc. and affiliates.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#include <cstddef>
#include <cstdint>
#include <string>
#include <vector>
#include "rocksdb/db.h"
#include "table/multiget_context.h"
#include "util/autovector.h"
namespace ROCKSDB_NAMESPACE {
inline std::string* GetOutputTimestamp(OutputMetadata* output_metadata) {
return output_metadata != nullptr && output_metadata->timestamp.has_value()
? &*output_metadata->timestamp
: nullptr;
}
inline std::vector<std::string>* GetOutputTimestamps(
MultiGetOutputMetadata* output_metadata) {
return output_metadata != nullptr && output_metadata->timestamps.has_value()
? &*output_metadata->timestamps
: nullptr;
}
inline bool* GetOutputNewerVersionPresent(OutputMetadata* output_metadata) {
return output_metadata != nullptr &&
output_metadata->newer_version_present.has_value()
? &*output_metadata->newer_version_present
: nullptr;
}
inline std::vector<uint8_t>* GetOutputNewerVersionPresent(
MultiGetOutputMetadata* output_metadata) {
return output_metadata != nullptr &&
output_metadata->newer_version_present.has_value()
? &*output_metadata->newer_version_present
: nullptr;
}
inline ColumnFamilyHandle** MakeMutableCfHandles(
ColumnFamilyHandle* const* column_families, size_t num_keys,
autovector<ColumnFamilyHandle*, MultiGetContext::MAX_BATCH_SIZE>*
stack_column_families,
std::vector<ColumnFamilyHandle*>* heap_column_families) {
ColumnFamilyHandle** mutable_column_families;
if (num_keys <= MultiGetContext::MAX_BATCH_SIZE) {
stack_column_families->resize(num_keys);
mutable_column_families =
num_keys == 0 ? nullptr : &(*stack_column_families)[0];
} else {
heap_column_families->resize(num_keys);
mutable_column_families = heap_column_families->data();
}
for (size_t i = 0; i < num_keys; ++i) {
mutable_column_families[i] = column_families[i];
}
return mutable_column_families;
}
} // namespace ROCKSDB_NAMESPACE
+58 -14
View File
@@ -683,7 +683,7 @@ Status DBImpl::Recover(
f->marked_for_compaction,
f->temperature, // this can be different from
// `last_level_temperature`
f->oldest_blob_file_number, f->oldest_ancester_time,
f->oldest_blob_file_number, f->oldest_ancestor_time,
f->file_creation_time, f->epoch_number,
f->file_checksum, f->file_checksum_func_name,
f->unique_id, f->compensated_range_deletion_size,
@@ -1034,7 +1034,7 @@ Status DBImpl::InitPersistStatsColumnFamily() {
return s;
}
Status DBImpl::LogAndApplyForRecovery(const RecoveryContext& recovery_ctx) {
Status DBImpl::LogAndApplyForRecovery(RecoveryContext& recovery_ctx) {
mutex_.AssertHeld();
// descriptor_log_ is normally null after Recover, but when
// reuse_manifest_on_open is set VersionSet::Recover may have already
@@ -1044,9 +1044,21 @@ Status DBImpl::LogAndApplyForRecovery(const RecoveryContext& recovery_ctx) {
const ReadOptions read_options(Env::IOActivity::kDBOpen);
const WriteOptions write_options(Env::IOActivity::kDBOpen);
if (versions_->force_new_manifest_on_open_ &&
!recovery_ctx.HasVersionEdits()) {
VersionEdit edit;
ColumnFamilyData* default_cfd =
versions_->GetColumnFamilySet()->GetDefault();
assert(default_cfd);
recovery_ctx.UpdateVersionEdits(default_cfd, edit);
}
Status s = versions_->LogAndApply(recovery_ctx.cfds_, read_options,
write_options, recovery_ctx.edit_lists_,
&mutex_, directories_.GetDbDir());
if (s.ok()) {
versions_->force_new_manifest_on_open_ = false;
}
return s;
}
@@ -2131,7 +2143,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
immutable_db_options_.clock->GetCurrentTime(&_current_time)
.PermitUncheckedError(); // ignore error
const uint64_t current_time = static_cast<uint64_t>(_current_time);
meta.oldest_ancester_time = current_time;
meta.oldest_ancestor_time = current_time;
meta.epoch_number = cfd->NewEpochNumber();
{
auto write_hint = cfd->current()->storage_info()->CalculateSSTWriteHint(
@@ -2265,7 +2277,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
meta.fd.GetFileSize(), meta.smallest, meta.largest,
meta.fd.smallest_seqno, meta.fd.largest_seqno,
meta.marked_for_compaction, meta.temperature,
meta.oldest_blob_file_number, meta.oldest_ancester_time,
meta.oldest_blob_file_number, meta.oldest_ancestor_time,
meta.file_creation_time, meta.epoch_number,
meta.file_checksum, meta.file_checksum_func_name,
meta.unique_id, meta.compensated_range_deletion_size,
@@ -2699,6 +2711,12 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
if (s.ok()) {
s = impl->CreateArchivalDirectory();
}
if (s.ok() &&
impl->immutable_db_options_.use_session_tmp_dir_for_remote_compaction) {
// Create the session-scoped temporary directory before a CompactionService
// can create any per-job directory under it.
s = impl->env_->CreateDirIfMissing(SessionTmpDir(dbname));
}
if (!s.ok()) {
return s;
}
@@ -2773,8 +2791,9 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
s = WritableFileWriter::PrepareIOOptions(write_options, opts);
}
if (s.ok()) {
s = log_writer->file()->Sync(opts,
impl->immutable_db_options_.use_fsync);
s = impl->immutable_db_options_.use_fsync
? log_writer->file()->Fsync(opts)
: log_writer->file()->Sync(opts);
}
}
}
@@ -2913,6 +2932,13 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
impl->immutable_db_options_.db_paths[0].path);
}
if (s.ok() &&
impl->immutable_db_options_.use_session_tmp_dir_for_remote_compaction) {
// Finish deleting data from the previous DB session before this open can
// schedule a new compaction in the directory.
impl->CleanupSessionTmpDir();
}
if (s.ok()) {
// When the DB is stopped, it's possible that there are some .trash files
// that were not deleted yet, when we open the DB we will find these .trash
@@ -2954,8 +2980,9 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
IOOptions opts;
s = WritableFileWriter::PrepareIOOptions(write_options, opts);
if (s.ok()) {
s = log_writer->file()->Sync(opts,
impl->immutable_db_options_.use_fsync);
s = impl->immutable_db_options_.use_fsync
? log_writer->file()->Fsync(opts)
: log_writer->file()->Sync(opts);
}
}
}
@@ -2988,6 +3015,8 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
}
}
if (s.ok()) {
// Register only after open succeeds and without holding the DB mutex.
impl->MaybeRegisterFlushInitiator();
*dbptr = std::move(impl);
} else {
for (auto* h : *handles) {
@@ -3045,8 +3074,18 @@ void DBImpl::ScheduleAsyncFileOpening() {
bg_async_file_open_state_ = AsyncFileOpenState::kScheduled;
// since this is a one time job, best to schedule it with high priority
auto unschedule_callback = [](void* arg) {
auto* unscheduled_ctx = static_cast<AsyncFileOpenContext*>(arg);
DBImpl* db = unscheduled_ctx->db;
db->mutex_.AssertHeld();
delete unscheduled_ctx;
db->bg_async_file_open_state_ = AsyncFileOpenState::kComplete;
db->bg_cv_.SignalAll();
TEST_SYNC_POINT("DBImpl::UnscheduleAsyncFileOpenCallback");
TEST_SYNC_POINT("DBImpl::BGWorkAsyncFileOpen:Done");
};
env_->Schedule(&DBImpl::BGWorkAsyncFileOpen, ctx, Env::Priority::HIGH,
nullptr);
&bg_async_file_open_state_, unschedule_callback);
}
void DBImpl::MarkAsyncFileOpenNotNeeded() {
@@ -3076,8 +3115,9 @@ void DBImpl::BGWorkAsyncFileOpen(void* arg) {
auto* version = ctx->versions[i];
ColumnFamilyData* cfd = version->cfd();
// Skip column families that were dropped after scheduling
if (cfd->IsDropped()) {
// Retained secondary handles remain readable after the primary drops the
// column family, so their captured Versions still need pinned readers.
if (cfd->IsDropped() && !db->read_only_) {
continue;
}
@@ -3107,11 +3147,15 @@ void DBImpl::BGWorkAsyncFileOpen(void* arg) {
if (!s.ok()) {
ROCKS_LOG_ERROR(
db->immutable_db_options_.info_log,
"BGWorkAsyncFileOpen: LoadTableHandlers failed for CF %s: "
"%s",
"BGWorkAsyncFileOpen: file validation/opening failed for CF %s: %s",
cfd->GetName().c_str(), s.ToString().c_str());
InstrumentedMutexLock l(&db->mutex_);
db->error_handler_.SetBGError(s, BackgroundErrorReason::kAsyncFileOpen);
if (db->read_only_) {
db->error_handler_.SetBGErrorNoAutoRecovery(
s, BackgroundErrorReason::kAsyncFileOpen);
} else {
db->error_handler_.SetBGError(s, BackgroundErrorReason::kAsyncFileOpen);
}
break;
}
}
+114 -43
View File
@@ -11,6 +11,7 @@
#include "db/blob/blob_fetcher.h"
#include "db/db_impl/compacted_db_impl.h"
#include "db/db_impl/db_impl.h"
#include "db/db_impl/db_impl_metadata.h"
#include "db/manifest_ops.h"
#include "db/merge_context.h"
#include "logging/logging.h"
@@ -30,6 +31,30 @@ DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options,
DBImplReadOnly::~DBImplReadOnly() = default;
void DBImplReadOnly::MultiGetWithMetadata(
const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle* const* column_families, const Slice* keys,
PinnableSlice* values, Status* statuses,
MultiGetOutputMetadata* output_metadata, const bool sorted_input) {
std::vector<std::string>* timestamps = GetOutputTimestamps(output_metadata);
if (timestamps != nullptr) {
timestamps->resize(num_keys);
}
std::vector<uint8_t>* newer_version_present =
GetOutputNewerVersionPresent(output_metadata);
if (newer_version_present != nullptr) {
newer_version_present->assign(num_keys, false);
}
autovector<ColumnFamilyHandle*, MultiGetContext::MAX_BATCH_SIZE>
stack_column_families;
std::vector<ColumnFamilyHandle*> heap_column_families;
ColumnFamilyHandle** mutable_column_families = MakeMutableCfHandles(
column_families, num_keys, &stack_column_families, &heap_column_families);
DBImpl::MultiGet(options, num_keys, mutable_column_families, keys, values,
timestamps != nullptr ? timestamps->data() : nullptr,
statuses, sorted_input);
}
Iterator* DBImplReadOnly::NewIterator(const ReadOptions& _read_options,
ColumnFamilyHandle* column_family) {
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
@@ -82,65 +107,111 @@ Status DBImplReadOnly::NewIterators(
const ReadOptions& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) {
if (read_options.timestamp) {
for (auto* cf : column_families) {
assert(cf);
const Status s = FailIfTsMismatchCf(cf, *(read_options.timestamp));
if (!s.ok()) {
return s;
}
}
} else {
for (auto* cf : column_families) {
assert(cf);
const Status s = FailIfCfHasTs(cf);
if (!s.ok()) {
return s;
}
}
}
return NewIterators(
std::vector<ReadOptions>(column_families.size(), read_options),
column_families, iterators);
}
ReadCallback* read_callback = nullptr; // No read callback provided.
Status DBImplReadOnly::NewIterators(
const std::vector<ReadOptions>& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) {
if (read_options.size() != column_families.size()) {
return Status::InvalidArgument(
"read_options and column_families must have the same size");
}
if (iterators == nullptr) {
return Status::InvalidArgument("iterators not allowed to be nullptr");
}
if (column_families.empty()) {
iterators->clear();
return Status::OK();
}
const Snapshot* const snapshot = read_options.front().snapshot;
const bool tailing = read_options.front().tailing;
std::vector<ReadOptions> normalized_read_options;
normalized_read_options.reserve(read_options.size());
for (size_t i = 0; i < read_options.size(); ++i) {
const ReadOptions& options = read_options[i];
if (options.io_activity != Env::IOActivity::kUnknown &&
options.io_activity != Env::IOActivity::kDBIterator) {
return Status::InvalidArgument(
"Can only call NewIterators with `ReadOptions::io_activity` is "
"`Env::IOActivity::kUnknown` or `Env::IOActivity::kDBIterator`");
}
if (options.read_tier == kPersistedTier) {
return Status::NotSupported(
"ReadTier::kPersistedData is not yet supported in iterators.");
}
if (options.snapshot != snapshot) {
return Status::InvalidArgument(
"All ReadOptions must use the same snapshot");
}
if (options.tailing != tailing) {
return Status::InvalidArgument(
"All ReadOptions must use the same tailing setting");
}
auto* cf = column_families[i];
assert(cf);
Status s;
if (options.timestamp) {
s = FailIfTsMismatchCf(cf, *(options.timestamp));
} else {
s = FailIfCfHasTs(cf);
}
if (!s.ok()) {
return s;
}
normalized_read_options.emplace_back(options);
if (normalized_read_options.back().io_activity ==
Env::IOActivity::kUnknown) {
normalized_read_options.back().io_activity = Env::IOActivity::kDBIterator;
}
}
iterators->clear();
iterators->reserve(column_families.size());
SequenceNumber latest_snapshot = versions_->LastSequence();
SequenceNumber read_seq =
read_options.snapshot != nullptr
? static_cast<const SnapshotImpl*>(read_options.snapshot)->number_
: latest_snapshot;
snapshot != nullptr ? static_cast<const SnapshotImpl*>(snapshot)->number_
: versions_->LastSequence();
autovector<std::tuple<ColumnFamilyHandleImpl*, SuperVersion*>> cfh_to_sv;
for (auto* cf : column_families) {
auto* cfh = static_cast_with_check<ColumnFamilyHandleImpl>(cf);
cfh_to_sv.emplace_back(cfh, cfh->cfd()->GetSuperVersion()->Ref());
}
const bool check_read_ts =
read_options.timestamp && read_options.timestamp->size() > 0;
for (auto cfh : column_families) {
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(cfh)->cfd();
auto* sv = cfd->GetSuperVersion()->Ref();
cfh_to_sv.emplace_back(static_cast_with_check<ColumnFamilyHandleImpl>(cfh),
sv);
if (check_read_ts) {
const auto unref_super_versions = [&]() {
for (const auto& entry : cfh_to_sv) {
std::get<1>(entry)->Unref();
}
};
for (size_t i = 0; i < cfh_to_sv.size(); ++i) {
const ReadOptions& options = normalized_read_options[i];
if (options.timestamp && !options.timestamp->empty()) {
auto* cfd = std::get<0>(cfh_to_sv[i])->cfd();
auto* sv = std::get<1>(cfh_to_sv[i]);
const Status s =
FailIfReadCollapsedHistory(cfd, sv, *(read_options.timestamp));
FailIfReadCollapsedHistory(cfd, sv, *(options.timestamp));
if (!s.ok()) {
for (auto prev_entry : cfh_to_sv) {
std::get<1>(prev_entry)->Unref();
}
unref_super_versions();
return s;
}
}
}
assert(cfh_to_sv.size() == column_families.size());
for (auto [cfh, sv] : cfh_to_sv) {
auto* db_iter = NewArenaWrappedDbIterator(
env_, read_options, cfh, sv, read_seq, read_callback, this,
/*expose_blob_index=*/false, /*allow_refresh=*/false,
/*allow_mark_memtable_for_flush=*/false);
iterators->push_back(db_iter);
}
for (size_t i = 0; i < cfh_to_sv.size(); ++i) {
auto* cfh = std::get<0>(cfh_to_sv[i]);
auto* sv = std::get<1>(cfh_to_sv[i]);
iterators->push_back(NewArenaWrappedDbIterator(
env_, normalized_read_options[i], cfh, sv, read_seq,
nullptr /* read_callback */, this, /* expose_blob_index */ false,
/* allow_refresh */ false, /* allow_mark_memtable_for_flush */ false));
}
return Status::OK();
}
+10 -1
View File
@@ -28,7 +28,13 @@ class DBImplReadOnly : public DBImpl {
const Slice& key,
GetImplOptions& get_impl_options);
// TODO: Implement ReadOnly MultiGet?
using DBImpl::MultiGetWithMetadata;
void MultiGetWithMetadata(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle* const* column_families,
const Slice* keys, PinnableSlice* values,
Status* statuses,
MultiGetOutputMetadata* output_metadata,
const bool sorted_input = false) override;
using DBImpl::NewIterator;
Iterator* NewIterator(const ReadOptions& _read_options,
@@ -37,6 +43,9 @@ class DBImplReadOnly : public DBImpl {
Status NewIterators(const ReadOptions& options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
Status NewIterators(const std::vector<ReadOptions>& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
using DBImpl::Put;
Status Put(const WriteOptions& /*options*/,
+27 -10
View File
@@ -20,6 +20,10 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplReadOnly::GetImpl)
get_impl_options.merge_operands != nullptr);
assert(get_impl_options.column_family);
// This instance cannot observe writes after one of its snapshots, so the
// false result initialized by GetWithMetadata is definitive.
get_impl_options.newer_version_present = nullptr;
Status s;
if (read_options.timestamp) {
@@ -49,7 +53,13 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplReadOnly::GetImpl)
const Comparator* ucmp = get_impl_options.column_family->GetComparator();
assert(ucmp);
SequenceNumber snapshot = versions_->LastSequence();
// For a batched lazy read (MultiGetEntityLazy), use the SuperVersion and
// consistent sequence number the batch already fixed (see
// GetImplOptions::lazy_columns_shared_sv); otherwise this read-only
// instance's stable current SuperVersion and last sequence.
SequenceNumber snapshot = get_impl_options.lazy_columns_shared_sv != nullptr
? get_impl_options.lazy_columns_snapshot_seq
: versions_->LastSequence();
GetWithTimestampReadCallback read_cb(snapshot);
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(
get_impl_options.column_family);
@@ -77,7 +87,10 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplReadOnly::GetImpl)
// In read-only mode Get(), no super version operation is needed (i.e.
// GetAndRefSuperVersion and ReturnAndCleanupSuperVersion)
SuperVersion* super_version = cfd->GetSuperVersion();
SuperVersion* super_version =
get_impl_options.lazy_columns_shared_sv != nullptr
? get_impl_options.lazy_columns_shared_sv
: cfd->GetSuperVersion();
if (read_options.timestamp && read_options.timestamp->size() > 0) {
s = FailIfReadCollapsedHistory(cfd, super_version,
*(read_options.timestamp));
@@ -157,15 +170,19 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplReadOnly::GetImpl)
PERF_COUNTER_ADD(get_read_bytes, size);
}
}
if (get_impl_options.lazy_columns_version != nullptr && s.ok()) {
// Lazy result (GetEntityLazy / MultiGetEntityLazy): hand back the Version
// the entity's blob references must be resolved against (both the
// single-key and batched paths).
*get_impl_options.lazy_columns_version = super_version->current;
}
if (get_impl_options.lazy_columns_pin != nullptr && s.ok()) {
// Lazy result (GetEntityLazy): hand back the Version the entity's blob
// references must be resolved against, and transfer a SuperVersion pin into
// the result so it -- and deferred blob reads -- stay valid after this call
// (as an iterator's pin does). Read-only borrows the current SuperVersion
// without a reference, so the pin takes its own.
if (get_impl_options.lazy_columns_version != nullptr) {
*get_impl_options.lazy_columns_version = super_version->current;
}
// Single-key GetEntityLazy: transfer a SuperVersion pin into the result so
// it -- and deferred blob reads -- stay valid after this call (as an
// iterator's pin does). Read-only borrows the current SuperVersion without
// a reference, so the pin takes its own. The batched path passes no pin (it
// uses lazy_columns_shared_sv); the batch holds one shared pin per column
// family instead.
TransferSuperVersionPin(super_version, get_impl_options.lazy_columns_pin);
}
CO_RETURN s;
File diff suppressed because it is too large. Load diff
+170 -5
View File
@@ -5,11 +5,19 @@
#pragma once
#include <condition_variable>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <unordered_map>
#include <unordered_set>
#include <utility>
#include <vector>
#include "db/db_impl/db_impl.h"
#include "logging/logging.h"
#include "port/port.h"
namespace ROCKSDB_NAMESPACE {
@@ -101,6 +109,14 @@ class DBImplSecondary : public DBImpl {
const Slice& key,
GetImplOptions& get_impl_options);
using DBImpl::MultiGetWithMetadata;
void MultiGetWithMetadata(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle* const* column_families,
const Slice* keys, PinnableSlice* values,
Status* statuses,
MultiGetOutputMetadata* output_metadata,
const bool sorted_input = false) override;
using DBImpl::NewIterator;
// Operations on the created iterators can return IOError due to files being
// deleted by the primary. To avoid IOError in this case, application can
@@ -124,6 +140,9 @@ class DBImplSecondary : public DBImpl {
Status NewIterators(const ReadOptions& _read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
Status NewIterators(const std::vector<ReadOptions>& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators) override;
using DBImpl::Put;
Status Put(const WriteOptions& /*options*/,
@@ -274,6 +293,8 @@ class DBImplSecondary : public DBImpl {
#endif // NDEBUG
protected:
Status CloseImpl() override;
Status FlushForGetLiveFiles(bool /*force_atomic_flush*/) override {
// No-op for read-only DB
return Status::OK();
@@ -287,13 +308,76 @@ class DBImplSecondary : public DBImpl {
return false;
}
// Seals `cfd`'s active memtable: adds it to the immutable memtable list and
// puts a fresh, empty active memtable in its place. This is the counterpart
// of DBImpl::SwitchMemtable(), which cannot be used here because there is no
// writer queue and no WAL of this instance's own.
//
// `next_log_number` is recorded on the sealed memtable. A subsequent
// MemTableList::RemoveOldMemTables(log_number) call considers the sealed
// memtable collectible once `next_log_number <= log_number`, so pass a log
// number above every WAL whose entries the memtable holds.
// `new_mem_earliest_seq` becomes the replacement memtable's earliest
// sequence number.
//
// REQUIRES: mutex_ held
void SealActiveMemtable(ColumnFamilyData* cfd, uint64_t next_log_number,
SequenceNumber new_mem_earliest_seq,
JobContext* job_context);
std::unique_ptr<log::FragmentBufferedReader> manifest_reader_;
std::unique_ptr<log::Reader::Reporter> manifest_reporter_;
std::unique_ptr<Status> manifest_reader_status_;
// Publishes a view of every initialized, non-dropped column family while
// mutex_ protects their SuperVersions. The previous view is retained for
// cleanup outside mutex_ and outside reader threads. No-op once the view has
// been torn down for close.
// REQUIRES: mutex_ held
void PublishSecondaryReadView();
// Reclaims retired views that have no readers. This must be called without
// mutex_ because releasing a view's final SuperVersion can lock it.
void CleanupRetiredSecondaryReadViews();
private:
friend class DB;
struct SecondaryReadView {
using SuperVersions = std::unordered_map<uint32_t, SuperVersion*>;
SecondaryReadView(DBImplSecondary* db_arg, SequenceNumber sequence_arg)
: db(db_arg), sequence(sequence_arg) {}
~SecondaryReadView();
SecondaryReadView(const SecondaryReadView&) = delete;
SecondaryReadView& operator=(const SecondaryReadView&) = delete;
SecondaryReadView(SecondaryReadView&&) = delete;
SecondaryReadView& operator=(SecondaryReadView&&) = delete;
DBImplSecondary* db;
SequenceNumber sequence;
SuperVersions super_versions;
};
void RetireSecondaryReadView(
std::shared_ptr<const SecondaryReadView> read_view);
// REQUIRES: retired_secondary_read_views_mutex_ held
bool HasReclaimableSecondaryReadView() const;
// REQUIRES: retired_secondary_read_views_mutex_ held
bool AllSecondaryReadViewsReclaimed() const;
bool ReclaimRetiredSecondaryReadViews();
void ReclaimRetiredSecondaryReadViewsLoop();
void DrainRetiredSecondaryReadViews();
void StopSecondaryReadViewReclaimer();
void ReleaseSecondaryReadView(
std::shared_ptr<const SecondaryReadView>* read_view);
void ResetSecondaryReadView();
Status NewIteratorsImpl(
const std::vector<ReadOptions>& read_options,
const std::vector<ColumnFamilyHandle*>& column_families,
std::vector<Iterator*>* iterators);
// No copying allowed
DBImplSecondary(const DBImplSecondary&);
void operator=(const DBImplSecondary&);
@@ -304,13 +388,63 @@ class DBImplSecondary : public DBImpl {
std::unordered_set<ColumnFamilyData*>* cfds_changed,
JobContext* job_context);
Status FindNewLogNumbers(std::vector<uint64_t>* logs);
// After manifest recovery, replay WALs and refresh log_readers_ if necessary
// Replays WALs after manifest recovery, refreshes log_readers_ as needed, and
// drops recovered transactions the primary resolved. Cleanup runs even when
// `log_numbers` is empty.
// REQUIRES: log_numbers are sorted in ascending order
Status RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
SequenceNumber* next_sequence,
std::unordered_set<ColumnFamilyData*>* cfds_changed,
JobContext* job_context);
// Records `log_number` as the newest WAL whose entries `cf_id`'s active
// memtable may hold, never lowering what is already recorded. Returns what
// was recorded before this call, or nullopt if nothing was.
//
// REQUIRES: mutex_ held
std::optional<uint64_t> RecordCurrentLog(uint32_t cf_id, uint64_t log_number);
// Seals the column family's active memtable if everything it holds is
// already readable through the installed Version, so that a following
// RemoveOldMemTables(installed_log_number) call drops it. Returns whether the
// memtable was sealed; in the common case it is not, and this is a no-op.
//
// `installed_log_number` is the column family's log number as of the Version
// most recently installed for it, from
// ReactiveVersionSet::GetInstalledVersionLogNumber(). It gates the seal, and
// the sealed memtable's next log number is at most that value.
//
// REQUIRES: mutex_ held
// REQUIRES: installed_log_number <= cfd->GetLogNumber()
// REQUIRES: cf_id_to_current_log_[cfd->GetID()] is at least as new as the
// newest WAL whose entries the active memtable holds, i.e. any WAL replay for
// this round has run. Too low a value drops a memtable the installed Version
// does not cover; too high only retains it for longer. RecoverLogFiles()
// records both before and after each insert to keep that true; see there.
bool MaybeSealFullyFlushedActiveMemtable(ColumnFamilyData* cfd,
uint64_t installed_log_number,
JobContext* job_context);
// Warns while `installed_log_number` lags behind cfd->GetLogNumber() and the
// column family is holding immutable memtables that RemoveOldMemTables()
// cannot collect as a result. Nothing on a secondary flushes them, so their
// number grows by one per WAL switch until the primary's flushed files become
// readable or the instance is reopened.
//
// Warns only when what it would report has changed, since the condition can
// persist across arbitrarily many catch-up calls.
//
// REQUIRES: mutex_ held
void MaybeWarnAboutRetainedMemtables(ColumnFamilyData* cfd,
uint64_t installed_log_number);
// Drops recovered transactions the primary resolved and flushed.
//
// REQUIRES: mutex_ held
// REQUIRES: every WAL reader for the round opened successfully
// REQUIRES: called before replaying records from the round
void DeleteResolvedRecoveredTransactions();
// Run compaction without installation, the output files will be placed in the
// secondary DB path. The LSM tree won't be changed, the secondary DB is still
// in read-only mode.
@@ -350,7 +484,8 @@ class DBImplSecondary : public DBImpl {
};
Status InitializeCompactionWorkspace(
bool allow_resumption, std::unique_ptr<FSDirectory>* output_dir,
bool allow_resumption, bool resumption_requested,
std::unique_ptr<FSDirectory>* output_dir,
std::unique_ptr<log::Writer>* compaction_progress_writer);
Status PrepareCompactionProgressState();
@@ -404,19 +539,49 @@ class DBImplSecondary : public DBImpl {
// MANIFEST only. When recover_wal is true, WAL files are also replayed
// (needed by DB::OpenAsSecondary). When false, WAL replay is skipped
// (used by DB::OpenAndCompact which only needs LSM state).
//
// When trust_manifest_recovery is true (only DB::OpenAndCompact), MANIFEST
// recovery trusts the manifest and does not stat/open SST or blob files, so
// only the compaction's (deletion-protected) input files are opened later, on
// demand. Must be false for normal secondary opens.
static Status OpenAsSecondaryImpl(
const DBOptions& db_options, const std::string& dbname,
const std::string& secondary_path,
const std::vector<ColumnFamilyDescriptor>& column_families,
std::vector<ColumnFamilyHandle*>* handles, std::unique_ptr<DB>* dbptr,
bool recover_wal);
bool recover_wal, bool trust_manifest_recovery);
// Cache log readers for each log number, used for continue WAL replay
// after recovery
std::map<uint64_t, std::unique_ptr<LogReaderContainer>> log_readers_;
// Current WAL number replayed for each column family.
std::unordered_map<ColumnFamilyData*, uint64_t> cfd_to_current_log_;
// The newest WAL whose entries each column family's active memtable may
// hold, keyed by column family id. An upper bound, not necessarily a WAL the
// memtable took entries from. Column family ids are never reused, so an entry
// left behind by a dropped column family can never be mistaken for a new
// column family's.
std::unordered_map<uint32_t, uint64_t> cf_id_to_current_log_;
// The installed Version's log number and retained memtable count last
// reported by MaybeWarnAboutRetainedMemtables() for each column family.
std::unordered_map<uint32_t, std::pair<uint64_t, int>>
cf_id_to_retention_warning_;
// Mutated only under mutex_, so that load-then-store stays atomic between
// publication and teardown. Readers load it without mutex_, so every access
// must still go through AtomicSharedPtrLoad/Store.
std::shared_ptr<const SecondaryReadView> secondary_read_view_;
// Set by ResetSecondaryReadView() so a later catch-up cannot publish a view
// that would outlive the drain close waits on.
// Guarded by mutex_, which every mutation of secondary_read_view_ holds.
bool secondary_read_view_closed_ = false;
std::mutex retired_secondary_read_views_mutex_;
std::condition_variable retired_secondary_read_views_cv_;
std::vector<std::shared_ptr<const SecondaryReadView>>
retired_secondary_read_views_;
size_t secondary_read_view_reclaims_in_flight_ = 0;
bool secondary_read_view_reclaimer_stopping_ = false;
std::unique_ptr<port::Thread> secondary_read_view_reclaimer_;
const std::string secondary_path_;
+49 -17
View File
@@ -19,6 +19,14 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplSecondary::GetImpl)
get_impl_options.merge_operands != nullptr);
assert(get_impl_options.column_family);
if (get_impl_options.newer_version_present != nullptr) {
if (read_options.snapshot != nullptr) {
CO_RETURN Status::NotSupported(
"GetWithMetadata is not supported in secondary DB mode");
}
get_impl_options.newer_version_present = nullptr;
}
Status s;
if (read_options.timestamp) {
@@ -48,7 +56,12 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplSecondary::GetImpl)
const Comparator* ucmp = get_impl_options.column_family->GetComparator();
assert(ucmp);
SequenceNumber snapshot = versions_->LastSequence();
// For a batched lazy read (MultiGetEntityLazy) use the sequence number the
// batch fixed (see GetImplOptions::lazy_columns_shared_sv); otherwise this
// instance's last sequence.
SequenceNumber snapshot = get_impl_options.lazy_columns_shared_sv != nullptr
? get_impl_options.lazy_columns_snapshot_seq
: versions_->LastSequence();
GetWithTimestampReadCallback read_cb(snapshot);
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(
get_impl_options.column_family);
@@ -74,21 +87,32 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplSecondary::GetImpl)
}
}
// Acquire SuperVersion
// Acquire SuperVersion, or use the batch's shared one for a batched lazy read
// (see GetImplOptions::lazy_columns_shared_sv), skipping the per-key
// reference and its release.
const bool own_super_version =
get_impl_options.lazy_columns_shared_sv == nullptr;
SuperVersion* super_version;
if (own_super_version) {
#if defined(WITH_COROUTINES)
SuperVersion* super_version = cfd->GetReferencedSuperVersion(this);
super_version = cfd->GetReferencedSuperVersion(this);
#else
SuperVersion* super_version = GetAndRefSuperVersion(cfd);
super_version = GetAndRefSuperVersion(cfd);
#endif
} else {
super_version = get_impl_options.lazy_columns_shared_sv;
}
if (read_options.timestamp && read_options.timestamp->size() > 0) {
s = FailIfReadCollapsedHistory(cfd, super_version,
*(read_options.timestamp));
if (!s.ok()) {
if (own_super_version) {
#if defined(WITH_COROUTINES)
CleanupSuperVersion(super_version);
CleanupSuperVersion(super_version);
#else
ReturnAndCleanupSuperVersion(cfd, super_version);
ReturnAndCleanupSuperVersion(cfd, super_version);
#endif
}
CO_RETURN s;
}
}
@@ -169,11 +193,13 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplSecondary::GetImpl)
}
if (!s.ok() && !s.IsMergeInProgress() && !s.IsNotFound()) {
assert(done);
if (own_super_version) {
#if defined(WITH_COROUTINES)
CleanupSuperVersion(super_version);
CleanupSuperVersion(super_version);
#else
ReturnAndCleanupSuperVersion(cfd, super_version);
ReturnAndCleanupSuperVersion(cfd, super_version);
#endif
}
CO_RETURN s;
}
if (!done) {
@@ -191,21 +217,27 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImplSecondary::GetImpl)
}
{
PERF_TIMER_GUARD(get_post_process_time);
if (get_impl_options.lazy_columns_version != nullptr && s.ok()) {
// Lazy result (GetEntityLazy / MultiGetEntityLazy): hand back the Version
// the entity's blob references must be resolved against (both the
// single-key and batched paths).
*get_impl_options.lazy_columns_version = super_version->current;
}
if (get_impl_options.lazy_columns_pin != nullptr && s.ok()) {
// Lazy result (GetEntityLazy): hand back the resolution Version and take
// a SuperVersion pin for the result before the borrowed reference below
// is released, so the result -- and deferred blob reads -- stay valid
// after this call (as an iterator's pin does).
if (get_impl_options.lazy_columns_version != nullptr) {
*get_impl_options.lazy_columns_version = super_version->current;
}
// Single-key GetEntityLazy: take a SuperVersion pin for the result before
// the borrowed reference below is released, so the result -- and deferred
// blob reads -- stay valid after this call (as an iterator's pin does).
// The batched path passes no pin (it uses lazy_columns_shared_sv); the
// batch holds one shared pin per column family instead.
TransferSuperVersionPin(super_version, get_impl_options.lazy_columns_pin);
}
if (own_super_version) {
#if defined(WITH_COROUTINES)
CleanupSuperVersion(super_version);
CleanupSuperVersion(super_version);
#else
ReturnAndCleanupSuperVersion(cfd, super_version);
ReturnAndCleanupSuperVersion(cfd, super_version);
#endif
}
RecordTick(stats_, NUMBER_KEYS_READ);
size_t size = 0;
// Mirror DBImpl::GetImpl: only produce merge-operand output and count bytes
+194 -34
View File
@@ -6,10 +6,6 @@
#include "util/coro_utils.h"
#if defined(USE_COROUTINES) && defined(WITH_COROUTINES)
#include "util/coro_stats_util.h"
#endif // USE_COROUTINES && WITH_COROUTINES
#if defined(WITHOUT_COROUTINES) || \
(defined(USE_COROUTINES) && defined(WITH_COROUTINES))
@@ -25,10 +21,6 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
DEFINE_SYNC_AND_ASYNC(Status, DBImpl::Get)
(const ReadOptions& _read_options, ColumnFamilyHandle* column_family,
const Slice& key, PinnableSlice* value, std::string* timestamp) {
#ifdef WITH_COROUTINES
INSTALL_COROUTINE_STATS_CONTEXT_SCOPE(
immutable_db_options_.fs->GetReadExecutor(), immutable_db_options_.env);
#endif
assert(value != nullptr);
value->Reset();
@@ -48,6 +40,106 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::Get)
timestamp);
}
DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetEntity)
(const ReadOptions& _read_options, ColumnFamilyHandle* column_family,
const Slice& key, PinnableWideColumns* columns) {
if (!column_family) {
CO_RETURN Status::InvalidArgument(
"Cannot call GetEntity without a column family handle");
}
if (!columns) {
CO_RETURN Status::InvalidArgument(
"Cannot call GetEntity without a PinnableWideColumns object");
}
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
_read_options.io_activity != Env::IOActivity::kGetEntity) {
CO_RETURN Status::InvalidArgument(
"Can only call GetEntity with `ReadOptions::io_activity` set to "
"`Env::IOActivity::kUnknown` or `Env::IOActivity::kGetEntity`");
}
ReadOptions read_options(_read_options);
if (read_options.io_activity == Env::IOActivity::kUnknown) {
read_options.io_activity = Env::IOActivity::kGetEntity;
}
columns->Reset();
GetImplOptions get_impl_options;
get_impl_options.column_family = column_family;
get_impl_options.columns = columns;
CO_RETURN CO_AWAIT(GetImpl, read_options, key, get_impl_options);
}
DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetEntity)
(const ReadOptions& _read_options, const Slice& key,
PinnableAttributeGroups* result) {
if (!result) {
CO_RETURN Status::InvalidArgument(
"Cannot call GetEntity without PinnableAttributeGroups object");
}
Status s;
const size_t num_column_families = result->size();
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
_read_options.io_activity != Env::IOActivity::kGetEntity) {
s = Status::InvalidArgument(
"Can only call GetEntity with `ReadOptions::io_activity` set to "
"`Env::IOActivity::kUnknown` or `Env::IOActivity::kGetEntity`");
for (size_t i = 0; i < num_column_families; ++i) {
(*result)[i].SetStatus(s);
}
CO_RETURN s;
}
// return early if no CF was passed in
if (num_column_families == 0) {
CO_RETURN s;
}
ReadOptions read_options(_read_options);
if (read_options.io_activity == Env::IOActivity::kUnknown) {
read_options.io_activity = Env::IOActivity::kGetEntity;
}
std::vector<Slice> keys;
std::vector<ColumnFamilyHandle*> column_families;
for (size_t i = 0; i < num_column_families; ++i) {
// If any of the CFH is null, break early since the entire query will fail
if (!(*result)[i].column_family()) {
s = Status::InvalidArgument(
"DB failed to query because one or more group(s) have null column "
"family handle");
(*result)[i].SetStatus(
Status::InvalidArgument("Column family handle cannot be null"));
break;
}
// Adding the same key slice for different CFs
keys.emplace_back(key);
column_families.emplace_back((*result)[i].column_family());
}
if (!s.ok()) {
for (size_t i = 0; i < num_column_families; ++i) {
if ((*result)[i].status().ok()) {
(*result)[i].SetStatus(
Status::Incomplete("DB not queried due to invalid argument(s) in "
"one or more of the attribute groups"));
}
}
CO_RETURN s;
}
std::vector<PinnableWideColumns> columns(num_column_families);
std::vector<Status> statuses(num_column_families);
CO_AWAIT(MultiGetCommon, read_options, num_column_families,
column_families.data(), keys.data(),
/* values */ nullptr, columns.data(),
/* timestamps */ nullptr, statuses.data(),
/* newer_version_present */ nullptr,
/* sorted_input */ false);
// Set results
for (size_t i = 0; i < num_column_families; ++i) {
(*result)[i].Reset();
(*result)[i].SetStatus(statuses[i]);
(*result)[i].SetColumns(std::move(columns[i]));
}
CO_RETURN s;
}
DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
(const ReadOptions& read_options, ColumnFamilyHandle* column_family,
const Slice& key, PinnableSlice* value, std::string* timestamp) {
@@ -69,6 +161,13 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
assert(get_impl_options.column_family);
if (get_impl_options.newer_version_present != nullptr &&
read_options.snapshot != nullptr &&
get_impl_options.callback != nullptr) {
CO_RETURN Status::NotSupported(
"Newer-version metadata is not supported with a read callback");
}
if (read_options.timestamp) {
const Status s = FailIfTsMismatchCf(get_impl_options.column_family,
*(read_options.timestamp));
@@ -118,13 +217,33 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
}
}
// Acquire SuperVersion
// Acquire SuperVersion (or, for a batched lazy read, use the shared one the
// batch already referenced -- see GetImplOptions::lazy_columns_shared_sv).
const bool own_super_version =
get_impl_options.lazy_columns_shared_sv == nullptr;
SuperVersion* sv;
#if defined(WITH_COROUTINES)
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
Defer sv_cleanup([&] { CleanupSuperVersion(sv); });
if (own_super_version) {
sv = cfd->GetReferencedSuperVersion(this);
} else {
sv = get_impl_options.lazy_columns_shared_sv;
}
Defer sv_cleanup([&] {
if (own_super_version) {
CleanupSuperVersion(sv);
}
});
#else
SuperVersion* sv = GetAndRefSuperVersion(cfd);
Defer sv_cleanup([&] { ReturnAndCleanupSuperVersion(cfd, sv); });
if (own_super_version) {
sv = GetAndRefSuperVersion(cfd);
} else {
sv = get_impl_options.lazy_columns_shared_sv;
}
Defer sv_cleanup([&] {
if (own_super_version) {
ReturnAndCleanupSuperVersion(cfd, sv);
}
});
#endif
if (read_options.timestamp && read_options.timestamp->size() > 0) {
const Status s =
@@ -139,7 +258,11 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
TEST_SYNC_POINT("DBImpl::GetImpl:2");
SequenceNumber snapshot;
if (read_options.snapshot != nullptr) {
if (!own_super_version) {
// Batched lazy read: use the batch's consistent sequence number (derived
// once, with the shared SuperVersion, by MultiCFSnapshot).
snapshot = get_impl_options.lazy_columns_snapshot_seq;
} else if (read_options.snapshot != nullptr) {
if (get_impl_options.callback) {
// Already calculated based on read_options.snapshot
snapshot = get_impl_options.callback->max_visible_seq();
@@ -173,6 +296,17 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
snapshot = get_impl_options.callback->max_visible_seq();
}
}
SequenceNumber lookup_snapshot = snapshot;
const bool newer_version_present_requested =
get_impl_options.newer_version_present != nullptr &&
read_options.snapshot != nullptr;
const SequenceNumber newer_version_upper_bound_seq =
newer_version_present_requested ? GetLastPublishedSequence() : snapshot;
const bool track_newer_versions = newer_version_present_requested &&
snapshot < newer_version_upper_bound_seq;
if (track_newer_versions) {
lookup_snapshot = newer_version_upper_bound_seq;
}
// If timestamp is used, we use read callback to ensure <key,t,s> is returned
// only if t <= read_opts.timestamp and s <= snapshot.
// HACK: temporarily overwrite input struct field but restore
@@ -184,6 +318,13 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
.callback); // timestamp with callback is not supported
read_cb.Refresh(snapshot);
get_impl_options.callback = &read_cb;
} else if (track_newer_versions && get_impl_options.callback == nullptr) {
read_cb.Refresh(snapshot);
get_impl_options.callback = &read_cb;
}
if (track_newer_versions) {
read_cb.EnableNewerVersionTracking(snapshot, newer_version_upper_bound_seq,
get_impl_options.newer_version_present);
}
TEST_SYNC_POINT("DBImpl::GetImpl:3");
TEST_SYNC_POINT("DBImpl::GetImpl:4");
@@ -198,7 +339,7 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
// First look in the memtable, then in the immutable memtable (if any).
// s is both in/out. When in, s could either be OK or MergeInProgress.
// merge_operands will contain the sequence of merges in the latter case.
LookupKey lkey(key, snapshot, read_options.timestamp);
LookupKey lkey(key, lookup_snapshot, read_options.timestamp);
PERF_TIMER_STOP(get_snapshot_time);
bool skip_memtable = (read_options.read_tier == kPersistedTier &&
@@ -403,17 +544,21 @@ DEFINE_SYNC_AND_ASYNC(Status, DBImpl::GetImpl)
PERF_COUNTER_ADD(get_read_bytes, size);
}
if (get_impl_options.lazy_columns_version != nullptr && s.ok()) {
// Lazy result (GetEntityLazy / MultiGetEntityLazy): the entity's blob
// references were left unresolved. Hand back the Version they must be
// resolved against (for both the single-key and batched paths).
*get_impl_options.lazy_columns_version = sv->current;
}
if (get_impl_options.lazy_columns_pin != nullptr && s.ok()) {
// Lazy result (GetEntityLazy): the entity's blob references were left
// unresolved. Hand back the Version they must be resolved against and
// transfer an extra SuperVersion reference into the result's pin, so the
// result -- and thus deferred blob reads -- stay valid after this call
// returns (as an iterator's SuperVersion pin does). The reference is
// released when the pin's cleanup runs (result destruction); the
// borrowed reference for this call is dropped by the sv_cleanup Defer.
if (get_impl_options.lazy_columns_version != nullptr) {
*get_impl_options.lazy_columns_version = sv->current;
}
// Single-key GetEntityLazy: transfer an extra SuperVersion reference into
// the result's pin, so the result -- and thus deferred blob reads -- stay
// valid after this call returns (as an iterator's SuperVersion pin does).
// The reference is released when the pin's cleanup runs (result
// destruction); the borrowed reference for this call is dropped by the
// sv_cleanup Defer. The batched path passes no pin here (it uses
// lazy_columns_shared_sv) because the batch holds one shared pin per
// column family instead.
TransferSuperVersionPin(sv, get_impl_options.lazy_columns_pin);
}
RecordInHistogram(stats_, BYTES_PER_READ, size);
@@ -635,7 +780,7 @@ DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGetCommon)
(const ReadOptions& read_options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys, PinnableSlice* values,
PinnableWideColumns* columns, std::string* timestamps, Status* statuses,
const bool sorted_input) {
std::vector<uint8_t>* newer_version_present, const bool sorted_input) {
if (num_keys == 0) {
CO_RETURN;
}
@@ -754,8 +899,26 @@ DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGetCommon)
GetWithTimestampReadCallback timestamp_read_callback(0);
ReadCallback* read_callback = nullptr;
if (read_options.timestamp && read_options.timestamp->size() > 0) {
SequenceNumber lookup_seqnum = consistent_seqnum;
const bool newer_version_present_requested =
newer_version_present != nullptr && read_options.snapshot != nullptr;
const SequenceNumber newer_version_upper_bound_seq =
newer_version_present_requested ? GetLastPublishedSequence()
: consistent_seqnum;
const bool track_newer_versions =
newer_version_present_requested &&
consistent_seqnum < newer_version_upper_bound_seq;
if (track_newer_versions) {
lookup_seqnum = newer_version_upper_bound_seq;
}
if ((read_options.timestamp && read_options.timestamp->size() > 0) ||
track_newer_versions) {
timestamp_read_callback.Refresh(consistent_seqnum);
if (track_newer_versions) {
timestamp_read_callback.EnableNewerVersionTracking(
consistent_seqnum, newer_version_upper_bound_seq,
/*single_key_result=*/nullptr);
}
read_callback = &timestamp_read_callback;
}
@@ -766,8 +929,7 @@ DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGetCommon)
cf_sv_pair_iter != cf_sv_pairs.end()) {
s = CO_AWAIT(MultiGetImpl, read_options, key_range_per_cf_iter->start,
key_range_per_cf_iter->num_keys, &sorted_keys,
cf_sv_pair_iter->super_version, consistent_seqnum,
read_callback);
cf_sv_pair_iter->super_version, lookup_seqnum, read_callback);
if (!s.ok()) {
break;
}
@@ -795,16 +957,13 @@ DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGetCommon)
CleanupSuperVersion(cf_sv_pair.super_version);
}
}
CopyNewerVersionPresent(key_context, newer_version_present);
}
DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGet)
(const ReadOptions& _read_options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys, PinnableSlice* values,
std::string* timestamps, Status* statuses, const bool sorted_input) {
#ifdef WITH_COROUTINES
INSTALL_COROUTINE_STATS_CONTEXT_SCOPE(
immutable_db_options_.fs->GetReadExecutor(), immutable_db_options_.env);
#endif
if (_read_options.io_activity != Env::IOActivity::kUnknown &&
_read_options.io_activity != Env::IOActivity::kMultiGet) {
Status s = Status::InvalidArgument(
@@ -823,7 +982,8 @@ DEFINE_SYNC_AND_ASYNC(void, DBImpl::MultiGet)
}
CO_AWAIT(MultiGetCommon, read_options, num_keys, column_families, keys,
values,
/* columns */ nullptr, timestamps, statuses, sorted_input);
/* columns */ nullptr, timestamps, statuses,
/* newer_version_present */ nullptr, sorted_input);
CO_RETURN;
}
+244 -63
View File
@@ -380,15 +380,26 @@ Status DBImpl::IngestWBWIAsMemtable(
// Switch memtable and add WBWIMemTables
Status s;
const bool defer_wbm_accounting_refresh =
immutable_db_options_.atomic_flush && cfds.size() > 1;
bool switch_attempted = false;
for (size_t i = 0; i < cfds.size(); ++i) {
WriteContext write_context;
switch_attempted = true;
// TODO: not switch on empty memtable, may need to update metadata
// like NextLogNumber(), earliest_seqno and memtable id.
if (i < memtables.size()) {
s = SwitchMemtable(cfds[i], &write_context, memtables[i],
last_seqno_after_ingest);
s = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfds[i], &write_context, memtables[i],
last_seqno_after_ingest,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfds[i], &write_context, memtables[i],
last_seqno_after_ingest);
} else {
s = SwitchMemtable(cfds[i], &write_context);
s = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfds[i], &write_context, nullptr, 0,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfds[i], &write_context);
}
if (!s.ok()) {
// SwitchMemtable() can only fail if a new WAL is to be created, this
@@ -420,6 +431,9 @@ Status DBImpl::IngestWBWIAsMemtable(
break;
}
}
if (defer_wbm_accounting_refresh && switch_attempted) {
UpdateWBMAccountingAfterMemtableSwitch(true /* refresh_now */);
}
for (size_t i = 0; i < cfds.size(); ++i) {
if (cfds[i]->UnrefAndTryDelete()) {
cfds[i] = nullptr;
@@ -1201,6 +1215,7 @@ Status DBImpl::WriteImpl(
IOStatus io_s;
Status pre_release_cb_status;
size_t seq_inc = 0;
uint64_t wal_write_number = 0;
if (status.ok()) {
if (deferred_put_entities != nullptr) {
my_batch->Clear();
@@ -1362,6 +1377,7 @@ Status DBImpl::WriteImpl(
if (!two_write_queues_) {
if (status.ok() && !write_options.disableWAL) {
assert(wal_context.wal_file_number_size);
wal_write_number = wal_context.wal_file_number_size->number;
wal_context.prev_size = wal_context.writer->file()->GetFileSize();
PERF_TIMER_GUARD(write_wal_time);
io_s = WriteGroupToWAL(write_group, wal_context.writer, wal_used,
@@ -1486,7 +1502,9 @@ Status DBImpl::WriteImpl(
if (!io_s.ok()) {
// Check WriteToWAL status
WALIOStatusCheck(io_s);
TEST_SYNC_POINT_CALLBACK("DBImpl::WriteImpl:BeforeWALIOStatusCheck", &io_s);
WALIOStatusCheck(io_s, wal_write_number, last_sequence,
last_sequence + 1 - seq_inc);
}
if (!w.CallbackFailed()) {
if (!io_s.ok()) {
@@ -2069,7 +2087,10 @@ void DBImpl::WriteStatusCheck(const Status& status) {
}
}
void DBImpl::WALIOStatusCheck(const IOStatus& io_status) {
void DBImpl::WALIOStatusCheck(const IOStatus& io_status,
uint64_t failed_wal_number,
SequenceNumber failed_wal_sequence,
SequenceNumber failed_wal_sequence_start) {
// Is setting bg_error_ enough here? This will at least stop
// compaction and fail any further writes.
if ((immutable_db_options_.paranoid_checks && !io_status.ok() &&
@@ -2077,8 +2098,33 @@ void DBImpl::WALIOStatusCheck(const IOStatus& io_status) {
io_status.IsIOFenced()) {
mutex_.Lock();
// Maybe change the return status to void?
error_handler_.SetBGError(io_status, BackgroundErrorReason::kWriteCallback,
/*wal_related=*/true);
if (io_status.IsIOError() && !io_status.IsIOFenced()) {
DBRecoverContext context;
context.failed_wal_sequence = failed_wal_sequence;
context.failed_wal_sequence_start = failed_wal_sequence_start;
const uint64_t resolved_failed_wal_number =
failed_wal_number != 0 ? failed_wal_number : cur_wal_number_;
if (io_status.GetScope() == IOStatus::IOErrorScope::kIOErrorScopeFile) {
context.failed_wal_number = resolved_failed_wal_number;
if (failed_wal_sequence_start != 0 &&
failed_wal_sequence_start <= failed_wal_sequence) {
// Successful live resumption deliberately omits this write from the
// database even if the physical WAL record is complete. Keep a
// DB-lifetime fence so TransactionLogIterator does not expose it.
wal_manager_.RecordIndeterminateWriteSequence(
resolved_failed_wal_number, failed_wal_sequence_start,
failed_wal_sequence);
}
ExtendWALRecoveryCutoff(context.failed_wal_number);
}
error_handler_.SetBGError(io_status,
BackgroundErrorReason::kWriteCallback,
/*wal_related=*/true, context);
} else {
error_handler_.SetBGError(io_status,
BackgroundErrorReason::kWriteCallback,
/*wal_related=*/true);
}
mutex_.Unlock();
} else {
// Force writable file to be continue writable.
@@ -2086,6 +2132,16 @@ void DBImpl::WALIOStatusCheck(const IOStatus& io_status) {
}
}
void DBImpl::ExtendWALRecoveryCutoff(uint64_t failed_wal_number) {
assert(failed_wal_number != 0);
uint64_t current = wal_recovery_cutoff_.load(std::memory_order_relaxed);
while (current < failed_wal_number &&
!wal_recovery_cutoff_.compare_exchange_weak(
current, failed_wal_number, std::memory_order_release,
std::memory_order_relaxed)) {
}
}
void DBImpl::HandleMemTableInsertFailure(const Status& status) {
assert(!status.ok());
// A non-OK status on memtable insert indicates that the state implied by the
@@ -2127,14 +2183,11 @@ Status DBImpl::PreprocessWrite(const WriteOptions& write_options,
}
if (UNLIKELY(status.ok() && write_buffer_manager_->ShouldFlush())) {
// Before a new memtable is added in SwitchMemtable(),
// write_buffer_manager_->ShouldFlush() will keep returning true. If another
// thread is writing to another DB with the same write buffer, they may also
// be flushed. We may end up with flushing much more DBs than needed. It's
// suboptimal but still correct.
InstrumentedMutexLock l(&mutex_);
WaitForPendingWrites();
status = HandleWriteBufferManagerFlush(write_context);
if (write_buffer_manager_->TryAcquireLocalFlush()) {
InstrumentedMutexLock l(&mutex_);
WaitForPendingWrites();
status = HandleWriteBufferManagerFlush(write_context);
}
}
if (UNLIKELY(status.ok() && !trim_history_scheduler_.Empty())) {
@@ -2178,7 +2231,7 @@ Status DBImpl::PreprocessWrite(const WriteOptions& write_options,
status = Status::Incomplete("Write stall");
} else {
InstrumentedMutexLock l(&mutex_);
WriteBufferManagerStallWrites();
status = WriteBufferManagerStallWrites(write_context);
}
}
InstrumentedMutexLock l(&wal_write_mutex_);
@@ -2287,10 +2340,13 @@ IOStatus DBImpl::WriteToWAL(const WriteBatch& merged_batch,
}
IOStatus io_s = log_writer->MaybeAddUserDefinedTimestampSizeRecord(
write_options, versions_->GetColumnFamiliesTimestampSizeForRecord());
TEST_SYNC_POINT_CALLBACK(
"DBImpl::WriteToWAL:AfterMaybeAddUserDefinedTimestampSizeRecord", &io_s);
if (!io_s.ok()) {
return io_s;
}
io_s = log_writer->AddRecord(write_options, log_entry, sequence);
TEST_SYNC_POINT_CALLBACK("DBImpl::WriteToWAL:AfterAddRecord", &io_s);
if (UNLIKELY(needs_locking)) {
wal_write_mutex_.Unlock();
@@ -2379,7 +2435,8 @@ IOStatus DBImpl::WriteGroupToWAL(const WriteThread::WriteGroup& write_group,
// and closed WAL that just needs to be recorded as synced in the
// manifest.
if (auto* f = log.writer->file()) {
io_s = f->Sync(opts, immutable_db_options_.use_fsync);
io_s =
immutable_db_options_.use_fsync ? f->Fsync(opts) : f->Sync(opts);
if (!io_s.ok()) {
break;
}
@@ -2671,14 +2728,24 @@ Status DBImpl::SwitchWAL(WriteContext* write_context) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
const bool defer_wbm_accounting_refresh =
immutable_db_options_.atomic_flush && cfds.size() > 1;
bool switch_attempted = false;
for (const auto cfd : cfds) {
cfd->Ref();
status = SwitchMemtable(cfd, write_context);
switch_attempted = true;
status = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfd, write_context, nullptr, 0,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfd, write_context);
cfd->UnrefAndTryDelete();
if (!status.ok()) {
break;
}
}
if (defer_wbm_accounting_refresh && switch_attempted) {
UpdateWBMAccountingAfterMemtableSwitch(true /* refresh_now */);
}
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
@@ -2722,25 +2789,17 @@ Status DBImpl::HandleWriteBufferManagerFlush(WriteContext* write_context) {
if (immutable_db_options_.atomic_flush) {
SelectColumnFamiliesForAtomicFlush(&cfds);
} else {
ColumnFamilyData* cfd_picked = nullptr;
SequenceNumber seq_num_for_cf_picked = kMaxSequenceNumber;
for (auto cfd : *versions_->GetColumnFamilySet()) {
if (cfd->IsDropped()) {
continue;
}
if (!cfd->mem()->IsEmpty() && !cfd->imm()->IsFlushPendingOrRunning()) {
// We only consider flush on CFs with bytes in the mutable memtable,
// and no immutable memtables for which flush has yet to finish. If
// we triggered flush on CFs already trying to flush, we would risk
// creating too many immutable memtables leading to write stalls.
uint64_t seq = cfd->mem()->GetCreationSeq();
if (cfd_picked == nullptr || seq < seq_num_for_cf_picked) {
cfd_picked = cfd;
seq_num_for_cf_picked = seq;
}
}
}
const WriteBufferFlushPolicy policy = write_buffer_manager_->flush_policy();
// Cross-DB selection still flushes the largest CF within the chosen DB.
const bool flush_largest =
policy == WriteBufferFlushPolicy::kFlushLargest ||
policy == WriteBufferFlushPolicy::kFlushLargestAcrossDBs;
const bool include_waiting_immutable =
policy == WriteBufferFlushPolicy::kFlushLargestAcrossDBs;
const FlushableCFs flushable =
CollectFlushableCFs(include_waiting_immutable);
ColumnFamilyData* cfd_picked =
flush_largest ? flushable.largest : flushable.oldest;
if (cfd_picked != nullptr) {
cfds.push_back(cfd_picked);
}
@@ -2760,12 +2819,17 @@ Status DBImpl::HandleWriteBufferManagerFlush(WriteContext* write_context) {
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
const bool defer_wbm_accounting_refresh =
immutable_db_options_.atomic_flush && cfds.size() > 1;
for (const auto cfd : cfds) {
if (cfd->mem()->IsEmpty()) {
continue;
}
cfd->Ref();
status = SwitchMemtable(cfd, write_context);
status = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfd, write_context, nullptr, 0,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfd, write_context);
cfd->UnrefAndTryDelete();
if (!status.ok()) {
break;
@@ -2794,6 +2858,11 @@ Status DBImpl::HandleWriteBufferManagerFlush(WriteContext* write_context) {
GenerateFlushRequest(cfds, FlushReason::kWriteBufferManager, &flush_req);
EnqueuePendingFlush(flush_req);
}
}
if (write_buffer_manager_->ShouldTrackFlushInitiator()) {
RefreshFlushableMemAccounting();
}
if (status.ok()) {
MaybeScheduleFlushOrCompaction();
}
return status;
@@ -2914,25 +2983,63 @@ Status DBImpl::DelayWrite(uint64_t num_bytes, WriteThread& write_thread,
// REQUIRES: mutex_ is held
// REQUIRES: this thread is currently at the front of the writer queue
void DBImpl::WriteBufferManagerStallWrites() {
Status DBImpl::WriteBufferManagerStallWrites(WriteContext* write_context) {
mutex_.AssertHeld();
// First block future writer threads who want to add themselves to the queue
// of WriteThread.
write_thread_.BeginWriteStall();
mutex_.Unlock();
for (;;) {
// First block future writer threads who want to add themselves to the
// queue of WriteThread.
write_thread_.BeginWriteStall();
const bool hid_flush_initiator =
wbm_flush_initiator_ != nullptr &&
write_buffer_manager_->ShouldTrackFlushInitiator();
if (hid_flush_initiator) {
wbm_flush_initiator_->SetFlushable(false);
}
mutex_.Unlock();
// Change the state to State::Blocked.
static_cast<WBMStallInterface*>(wbm_stall_.get())
->SetState(WBMStallInterface::State::BLOCKED);
// Then WriteBufferManager will add DB instance to its queue
// and block this thread by calling WBMStallInterface::Block().
write_buffer_manager_->BeginWriteStall(wbm_stall_.get());
wbm_stall_->Block();
WBMStallInterface* const stall =
static_cast<WBMStallInterface*>(wbm_stall_.get());
stall->SetState(WBMStallInterface::State::BLOCKED);
write_buffer_manager_->BeginWriteStall(wbm_stall_.get());
bool retry_local_flush = false;
if (write_buffer_manager_->ShouldTrackFlushInitiator()) {
for (;;) {
if (stall->BlockFor(
write_buffer_manager_->GetLocalFlushRetryMicros())) {
break;
}
if (write_buffer_manager_->TryAcquireLocalFlush(wbm_stall_.get())) {
write_buffer_manager_->RemoveDBFromQueue(wbm_stall_.get());
retry_local_flush = true;
break;
}
}
} else {
stall->Block();
}
mutex_.Lock();
// Stall has ended. Signal writer threads so that they can add
// themselves to the WriteThread queue for writes.
write_thread_.EndWriteStall();
mutex_.Lock();
// Allow this front writer to retry the manager-wide emergency lease. It
// must not let its write through while the hard limit is still exceeded.
write_thread_.EndWriteStall();
// Restore the state hidden by this stall even if the runtime policy
// changed while the writer was blocked. Otherwise switching back to the
// across-DB policy can leave this DB permanently ineligible.
if (hid_flush_initiator) {
wbm_flush_initiator_->SetFlushable(true);
if (write_buffer_manager_->ShouldTrackFlushInitiator()) {
write_buffer_manager_->NotifyFlushInitiatorChanged();
}
}
if (!retry_local_flush || !write_buffer_manager_->ShouldStall()) {
return Status::OK();
}
WaitForPendingWrites();
Status status = HandleWriteBufferManagerFlush(write_context);
if (!status.ok()) {
return status;
}
}
}
Status DBImpl::ThrottleLowPriWritesIfNeeded(const WriteOptions& write_options,
@@ -3063,17 +3170,27 @@ Status DBImpl::ScheduleFlushes(WriteContext* context) {
nullptr);
autovector<FlushReason> flush_reasons;
flush_reasons.reserve(cfds.size());
const bool defer_wbm_accounting_refresh =
immutable_db_options_.atomic_flush && cfds.size() > 1;
bool switch_attempted = false;
for (auto& cfd : cfds) {
FlushReason flush_reason = FlushReason::kWriteBufferFull;
if (status.ok() && !cfd->mem()->IsEmpty()) {
flush_reason = cfd->mem()->GetFlushReason();
status = SwitchMemtable(cfd, context);
switch_attempted = true;
status = defer_wbm_accounting_refresh
? SwitchMemtableImpl(cfd, context, nullptr, 0,
false /* refresh_wbm_accounting */)
: SwitchMemtable(cfd, context);
}
flush_reasons.push_back(flush_reason);
if (cfd->UnrefAndTryDelete()) {
cfd = nullptr;
}
}
if (defer_wbm_accounting_refresh && switch_attempted) {
UpdateWBMAccountingAfterMemtableSwitch(true /* refresh_now */);
}
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
@@ -3118,6 +3235,31 @@ void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
ReadOnlyMemTable* new_imm,
SequenceNumber last_seqno) {
return SwitchMemtableImpl(cfd, context, new_imm, last_seqno,
true /* refresh_wbm_accounting */);
}
void DBImpl::UpdateWBMAccountingAfterMemtableSwitch(bool refresh_now) {
mutex_.AssertHeld();
if (wbm_flush_initiator_ == nullptr ||
!write_buffer_manager_->ShouldTrackFlushInitiator()) {
return;
}
if (refresh_now) {
TEST_SYNC_POINT("DBImpl::UpdateWBMAccountingAfterMemtableSwitch:Refresh");
RefreshFlushableMemAccounting();
} else {
// Do not let the sorter consume an intermediate snapshot while a caller is
// switching a batch of memtables. The caller refreshes once after the
// complete batch, including a partial-error exit.
wbm_flush_initiator_->InvalidateLargestFlushableCFMem();
}
}
Status DBImpl::SwitchMemtableImpl(ColumnFamilyData* cfd, WriteContext* context,
ReadOnlyMemTable* new_imm,
SequenceNumber last_seqno,
bool refresh_wbm_accounting) {
mutex_.AssertHeld();
assert(lock_wal_owner_thread_id_counts_.empty());
@@ -3128,6 +3270,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
UnpublishedWAL prepared_wal;
MemTable* new_mem = nullptr;
IOStatus io_s;
bool current_wal_write_buffer_failed = false;
// Recoverable state is persisted in WAL. After memtable switch, WAL might
// be deleted, so we write the state to memtable to be persisted as well.
@@ -3142,7 +3285,9 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
if (two_write_queues_) {
wal_write_mutex_.Lock();
}
bool creating_new_log = !wal_empty_;
bool creating_new_log =
!wal_empty_ || (recovering_from_wal_write_error_ &&
cur_wal_number_ <= recover_wal_through_number_);
if (two_write_queues_) {
wal_write_mutex_.Unlock();
}
@@ -3273,13 +3418,28 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
InstrumentedMutexLock l(&wal_write_mutex_);
assert(new_log != nullptr);
if (!logs_.empty()) {
// Alway flush the buffer of the last log before switching to a new one
// Always flush the buffer of the last log before switching to a new one,
// except when recovery is replacing a file-scoped failed WAL. Its
// accepted contents are persisted by the atomic recovery flush.
log::Writer* cur_log_writer = logs_.back().writer;
if (error_handler_.IsRecoveryInProgress()) {
const bool abandon_current_wal =
recovering_from_wal_write_error_ &&
cur_log_writer->get_log_number() <= recover_wal_through_number_;
if (abandon_current_wal) {
TEST_SYNC_POINT("DBImpl::SwitchMemtable:AbandonWAL");
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"[WAL recovery] [%s] Replacing failed WAL #%" PRIu64
" with WAL #%" PRIu64,
cfd->GetName().c_str(), cur_log_writer->get_log_number(),
new_log_number);
} else if (error_handler_.IsRecoveryInProgress()) {
// In recovery path, we force another try of writing WAL buffer.
cur_log_writer->file()->reset_seen_error();
}
io_s = cur_log_writer->WriteBuffer(write_options);
if (!abandon_current_wal) {
io_s = cur_log_writer->WriteBuffer(write_options);
current_wal_write_buffer_failed = !io_s.ok();
}
if (s.ok()) {
s = io_s;
}
@@ -3306,16 +3466,35 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
delete new_mem;
delete new_log;
context->superversion_context.new_superversion.reset();
// We may have lost data from the WritableFileBuffer in-memory buffer for
// the current log, so treat it as a fatal error and set bg_error
// Only a WriteBuffer() failure belongs to the current WAL. CreateWAL() and
// StartWALFile() operate on an unpublished replacement, so a failure there
// must not quarantine an otherwise healthy current WAL.
if (!io_s.ok()) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable,
/*wal_related=*/true);
if (!current_wal_write_buffer_failed) {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"Failed to create or start replacement WAL #%" PRIu64
"; current WAL #%" PRIu64 " remains active: %s",
new_log_number, cur_wal_number_,
io_s.ToString().c_str());
}
if (current_wal_write_buffer_failed && io_s.IsIOError() &&
!io_s.IsIOFenced() &&
io_s.GetScope() == IOStatus::IOErrorScope::kIOErrorScopeFile) {
DBRecoverContext recovery_context;
recovery_context.failed_wal_number = cur_wal_number_;
ExtendWALRecoveryCutoff(recovery_context.failed_wal_number);
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable,
/*wal_related=*/true, recovery_context);
} else {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable,
/*wal_related=*/true);
}
} else {
error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
}
// Read back bg_error in order to get the right severity
s = error_handler_.GetBGError();
UpdateWBMAccountingAfterMemtableSwitch(refresh_wbm_accounting);
return s;
}
@@ -3357,6 +3536,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
BackgroundErrorReason::kManifestWrite);
}
if (!s.ok()) {
UpdateWBMAccountingAfterMemtableSwitch(refresh_wbm_accounting);
return s;
}
@@ -3409,6 +3589,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
}
new_mem->Ref();
cfd->SetMemtable(new_mem);
UpdateWBMAccountingAfterMemtableSwitch(refresh_wbm_accounting);
InstallSuperVersionAndScheduleWork(cfd, &context->superversion_context);
MaybeScheduleAsyncWALPrecreate(preallocate_block_size);
+103
View File
@@ -4308,6 +4308,109 @@ TEST_F(DBIteratorTest, IteratorsConsistentViewExplicitSnapshot) {
}
}
TEST_F(DBIteratorTest, NewIteratorsWithPerColumnFamilyReadOptions) {
Options options = GetDefaultOptions();
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
BlockBasedTableOptions table_options;
table_options.filter_policy.reset(NewBloomFilterPolicy(10));
table_options.whole_key_filtering = false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
CreateAndReopenWithCF({"cf_1"}, options);
ASSERT_OK(Put(0, "b1", "default_value"));
ASSERT_OK(Put(1, "b1", "cf_1_value"));
ASSERT_OK(Flush(0));
ASSERT_OK(Flush(1));
ReadOptions prefix_read_options;
prefix_read_options.total_order_seek = false;
ReadOptions total_order_read_options;
total_order_read_options.total_order_seek = true;
std::vector<ReadOptions> read_options = {prefix_read_options,
total_order_read_options};
std::vector<Iterator*> iterators;
ASSERT_OK(db_->NewIterators(read_options, handles_, &iterators));
ASSERT_EQ(2U, iterators.size());
iterators[0]->Seek("a9");
ASSERT_FALSE(iterators[0]->Valid());
ASSERT_OK(iterators[0]->status());
iterators[1]->Seek("a9");
ASSERT_TRUE(iterators[1]->Valid());
ASSERT_EQ("b1", iterators[1]->key());
ASSERT_OK(iterators[1]->status());
for (auto* iterator : iterators) {
delete iterator;
}
}
TEST_F(DBIteratorTest, NewIteratorsPerColumnFamilyOptionsValidation) {
Options options = GetDefaultOptions();
CreateAndReopenWithCF({"cf_1"}, options);
std::vector<Iterator*> iterators;
ASSERT_TRUE(
db_->NewIterators(std::vector<ReadOptions>(1), handles_, &iterators)
.IsInvalidArgument());
std::vector<ReadOptions> read_options(2);
read_options[1].tailing = true;
ASSERT_TRUE(db_->NewIterators(read_options, handles_, &iterators)
.IsInvalidArgument());
read_options[1].tailing = false;
const Snapshot* first_snapshot = db_->GetSnapshot();
ASSERT_OK(Put(0, "key", "value"));
const Snapshot* second_snapshot = db_->GetSnapshot();
read_options[0].snapshot = second_snapshot;
read_options[1].snapshot = second_snapshot;
ASSERT_OK(db_->NewIterators(read_options, handles_, &iterators));
for (auto* iterator : iterators) {
delete iterator;
}
read_options[0].snapshot = first_snapshot;
read_options[1].snapshot = second_snapshot;
ASSERT_TRUE(db_->NewIterators(read_options, handles_, &iterators)
.IsInvalidArgument());
db_->ReleaseSnapshot(first_snapshot);
db_->ReleaseSnapshot(second_snapshot);
}
TEST_F(DBIteratorTest,
ReadOnlyNewIteratorsAllowsTableFilterWithRangeConversion) {
Options options = CurrentOptions();
options.min_tombstones_for_range_conversion = 2;
DestroyAndReopen(options);
ASSERT_OK(Put("key", "value"));
ASSERT_OK(Flush());
ASSERT_OK(ReadOnlyReopen(options));
std::function<bool(const TableProperties&)> table_filter =
[](const TableProperties&) { return true; };
ReadOptions read_options;
read_options.table_filter = &table_filter;
const std::vector<ReadOptions> per_cf_read_options = {read_options};
const std::vector<ColumnFamilyHandle*> column_families = {
db_->DefaultColumnFamily()};
for (bool use_per_cf_read_options : {false, true}) {
std::vector<Iterator*> iterators;
const Status s =
use_per_cf_read_options
? db_->NewIterators(per_cf_read_options, column_families,
&iterators)
: db_->NewIterators(read_options, column_families, &iterators);
ASSERT_OK(s);
ASSERT_EQ(1U, iterators.size());
iterators[0]->SeekToFirst();
ASSERT_OK(iterators[0]->status());
ASSERT_TRUE(iterators[0]->Valid());
ASSERT_EQ("key", iterators[0]->key());
delete iterators[0];
}
}
TEST_P(DBIteratorTest, MemtableOpsScanFlushTriggerWithSeek) {
// Tests that option memtable_op_scan_flush_trigger works when the limit
// is reached during a Seek() operation.
+8 -4
View File
@@ -140,7 +140,8 @@ TEST_F(DBMemTableTest, DuplicateSeq) {
ImmutableOptions ioptions(options);
WriteBufferManager wb(options.db_write_buffer_size);
MemTable* mem = new MemTable(cmp, ioptions, MutableCFOptions(options), &wb,
kMaxSequenceNumber, 0 /* column_family_id */);
kMaxSequenceNumber, 0 /* column_family_id */,
nullptr /* flush_initiator */);
// Write some keys and make sure it returns false on duplicates
ASSERT_OK(
@@ -188,7 +189,8 @@ TEST_F(DBMemTableTest, DuplicateSeq) {
new TestPrefixExtractor()); // which uses _ to extract the prefix
ioptions = ImmutableOptions(options);
mem = new MemTable(cmp, ioptions, MutableCFOptions(options), &wb,
kMaxSequenceNumber, 0 /* column_family_id */);
kMaxSequenceNumber, 0 /* column_family_id */,
nullptr /* flush_initiator */);
// Insert a duplicate key with _ in it
ASSERT_OK(
mem->Add(seq, kTypeValue, "key_1", "value", nullptr /* kv_prot_info */));
@@ -201,7 +203,8 @@ TEST_F(DBMemTableTest, DuplicateSeq) {
options.allow_concurrent_memtable_write = true;
ioptions = ImmutableOptions(options);
mem = new MemTable(cmp, ioptions, MutableCFOptions(options), &wb,
kMaxSequenceNumber, 0 /* column_family_id */);
kMaxSequenceNumber, 0 /* column_family_id */,
nullptr /* flush_initiator */);
MemTablePostProcessInfo post_process_info;
ASSERT_OK(mem->Add(seq, kTypeValue, "key", "value",
nullptr /* kv_prot_info */, true, &post_process_info));
@@ -234,7 +237,8 @@ TEST_F(DBMemTableTest, ConcurrentMergeWrite) {
ImmutableOptions ioptions(options);
WriteBufferManager wb(options.db_write_buffer_size);
MemTable* mem = new MemTable(cmp, ioptions, MutableCFOptions(options), &wb,
kMaxSequenceNumber, 0 /* column_family_id */);
kMaxSequenceNumber, 0 /* column_family_id */,
nullptr /* flush_initiator */);
// Put 0 as the base
PutFixed64(&value, static_cast<uint64_t>(0));
+1707 -12
View File
File diff suppressed because it is too large. Load diff
+81 -5
View File
@@ -1183,10 +1183,43 @@ TEST_F(DBSSTTest, DeleteSchedulerMultipleDBPaths) {
}
TEST_F(DBSSTTest, DestroyDBWithRateLimitedDelete) {
Destroy(last_options_);
int bg_delete_file = 0;
int preexisting_trash_files_scheduled = 0;
uint64_t preexisting_trash_penalty = 0;
std::string file_being_deleted;
std::string preexisting_trash_file;
std::vector<uint64_t> penalties;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteTrashFile:DeleteFile",
[&](void* /*arg*/) { bg_delete_file++; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteTrashFile::cb",
[&](void* arg) { file_being_deleted = *static_cast<std::string*>(arg); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::BackgroundEmptyTrash:Wait", [&](void* arg) {
uint64_t penalty = *(static_cast<uint64_t*>(arg));
penalties.push_back(penalty);
if (file_being_deleted == preexisting_trash_file) {
preexisting_trash_penalty = penalty;
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SstFileManagerImpl::ScheduleExistingTrashFileDeletion", [&](void* arg) {
const std::string& file_path = *static_cast<std::string*>(arg);
if (EndsWith(file_path, ".trash")) {
preexisting_trash_files_scheduled++;
}
});
std::atomic<bool> fail_preexisting_trash_delete{true};
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteTrashFile:BeforeDeleteFile", [&](void* arg) {
if (file_being_deleted == preexisting_trash_file &&
fail_preexisting_trash_delete.exchange(false)) {
*static_cast<Status*>(arg) = Status::IOError("injected delete error");
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Status s;
@@ -1196,7 +1229,7 @@ TEST_F(DBSSTTest, DestroyDBWithRateLimitedDelete) {
options.sst_file_manager.reset(
NewSstFileManager(env_, nullptr, "", 0, false, &s, 0));
ASSERT_OK(s);
DestroyAndReopen(options);
Reopen(options);
// Create 4 files in L0
for (int i = 0; i < 4; i++) {
@@ -1211,27 +1244,70 @@ TEST_F(DBSSTTest, DestroyDBWithRateLimitedDelete) {
int num_sst_files = 0;
int num_wal_files = 0;
std::string sst_file_to_mark_trash;
std::vector<std::string> db_files;
ASSERT_OK(env_->GetChildren(dbname_, &db_files));
for (const std::string& f : db_files) {
if (f.substr(f.find_last_of('.') + 1) == "sst") {
num_sst_files++;
if (sst_file_to_mark_trash.empty()) {
sst_file_to_mark_trash = dbname_;
sst_file_to_mark_trash.append("/").append(f);
}
} else if (f.substr(f.find_last_of('.') + 1) == "log") {
num_wal_files++;
} else {
continue;
}
}
ASSERT_GT(num_sst_files, 0);
ASSERT_GT(num_wal_files, 0);
ASSERT_FALSE(sst_file_to_mark_trash.empty());
uint64_t preexisting_trash_file_size = 0;
ASSERT_OK(
env_->GetFileSize(sst_file_to_mark_trash, &preexisting_trash_file_size));
preexisting_trash_file = sst_file_to_mark_trash;
preexisting_trash_file.append(".trash");
auto sfm = static_cast<SstFileManagerImpl*>(options.sst_file_manager.get());
sfm->SetDeleteRateBytesPerSecond(1024 * 1024);
const int64_t rate_bytes_per_sec = 1024 * 1024;
sfm->SetDeleteRateBytesPerSecond(rate_bytes_per_sec);
// Set an extra high trash ratio to prevent immediate/non-rate limited
// deletions
sfm->delete_scheduler()->SetMaxTrashDBRatio(1000.0);
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(
sfm->OnAddFile(sst_file_to_mark_trash, preexisting_trash_file_size));
ASSERT_OK(sfm->ScheduleFileDeletion(sst_file_to_mark_trash, dbname_,
/*force_bg=*/true));
sfm->WaitForEmptyTrashInDirectory(dbname_);
ASSERT_FALSE(fail_preexisting_trash_delete);
ASSERT_OK(env_->FileExists(preexisting_trash_file));
ASSERT_EQ(sfm->GetTotalSize(), preexisting_trash_file_size);
ASSERT_EQ(sfm->GetTotalTrashSize(), preexisting_trash_file_size);
const std::string unaccounted_trash_file = dbname_ + "/unaccounted.sst.trash";
ASSERT_OK(WriteStringToFile(env_, "trash", unaccounted_trash_file,
/*should_sync=*/false));
std::string dbname_alias = dbname_;
dbname_alias.insert(dbname_alias.find_last_of('/'), "/");
ASSERT_OK(DestroyDB(dbname_alias, options));
sfm->WaitForEmptyTrash();
ASSERT_EQ(bg_delete_file, num_sst_files + num_wal_files);
ASSERT_EQ(bg_delete_file, num_sst_files + num_wal_files + 2);
ASSERT_EQ(preexisting_trash_files_scheduled, 2);
ASSERT_GE(preexisting_trash_penalty,
preexisting_trash_file_size * 1000000 / rate_bytes_per_sec);
ASSERT_EQ(penalties.size(), num_sst_files + num_wal_files + 2);
ASSERT_TRUE(sfm->GetTrackedFiles().empty());
ASSERT_EQ(sfm->GetTotalSize(), 0);
ASSERT_EQ(sfm->GetTotalTrashSize(), 0);
auto bg_errors = sfm->delete_scheduler()->GetBackgroundErrors();
ASSERT_EQ(bg_errors.size(), 1);
for (auto& error : bg_errors) {
ASSERT_NOK(error.second);
}
ASSERT_TRUE(env_->FileExists(dbname_).IsNotFound());
}
TEST_F(DBSSTTest, DBWithMaxSpaceAllowed) {
@@ -1672,7 +1748,7 @@ TEST_F(DBSSTTest, GetTotalSstFilesSize) {
// it. Should remove the workaround after we propagate the property on
// compaction.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FlushJob::WriteLevel0Table:oldest_ancester_time", [&](void* arg) {
"FlushJob::WriteLevel0Table:oldest_ancestor_time", [&](void* arg) {
uint64_t* current_time = static_cast<uint64_t*>(arg);
*current_time = 0;
});
+169
View File
@@ -17,6 +17,8 @@
#include "port/stack_trace.h"
#include "rocksdb/advanced_compression.h"
#include "rocksdb/db.h"
#include "rocksdb/sst_file_reader.h"
#include "rocksdb/sst_file_writer.h"
#include "rocksdb/types.h"
#include "rocksdb/utilities/object_registry.h"
#include "rocksdb/utilities/table_properties_collectors.h"
@@ -851,6 +853,173 @@ TEST_F(DBTablePropertiesTest, KeyLargestSmallestSeqno) {
}
}
TEST_F(DBTablePropertiesTest, LsmInfoAtCreationPacking) {
using Applicability = LsmInfoAtCreation::Applicability;
auto round_trip = [](const LsmInfoAtCreation& info) {
return LsmInfoAtCreation::DecodeFrom(info.Encode());
};
// Default is "unknown", which encodes to 0 and is distinct from every other
// state.
LsmInfoAtCreation unknown;
EXPECT_EQ(unknown.applicability, Applicability::kUnknown);
EXPECT_EQ(unknown.Encode(), 0U);
EXPECT_EQ(round_trip(unknown), unknown);
// "not applicable" is non-zero (so it persists) and distinct from "unknown".
LsmInfoAtCreation na = LsmInfoAtCreation::NotApplicable();
EXPECT_EQ(na.applicability, Applicability::kNotApplicable);
EXPECT_NE(na.Encode(), 0U);
EXPECT_NE(na.Encode(), unknown.Encode());
EXPECT_EQ(round_trip(na), na);
// "applicable" round-trips natural level values and bottommost. Level 0 / not
// bottommost is still non-zero and thus distinct from "unknown"/"N/A".
for (int level : {0, 1, 5, 63, 200}) {
for (bool bottommost : {false, true}) {
LsmInfoAtCreation info = LsmInfoAtCreation::Applicable(level, bottommost);
EXPECT_NE(info.Encode(), 0U);
LsmInfoAtCreation decoded = round_trip(info);
EXPECT_EQ(decoded.applicability, Applicability::kApplicable);
EXPECT_EQ(decoded.level_at_creation, level);
EXPECT_EQ(decoded.is_bottommost, bottommost);
EXPECT_EQ(decoded, info);
}
}
// Applicable but with an unknown level (e.g. the DB repairer): still
// applicable, bottommost preserved, level decodes back to kUnknownLevel.
LsmInfoAtCreation unknown_level = LsmInfoAtCreation::Applicable(
LsmInfoAtCreation::kUnknownLevel, /*is_bottommost=*/true);
LsmInfoAtCreation decoded = round_trip(unknown_level);
EXPECT_EQ(decoded.applicability, Applicability::kApplicable);
EXPECT_EQ(decoded.level_at_creation, LsmInfoAtCreation::kUnknownLevel);
EXPECT_TRUE(decoded.is_bottommost);
}
TEST_F(DBTablePropertiesTest, CreationTimeAliasesOldestAncestorTime) {
TableProperties props;
ASSERT_EQ(&props.creation_time, &props.oldest_ancestor_time);
props.creation_time = 12345;
ASSERT_EQ(props.oldest_ancestor_time, 12345U);
ASSERT_EQ(&TablePropertiesNames::kCreationTime,
&TablePropertiesNames::kOldestAncestorTime);
}
TEST_F(DBTablePropertiesTest, SstFileMetaDataOldestAncesterTimeAlias) {
// Intentionally uses the deprecated misspelled alias
LiveFileMetaData meta;
ASSERT_EQ(&meta.oldest_ancester_time, &meta.oldest_ancestor_time);
meta.oldest_ancester_time = 12345;
ASSERT_EQ(meta.oldest_ancestor_time, 12345U);
}
TEST_F(DBTablePropertiesTest, UncompressedDataSizeAndLsmInfoPersisted) {
Options options = CurrentOptions();
options.compression = kNoCompression;
DestroyAndReopen(options);
ASSERT_OK(db_->Put(WriteOptions(), "key1", "value1"));
ASSERT_OK(db_->Put(WriteOptions(), "key2", "value2"));
ASSERT_OK(db_->Put(WriteOptions(), "key3", "value3"));
ASSERT_OK(db_->Flush(FlushOptions()));
// Freshly flushed L0 file. uncompressed_data_size is now persisted (it used
// to always read back as 0), and LSM-at-creation info is populated.
{
TablePropertiesCollection props;
ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
ASSERT_EQ(1U, props.size());
auto table_props = props.begin()->second;
EXPECT_GT(table_props->uncompressed_data_size, 0U);
LsmInfoAtCreation info =
LsmInfoAtCreation::DecodeFrom(table_props->lsm_info_at_creation);
EXPECT_EQ(info.applicability,
LsmInfoAtCreation::Applicability::kApplicable);
EXPECT_EQ(info.level_at_creation, 0);
EXPECT_FALSE(info.is_bottommost);
}
// After forcing a bottommost compaction, the (re)created file records
// is_bottommost = true and a deeper creation level.
{
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
TablePropertiesCollection props;
ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
ASSERT_EQ(1U, props.size());
auto table_props = props.begin()->second;
EXPECT_GT(table_props->uncompressed_data_size, 0U);
LsmInfoAtCreation info =
LsmInfoAtCreation::DecodeFrom(table_props->lsm_info_at_creation);
EXPECT_EQ(info.applicability,
LsmInfoAtCreation::Applicability::kApplicable);
EXPECT_TRUE(info.is_bottommost);
EXPECT_GT(info.level_at_creation, 0);
}
}
TEST_F(DBTablePropertiesTest, LsmInfoNotApplicableForSstFileWriter) {
// A file written by SstFileWriter has no LSM position, so its
// lsm_info_at_creation records "not applicable" (distinct from "unknown").
Options options = CurrentOptions();
options.compression = kNoCompression;
std::string file = dbname_ + "/for_ingestion.sst";
{
SstFileWriter writer(EnvOptions(), options);
ASSERT_OK(writer.Open(file));
ASSERT_OK(writer.Put("a", "1"));
ASSERT_OK(writer.Put("b", "2"));
ASSERT_OK(writer.Finish());
}
SstFileReader reader(options);
ASSERT_OK(reader.Open(file));
ASSERT_OK(reader.VerifyChecksum());
std::shared_ptr<const TableProperties> table_props =
reader.GetTableProperties();
ASSERT_NE(table_props, nullptr);
LsmInfoAtCreation info =
LsmInfoAtCreation::DecodeFrom(table_props->lsm_info_at_creation);
EXPECT_EQ(info.applicability,
LsmInfoAtCreation::Applicability::kNotApplicable);
EXPECT_EQ(info.level_at_creation, LsmInfoAtCreation::kUnknownLevel);
EXPECT_FALSE(info.is_bottommost);
// uncompressed_data_size is persisted for SstFileWriter files too.
EXPECT_GT(table_props->uncompressed_data_size, 0U);
}
TEST_F(DBTablePropertiesTest, CompressionManagerFullDetailsPersisted) {
// With an explicit CompressionManager, compression_options records the
// manager's full details via ToString() (id plus any configured options),
// not just GetId(). This manager has no options, so ToString() is its id.
Options options = CurrentOptions();
options.compression = kNoCompression;
options.compression_manager =
std::make_shared<ParseCompressionDisplayNameManager>();
DestroyAndReopen(options);
ASSERT_OK(db_->Put(WriteOptions(), "k", "v"));
ASSERT_OK(db_->Flush(FlushOptions()));
TablePropertiesCollection props;
ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
ASSERT_EQ(1U, props.size());
auto table_props = props.begin()->second;
EXPECT_NE(
table_props->compression_options.find(
"_compression_manager=" +
std::string(ParseCompressionDisplayNameManager::kCompatibilityName)),
std::string::npos)
<< table_props->compression_options;
}
TEST_F(DBTablePropertiesTest, ParseCompressionNameForDisplay) {
auto custom_mgr = std::make_shared<ParseCompressionDisplayNameManager>();
Loaded 100 of 413 files, more files were not shown because too many files have changed in this diff. Show more