mirror of
https://github.com/facebook/rocksdb.git
synced 2026-11-06 11:02:36 -05:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ec93377878 | ||
|
|
0e1c7f9e62 | ||
|
|
c12d364ad1 | ||
|
|
766ca8f985 | ||
|
|
3c8f8f417d |
No files matched your search
@@ -310,6 +310,7 @@ build (and where noted, run tests) across:
|
||||
| Axis | Must support |
|
||||
|------|--------------|
|
||||
| OS | Linux (x86_64 + ARM), macOS, Windows |
|
||||
| Architecture | Little-endian and big-endian targets, including Linux on s390x |
|
||||
| Compiler | GCC, Clang (libstdc++ **and** libc++), AppleClang, **MSVC (VS2022)**, MinGW (Linux cross-compile, build-only, no gflags) |
|
||||
| Build system | Make, CMake, and BUCK (internal) -- keep all in sync (see "Build system" above) |
|
||||
| Config | release (`-fno-rtti`), `ASSERT_STATUS_CHECKED`, ASAN/UBSAN/TSAN, folly, unity build, JNI/Java |
|
||||
@@ -324,6 +325,12 @@ abstractions; otherwise guard with `#ifdef OS_WIN` (POSIX `<unistd.h>` ->
|
||||
Windows `<process.h>`). Because libc++ is also tested, include what you use
|
||||
rather than relying on libstdc++ transitive includes.
|
||||
|
||||
Do not use `static_assert` to reject a supported architecture for an optional
|
||||
feature. Check an existing `port::` capability such as `port::kLittleEndian`
|
||||
and disable only the unsupported feature so the rest of RocksDB still builds.
|
||||
Use a portable representation when the feature itself must work across
|
||||
architectures.
|
||||
|
||||
### Unit Test
|
||||
* After all of the unit tests are added, review them and try to extract common
|
||||
reusable utility functions to reduce code duplication due to copy past between
|
||||
|
||||
+37
@@ -1,6 +1,43 @@
|
||||
# Rocksdb Change Log
|
||||
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
|
||||
|
||||
## 11.11.1 (09/24/2026)
|
||||
### Behavior Changes
|
||||
* `Iterator::Refresh()` now returns `NotSupported` on iterators from a secondary DB's `DB::NewIterators()` only, because those iterators are pinned to one cross-column-family read view. Iterators from `DB::NewIterator()` are unaffected; to advance a batch, call `DB::NewIterators()` again after `TryCatchUpWithPrimary()`.
|
||||
|
||||
### Bug Fixes
|
||||
* Fixed secondary DB `NewIterators()` returning iterators from different database states or blocking for `TryCatchUpWithPrimary()` by serving the last completed catch-up view.
|
||||
|
||||
## 11.11.0 (09/21/2026)
|
||||
### New Features
|
||||
* External tables now support DB-assigned global sequence numbers when `IngestExternalFileOptions::allow_global_seqno` is enabled, allowing overlapping external tables to be ingested across multiple levels when compactions are disabled. This includes the deprecated `write_global_seqno` path when the reader supplies the properties block's file offset.
|
||||
* Added checksum handoff support for SST file writes when `BlockBasedTableOptions::checksum` is `kCRC32c`, allowing block and trailer CRC32C checksums to be passed from the table builder to the file writer for buffered-data validation and filesystem in-flight write protection.
|
||||
* Enhanced `sst_dump --command=recompress` to be more powerful for benchmarking, including through automation with a new `--json` option for machine-readable output.
|
||||
|
||||
### Public API Changes
|
||||
* Changed the experimental async read statistics API by removing `DB::AsyncCallback::EnableStats()` and changing `OnComplete(const PerfContext*, const IOStatsContext*)` to `OnComplete()`. Callers must configure TLS statistics before each async or coroutine read. `DB::GetAsync()` and `DB::MultiGetAsync()` callbacks must copy per-operation statistics from TLS during `OnComplete()`; coroutine users must copy them from inside a `CoroStackableDB` hook before suspending again. Each operation leaves TLS statistics collection disabled.
|
||||
* Added `Compressor::MaybeOverrideCompressionOptions()` (EXPERIMENTAL), an omnibus hook letting a custom compressor adjust the subset of `CompressionOptions` that the SST builder applies rather than the compressor itself: `parallel_threads`, `max_compressed_bytes_per_kb`, `auto_skip`, and `auto_skip_min_sample_every` (and potentially more in the future). This deprecates `Compressor::GetRecommendedParallelThreads()`, which it subsumes; the default implementation still honors an overridden `GetRecommendedParallelThreads()` for compatibility.
|
||||
* Added experimental `CoroDB::CoGet()` and `CoroDB::CoMultiGet()` overloads for point reads from an `SstFileReader`.
|
||||
* Added experimental `DB::GetWithMetadata()` and `DB::MultiGetWithMetadata()` APIs with opt-in output metadata for key timestamps and explicit-snapshot newer-version detection.
|
||||
* Added a `DB::NewIterators()` overload that accepts parallel vectors of `ReadOptions` and column family handles, allowing each returned iterator to use independent read settings while sharing a consistent database state. The vectors must have the same size, all snapshots must be null or identical, and all `tailing` settings must match.
|
||||
* Added `ReadOptions::custom_context`, an opaque, non-owning pointer passed through table reads so external table implementations can consume application-defined read options.
|
||||
|
||||
### Behavior Changes
|
||||
* `ExternalTableReader::MultiGet()` is now used by RocksDB's `DB::MultiGet()` path. Previously, the external table adapter used `TableReader`'s fallback, which called `ExternalTableReader::Get()` separately for each key.
|
||||
* `DB::OpenAndCompact()` now requires `output_directory` to be a simple name for a new subdirectory when the new `use_session_tmp_dir_for_remote_compaction` DB option is enabled. The option places that subdirectory under the canonical `<dbname>/session_tmp/` root, and `DB::Open()` deletes stale contents under the root after recovery. The disabled default preserves the existing `<dbname>/<output_directory>` layout without automatic cleanup. Previously, remote-compaction output directories were created directly under `<dbname>`, which made it challenging to safely identify and delete obsolete files.
|
||||
* Several block-based SST table properties now better support compression benchmarking and optimization: `uncompressed_data_size` is now persisted in the file (previously it was computed only at write time and read back as 0); a new `lsm_info_at_creation` property records the LSM level and "bottommost" status at the file's creation, distinguishing "not applicable" (e.g. SstFileWriter files) from "unknown" (older files); and the `compression_options` property now records the full compression manager configuration instead of only its name.
|
||||
|
||||
### Bug Fixes
|
||||
* Fixed a usually-minor performance bug where `Cache::ApplyToAllEntries()` used single-entry batches instead of the intended 256-entry default for `ApplyToAllEntriesOptions::average_entries_per_lock`. All of the Cache implementations (LRU and HCC) benefit from the fix, but the disfavored LRU cache was especially impacted by repeated mutex acquisition and release.
|
||||
* Fixed stats-disabled coroutine and asynchronous reads inheriting enabled TLS statistics state when interleaved with stats-enabled reads on the same executor. Each read now uses its captured per-call configuration, while enabled request counters remain isolated across suspensions.
|
||||
* Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) requested with `OpenAndCompactOptions::allow_resumption=true` but for which resumption was internally disabled (incompatible with `paranoid_file_checks` or output-iteration verification) could fail a retried attempt, because the output directory was not cleaned before starting fresh and output files left by a previously interrupted attempt collided with the reused file numbers.
|
||||
* Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) could fail when the primary rotated its MANIFEST while the worker was opening the primary's directory read-only. The open reads `CURRENT`, and the primary publishes a new MANIFEST by renaming a freshly written `CURRENT` over the old one; on file systems without POSIX read-after-unlink semantics (remote/object stores) the replaced object is then reported as gone (`PathNotFound`/`NotFound`, or `TryAgain` for the MANIFEST equivalent) instead of remaining readable. `DB::OpenAndCompact()` now retries the secondary open a bounded, configurable number of times for those transient statuses, so a concurrent rotation no longer fails the compaction (which, for a `CompactionService` that does not fall back to a local compaction, became a background error on the primary).
|
||||
* Fixed secondary DB `NewIterators()` returning iterators from different database states when `TryCatchUpWithPrimary()` runs concurrently.
|
||||
* Fixed a bug where a `SeekToFirst()` on an iterator previously used for a `Seek()` could end the scan early and silently omit keys with `ReadOptions::async_io`, or otherwise run with readahead effectively disabled. Affects scans using `ReadOptions::prefix_same_as_start` together with `ReadOptions::auto_readahead_size` (enabled by default); memory-mapped reads are unaffected.
|
||||
|
||||
### Performance Improvements
|
||||
* `DB::MultiGetEntityLazy()` now reads all keys of a batch under a single shared per-column-family SuperVersion pin, and a single `LazyWideColumnsBatch::MultiResolve()` coalesces the resulting blob reads across keys: one read per blob file (and one per SST for embedded blobs), for whole-column and byte-range reads alike, instead of reading each key's blobs separately.
|
||||
|
||||
## 11.10.0 (08/28/2026)
|
||||
### New Features
|
||||
* Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
|
||||
|
||||
@@ -204,6 +204,19 @@ class DBFollowerTest : public DBTestBase {
|
||||
return result;
|
||||
}
|
||||
|
||||
void VerifyFollowerBatchValue(const std::string& key,
|
||||
const std::string& expected) {
|
||||
std::vector<Iterator*> iterators;
|
||||
ASSERT_OK(follower()->NewIterators(
|
||||
ReadOptions(), {follower()->DefaultColumnFamily()}, &iterators));
|
||||
ASSERT_EQ(1U, iterators.size());
|
||||
std::unique_ptr<Iterator> iterator(iterators[0]);
|
||||
iterator->Seek(key);
|
||||
ASSERT_OK(iterator->status());
|
||||
ASSERT_TRUE(iterator->Valid());
|
||||
ASSERT_EQ(expected, iterator->value());
|
||||
}
|
||||
|
||||
DB* follower() { return follower_.get(); }
|
||||
DBFollowerTestFS* follower_fs() {
|
||||
return static_cast<DBFollowerTestFS*>(follower_env_->GetFileSystem().get());
|
||||
@@ -253,6 +266,32 @@ TEST_F(DBFollowerTest, Basic) {
|
||||
CheckDirs();
|
||||
}
|
||||
|
||||
TEST_F(DBFollowerTest, NewIteratorsRefreshesAfterCatchUp) {
|
||||
ASSERT_OK(Put("key", "v1"));
|
||||
ASSERT_OK(Flush());
|
||||
|
||||
SyncPoint::GetInstance()->LoadDependency({
|
||||
{"DBImplFollower::TryCatchupWithLeader:Begin1", "Leader::Start"},
|
||||
{"Leader::Done", "DBImplFollower::TryCatchupWithLeader:Begin2"},
|
||||
{"DBImplFollower::TryCatchupWithLeader:End", "Follower::WaitForCatchup"},
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
ASSERT_OK(OpenAsFollower());
|
||||
VerifyFollowerBatchValue("key", "v1");
|
||||
|
||||
TEST_SYNC_POINT("Leader::Start");
|
||||
ASSERT_OK(Put("key", "v2"));
|
||||
ASSERT_OK(Flush());
|
||||
TEST_SYNC_POINT("Leader::Done");
|
||||
TEST_SYNC_POINT("Follower::WaitForCatchup");
|
||||
|
||||
VerifyFollowerBatchValue("key", "v2");
|
||||
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearTrace();
|
||||
}
|
||||
|
||||
TEST_F(DBFollowerTest, Flush) {
|
||||
SyncPoint::GetInstance()->LoadDependency({
|
||||
{"DBImplFollower::TryCatchupWithLeader:Begin1", "Leader::Start"},
|
||||
|
||||
@@ -153,6 +153,7 @@ Status DBImplFollower::TryCatchUpWithLeader() {
|
||||
cfd->InstallSuperVersion(&sv_context, &mutex_);
|
||||
sv_context.NewSuperVersion();
|
||||
}
|
||||
PublishSecondaryReadView();
|
||||
}
|
||||
|
||||
for (auto& file : files_to_delete) {
|
||||
@@ -164,6 +165,7 @@ Status DBImplFollower::TryCatchUpWithLeader() {
|
||||
}
|
||||
}
|
||||
}
|
||||
CleanupRetiredSecondaryReadViews();
|
||||
job_context.Clean();
|
||||
|
||||
// Cleanup unused, obsolete files.
|
||||
@@ -322,9 +324,11 @@ Status DB::OpenAsFollower(
|
||||
sv_context.NewSuperVersion();
|
||||
cfd->InstallSuperVersion(&sv_context, &impl->mutex_);
|
||||
}
|
||||
impl->PublishSecondaryReadView();
|
||||
impl->MarkAsyncFileOpenNotNeeded();
|
||||
}
|
||||
impl->mutex_.Unlock();
|
||||
impl->CleanupRetiredSecondaryReadViews();
|
||||
sv_context.Clean();
|
||||
if (s.ok()) {
|
||||
dbptr->reset(impl);
|
||||
|
||||
+253
-12
@@ -23,6 +23,7 @@
|
||||
#include "monitoring/perf_context_imp.h"
|
||||
#include "rocksdb/convenience.h"
|
||||
#include "rocksdb/utilities/options_util.h"
|
||||
#include "util/atomic.h"
|
||||
#include "util/cast_util.h"
|
||||
#include "util/write_batch_util.h"
|
||||
|
||||
@@ -33,12 +34,222 @@ DBImplSecondary::DBImplSecondary(const DBOptions& db_options,
|
||||
std::string secondary_path)
|
||||
: DBImpl(db_options, dbname, false, true, true),
|
||||
secondary_path_(std::move(secondary_path)) {
|
||||
secondary_read_view_reclaimer_.reset(new port::Thread(
|
||||
&DBImplSecondary::ReclaimRetiredSecondaryReadViewsLoop, this));
|
||||
ROCKS_LOG_INFO(immutable_db_options_.info_log,
|
||||
"Opening the db in secondary mode");
|
||||
LogFlush(immutable_db_options_.info_log);
|
||||
}
|
||||
|
||||
DBImplSecondary::~DBImplSecondary() = default;
|
||||
DBImplSecondary::~DBImplSecondary() {
|
||||
ResetSecondaryReadView();
|
||||
StopSecondaryReadViewReclaimer();
|
||||
}
|
||||
|
||||
// REQUIRES: mutex_ not held. CleanupSuperVersion() locks mutex_ to release a
|
||||
// SuperVersion's last reference, so dropping the last reference to a view while
|
||||
// mutex_ is held self-deadlocks on the non-reentrant mutex.
|
||||
DBImplSecondary::SecondaryReadView::~SecondaryReadView() {
|
||||
for (const SuperVersions::value_type& entry : super_versions) {
|
||||
db->CleanupSuperVersion(entry.second);
|
||||
}
|
||||
}
|
||||
|
||||
void DBImplSecondary::PublishSecondaryReadView() {
|
||||
mutex_.AssertHeld();
|
||||
|
||||
if (secondary_read_view_closed_) {
|
||||
// A catch-up that raced ResetSecondaryReadView() must not republish, or the
|
||||
// new view would outlive the drain that close waits on.
|
||||
return;
|
||||
}
|
||||
|
||||
std::shared_ptr<SecondaryReadView> new_view =
|
||||
std::make_shared<SecondaryReadView>(this, versions_->LastSequence());
|
||||
for (ColumnFamilyData* cfd : *versions_->GetColumnFamilySet()) {
|
||||
if (cfd->IsDropped() || !cfd->initialized()) {
|
||||
continue;
|
||||
}
|
||||
SuperVersion* super_version = cfd->GetSuperVersion();
|
||||
if (super_version == nullptr) {
|
||||
continue;
|
||||
}
|
||||
new_view->super_versions.emplace(cfd->GetID(), super_version->Ref());
|
||||
}
|
||||
|
||||
// Load the outgoing view before the store, and keep that reference until it
|
||||
// reaches the retirement queue. The store drops the published reference, so
|
||||
// without this local it could be the last one and ~SecondaryReadView() would
|
||||
// run here, under mutex_. Retiring before the store would leave the view
|
||||
// reachable by readers after it was queued for reclamation.
|
||||
std::shared_ptr<const SecondaryReadView> previous_view =
|
||||
AtomicSharedPtrLoad(&secondary_read_view_, std::memory_order_acquire);
|
||||
AtomicSharedPtrStore(
|
||||
&secondary_read_view_,
|
||||
std::shared_ptr<const SecondaryReadView>(std::move(new_view)),
|
||||
std::memory_order_release);
|
||||
RetireSecondaryReadView(std::move(previous_view));
|
||||
}
|
||||
|
||||
void DBImplSecondary::RetireSecondaryReadView(
|
||||
std::shared_ptr<const SecondaryReadView> read_view) {
|
||||
if (read_view == nullptr) {
|
||||
return;
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
retired_secondary_read_views_.emplace_back(std::move(read_view));
|
||||
}
|
||||
retired_secondary_read_views_cv_.notify_all();
|
||||
}
|
||||
|
||||
bool DBImplSecondary::HasReclaimableSecondaryReadView() const {
|
||||
for (const auto& read_view : retired_secondary_read_views_) {
|
||||
if (read_view.use_count() == 1) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool DBImplSecondary::AllSecondaryReadViewsReclaimed() const {
|
||||
return retired_secondary_read_views_.empty() &&
|
||||
secondary_read_view_reclaims_in_flight_ == 0;
|
||||
}
|
||||
|
||||
bool DBImplSecondary::ReclaimRetiredSecondaryReadViews() {
|
||||
// Keep moved views alive until after the retirement lock is released. Their
|
||||
// destructors call CleanupSuperVersion(), which can lock mutex_.
|
||||
std::vector<std::shared_ptr<const SecondaryReadView>> reclaimable_views;
|
||||
size_t num_reclaiming;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
for (auto iter = retired_secondary_read_views_.begin();
|
||||
iter != retired_secondary_read_views_.end();) {
|
||||
// The view is no longer published, so its reference count can only
|
||||
// decrease. A count of one means the retirement queue is the sole owner.
|
||||
if (iter->use_count() == 1) {
|
||||
reclaimable_views.emplace_back(std::move(*iter));
|
||||
iter = retired_secondary_read_views_.erase(iter);
|
||||
} else {
|
||||
++iter;
|
||||
}
|
||||
}
|
||||
num_reclaiming = reclaimable_views.size();
|
||||
secondary_read_view_reclaims_in_flight_ += num_reclaiming;
|
||||
if (num_reclaiming == 0) {
|
||||
return AllSecondaryReadViewsReclaimed();
|
||||
}
|
||||
}
|
||||
|
||||
TEST_SYNC_POINT("DBImplSecondary::ReclaimRetiredReadViews:BeforeReclaim");
|
||||
reclaimable_views.clear();
|
||||
bool all_reclaimed;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
assert(secondary_read_view_reclaims_in_flight_ >= num_reclaiming);
|
||||
secondary_read_view_reclaims_in_flight_ -= num_reclaiming;
|
||||
all_reclaimed = AllSecondaryReadViewsReclaimed();
|
||||
}
|
||||
retired_secondary_read_views_cv_.notify_all();
|
||||
TEST_SYNC_POINT("DBImplSecondary::ReclaimRetiredReadViews:AfterReclaim");
|
||||
return all_reclaimed;
|
||||
}
|
||||
|
||||
void DBImplSecondary::ReclaimRetiredSecondaryReadViewsLoop() {
|
||||
while (true) {
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
retired_secondary_read_views_cv_.wait(lock, [this]() {
|
||||
if (secondary_read_view_reclaimer_stopping_ &&
|
||||
AllSecondaryReadViewsReclaimed()) {
|
||||
return true;
|
||||
}
|
||||
return HasReclaimableSecondaryReadView();
|
||||
});
|
||||
if (secondary_read_view_reclaimer_stopping_ &&
|
||||
AllSecondaryReadViewsReclaimed()) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
TEST_SYNC_POINT(
|
||||
"DBImplSecondary::ReclaimRetiredReadViewsLoop:BeforeReclaim");
|
||||
ReclaimRetiredSecondaryReadViews();
|
||||
}
|
||||
}
|
||||
|
||||
void DBImplSecondary::CleanupRetiredSecondaryReadViews() {
|
||||
ReclaimRetiredSecondaryReadViews();
|
||||
}
|
||||
|
||||
void DBImplSecondary::DrainRetiredSecondaryReadViews() {
|
||||
while (!ReclaimRetiredSecondaryReadViews()) {
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
TEST_SYNC_POINT("DBImplSecondary::DrainRetiredReadViews:Wait");
|
||||
retired_secondary_read_views_cv_.wait(lock, [this]() {
|
||||
if (AllSecondaryReadViewsReclaimed()) {
|
||||
return true;
|
||||
}
|
||||
return HasReclaimableSecondaryReadView();
|
||||
});
|
||||
}
|
||||
TEST_SYNC_POINT("DBImplSecondary::DrainRetiredReadViews:AfterWait");
|
||||
}
|
||||
}
|
||||
|
||||
void DBImplSecondary::ReleaseSecondaryReadView(
|
||||
std::shared_ptr<const SecondaryReadView>* read_view) {
|
||||
read_view->reset();
|
||||
{
|
||||
// Synchronize the reference-count change with the wait predicate so its
|
||||
// notification cannot be lost.
|
||||
std::lock_guard<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
retired_secondary_read_views_cv_.notify_all();
|
||||
}
|
||||
}
|
||||
|
||||
void DBImplSecondary::StopSecondaryReadViewReclaimer() {
|
||||
if (secondary_read_view_reclaimer_ == nullptr) {
|
||||
return;
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(retired_secondary_read_views_mutex_);
|
||||
secondary_read_view_reclaimer_stopping_ = true;
|
||||
}
|
||||
retired_secondary_read_views_cv_.notify_all();
|
||||
TEST_SYNC_POINT("DBImplSecondary::StopSecondaryReadViewReclaimer:BeforeJoin");
|
||||
secondary_read_view_reclaimer_->join();
|
||||
secondary_read_view_reclaimer_.reset();
|
||||
}
|
||||
|
||||
void DBImplSecondary::ResetSecondaryReadView() {
|
||||
std::shared_ptr<const SecondaryReadView> read_view;
|
||||
{
|
||||
// mutex_ makes this read-modify-write atomic against
|
||||
// PublishSecondaryReadView(), which holds it too. Without that, a catch-up
|
||||
// racing close can read the same outgoing view and retire it twice, leaving
|
||||
// two queue entries that each see use_count() == 2 so the drain below never
|
||||
// finishes.
|
||||
InstrumentedMutexLock lock_guard(&mutex_);
|
||||
secondary_read_view_closed_ = true;
|
||||
read_view =
|
||||
AtomicSharedPtrLoad(&secondary_read_view_, std::memory_order_acquire);
|
||||
AtomicSharedPtrStore(&secondary_read_view_,
|
||||
std::shared_ptr<const SecondaryReadView>(),
|
||||
std::memory_order_release);
|
||||
}
|
||||
RetireSecondaryReadView(std::move(read_view));
|
||||
DrainRetiredSecondaryReadViews();
|
||||
}
|
||||
|
||||
Status DBImplSecondary::CloseImpl() {
|
||||
// The view's SuperVersions pin their column families, so release them before
|
||||
// DBImpl::CloseImpl() destroys the VersionSet.
|
||||
ResetSecondaryReadView();
|
||||
StopSecondaryReadViewReclaimer();
|
||||
return DBImpl::CloseImpl();
|
||||
}
|
||||
|
||||
Status DBImplSecondary::Recover(
|
||||
const std::vector<ColumnFamilyDescriptor>& column_families,
|
||||
@@ -738,23 +949,44 @@ Status DBImplSecondary::NewIteratorsImpl(
|
||||
iterators->clear();
|
||||
iterators->reserve(column_families.size());
|
||||
|
||||
SequenceNumber read_seq;
|
||||
std::shared_ptr<const SecondaryReadView> read_view =
|
||||
AtomicSharedPtrLoad(&secondary_read_view_, std::memory_order_acquire);
|
||||
if (read_view == nullptr) {
|
||||
return Status::InvalidArgument("Secondary read view is not initialized");
|
||||
}
|
||||
TEST_SYNC_POINT("DBImplSecondary::NewIterators:AfterLoadReadView");
|
||||
|
||||
const SequenceNumber read_seq = read_view->sequence;
|
||||
autovector<ColumnFamilySuperVersionPair, MultiGetContext::MAX_BATCH_SIZE>
|
||||
cf_sv_pairs;
|
||||
{
|
||||
InstrumentedMutexLock lock_guard(&mutex_);
|
||||
read_seq = versions_->LastSequence();
|
||||
for (auto* cf : column_families) {
|
||||
auto* cfh = static_cast_with_check<ColumnFamilyHandleImpl>(cf);
|
||||
cf_sv_pairs.emplace_back(cfh, cfh->cfd()->GetSuperVersion()->Ref());
|
||||
}
|
||||
}
|
||||
|
||||
const auto cleanup_super_versions = [&]() {
|
||||
for (const auto& cf_sv_pair : cf_sv_pairs) {
|
||||
CleanupSuperVersion(cf_sv_pair.super_version);
|
||||
}
|
||||
};
|
||||
for (ColumnFamilyHandle* cf : column_families) {
|
||||
ColumnFamilyHandleImpl* cfh =
|
||||
static_cast_with_check<ColumnFamilyHandleImpl>(cf);
|
||||
const SecondaryReadView::SuperVersions::const_iterator sv_iter =
|
||||
read_view->super_versions.find(cfh->GetID());
|
||||
if (sv_iter == read_view->super_versions.end()) {
|
||||
if (!cfh->cfd()->IsDropped()) {
|
||||
cleanup_super_versions();
|
||||
ReleaseSecondaryReadView(&read_view);
|
||||
return Status::InvalidArgument(
|
||||
"Column family is not part of the secondary read view");
|
||||
}
|
||||
// A retained handle remains readable after its column family is dropped.
|
||||
// Its immutable final SuperVersion is intentionally not published, since
|
||||
// doing so would keep the dropped column family alive indefinitely.
|
||||
cf_sv_pairs.emplace_back(cfh,
|
||||
cfh->cfd()->GetReferencedSuperVersion(this));
|
||||
} else {
|
||||
cf_sv_pairs.emplace_back(cfh, sv_iter->second->Ref());
|
||||
}
|
||||
}
|
||||
ReleaseSecondaryReadView(&read_view);
|
||||
TEST_SYNC_POINT("DBImplSecondary::NewIterators:AfterReleaseReadView");
|
||||
for (size_t i = 0; i < cf_sv_pairs.size(); ++i) {
|
||||
const auto& cf_sv_pair = cf_sv_pairs[i];
|
||||
const ReadOptions& options = normalized_read_options[i];
|
||||
@@ -771,7 +1003,8 @@ Status DBImplSecondary::NewIteratorsImpl(
|
||||
const auto& cf_sv_pair = cf_sv_pairs[i];
|
||||
iterators->push_back(NewIteratorImpl(
|
||||
normalized_read_options[i], cf_sv_pair.cfh, cf_sv_pair.super_version,
|
||||
read_seq, nullptr /* read_callback */));
|
||||
read_seq, nullptr /* read_callback */, false /* expose_blob_index */,
|
||||
false /* allow_refresh */));
|
||||
TEST_SYNC_POINT("DBImplSecondary::NewIterators:AfterCreateIterator");
|
||||
}
|
||||
return Status::OK();
|
||||
@@ -881,8 +1114,14 @@ Status DBImplSecondary::TryCatchUpWithPrimary() {
|
||||
// completions; an outstanding prepare stops the scan.
|
||||
(void)logs_with_prep_tracker_.FindMinLogContainingOutstandingPrep();
|
||||
}
|
||||
TEST_SYNC_POINT(
|
||||
"DBImplSecondary::TryCatchUpWithPrimary:BeforePublishReadView");
|
||||
TEST_SYNC_POINT(
|
||||
"DBImplSecondary::TryCatchUpWithPrimary:AllowPublishReadView");
|
||||
PublishSecondaryReadView();
|
||||
}
|
||||
}
|
||||
CleanupRetiredSecondaryReadViews();
|
||||
job_context.Clean();
|
||||
|
||||
// Cleanup unused, obsolete files.
|
||||
@@ -1015,9 +1254,11 @@ Status DBImplSecondary::OpenAsSecondaryImpl(
|
||||
sv_context.NewSuperVersion();
|
||||
cfd->InstallSuperVersion(&sv_context, &impl->mutex_);
|
||||
}
|
||||
impl->PublishSecondaryReadView();
|
||||
impl->MarkAsyncFileOpenNotNeeded();
|
||||
}
|
||||
impl->mutex_.Unlock();
|
||||
impl->CleanupRetiredSecondaryReadViews();
|
||||
sv_context.Clean();
|
||||
job_context.Clean();
|
||||
if (s.ok()) {
|
||||
|
||||
@@ -5,14 +5,19 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <condition_variable>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <unordered_map>
|
||||
#include <unordered_set>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "db/db_impl/db_impl.h"
|
||||
#include "logging/logging.h"
|
||||
#include "port/port.h"
|
||||
|
||||
namespace ROCKSDB_NAMESPACE {
|
||||
|
||||
@@ -288,6 +293,8 @@ class DBImplSecondary : public DBImpl {
|
||||
#endif // NDEBUG
|
||||
|
||||
protected:
|
||||
Status CloseImpl() override;
|
||||
|
||||
Status FlushForGetLiveFiles(bool /*force_atomic_flush*/) override {
|
||||
// No-op for read-only DB
|
||||
return Status::OK();
|
||||
@@ -322,9 +329,50 @@ class DBImplSecondary : public DBImpl {
|
||||
std::unique_ptr<log::Reader::Reporter> manifest_reporter_;
|
||||
std::unique_ptr<Status> manifest_reader_status_;
|
||||
|
||||
// Publishes a view of every initialized, non-dropped column family while
|
||||
// mutex_ protects their SuperVersions. The previous view is retained for
|
||||
// cleanup outside mutex_ and outside reader threads. No-op once the view has
|
||||
// been torn down for close.
|
||||
// REQUIRES: mutex_ held
|
||||
void PublishSecondaryReadView();
|
||||
|
||||
// Reclaims retired views that have no readers. This must be called without
|
||||
// mutex_ because releasing a view's final SuperVersion can lock it.
|
||||
void CleanupRetiredSecondaryReadViews();
|
||||
|
||||
private:
|
||||
friend class DB;
|
||||
|
||||
struct SecondaryReadView {
|
||||
using SuperVersions = std::unordered_map<uint32_t, SuperVersion*>;
|
||||
|
||||
SecondaryReadView(DBImplSecondary* db_arg, SequenceNumber sequence_arg)
|
||||
: db(db_arg), sequence(sequence_arg) {}
|
||||
~SecondaryReadView();
|
||||
SecondaryReadView(const SecondaryReadView&) = delete;
|
||||
SecondaryReadView& operator=(const SecondaryReadView&) = delete;
|
||||
SecondaryReadView(SecondaryReadView&&) = delete;
|
||||
SecondaryReadView& operator=(SecondaryReadView&&) = delete;
|
||||
|
||||
DBImplSecondary* db;
|
||||
SequenceNumber sequence;
|
||||
SuperVersions super_versions;
|
||||
};
|
||||
|
||||
void RetireSecondaryReadView(
|
||||
std::shared_ptr<const SecondaryReadView> read_view);
|
||||
// REQUIRES: retired_secondary_read_views_mutex_ held
|
||||
bool HasReclaimableSecondaryReadView() const;
|
||||
// REQUIRES: retired_secondary_read_views_mutex_ held
|
||||
bool AllSecondaryReadViewsReclaimed() const;
|
||||
bool ReclaimRetiredSecondaryReadViews();
|
||||
void ReclaimRetiredSecondaryReadViewsLoop();
|
||||
void DrainRetiredSecondaryReadViews();
|
||||
void StopSecondaryReadViewReclaimer();
|
||||
void ReleaseSecondaryReadView(
|
||||
std::shared_ptr<const SecondaryReadView>* read_view);
|
||||
void ResetSecondaryReadView();
|
||||
|
||||
Status NewIteratorsImpl(
|
||||
const std::vector<ReadOptions>& read_options,
|
||||
const std::vector<ColumnFamilyHandle*>& column_families,
|
||||
@@ -519,6 +567,22 @@ class DBImplSecondary : public DBImpl {
|
||||
std::unordered_map<uint32_t, std::pair<uint64_t, int>>
|
||||
cf_id_to_retention_warning_;
|
||||
|
||||
// Mutated only under mutex_, so that load-then-store stays atomic between
|
||||
// publication and teardown. Readers load it without mutex_, so every access
|
||||
// must still go through AtomicSharedPtrLoad/Store.
|
||||
std::shared_ptr<const SecondaryReadView> secondary_read_view_;
|
||||
// Set by ResetSecondaryReadView() so a later catch-up cannot publish a view
|
||||
// that would outlive the drain close waits on.
|
||||
// Guarded by mutex_, which every mutation of secondary_read_view_ holds.
|
||||
bool secondary_read_view_closed_ = false;
|
||||
std::mutex retired_secondary_read_views_mutex_;
|
||||
std::condition_variable retired_secondary_read_views_cv_;
|
||||
std::vector<std::shared_ptr<const SecondaryReadView>>
|
||||
retired_secondary_read_views_;
|
||||
size_t secondary_read_view_reclaims_in_flight_ = 0;
|
||||
bool secondary_read_view_reclaimer_stopping_ = false;
|
||||
std::unique_ptr<port::Thread> secondary_read_view_reclaimer_;
|
||||
|
||||
const std::string secondary_path_;
|
||||
|
||||
CompactionProgress compaction_progress_;
|
||||
|
||||
+451
-13
@@ -7,6 +7,10 @@
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
||||
|
||||
#include <condition_variable>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
|
||||
#include "db/blob/blob_index.h"
|
||||
#include "db/db_impl/db_impl_secondary.h"
|
||||
#include "db/db_test_util.h"
|
||||
@@ -173,6 +177,19 @@ class DBSecondaryTestBase : public DBBasicTestWithTimestampBase {
|
||||
ASSERT_EQ(expected, iter->value().ToString());
|
||||
}
|
||||
|
||||
void VerifySecondaryBatchValue(ColumnFamilyHandle* cfh,
|
||||
const std::string& key,
|
||||
const std::string& expected) {
|
||||
std::vector<Iterator*> iterators;
|
||||
ASSERT_OK(db_secondary_->NewIterators(ReadOptions(), {cfh}, &iterators));
|
||||
ASSERT_EQ(1U, iterators.size());
|
||||
std::unique_ptr<Iterator> iterator(iterators[0]);
|
||||
iterator->Seek(key);
|
||||
ASSERT_OK(iterator->status());
|
||||
ASSERT_TRUE(iterator->Valid());
|
||||
ASSERT_EQ(expected, iterator->value());
|
||||
}
|
||||
|
||||
// Returns the path of the primary's newest table file for `cf_name`.
|
||||
std::string GetNewestTableFilePath(const std::string& cf_name) {
|
||||
std::vector<LiveFileMetaData> live_files;
|
||||
@@ -1215,8 +1232,11 @@ TEST_F(DBSecondaryCatchUpFaultTest, ReconcilesMemtableAfterFailedRound) {
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
fault_fs_->SetFilesystemActive(true);
|
||||
|
||||
VerifySecondaryBatchValue(db_secondary_->DefaultColumnFamily(), "foo", "v1");
|
||||
|
||||
ASSERT_OK(db_secondary_->TryCatchUpWithPrimary());
|
||||
VerifySecondaryValue("foo", "v2");
|
||||
VerifySecondaryBatchValue(db_secondary_->DefaultColumnFamily(), "foo", "v2");
|
||||
}
|
||||
|
||||
// A round whose WAL discovery reports a purged path replays nothing, yet
|
||||
@@ -1775,26 +1795,25 @@ TEST_F(DBSecondaryTest, NewIteratorsConsistentViewDuringCatchUp) {
|
||||
ASSERT_OK(
|
||||
db_->CompactRange(CompactRangeOptions(), handles_[1], nullptr, nullptr));
|
||||
|
||||
bool caught_up = false;
|
||||
Status catch_up_status;
|
||||
const auto catch_up = [&](void*) {
|
||||
if (!caught_up) {
|
||||
caught_up = true;
|
||||
catch_up_status = db_secondary_->TryCatchUpWithPrimary();
|
||||
}
|
||||
};
|
||||
SyncPoint::GetInstance()->SetCallBack("DBImpl::MultiCFSnapshot::AfterRefSV",
|
||||
catch_up);
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::NewIterators:AfterCreateIterator", catch_up);
|
||||
SyncPoint::GetInstance()->LoadDependency({
|
||||
{"DBImplSecondary::TryCatchUpWithPrimary:BeforePublishReadView",
|
||||
"DBImplSecondary::NewIterators:AfterLoadReadView"},
|
||||
{"DBImplSecondary::NewIterators:AfterLoadReadView",
|
||||
"DBImplSecondary::TryCatchUpWithPrimary:AllowPublishReadView"},
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
port::Thread catch_up_thread(
|
||||
[&]() { catch_up_status = db_secondary_->TryCatchUpWithPrimary(); });
|
||||
|
||||
std::vector<Iterator*> iterators;
|
||||
const Status new_iterators_status = db_secondary_->NewIterators(
|
||||
ReadOptions(), handles_secondary_, &iterators);
|
||||
catch_up_thread.join();
|
||||
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
SyncPoint::GetInstance()->ClearTrace();
|
||||
|
||||
std::vector<std::unique_ptr<Iterator>> owned_iterators;
|
||||
owned_iterators.reserve(iterators.size());
|
||||
@@ -1803,7 +1822,6 @@ TEST_F(DBSecondaryTest, NewIteratorsConsistentViewDuringCatchUp) {
|
||||
}
|
||||
|
||||
ASSERT_OK(catch_up_status);
|
||||
ASSERT_TRUE(caught_up);
|
||||
ASSERT_OK(new_iterators_status);
|
||||
ASSERT_EQ(2, owned_iterators.size());
|
||||
for (const auto& iterator : owned_iterators) {
|
||||
@@ -1817,6 +1835,97 @@ TEST_F(DBSecondaryTest, NewIteratorsConsistentViewDuringCatchUp) {
|
||||
VerifySecondaryValue(handles_secondary_[1], "key", "new");
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, RetiredReadViewIsNotReclaimedByReader) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
options.disable_auto_compactions = true;
|
||||
CreateAndReopenWithCF({"cf1"}, options);
|
||||
|
||||
ASSERT_OK(Put(0, "key", "old"));
|
||||
ASSERT_OK(Put(1, "key", "old"));
|
||||
ASSERT_OK(Flush(0));
|
||||
ASSERT_OK(Flush(1));
|
||||
|
||||
Options secondary_options = options;
|
||||
secondary_options.max_open_files = -1;
|
||||
OpenSecondaryWithColumnFamilies({"cf1"}, secondary_options);
|
||||
|
||||
WriteOptions write_options;
|
||||
write_options.disableWAL = true;
|
||||
ASSERT_OK(db_->Put(write_options, handles_[1], "key", "new"));
|
||||
ASSERT_OK(Flush(1));
|
||||
|
||||
std::mutex state_mutex;
|
||||
std::condition_variable state_cv;
|
||||
bool reader_loaded = false;
|
||||
bool allow_reader = false;
|
||||
bool second_catch_up_locked = false;
|
||||
bool reader_done = false;
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::NewIterators:AfterLoadReadView", [&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
reader_loaded = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reader; });
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
Status reader_status;
|
||||
std::vector<Iterator*> iterators;
|
||||
port::Thread reader_thread([&]() {
|
||||
reader_status = db_secondary_->NewIterators(
|
||||
ReadOptions(), {handles_secondary_[0]}, &iterators);
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
reader_done = true;
|
||||
state_cv.notify_all();
|
||||
});
|
||||
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return reader_loaded; });
|
||||
}
|
||||
const Status first_catch_up_status = db_secondary_->TryCatchUpWithPrimary();
|
||||
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::TryCatchUpWithPrimary:BeforePublishReadView",
|
||||
[&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
second_catch_up_locked = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return reader_done; });
|
||||
});
|
||||
Status second_catch_up_status;
|
||||
port::Thread second_catch_up_thread([&]() {
|
||||
second_catch_up_status = db_secondary_->TryCatchUpWithPrimary();
|
||||
});
|
||||
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return second_catch_up_locked; });
|
||||
allow_reader = true;
|
||||
state_cv.notify_all();
|
||||
}
|
||||
|
||||
reader_thread.join();
|
||||
second_catch_up_thread.join();
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
|
||||
std::vector<std::unique_ptr<Iterator>> owned_iterators;
|
||||
owned_iterators.reserve(iterators.size());
|
||||
for (Iterator* iterator : iterators) {
|
||||
owned_iterators.emplace_back(iterator);
|
||||
}
|
||||
ASSERT_OK(first_catch_up_status);
|
||||
ASSERT_OK(second_catch_up_status);
|
||||
ASSERT_OK(reader_status);
|
||||
ASSERT_EQ(1U, owned_iterators.size());
|
||||
owned_iterators[0]->Seek("key");
|
||||
ASSERT_OK(owned_iterators[0]->status());
|
||||
ASSERT_TRUE(owned_iterators[0]->Valid());
|
||||
ASSERT_EQ("old", owned_iterators[0]->value());
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, RefreshIterator) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
@@ -2070,6 +2179,334 @@ TEST_F(DBSecondaryTest, PrimaryDropColumnFamily) {
|
||||
value.clear();
|
||||
ASSERT_OK(db_secondary_->Get(ropts, handles_secondary_[1], "foo", &value));
|
||||
ASSERT_EQ("foo_val_1", value);
|
||||
|
||||
std::vector<Iterator*> raw_iterators;
|
||||
Status new_iterators_status = db_secondary_->NewIterators(
|
||||
ropts, {handles_secondary_[1]}, &raw_iterators);
|
||||
std::vector<std::unique_ptr<Iterator>> iterators;
|
||||
for (Iterator* iterator : raw_iterators) {
|
||||
iterators.emplace_back(iterator);
|
||||
}
|
||||
ASSERT_OK(new_iterators_status);
|
||||
ASSERT_EQ(1U, iterators.size());
|
||||
iterators[0]->SeekToFirst();
|
||||
ASSERT_TRUE(iterators[0]->Valid());
|
||||
ASSERT_EQ("foo", iterators[0]->key().ToString());
|
||||
ASSERT_EQ("foo_val_1", iterators[0]->value().ToString());
|
||||
|
||||
// A later publication must keep skipping the dropped column family after
|
||||
// releasing the previous view's SuperVersion.
|
||||
ASSERT_OK(db_secondary_->TryCatchUpWithPrimary());
|
||||
|
||||
raw_iterators.clear();
|
||||
new_iterators_status = db_secondary_->NewIterators(
|
||||
ropts, {handles_secondary_[0], handles_secondary_[1]}, &raw_iterators);
|
||||
iterators.clear();
|
||||
for (Iterator* iterator : raw_iterators) {
|
||||
iterators.emplace_back(iterator);
|
||||
}
|
||||
ASSERT_OK(new_iterators_status);
|
||||
ASSERT_EQ(2U, iterators.size());
|
||||
iterators[0]->SeekToFirst();
|
||||
ASSERT_FALSE(iterators[0]->Valid());
|
||||
ASSERT_OK(iterators[0]->status());
|
||||
iterators[1]->SeekToFirst();
|
||||
ASSERT_TRUE(iterators[1]->Valid());
|
||||
ASSERT_EQ("foo", iterators[1]->key().ToString());
|
||||
ASSERT_EQ("foo_val_1", iterators[1]->value().ToString());
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, ReclaimsRetiredReadViewAfterLastReader) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
CreateAndReopenWithCF({"cf1"}, options);
|
||||
|
||||
options.max_open_files = -1;
|
||||
OpenSecondaryWithColumnFamilies({"cf1"}, options);
|
||||
|
||||
std::mutex state_mutex;
|
||||
std::condition_variable state_cv;
|
||||
bool reader_loaded = false;
|
||||
bool allow_reader = false;
|
||||
bool view_reclaimed = false;
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::NewIterators:AfterLoadReadView", [&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
reader_loaded = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reader; });
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::ReclaimRetiredReadViews:AfterReclaim",
|
||||
[&](void* /*arg*/) {
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
view_reclaimed = true;
|
||||
state_cv.notify_all();
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
Status reader_status;
|
||||
std::vector<Iterator*> iterators;
|
||||
port::Thread reader_thread([&]() {
|
||||
reader_status = db_secondary_->NewIterators(ReadOptions(),
|
||||
handles_secondary_, &iterators);
|
||||
});
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return reader_loaded; });
|
||||
}
|
||||
|
||||
const Status catch_up_status = db_secondary_->TryCatchUpWithPrimary();
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
allow_reader = true;
|
||||
state_cv.notify_all();
|
||||
if (catch_up_status.ok()) {
|
||||
state_cv.wait(lock, [&]() { return view_reclaimed; });
|
||||
}
|
||||
}
|
||||
|
||||
reader_thread.join();
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
|
||||
std::vector<std::unique_ptr<Iterator>> owned_iterators;
|
||||
for (Iterator* iterator : iterators) {
|
||||
owned_iterators.emplace_back(iterator);
|
||||
}
|
||||
ASSERT_OK(catch_up_status);
|
||||
ASSERT_OK(reader_status);
|
||||
ASSERT_EQ(handles_secondary_.size(), owned_iterators.size());
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, CloseWaitsForInFlightReadViewReclamation) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
CreateAndReopenWithCF({"cf1"}, options);
|
||||
|
||||
options.max_open_files = -1;
|
||||
OpenSecondaryWithColumnFamilies({"cf1"}, options);
|
||||
|
||||
std::mutex state_mutex;
|
||||
std::condition_variable state_cv;
|
||||
std::thread::id reclaimer_thread_id;
|
||||
bool reader_loaded = false;
|
||||
bool allow_reader = false;
|
||||
bool reclaimer_ready = false;
|
||||
bool allow_reclaimer = false;
|
||||
bool reclamation_in_flight = false;
|
||||
bool allow_reclamation = false;
|
||||
bool close_waiting = false;
|
||||
bool close_reached_stop = false;
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::NewIterators:AfterLoadReadView", [&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
reader_loaded = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reader; });
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::ReclaimRetiredReadViewsLoop:BeforeReclaim",
|
||||
[&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
reclaimer_thread_id = std::this_thread::get_id();
|
||||
reclaimer_ready = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reclaimer; });
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::ReclaimRetiredReadViews:BeforeReclaim",
|
||||
[&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
if (std::this_thread::get_id() != reclaimer_thread_id) {
|
||||
return;
|
||||
}
|
||||
reclamation_in_flight = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reclamation; });
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::DrainRetiredReadViews:Wait", [&](void* /*arg*/) {
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
close_waiting = true;
|
||||
state_cv.notify_all();
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::StopSecondaryReadViewReclaimer:BeforeJoin",
|
||||
[&](void* /*arg*/) {
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
close_reached_stop = true;
|
||||
state_cv.notify_all();
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
Status reader_status;
|
||||
std::vector<Iterator*> iterators;
|
||||
port::Thread reader_thread([&]() {
|
||||
reader_status = db_secondary_->NewIterators(ReadOptions(),
|
||||
handles_secondary_, &iterators);
|
||||
});
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return reader_loaded; });
|
||||
}
|
||||
const Status catch_up_status = db_secondary_->TryCatchUpWithPrimary();
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
allow_reader = true;
|
||||
state_cv.notify_all();
|
||||
}
|
||||
reader_thread.join();
|
||||
|
||||
for (Iterator* iterator : iterators) {
|
||||
delete iterator;
|
||||
}
|
||||
iterators.clear();
|
||||
Status destroy_handles_status;
|
||||
for (ColumnFamilyHandle* handle : handles_secondary_) {
|
||||
const Status s = db_secondary_->DestroyColumnFamilyHandle(handle);
|
||||
if (!s.ok() && destroy_handles_status.ok()) {
|
||||
destroy_handles_status = s;
|
||||
}
|
||||
}
|
||||
handles_secondary_.clear();
|
||||
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return reclaimer_ready; });
|
||||
allow_reclaimer = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return reclamation_in_flight; });
|
||||
}
|
||||
|
||||
Status close_status;
|
||||
port::Thread close_thread([&]() { close_status = db_secondary_->Close(); });
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return close_waiting || close_reached_stop; });
|
||||
EXPECT_TRUE(close_waiting);
|
||||
EXPECT_FALSE(close_reached_stop);
|
||||
allow_reclamation = true;
|
||||
state_cv.notify_all();
|
||||
}
|
||||
close_thread.join();
|
||||
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
|
||||
ASSERT_OK(catch_up_status);
|
||||
ASSERT_OK(reader_status);
|
||||
ASSERT_OK(destroy_handles_status);
|
||||
ASSERT_OK(close_status);
|
||||
db_secondary_.reset();
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, ExplicitCloseReleasesPublishedReadView) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
CreateAndReopenWithCF({"cf1"}, options);
|
||||
|
||||
options.max_open_files = -1;
|
||||
OpenSecondaryWithColumnFamilies({"cf1"}, options);
|
||||
ASSERT_OK(Put(0, "key", "value"));
|
||||
ASSERT_OK(Put(1, "key", "value"));
|
||||
ASSERT_OK(db_->FlushWAL(/*sync=*/true));
|
||||
ASSERT_OK(db_secondary_->TryCatchUpWithPrimary());
|
||||
|
||||
for (ColumnFamilyHandle* handle : handles_secondary_) {
|
||||
ASSERT_OK(db_secondary_->DestroyColumnFamilyHandle(handle));
|
||||
}
|
||||
handles_secondary_.clear();
|
||||
ASSERT_OK(db_secondary_->Close());
|
||||
db_secondary_.reset();
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, ExplicitCloseWaitsForRetiredReadView) {
|
||||
Options options;
|
||||
options.env = env_;
|
||||
CreateAndReopenWithCF({"cf1"}, options);
|
||||
|
||||
options.max_open_files = -1;
|
||||
OpenSecondaryWithColumnFamilies({"cf1"}, options);
|
||||
|
||||
std::mutex state_mutex;
|
||||
std::condition_variable state_cv;
|
||||
bool reader_loaded = false;
|
||||
bool allow_reader = false;
|
||||
bool close_waiting = false;
|
||||
bool close_reclaiming = false;
|
||||
bool allow_close = false;
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::NewIterators:AfterLoadReadView", [&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
reader_loaded = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_reader; });
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::DrainRetiredReadViews:Wait", [&](void* /*arg*/) {
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
close_waiting = true;
|
||||
state_cv.notify_all();
|
||||
});
|
||||
SyncPoint::GetInstance()->SetCallBack(
|
||||
"DBImplSecondary::DrainRetiredReadViews:AfterWait", [&](void* /*arg*/) {
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
close_reclaiming = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return allow_close; });
|
||||
});
|
||||
SyncPoint::GetInstance()->EnableProcessing();
|
||||
|
||||
Status reader_status;
|
||||
std::vector<Iterator*> iterators;
|
||||
port::Thread reader_thread([&]() {
|
||||
reader_status = db_secondary_->NewIterators(ReadOptions(),
|
||||
handles_secondary_, &iterators);
|
||||
});
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return reader_loaded; });
|
||||
}
|
||||
|
||||
Status close_status;
|
||||
port::Thread close_thread([&]() { close_status = db_secondary_->Close(); });
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state_mutex);
|
||||
state_cv.wait(lock, [&]() { return close_waiting; });
|
||||
allow_reader = true;
|
||||
state_cv.notify_all();
|
||||
state_cv.wait(lock, [&]() { return close_reclaiming; });
|
||||
}
|
||||
|
||||
reader_thread.join();
|
||||
const size_t num_iterators = iterators.size();
|
||||
for (Iterator* iterator : iterators) {
|
||||
delete iterator;
|
||||
}
|
||||
iterators.clear();
|
||||
Status destroy_handles_status;
|
||||
for (ColumnFamilyHandle* handle : handles_secondary_) {
|
||||
const Status s = db_secondary_->DestroyColumnFamilyHandle(handle);
|
||||
if (!s.ok() && destroy_handles_status.ok()) {
|
||||
destroy_handles_status = s;
|
||||
}
|
||||
}
|
||||
handles_secondary_.clear();
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(state_mutex);
|
||||
allow_close = true;
|
||||
state_cv.notify_all();
|
||||
}
|
||||
close_thread.join();
|
||||
SyncPoint::GetInstance()->DisableProcessing();
|
||||
SyncPoint::GetInstance()->ClearAllCallBacks();
|
||||
|
||||
ASSERT_OK(reader_status);
|
||||
ASSERT_EQ(2U, num_iterators);
|
||||
ASSERT_OK(destroy_handles_status);
|
||||
ASSERT_OK(close_status);
|
||||
db_secondary_.reset();
|
||||
}
|
||||
|
||||
TEST_F(DBSecondaryTest, SwitchManifest) {
|
||||
@@ -2395,6 +2832,7 @@ TEST_F(DBSecondaryTest, NewIteratorsPerColumnFamilyOptionsConsistentView) {
|
||||
ASSERT_OK(iterators[i]->status());
|
||||
ASSERT_TRUE(iterators[i]->Valid());
|
||||
ASSERT_EQ(expected_values[i], iterators[i]->value());
|
||||
ASSERT_TRUE(iterators[i]->Refresh().IsNotSupported());
|
||||
delete iterators[i];
|
||||
}
|
||||
|
||||
|
||||
@@ -1204,17 +1204,6 @@ class CfConsistencyStressTest : public StressTest {
|
||||
SharedState* shared = thread->shared;
|
||||
assert(shared);
|
||||
|
||||
if (secondary_db_) {
|
||||
status = secondary_db_->TryCatchUpWithPrimary();
|
||||
if (!status.ok()) {
|
||||
fprintf(stderr, "TryCatchUpWithPrimary: %s\n",
|
||||
status.ToString().c_str());
|
||||
shared->SetShouldStopTest();
|
||||
assert(false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const auto checksum_column_family = [](Iterator* iter,
|
||||
uint32_t* checksum) -> Status {
|
||||
assert(nullptr != checksum);
|
||||
@@ -1238,46 +1227,78 @@ class CfConsistencyStressTest : public StressTest {
|
||||
// `FLAGS_rate_limit_user_ops` to avoid slowing any validation.
|
||||
ReadOptions ropts(FLAGS_verify_checksum, true);
|
||||
ropts.total_order_seek = true;
|
||||
if (nullptr == secondary_db_ || FLAGS_auto_refresh_iterator_with_snapshot) {
|
||||
if (nullptr == secondary_db_) {
|
||||
ropts.snapshot = snapshot_guard.snapshot();
|
||||
ropts.auto_refresh_iterator_with_snapshot = true;
|
||||
}
|
||||
uint32_t crc = 0;
|
||||
{
|
||||
// Compute crc for all key-values of default column family.
|
||||
std::unique_ptr<Iterator> it(db_ptr->NewIterator(ropts));
|
||||
status = checksum_column_family(it.get(), &crc);
|
||||
if (!status.ok()) {
|
||||
fprintf(stderr, "Computing checksum of default cf: %s\n",
|
||||
status.ToString().c_str());
|
||||
assert(false);
|
||||
}
|
||||
|
||||
std::vector<Iterator*> raw_iterators;
|
||||
Status catch_up_status;
|
||||
if (secondary_db_) {
|
||||
// The separate thread intentionally overlaps catch-up with
|
||||
// NewIterators(). Fault injection is disabled separately because its
|
||||
// state is thread-local.
|
||||
port::Thread catch_up_thread([&]() {
|
||||
if (db_fault_injection_fs_) {
|
||||
db_fault_injection_fs_->DisableAllThreadLocalErrorInjection();
|
||||
}
|
||||
catch_up_status = secondary_db_->TryCatchUpWithPrimary();
|
||||
if (db_fault_injection_fs_) {
|
||||
db_fault_injection_fs_->EnableAllThreadLocalErrorInjection();
|
||||
}
|
||||
});
|
||||
status = db_ptr->NewIterators(ropts, cfhs, &raw_iterators);
|
||||
catch_up_thread.join();
|
||||
} else {
|
||||
status = db_ptr->NewIterators(ropts, cfhs, &raw_iterators);
|
||||
}
|
||||
// Since we currently intentionally disallow reading from the secondary
|
||||
// instance with snapshot, we cannot achieve cross-cf consistency if WAL is
|
||||
// enabled because there is no guarantee that secondary instance replays
|
||||
// the primary's WAL to a consistent point where all cfs have the same
|
||||
// data.
|
||||
if (status.ok() && FLAGS_disable_wal) {
|
||||
uint32_t tmp_crc = 0;
|
||||
for (ColumnFamilyHandle* cfh : cfhs) {
|
||||
if (cfh == db_ptr->DefaultColumnFamily()) {
|
||||
continue;
|
||||
}
|
||||
std::unique_ptr<Iterator> it(db_ptr->NewIterator(ropts, cfh));
|
||||
status = checksum_column_family(it.get(), &tmp_crc);
|
||||
if (!status.ok() || tmp_crc != crc) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::unique_ptr<Iterator>> iterators;
|
||||
iterators.reserve(raw_iterators.size());
|
||||
for (Iterator* iterator : raw_iterators) {
|
||||
iterators.emplace_back(iterator);
|
||||
}
|
||||
|
||||
if (secondary_db_ && !catch_up_status.ok()) {
|
||||
fprintf(stderr, "TryCatchUpWithPrimary: %s\n",
|
||||
catch_up_status.ToString().c_str());
|
||||
shared->SetShouldStopTest();
|
||||
assert(false);
|
||||
return;
|
||||
}
|
||||
if (!status.ok()) {
|
||||
fprintf(stderr, "NewIterators: %s\n", status.ToString().c_str());
|
||||
shared->SetShouldStopTest();
|
||||
assert(false);
|
||||
return;
|
||||
}
|
||||
|
||||
// With WAL enabled, secondary MANIFEST recovery can expose flushed state
|
||||
// beyond the point reached by WAL replay. NewIterators() keeps one
|
||||
// published view but cannot make those column families logically
|
||||
// consistent.
|
||||
const size_t num_cfs_to_verify =
|
||||
FLAGS_disable_wal ? iterators.size()
|
||||
: std::min<size_t>(1, iterators.size());
|
||||
uint32_t expected_crc = 0;
|
||||
for (size_t i = 0; i < num_cfs_to_verify; ++i) {
|
||||
uint32_t crc = 0;
|
||||
status = checksum_column_family(iterators[i].get(), &crc);
|
||||
if (!status.ok()) {
|
||||
fprintf(stderr, "status: %s\n", status.ToString().c_str());
|
||||
fprintf(stderr, "Computing checksum of cf %zu: %s\n", i,
|
||||
status.ToString().c_str());
|
||||
shared->SetShouldStopTest();
|
||||
assert(false);
|
||||
} else if (tmp_crc != crc) {
|
||||
fprintf(stderr, "tmp_crc=%" PRIu32 " crc=%" PRIu32 "\n", tmp_crc, crc);
|
||||
return;
|
||||
}
|
||||
if (i == 0) {
|
||||
expected_crc = crc;
|
||||
} else if (FLAGS_disable_wal && crc != expected_crc) {
|
||||
fprintf(stderr, "cf %zu crc=%" PRIu32 " expected=%" PRIu32 "\n", i, crc,
|
||||
expected_crc);
|
||||
shared->SetShouldStopTest();
|
||||
assert(false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+10
-2
@@ -1267,7 +1267,11 @@ class DB {
|
||||
}
|
||||
// Returns iterators from a consistent database state across multiple
|
||||
// column families. Iterators are heap allocated and need to be deleted
|
||||
// before the db is destroyed
|
||||
// before the db is destroyed. For a secondary DB, the iterators use one
|
||||
// published read view without waiting for a concurrent catch-up and do not
|
||||
// support Iterator::Refresh(). This does not strengthen secondary recovery's
|
||||
// consistency guarantees; WAL replay can stop before all column families
|
||||
// reach the same sequence.
|
||||
virtual Status NewIterators(
|
||||
const ReadOptions& options,
|
||||
const std::vector<ColumnFamilyHandle*>& column_families,
|
||||
@@ -1279,7 +1283,11 @@ class DB {
|
||||
// be null or point to the same Snapshot, and all tailing settings must be
|
||||
// the same. Other ReadOptions fields may differ between column families.
|
||||
// Iterators are heap allocated and need to be deleted before the db is
|
||||
// destroyed.
|
||||
// destroyed. For a secondary DB, the iterators use one published read view
|
||||
// without waiting for a concurrent catch-up and do not support
|
||||
// Iterator::Refresh(). This does not strengthen secondary recovery's
|
||||
// consistency guarantees; WAL replay can stop before all column families
|
||||
// reach the same sequence.
|
||||
virtual Status NewIterators(
|
||||
const std::vector<ReadOptions>& /*read_options*/,
|
||||
const std::vector<ColumnFamilyHandle*>& /*column_families*/,
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
// minor or major version number planned for release.
|
||||
#define ROCKSDB_MAJOR 11
|
||||
#define ROCKSDB_MINOR 11
|
||||
#define ROCKSDB_PATCH 0
|
||||
#define ROCKSDB_PATCH 1
|
||||
|
||||
// Make it easy to do conditional compilation based on version checks, i.e.
|
||||
// #if ROCKSDB_VERSION_GE(4, 5, 6)
|
||||
|
||||
@@ -1923,6 +1923,13 @@ def finalize_and_sanitize(src_params):
|
||||
dest_params["cache_size"] = 65536
|
||||
dest_params["use_write_buffer_manager"] = 0
|
||||
|
||||
# `db_stress` rejects either verification knob when `skip_verifydb` is set.
|
||||
# Must stay last: sanitization above can turn a verification knob back on
|
||||
# after `skip_verifydb` was decided.
|
||||
if dest_params.get("skip_verifydb", 0):
|
||||
dest_params["continuous_verification_interval"] = 0
|
||||
dest_params["verify_db_one_in"] = 0
|
||||
|
||||
return dest_params
|
||||
|
||||
|
||||
|
||||
@@ -343,6 +343,24 @@ class DBCrashTestTest(unittest.TestCase):
|
||||
command,
|
||||
)
|
||||
|
||||
def test_skip_verifydb_disables_verification_knobs(self):
|
||||
db_crashtest = self.load_db_crashtest()
|
||||
|
||||
unfinalized = db_crashtest.gen_cmd_params(
|
||||
self.build_mode_args(
|
||||
"blackbox",
|
||||
cf_consistency=True,
|
||||
test_best_efforts_recovery=True,
|
||||
)
|
||||
)
|
||||
unfinalized["db"] = self.test_tmpdir
|
||||
params = db_crashtest.finalize_and_sanitize(unfinalized)
|
||||
|
||||
# `db_stress` rejects either knob when `skip_verifydb` is set.
|
||||
self.assertEqual(1, params["skip_verifydb"])
|
||||
self.assertEqual(0, params["continuous_verification_interval"])
|
||||
self.assertEqual(0, params["verify_db_one_in"])
|
||||
|
||||
def test_stress_diagnostics_disabled_does_not_allocate_dir(self):
|
||||
db_crashtest = self.load_db_crashtest()
|
||||
params = {
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
`ExternalTableReader::MultiGet()` is now used by RocksDB's `DB::MultiGet()` path. Previously, the external table adapter used `TableReader`'s fallback, which called `ExternalTableReader::Get()` separately for each key.
|
||||
@@ -1 +0,0 @@
|
||||
`DB::OpenAndCompact()` now requires `output_directory` to be a simple name for a new subdirectory when the new `use_session_tmp_dir_for_remote_compaction` DB option is enabled. The option places that subdirectory under the canonical `<dbname>/session_tmp/` root, and `DB::Open()` deletes stale contents under the root after recovery. The disabled default preserves the existing `<dbname>/<output_directory>` layout without automatic cleanup. Previously, remote-compaction output directories were created directly under `<dbname>`, which made it challenging to safely identify and delete obsolete files.
|
||||
@@ -1 +0,0 @@
|
||||
Several block-based SST table properties now better support compression benchmarking and optimization: `uncompressed_data_size` is now persisted in the file (previously it was computed only at write time and read back as 0); a new `lsm_info_at_creation` property records the LSM level and "bottommost" status at the file's creation, distinguishing "not applicable" (e.g. SstFileWriter files) from "unknown" (older files); and the `compression_options` property now records the full compression manager configuration instead of only its name.
|
||||
@@ -0,0 +1 @@
|
||||
Fixed RocksDB builds on big-endian architectures such as s390x by disabling the optional persistent fault-injection log, whose binary format is little-endian-only.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a usually-minor performance bug where `Cache::ApplyToAllEntries()` used single-entry batches instead of the intended 256-entry default for `ApplyToAllEntriesOptions::average_entries_per_lock`. All of the Cache implementations (LRU and HCC) benefit from the fix, but the disfavored LRU cache was especially impacted by repeated mutex acquisition and release.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed stats-disabled coroutine and asynchronous reads inheriting enabled TLS statistics state when interleaved with stats-enabled reads on the same executor. Each read now uses its captured per-call configuration, while enabled request counters remain isolated across suspensions.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) requested with `OpenAndCompactOptions::allow_resumption=true` but for which resumption was internally disabled (incompatible with `paranoid_file_checks` or output-iteration verification) could fail a retried attempt, because the output directory was not cleaned before starting fresh and output files left by a previously interrupted attempt collided with the reused file numbers.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a bug where a remote compaction (`CompactionService` / `DB::OpenAndCompact`) could fail when the primary rotated its MANIFEST while the worker was opening the primary's directory read-only. The open reads `CURRENT`, and the primary publishes a new MANIFEST by renaming a freshly written `CURRENT` over the old one; on file systems without POSIX read-after-unlink semantics (remote/object stores) the replaced object is then reported as gone (`PathNotFound`/`NotFound`, or `TryAgain` for the MANIFEST equivalent) instead of remaining readable. `DB::OpenAndCompact()` now retries the secondary open a bounded, configurable number of times for those transient statuses, so a concurrent rotation no longer fails the compaction (which, for a `CompactionService` that does not fall back to a local compaction, became a background error on the primary).
|
||||
@@ -1 +0,0 @@
|
||||
Fixed secondary DB `NewIterators()` returning iterators from different database states when `TryCatchUpWithPrimary()` runs concurrently.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a bug where a `SeekToFirst()` on an iterator previously used for a `Seek()` could end the scan early and silently omit keys with `ReadOptions::async_io`, or otherwise run with readahead effectively disabled. Affects scans using `ReadOptions::prefix_same_as_start` together with `ReadOptions::auto_readahead_size` (enabled by default); memory-mapped reads are unaffected.
|
||||
@@ -1 +0,0 @@
|
||||
External tables now support DB-assigned global sequence numbers when `IngestExternalFileOptions::allow_global_seqno` is enabled, allowing overlapping external tables to be ingested across multiple levels when compactions are disabled. This includes the deprecated `write_global_seqno` path when the reader supplies the properties block's file offset.
|
||||
@@ -1 +0,0 @@
|
||||
Added checksum handoff support for SST file writes when `BlockBasedTableOptions::checksum` is `kCRC32c`, allowing block and trailer CRC32C checksums to be passed from the table builder to the file writer for buffered-data validation and filesystem in-flight write protection.
|
||||
@@ -1 +0,0 @@
|
||||
Enhanced `sst_dump --command=recompress` to be more powerful for benchmarking, including through automation with a new `--json` option for machine-readable output.
|
||||
@@ -1 +0,0 @@
|
||||
`DB::MultiGetEntityLazy()` now reads all keys of a batch under a single shared per-column-family SuperVersion pin, and a single `LazyWideColumnsBatch::MultiResolve()` coalesces the resulting blob reads across keys: one read per blob file (and one per SST for embedded blobs), for whole-column and byte-range reads alike, instead of reading each key's blobs separately.
|
||||
@@ -1 +0,0 @@
|
||||
Changed the experimental async read statistics API by removing `DB::AsyncCallback::EnableStats()` and changing `OnComplete(const PerfContext*, const IOStatsContext*)` to `OnComplete()`. Callers must configure TLS statistics before each async or coroutine read. `DB::GetAsync()` and `DB::MultiGetAsync()` callbacks must copy per-operation statistics from TLS during `OnComplete()`; coroutine users must copy them from inside a `CoroStackableDB` hook before suspending again. Each operation leaves TLS statistics collection disabled.
|
||||
@@ -1 +0,0 @@
|
||||
Added `Compressor::MaybeOverrideCompressionOptions()` (EXPERIMENTAL), an omnibus hook letting a custom compressor adjust the subset of `CompressionOptions` that the SST builder applies rather than the compressor itself: `parallel_threads`, `max_compressed_bytes_per_kb`, `auto_skip`, and `auto_skip_min_sample_every` (and potentially more in the future). This deprecates `Compressor::GetRecommendedParallelThreads()`, which it subsumes; the default implementation still honors an overridden `GetRecommendedParallelThreads()` for compatibility.
|
||||
@@ -1 +0,0 @@
|
||||
Added experimental `CoroDB::CoGet()` and `CoroDB::CoMultiGet()` overloads for point reads from an `SstFileReader`.
|
||||
@@ -1 +0,0 @@
|
||||
Added experimental `DB::GetWithMetadata()` and `DB::MultiGetWithMetadata()` APIs with opt-in output metadata for key timestamps and explicit-snapshot newer-version detection.
|
||||
@@ -1 +0,0 @@
|
||||
Added a `DB::NewIterators()` overload that accepts parallel vectors of `ReadOptions` and column family handles, allowing each returned iterator to use independent read settings while sharing a consistent database state. The vectors must have the same size, all snapshots must be null or identical, and all `tailing` settings must match.
|
||||
@@ -1 +0,0 @@
|
||||
Added `ReadOptions::custom_context`, an opaque, non-owning pointer passed through table reads so external table implementations can consume application-defined read options.
|
||||
@@ -26,6 +26,7 @@
|
||||
#include "env/io_posix.h"
|
||||
#include "monitoring/thread_status_util.h"
|
||||
#include "port/lang.h"
|
||||
#include "port/port.h"
|
||||
#include "port/stack_trace.h"
|
||||
#include "rocksdb/env.h"
|
||||
#include "rocksdb/io_status.h"
|
||||
@@ -79,7 +80,9 @@ uint8_t DetailPayloadSize(const InjectedErrorLog::TaggedEntryDetail& detail) {
|
||||
|
||||
int OpenInjectedErrorLogFile(const std::string& path) {
|
||||
#ifndef OS_WIN
|
||||
if (path.empty()) {
|
||||
// Entry is written in native layout, while the decoder expects little-endian
|
||||
// fields. Keep the optional log disabled on unsupported architectures.
|
||||
if (!port::kLittleEndian || path.empty()) {
|
||||
return -1;
|
||||
}
|
||||
int flags = O_WRONLY | O_CREAT | O_TRUNC;
|
||||
|
||||
@@ -44,6 +44,8 @@ namespace ROCKSDB_NAMESPACE {
|
||||
// A binary log that records injected errors directly to a file.
|
||||
// Thread-safe for concurrent writes. The log file can be flushed on exit or
|
||||
// from a signal handler so records survive clean exits and crash paths.
|
||||
// Disabled on big-endian platforms because the decoder expects little-endian
|
||||
// fields in the native Entry layout.
|
||||
class InjectedErrorLog {
|
||||
public:
|
||||
static constexpr size_t kMaxOpNameLen = 32;
|
||||
@@ -86,12 +88,13 @@ class InjectedErrorLog {
|
||||
char op_name[kMaxOpNameLen];
|
||||
char file_name[kMaxFileNameLen];
|
||||
char status_message[kMaxStatusMessageLen];
|
||||
// Explicit tail padding so the size does not depend on the alignment of
|
||||
// uint64_t (4 bytes on i386, 8 bytes on most other targets).
|
||||
uint8_t padding[4];
|
||||
};
|
||||
|
||||
static_assert(sizeof(Entry) == 256,
|
||||
"Injected error log entry size must stay stable");
|
||||
static_assert(port::kLittleEndian,
|
||||
"Injected error log requires little-endian platforms");
|
||||
|
||||
explicit InjectedErrorLog(const std::string& path);
|
||||
~InjectedErrorLog();
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "port/port.h"
|
||||
#include "test_util/testharness.h"
|
||||
|
||||
namespace ROCKSDB_NAMESPACE {
|
||||
@@ -79,6 +80,11 @@ class CloseCountingWritableFile : public FSWritableFileOwnerWrapper {
|
||||
// the expected fixed-width fields. The test writes one entry, flushes the
|
||||
// file, and decodes the raw bytes back into an entry struct.
|
||||
TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path = test::PerThreadDBPath("injected_error_log_basic.bin");
|
||||
{
|
||||
InjectedErrorLog log(path);
|
||||
@@ -114,6 +120,11 @@ TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
|
||||
// to the old in-memory ring size. The test writes more than 1,000 entries and
|
||||
// checks that both the first and last ones are still present in the file.
|
||||
TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path =
|
||||
test::PerThreadDBPath("injected_error_log_all_entries.bin");
|
||||
{
|
||||
@@ -144,6 +155,11 @@ TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
|
||||
// shared file. The test has several threads emit records concurrently and then
|
||||
// checks the raw file size matches the total number of writes.
|
||||
TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path = test::PerThreadDBPath("injected_error_log_concurrent.bin");
|
||||
constexpr int kNumThreads = 4;
|
||||
constexpr int kRecordsPerThread = 200;
|
||||
@@ -180,6 +196,11 @@ TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
|
||||
// in the fixed-width record. The test logs a path longer than the file-name
|
||||
// field and checks the stored sample matches the expected tail bytes.
|
||||
TEST_F(InjectedErrorLogTest, LongFileNameKeepsSuffix) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path =
|
||||
test::PerThreadDBPath("injected_error_log_suffix_truncation.bin");
|
||||
const std::string long_file_name =
|
||||
|
||||
Reference in new issue
Block a user