mirror of
https://github.com/facebook/rocksdb.git
synced 2026-11-06 11:02:36 -05:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a6ac302081 | ||
|
|
108d3bf062 | ||
|
|
d6cd1fc884 | ||
|
|
dfd356fa10 | ||
|
|
60d6f470e8 | ||
|
|
ea16e73a71 | ||
|
|
4c5486efec |
No files matched your search
@@ -308,6 +308,7 @@ build (and where noted, run tests) across:
|
||||
| Axis | Must support |
|
||||
|------|--------------|
|
||||
| OS | Linux (x86_64 + ARM), macOS, Windows |
|
||||
| Architecture | Little-endian and big-endian targets, including Linux on s390x |
|
||||
| Compiler | GCC, Clang (libstdc++ **and** libc++), AppleClang, **MSVC (VS2022)**, MinGW (Linux cross-compile, build-only, no gflags) |
|
||||
| Build system | Make, CMake, and BUCK (internal) -- keep all in sync (see "Build system" above) |
|
||||
| Config | release (`-fno-rtti`), `ASSERT_STATUS_CHECKED`, ASAN/UBSAN/TSAN, folly, unity build, JNI/Java |
|
||||
@@ -322,6 +323,12 @@ abstractions; otherwise guard with `#ifdef OS_WIN` (POSIX `<unistd.h>` ->
|
||||
Windows `<process.h>`). Because libc++ is also tested, include what you use
|
||||
rather than relying on libstdc++ transitive includes.
|
||||
|
||||
Do not use `static_assert` to reject a supported architecture for an optional
|
||||
feature. Check an existing `port::` capability such as `port::kLittleEndian`
|
||||
and disable only the unsupported feature so the rest of RocksDB still builds.
|
||||
Use a portable representation when the feature itself must work across
|
||||
architectures.
|
||||
|
||||
### Unit Test
|
||||
* After all of the unit tests are added, review them and try to extract common
|
||||
reusable utility functions to reduce code duplication due to copy past between
|
||||
|
||||
+22
@@ -1,6 +1,28 @@
|
||||
# Rocksdb Change Log
|
||||
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
|
||||
|
||||
## 11.10.0 (08/28/2026)
|
||||
### New Features
|
||||
* Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
|
||||
* A new experimental feature spreads out periodic (time-based) compaction trigger times across a fleet of similarly-configured DBs to avoid recurring "thundering herd" triggers, especially with remote compaction. See the new experimental DB option `periodic_compaction_phase_recovery_percent`.
|
||||
|
||||
### Public API Changes
|
||||
* `IngestExternalFileArg::atomic_replace_range` now supports a range that only partially overlaps existing files, instead of failing. Requires universal compaction and a table factory that supports range deletion; other configurations still fail on partial overlap. The range is also now specified as half-open (`start` inclusive, `limit` exclusive), resolving a previously documented ambiguity in the upper bound.
|
||||
* Added an immutable `external_table_config` option that is persisted in OPTIONS files and passed to external table factories during bootstrap.
|
||||
|
||||
### Bug Fixes
|
||||
* Fixed a liveness bug where RocksDB could remain indefinitely in recovery from a soft background error when concurrent writes continuously generated normal flush requests.
|
||||
* Fixed a race in online error recovery where a concurrent higher-severity I/O error could leave a stale retry-flush plan active, prevent the required WAL switch, and create duplicate internal WAL history detected on DB reopen.
|
||||
* Fix "bottommost files" infinite compaction loops (`kBottommostFiles`). This happens with the unsupported combination of leveled compaction and `preserve_internal_time_seconds` / `preclude_last_level_data_seconds`, and in an edge case with user-defined timestamps when `full_history_ts_low` is unset. The underlying logic is now more unified to reduce risk of a similar divergence in logic in the future.
|
||||
* Fixed `reuse_manifest_on_open` to create a fresh MANIFEST instead of appending after recovery sees an incomplete atomic group at the MANIFEST tail.
|
||||
* Starting remote compaction (`CompactionService` / `DB::OpenAndCompact`) is now more efficient and more reliable, by avoiding unnecessary file operations (latency) while using more accurate consistency / precondition checks, both avoiding some spurious failures and rejecting more legitimate inconsistencies. New DBOption `remote_compaction_manifest_floor` can be disabled in case of unexpected problems with the change.
|
||||
* Fixed unbounded retention of recovered transaction state by a long-lived secondary opened with `allow_2pc`.
|
||||
* Fixed stale reads on secondary instances, where entries replayed from a WAL that the primary had already flushed could shadow the newer values in the flushed files indefinitely. Also fixed a case where a secondary could stop returning such a key altogether while the primary's flushed file holding it was unreadable.
|
||||
* Fixed subset checkpoints to discard tolerated MANIFEST tail corruption before appending column-family drop records.
|
||||
|
||||
### Performance Improvements
|
||||
* Improved compaction CPU efficiency for column families using an `SstPartitioner` (such as the built-in `SstPartitionerFixedPrefix`): when the partitioner can express its next boundary as a key prefix via the new `SstPartitioner::ShouldPartitionByPrefix()` API, compaction now checks that cached prefix inline instead of copying every user key and making a virtual `ShouldPartition()` call for each key-value written to an SST.
|
||||
|
||||
## 11.9.0 (08/14/2026)
|
||||
### New Features
|
||||
* Added an integrated and improved compression auto-skip feature (new CompressionOptions, still experimental) that saves CPU by skipping compression when and where the overall ratio is inadequate. This improves on the previous auto-skip compression manager by (a) taking achieved compression ratios into account, (b) using a continuously evolving payoff estimate, and (c) combining natively with parallel compression.
|
||||
|
||||
+22
-12
@@ -21,6 +21,7 @@
|
||||
#include "rocksdb/merge_operator.h"
|
||||
#include "rocksdb/perf_context.h"
|
||||
#include "rocksdb/table.h"
|
||||
#include "rocksdb/utilities/checkpoint.h"
|
||||
#include "rocksdb/utilities/debug.h"
|
||||
#include "table/block_based/block_based_table_reader.h"
|
||||
#include "table/block_based/block_builder.h"
|
||||
@@ -356,10 +357,11 @@ TEST_F(DBBasicTest,
|
||||
}
|
||||
|
||||
TEST_F(DBBasicTest,
|
||||
OptimizeManifestForRecoveryIgnoresOptionsFilesForNextFileNumber) {
|
||||
OptimizeManifestForRecoveryKeepsCurrentOptionsFileWithStaleOptions) {
|
||||
Options options = CurrentOptions();
|
||||
options.create_if_missing = true;
|
||||
options.optimize_manifest_for_recovery = true;
|
||||
options.avoid_unnecessary_blocking_io = false;
|
||||
DestroyAndReopen(options);
|
||||
ASSERT_OK(Put("k", "v"));
|
||||
ASSERT_OK(Flush());
|
||||
@@ -368,20 +370,28 @@ TEST_F(DBBasicTest,
|
||||
dbfull()->GetVersionSet()->current_next_file_number();
|
||||
Close();
|
||||
|
||||
const uint64_t options_number = next_before_close + 100;
|
||||
ASSERT_OK(WriteStringToFile(env_, "" /*data*/,
|
||||
dbname_ + "/" + OptionsFileName(options_number),
|
||||
/*should_sync=*/true));
|
||||
ASSERT_OK(WriteStringToFile(env_, "" /*data*/,
|
||||
TempOptionsFileName(dbname_, options_number + 1),
|
||||
/*should_sync=*/true));
|
||||
const uint64_t stale_options_number = next_before_close + 100;
|
||||
ASSERT_OK(WriteStringToFile(
|
||||
env_, "" /*data*/, dbname_ + "/" + OptionsFileName(stale_options_number),
|
||||
/*should_sync=*/true));
|
||||
ASSERT_OK(WriteStringToFile(
|
||||
env_, "" /*data*/,
|
||||
dbname_ + "/" + OptionsFileName(stale_options_number + 1),
|
||||
/*should_sync=*/true));
|
||||
|
||||
RecoveryOptimizationCounters counters;
|
||||
counters.Install();
|
||||
Reopen(options);
|
||||
counters.Uninstall();
|
||||
const uint64_t current_options_number =
|
||||
dbfull()->GetVersionSet()->options_file_number();
|
||||
ASSERT_GT(current_options_number, 0u);
|
||||
ASSERT_OK(env_->FileExists(OptionsFileName(dbname_, current_options_number)));
|
||||
|
||||
ASSERT_EQ(1, counters.next_file_number.load());
|
||||
const std::string checkpoint_dir = dbname_ + "_checkpoint";
|
||||
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
|
||||
Checkpoint* checkpoint = nullptr;
|
||||
ASSERT_OK(Checkpoint::Create(db_.get(), &checkpoint));
|
||||
std::unique_ptr<Checkpoint> checkpoint_guard(checkpoint);
|
||||
ASSERT_OK(checkpoint_guard->CreateCheckpoint(checkpoint_dir));
|
||||
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
|
||||
ASSERT_EQ("v", Get("k"));
|
||||
}
|
||||
|
||||
|
||||
@@ -1211,9 +1211,7 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
|
||||
// Use >= so a crashed-mid-allocation file at exactly
|
||||
// prev_next_file_number triggers the advance -- otherwise the next
|
||||
// NewFileNumber() would collide with it.
|
||||
const bool must_reserve_file_number =
|
||||
type != kOptionsFile && type != kTempFile;
|
||||
if (must_reserve_file_number && number >= prev_next_file_number) {
|
||||
if (number >= prev_next_file_number) {
|
||||
on_disk_file_advanced_counter = true;
|
||||
}
|
||||
largest_file_number = std::max(largest_file_number, number);
|
||||
|
||||
@@ -5732,9 +5732,7 @@ void StressTest::RecordManifestStateBeforeReopen() {
|
||||
if (reuse_manifest && optimize_manifest) {
|
||||
// Check if ALL conditions for complete avoidance are met.
|
||||
// If so, use STRICT mode where failures are fatal.
|
||||
const bool no_fault_injection = !NeedsFaultInjection();
|
||||
const bool no_manifest_writes_expected =
|
||||
no_fault_injection &&
|
||||
FLAGS_avoid_flush_during_recovery && // No flush during recovery
|
||||
!FLAGS_write_dbid_to_manifest; // No DB_ID write on open
|
||||
// Note: avoid_flush_during_shutdown is NOT required for STRICT mode.
|
||||
|
||||
@@ -429,7 +429,8 @@ class StressTest {
|
||||
// - Not in best_efforts_recovery mode
|
||||
// - avoid_flush_during_recovery=true (no flush during recovery)
|
||||
// - write_dbid_to_manifest=0 (no DB_ID write on open)
|
||||
// - all fault injection flags are disabled
|
||||
// - metadata_write_fault_one_in=0 (no fault injection)
|
||||
// - open_metadata_write_fault_one_in=0 (no fault injection)
|
||||
// - MANIFEST not corrupted, not at size limit
|
||||
// Note: avoid_flush_during_shutdown is NOT required. If it leaves data
|
||||
// in WAL but avoid_flush_during_recovery=true prevents flushing it,
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a liveness bug where RocksDB could remain indefinitely in recovery from a soft background error when concurrent writes continuously generated normal flush requests.
|
||||
@@ -0,0 +1 @@
|
||||
Fixed RocksDB builds on big-endian architectures such as s390x by disabling the optional persistent fault-injection log, whose binary format is little-endian-only.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed a race in online error recovery where a concurrent higher-severity I/O error could leave a stale retry-flush plan active, prevent the required WAL switch, and create duplicate internal WAL history detected on DB reopen.
|
||||
@@ -1 +0,0 @@
|
||||
Fix "bottommost files" infinite compaction loops (`kBottommostFiles`). This happens with the unsupported combination of leveled compaction and `preserve_internal_time_seconds` / `preclude_last_level_data_seconds`, and in an edge case with user-defined timestamps when `full_history_ts_low` is unset. The underlying logic is now more unified to reduce risk of a similar divergence in logic in the future.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed `reuse_manifest_on_open` to create a fresh MANIFEST instead of appending after recovery sees an incomplete atomic group at the MANIFEST tail.
|
||||
@@ -1 +0,0 @@
|
||||
Starting remote compaction (`CompactionService` / `DB::OpenAndCompact`) is now more efficient and more reliable, by avoiding unnecessary file operations (latency) while using more accurate consistency / precondition checks, both avoiding some spurious failures and rejecting more legitimate inconsistencies. New DBOption `remote_compaction_manifest_floor` can be disabled in case of unexpected problems with the change.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed unbounded retention of recovered transaction state by a long-lived secondary opened with `allow_2pc`.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed stale reads on secondary instances, where entries replayed from a WAL that the primary had already flushed could shadow the newer values in the flushed files indefinitely. Also fixed a case where a secondary could stop returning such a key altogether while the primary's flushed file holding it was unreadable.
|
||||
@@ -1 +0,0 @@
|
||||
Fixed subset checkpoints to discard tolerated MANIFEST tail corruption before appending column-family drop records.
|
||||
@@ -1 +0,0 @@
|
||||
Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
|
||||
@@ -1 +0,0 @@
|
||||
A new experimental feature spreads out periodic (time-based) compaction trigger times across a fleet of similarly-configured DBs to avoid recurring "thundering herd" triggers, especially with remote compaction. See the new experimental DB option `periodic_compaction_phase_recovery_percent`.
|
||||
-1
@@ -1 +0,0 @@
|
||||
Improved compaction CPU efficiency for column families using an `SstPartitioner` (such as the built-in `SstPartitionerFixedPrefix`): when the partitioner can express its next boundary as a key prefix via the new `SstPartitioner::ShouldPartitionByPrefix()` API, compaction now checks that cached prefix inline instead of copying every user key and making a virtual `ShouldPartition()` call for each key-value written to an SST.
|
||||
@@ -1 +0,0 @@
|
||||
`IngestExternalFileArg::atomic_replace_range` now supports a range that only partially overlaps existing files, instead of failing. Requires universal compaction and a table factory that supports range deletion; other configurations still fail on partial overlap. The range is also now specified as half-open (`start` inclusive, `limit` exclusive), resolving a previously documented ambiguity in the upper bound.
|
||||
@@ -1 +0,0 @@
|
||||
Added an immutable `external_table_config` option that is persisted in OPTIONS files and passed to external table factories during bootstrap.
|
||||
@@ -26,6 +26,7 @@
|
||||
#include "env/io_posix.h"
|
||||
#include "monitoring/thread_status_util.h"
|
||||
#include "port/lang.h"
|
||||
#include "port/port.h"
|
||||
#include "port/stack_trace.h"
|
||||
#include "rocksdb/env.h"
|
||||
#include "rocksdb/io_status.h"
|
||||
@@ -79,7 +80,9 @@ uint8_t DetailPayloadSize(const InjectedErrorLog::TaggedEntryDetail& detail) {
|
||||
|
||||
int OpenInjectedErrorLogFile(const std::string& path) {
|
||||
#ifndef OS_WIN
|
||||
if (path.empty()) {
|
||||
// Entry is written in native layout, while the decoder expects little-endian
|
||||
// fields. Keep the optional log disabled on unsupported architectures.
|
||||
if (!port::kLittleEndian || path.empty()) {
|
||||
return -1;
|
||||
}
|
||||
int flags = O_WRONLY | O_CREAT | O_TRUNC;
|
||||
|
||||
@@ -44,6 +44,8 @@ namespace ROCKSDB_NAMESPACE {
|
||||
// A binary log that records injected errors directly to a file.
|
||||
// Thread-safe for concurrent writes. The log file can be flushed on exit or
|
||||
// from a signal handler so records survive clean exits and crash paths.
|
||||
// Disabled on big-endian platforms because the decoder expects little-endian
|
||||
// fields in the native Entry layout.
|
||||
class InjectedErrorLog {
|
||||
public:
|
||||
static constexpr size_t kMaxOpNameLen = 32;
|
||||
@@ -86,12 +88,13 @@ class InjectedErrorLog {
|
||||
char op_name[kMaxOpNameLen];
|
||||
char file_name[kMaxFileNameLen];
|
||||
char status_message[kMaxStatusMessageLen];
|
||||
// Explicit tail padding so the size does not depend on the alignment of
|
||||
// uint64_t (4 bytes on i386, 8 bytes on most other targets).
|
||||
uint8_t padding[4];
|
||||
};
|
||||
|
||||
static_assert(sizeof(Entry) == 256,
|
||||
"Injected error log entry size must stay stable");
|
||||
static_assert(port::kLittleEndian,
|
||||
"Injected error log requires little-endian platforms");
|
||||
|
||||
explicit InjectedErrorLog(const std::string& path);
|
||||
~InjectedErrorLog();
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "port/port.h"
|
||||
#include "test_util/testharness.h"
|
||||
|
||||
namespace ROCKSDB_NAMESPACE {
|
||||
@@ -79,6 +80,11 @@ class CloseCountingWritableFile : public FSWritableFileOwnerWrapper {
|
||||
// the expected fixed-width fields. The test writes one entry, flushes the
|
||||
// file, and decodes the raw bytes back into an entry struct.
|
||||
TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path = test::PerThreadDBPath("injected_error_log_basic.bin");
|
||||
{
|
||||
InjectedErrorLog log(path);
|
||||
@@ -114,6 +120,11 @@ TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
|
||||
// to the old in-memory ring size. The test writes more than 1,000 entries and
|
||||
// checks that both the first and last ones are still present in the file.
|
||||
TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path =
|
||||
test::PerThreadDBPath("injected_error_log_all_entries.bin");
|
||||
{
|
||||
@@ -144,6 +155,11 @@ TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
|
||||
// shared file. The test has several threads emit records concurrently and then
|
||||
// checks the raw file size matches the total number of writes.
|
||||
TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path = test::PerThreadDBPath("injected_error_log_concurrent.bin");
|
||||
constexpr int kNumThreads = 4;
|
||||
constexpr int kRecordsPerThread = 200;
|
||||
@@ -180,6 +196,11 @@ TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
|
||||
// in the fixed-width record. The test logs a path longer than the file-name
|
||||
// field and checks the stored sample matches the expected tail bytes.
|
||||
TEST_F(InjectedErrorLogTest, LongFileNameKeepsSuffix) {
|
||||
if (!port::kLittleEndian) {
|
||||
ROCKSDB_GTEST_BYPASS(
|
||||
"Persistent injected error log requires little-endian");
|
||||
return;
|
||||
}
|
||||
std::string path =
|
||||
test::PerThreadDBPath("injected_error_log_suffix_truncation.bin");
|
||||
const std::string long_file_name =
|
||||
|
||||
Reference in new issue
Block a user