Compare commits

...
7 Commits
Author SHA1 Message Date
anand76 a6ac302081 Revert "Fix Windows build issue in trie_index_factory.h"
This reverts commit dfd356fa10.
2026-10-15 10:41:13 -07:00
Anand Ananthabhotla 108d3bf062 Make InjectedErrorLog::Entry 256 bytes on 32-bit targets (#15388)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15388

`InjectedErrorLog::Entry` in `utilities/fault_injection_fs.h` is written to the injected error log in its native layout, and a `static_assert` requires `sizeof(Entry) == 256` so the record size stays stable. That assert fails on i386 builds.

Root cause: the fields add up to only 252 bytes. On targets where `alignof(uint64_t)` is 8 (x86_64, aarch64, armv7), the compiler adds 4 bytes of implicit tail padding, so the size is 256. On i386, `uint64_t` members are 4-byte aligned inside a struct, so there is no tail padding and the size is 252.

Fix: add an explicit trailing `uint8_t padding[4]` field so the size no longer depends on `uint64_t` alignment.

Compatibility: on 64-bit targets the new field takes the place of the old implicit tail padding, so every field offset and the 256-byte record layout are unchanged. `Record()` value-initializes the entry (`Entry entry{}`), so the padding bytes are always written as zero. No decoder in the repo depends on the padding bytes.

___

landed-with-radar-review

Differential Revision: D124873369

fbshipit-source-id: f5c6f43ca85013b0e769853ad13b8d4f3f89b958
2026-10-13 10:26:01 -07:00
xingbowang d6cd1fc884 Fix big-endian builds for the fault-injection filesystem (#15297)
Summary:
The persistent injected-error log writes `InjectedErrorLog::Entry` objects using their native in-memory layout, while the log format's consumer expects little-endian multi-byte fields. The existing `static_assert(port::kLittleEndian)` protected that format by rejecting every big-endian build that included the fault-injection filesystem, even though the log is an optional diagnostic.

This change keeps RocksDB and the fault-injection filesystem available on supported big-endian platforms while disabling only the incompatible persistent log. Log-file creation now returns an inactive descriptor when `port::kLittleEndian` is false, which makes `Record()` and `Flush()` remain safe no-ops. The fixed 256-byte entry-size assertion remains in place, and format-dependent tests are bypassed on big-endian targets.

The change also adds a release note and documents big-endian support plus the preferred capability-check pattern for optional architecture-specific features.

## Compatibility and limitations

- Little-endian behavior and the on-disk log layout are unchanged.
- On big-endian platforms, configuring an injected-error log path does not create or populate a log file; the rest of fault injection remains available.
- Windows already leaves this POSIX-backed diagnostic disabled.
- This does not introduce a portable log encoding. Supporting persistent injected-error logs on big-endian platforms would require a format change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/15297

Test Plan:
- `build_tools/rockstest.sh fault_injection_fs_test '--gtest_filter=InjectedErrorLogTest.*'` (little-endian macOS): all 5 tests passed.
- `make check-sources`
- `git diff --check upstream/main...2026_10_01_endian_fix`

Native big-endian build and test execution were not available locally; that path relies on the compile-time `port::kLittleEndian` capability and should be covered by big-endian CI.

Reviewed By: jaykorean

Differential Revision: D122880760

Pulled By: xingbowang

fbshipit-source-id: 59dae9661a47608aa12f41078ad1b3ef0fa7231e
2026-10-13 10:25:48 -07:00
anand76 dfd356fa10 Fix Windows build issue in trie_index_factory.h 2026-10-13 10:24:46 -07:00
Xingbo Wang 60d6f470e8 Update 11.10.0 release date to include the reverted commit 2026-08-28 09:47:49 -07:00
Anand Ananthabhotla ea16e73a71 Revert PR 15106 (Ignore OPTIONS files for manifest recovery advance) (#15158)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15158

Reverts `https://github.com/facebook/rocksdb/pull/15106`, which ignored `OPTIONS-*` and temp files when deciding whether recovery must advance `next_file_number_`.

Crash-test failures show backup/checkpoint failing to copy the current `OPTIONS-*` file after options cleanup, because `DeleteObsoleteOptionsFiles()` ignores `next_file_number_` and `options_file_number_` when choosing which files to keep. The failing commands use `optimize_manifest_for_recovery=1`, so restore the previous conservative file-number reservation behavior while we investigate a narrower fix.

Reviewed By: xingbowang

Differential Revision: D117807052

fbshipit-source-id: 89f8656f045be4ac36d2ec851980573ed83370fe
2026-08-28 04:44:28 -07:00
rocksdb-release-bot 4c5486efec Update HISTORY.md for 11.10.0 release 2026-08-27 16:03:42 -07:00
23 changed files with 85 additions and 34 deletions

No files matched your search

+7
View File
@@ -308,6 +308,7 @@ build (and where noted, run tests) across:
| Axis | Must support |
|------|--------------|
| OS | Linux (x86_64 + ARM), macOS, Windows |
| Architecture | Little-endian and big-endian targets, including Linux on s390x |
| Compiler | GCC, Clang (libstdc++ **and** libc++), AppleClang, **MSVC (VS2022)**, MinGW (Linux cross-compile, build-only, no gflags) |
| Build system | Make, CMake, and BUCK (internal) -- keep all in sync (see "Build system" above) |
| Config | release (`-fno-rtti`), `ASSERT_STATUS_CHECKED`, ASAN/UBSAN/TSAN, folly, unity build, JNI/Java |
@@ -322,6 +323,12 @@ abstractions; otherwise guard with `#ifdef OS_WIN` (POSIX `<unistd.h>` ->
Windows `<process.h>`). Because libc++ is also tested, include what you use
rather than relying on libstdc++ transitive includes.
Do not use `static_assert` to reject a supported architecture for an optional
feature. Check an existing `port::` capability such as `port::kLittleEndian`
and disable only the unsupported feature so the rest of RocksDB still builds.
Use a portable representation when the feature itself must work across
architectures.
### Unit Test
* After all of the unit tests are added, review them and try to extract common
reusable utility functions to reduce code duplication due to copy past between
+22
View File
@@ -1,6 +1,28 @@
# Rocksdb Change Log
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
## 11.10.0 (08/28/2026)
### New Features
* Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
* A new experimental feature spreads out periodic (time-based) compaction trigger times across a fleet of similarly-configured DBs to avoid recurring "thundering herd" triggers, especially with remote compaction. See the new experimental DB option `periodic_compaction_phase_recovery_percent`.
### Public API Changes
* `IngestExternalFileArg::atomic_replace_range` now supports a range that only partially overlaps existing files, instead of failing. Requires universal compaction and a table factory that supports range deletion; other configurations still fail on partial overlap. The range is also now specified as half-open (`start` inclusive, `limit` exclusive), resolving a previously documented ambiguity in the upper bound.
* Added an immutable `external_table_config` option that is persisted in OPTIONS files and passed to external table factories during bootstrap.
### Bug Fixes
* Fixed a liveness bug where RocksDB could remain indefinitely in recovery from a soft background error when concurrent writes continuously generated normal flush requests.
* Fixed a race in online error recovery where a concurrent higher-severity I/O error could leave a stale retry-flush plan active, prevent the required WAL switch, and create duplicate internal WAL history detected on DB reopen.
* Fix "bottommost files" infinite compaction loops (`kBottommostFiles`). This happens with the unsupported combination of leveled compaction and `preserve_internal_time_seconds` / `preclude_last_level_data_seconds`, and in an edge case with user-defined timestamps when `full_history_ts_low` is unset. The underlying logic is now more unified to reduce risk of a similar divergence in logic in the future.
* Fixed `reuse_manifest_on_open` to create a fresh MANIFEST instead of appending after recovery sees an incomplete atomic group at the MANIFEST tail.
* Starting remote compaction (`CompactionService` / `DB::OpenAndCompact`) is now more efficient and more reliable, by avoiding unnecessary file operations (latency) while using more accurate consistency / precondition checks, both avoiding some spurious failures and rejecting more legitimate inconsistencies. New DBOption `remote_compaction_manifest_floor` can be disabled in case of unexpected problems with the change.
* Fixed unbounded retention of recovered transaction state by a long-lived secondary opened with `allow_2pc`.
* Fixed stale reads on secondary instances, where entries replayed from a WAL that the primary had already flushed could shadow the newer values in the flushed files indefinitely. Also fixed a case where a secondary could stop returning such a key altogether while the primary's flushed file holding it was unreadable.
* Fixed subset checkpoints to discard tolerated MANIFEST tail corruption before appending column-family drop records.
### Performance Improvements
* Improved compaction CPU efficiency for column families using an `SstPartitioner` (such as the built-in `SstPartitionerFixedPrefix`): when the partitioner can express its next boundary as a key prefix via the new `SstPartitioner::ShouldPartitionByPrefix()` API, compaction now checks that cached prefix inline instead of copying every user key and making a virtual `ShouldPartition()` call for each key-value written to an SST.
## 11.9.0 (08/14/2026)
### New Features
* Added an integrated and improved compression auto-skip feature (new CompressionOptions, still experimental) that saves CPU by skipping compression when and where the overall ratio is inadequate. This improves on the previous auto-skip compression manager by (a) taking achieved compression ratios into account, (b) using a continuously evolving payoff estimate, and (c) combining natively with parallel compression.
+22 -12
View File
@@ -21,6 +21,7 @@
#include "rocksdb/merge_operator.h"
#include "rocksdb/perf_context.h"
#include "rocksdb/table.h"
#include "rocksdb/utilities/checkpoint.h"
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
@@ -356,10 +357,11 @@ TEST_F(DBBasicTest,
}
TEST_F(DBBasicTest,
OptimizeManifestForRecoveryIgnoresOptionsFilesForNextFileNumber) {
OptimizeManifestForRecoveryKeepsCurrentOptionsFileWithStaleOptions) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.optimize_manifest_for_recovery = true;
options.avoid_unnecessary_blocking_io = false;
DestroyAndReopen(options);
ASSERT_OK(Put("k", "v"));
ASSERT_OK(Flush());
@@ -368,20 +370,28 @@ TEST_F(DBBasicTest,
dbfull()->GetVersionSet()->current_next_file_number();
Close();
const uint64_t options_number = next_before_close + 100;
ASSERT_OK(WriteStringToFile(env_, "" /*data*/,
dbname_ + "/" + OptionsFileName(options_number),
/*should_sync=*/true));
ASSERT_OK(WriteStringToFile(env_, "" /*data*/,
TempOptionsFileName(dbname_, options_number + 1),
/*should_sync=*/true));
const uint64_t stale_options_number = next_before_close + 100;
ASSERT_OK(WriteStringToFile(
env_, "" /*data*/, dbname_ + "/" + OptionsFileName(stale_options_number),
/*should_sync=*/true));
ASSERT_OK(WriteStringToFile(
env_, "" /*data*/,
dbname_ + "/" + OptionsFileName(stale_options_number + 1),
/*should_sync=*/true));
RecoveryOptimizationCounters counters;
counters.Install();
Reopen(options);
counters.Uninstall();
const uint64_t current_options_number =
dbfull()->GetVersionSet()->options_file_number();
ASSERT_GT(current_options_number, 0u);
ASSERT_OK(env_->FileExists(OptionsFileName(dbname_, current_options_number)));
ASSERT_EQ(1, counters.next_file_number.load());
const std::string checkpoint_dir = dbname_ + "_checkpoint";
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
Checkpoint* checkpoint = nullptr;
ASSERT_OK(Checkpoint::Create(db_.get(), &checkpoint));
std::unique_ptr<Checkpoint> checkpoint_guard(checkpoint);
ASSERT_OK(checkpoint_guard->CreateCheckpoint(checkpoint_dir));
ASSERT_OK(DestroyDir(env_, checkpoint_dir));
ASSERT_EQ("v", Get("k"));
}
+1 -3
View File
@@ -1211,9 +1211,7 @@ Status DBImpl::MaybeUpdateNextFileNumber(RecoveryContext* recovery_ctx) {
// Use >= so a crashed-mid-allocation file at exactly
// prev_next_file_number triggers the advance -- otherwise the next
// NewFileNumber() would collide with it.
const bool must_reserve_file_number =
type != kOptionsFile && type != kTempFile;
if (must_reserve_file_number && number >= prev_next_file_number) {
if (number >= prev_next_file_number) {
on_disk_file_advanced_counter = true;
}
largest_file_number = std::max(largest_file_number, number);
-2
View File
@@ -5732,9 +5732,7 @@ void StressTest::RecordManifestStateBeforeReopen() {
if (reuse_manifest && optimize_manifest) {
// Check if ALL conditions for complete avoidance are met.
// If so, use STRICT mode where failures are fatal.
const bool no_fault_injection = !NeedsFaultInjection();
const bool no_manifest_writes_expected =
no_fault_injection &&
FLAGS_avoid_flush_during_recovery && // No flush during recovery
!FLAGS_write_dbid_to_manifest; // No DB_ID write on open
// Note: avoid_flush_during_shutdown is NOT required for STRICT mode.
+2 -1
View File
@@ -429,7 +429,8 @@ class StressTest {
// - Not in best_efforts_recovery mode
// - avoid_flush_during_recovery=true (no flush during recovery)
// - write_dbid_to_manifest=0 (no DB_ID write on open)
// - all fault injection flags are disabled
// - metadata_write_fault_one_in=0 (no fault injection)
// - open_metadata_write_fault_one_in=0 (no fault injection)
// - MANIFEST not corrupted, not at size limit
// Note: avoid_flush_during_shutdown is NOT required. If it leaves data
// in WAL but avoid_flush_during_recovery=true prevents flushing it,
@@ -1 +0,0 @@
Fixed a liveness bug where RocksDB could remain indefinitely in recovery from a soft background error when concurrent writes continuously generated normal flush requests.
@@ -0,0 +1 @@
Fixed RocksDB builds on big-endian architectures such as s390x by disabling the optional persistent fault-injection log, whose binary format is little-endian-only.
@@ -1 +0,0 @@
Fixed a race in online error recovery where a concurrent higher-severity I/O error could leave a stale retry-flush plan active, prevent the required WAL switch, and create duplicate internal WAL history detected on DB reopen.
@@ -1 +0,0 @@
Fix "bottommost files" infinite compaction loops (`kBottommostFiles`). This happens with the unsupported combination of leveled compaction and `preserve_internal_time_seconds` / `preclude_last_level_data_seconds`, and in an edge case with user-defined timestamps when `full_history_ts_low` is unset. The underlying logic is now more unified to reduce risk of a similar divergence in logic in the future.
@@ -1 +0,0 @@
Fixed `reuse_manifest_on_open` to create a fresh MANIFEST instead of appending after recovery sees an incomplete atomic group at the MANIFEST tail.
@@ -1 +0,0 @@
Starting remote compaction (`CompactionService` / `DB::OpenAndCompact`) is now more efficient and more reliable, by avoiding unnecessary file operations (latency) while using more accurate consistency / precondition checks, both avoiding some spurious failures and rejecting more legitimate inconsistencies. New DBOption `remote_compaction_manifest_floor` can be disabled in case of unexpected problems with the change.
@@ -1 +0,0 @@
Fixed unbounded retention of recovered transaction state by a long-lived secondary opened with `allow_2pc`.
@@ -1 +0,0 @@
Fixed stale reads on secondary instances, where entries replayed from a WAL that the primary had already flushed could shadow the newer values in the flushed files indefinitely. Also fixed a case where a secondary could stop returning such a key altogether while the primary's flushed file holding it was unreadable.
@@ -1 +0,0 @@
Fixed subset checkpoints to discard tolerated MANIFEST tail corruption before appending column-family drop records.
@@ -1 +0,0 @@
Added `BlockBasedTableOptions::optimize_key_common_prefix` as part of new `format_version=8`. When enabled, each data block and index block stores the common user-key prefix of its keys once instead of at every restart point, reducing block size and block-cache footprint for prefix-heavy data and speeding up Seek for bytewise-ordered keys.
@@ -1 +0,0 @@
A new experimental feature spreads out periodic (time-based) compaction trigger times across a fleet of similarly-configured DBs to avoid recurring "thundering herd" triggers, especially with remote compaction. See the new experimental DB option `periodic_compaction_phase_recovery_percent`.
@@ -1 +0,0 @@
Improved compaction CPU efficiency for column families using an `SstPartitioner` (such as the built-in `SstPartitionerFixedPrefix`): when the partitioner can express its next boundary as a key prefix via the new `SstPartitioner::ShouldPartitionByPrefix()` API, compaction now checks that cached prefix inline instead of copying every user key and making a virtual `ShouldPartition()` call for each key-value written to an SST.
@@ -1 +0,0 @@
`IngestExternalFileArg::atomic_replace_range` now supports a range that only partially overlaps existing files, instead of failing. Requires universal compaction and a table factory that supports range deletion; other configurations still fail on partial overlap. The range is also now specified as half-open (`start` inclusive, `limit` exclusive), resolving a previously documented ambiguity in the upper bound.
@@ -1 +0,0 @@
Added an immutable `external_table_config` option that is persisted in OPTIONS files and passed to external table factories during bootstrap.
+4 -1
View File
@@ -26,6 +26,7 @@
#include "env/io_posix.h"
#include "monitoring/thread_status_util.h"
#include "port/lang.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "rocksdb/env.h"
#include "rocksdb/io_status.h"
@@ -79,7 +80,9 @@ uint8_t DetailPayloadSize(const InjectedErrorLog::TaggedEntryDetail& detail) {
int OpenInjectedErrorLogFile(const std::string& path) {
#ifndef OS_WIN
if (path.empty()) {
// Entry is written in native layout, while the decoder expects little-endian
// fields. Keep the optional log disabled on unsupported architectures.
if (!port::kLittleEndian || path.empty()) {
return -1;
}
int flags = O_WRONLY | O_CREAT | O_TRUNC;
+5 -2
View File
@@ -44,6 +44,8 @@ namespace ROCKSDB_NAMESPACE {
// A binary log that records injected errors directly to a file.
// Thread-safe for concurrent writes. The log file can be flushed on exit or
// from a signal handler so records survive clean exits and crash paths.
// Disabled on big-endian platforms because the decoder expects little-endian
// fields in the native Entry layout.
class InjectedErrorLog {
public:
static constexpr size_t kMaxOpNameLen = 32;
@@ -86,12 +88,13 @@ class InjectedErrorLog {
char op_name[kMaxOpNameLen];
char file_name[kMaxFileNameLen];
char status_message[kMaxStatusMessageLen];
// Explicit tail padding so the size does not depend on the alignment of
// uint64_t (4 bytes on i386, 8 bytes on most other targets).
uint8_t padding[4];
};
static_assert(sizeof(Entry) == 256,
"Injected error log entry size must stay stable");
static_assert(port::kLittleEndian,
"Injected error log requires little-endian platforms");
explicit InjectedErrorLog(const std::string& path);
~InjectedErrorLog();
+21
View File
@@ -10,6 +10,7 @@
#include <thread>
#include <vector>
#include "port/port.h"
#include "test_util/testharness.h"
namespace ROCKSDB_NAMESPACE {
@@ -79,6 +80,11 @@ class CloseCountingWritableFile : public FSWritableFileOwnerWrapper {
// the expected fixed-width fields. The test writes one entry, flushes the
// file, and decodes the raw bytes back into an entry struct.
TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
if (!port::kLittleEndian) {
ROCKSDB_GTEST_BYPASS(
"Persistent injected error log requires little-endian");
return;
}
std::string path = test::PerThreadDBPath("injected_error_log_basic.bin");
{
InjectedErrorLog log(path);
@@ -114,6 +120,11 @@ TEST_F(InjectedErrorLogTest, BasicRecordAndFlush) {
// to the old in-memory ring size. The test writes more than 1,000 entries and
// checks that both the first and last ones are still present in the file.
TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
if (!port::kLittleEndian) {
ROCKSDB_GTEST_BYPASS(
"Persistent injected error log requires little-endian");
return;
}
std::string path =
test::PerThreadDBPath("injected_error_log_all_entries.bin");
{
@@ -144,6 +155,11 @@ TEST_F(InjectedErrorLogTest, DirectLogKeepsAllEntries) {
// shared file. The test has several threads emit records concurrently and then
// checks the raw file size matches the total number of writes.
TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
if (!port::kLittleEndian) {
ROCKSDB_GTEST_BYPASS(
"Persistent injected error log requires little-endian");
return;
}
std::string path = test::PerThreadDBPath("injected_error_log_concurrent.bin");
constexpr int kNumThreads = 4;
constexpr int kRecordsPerThread = 200;
@@ -180,6 +196,11 @@ TEST_F(InjectedErrorLogTest, ConcurrentRecord) {
// in the fixed-width record. The test logs a path longer than the file-name
// field and checks the stored sample matches the expected tail bytes.
TEST_F(InjectedErrorLogTest, LongFileNameKeepsSuffix) {
if (!port::kLittleEndian) {
ROCKSDB_GTEST_BYPASS(
"Persistent injected error log requires little-endian");
return;
}
std::string path =
test::PerThreadDBPath("injected_error_log_suffix_truncation.bin");
const std::string long_file_name =