Compare commits

..
11 Commits
Author SHA1 Message Date
anand76 3258b5c3e2 Update version.h to 7.10.2 2023-02-23 10:00:36 -08:00
anand76 3a04cd558e Update HISTORY.md for 7.10.2 2023-02-10 14:59:29 -08:00
anand76 0ffa8db9b1 Fix bug in WAL streaming uncompression (#11198)
Summary:
Fix a bug in the calculation of the input buffer address/offset in log_reader.cc. The bug is when consecutive fragments of a compressed record are located at the same offset in the log reader buffer, the second fragment input buffer is treated as a leftover from the previous input buffer. As a result, the offset in the `ZSTD_inBuffer` is not reset.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11198

Test Plan: Add a unit test in log_test.cc that fails without the fix and passes with it.

Reviewed By: ajkr, cbi42

Differential Revision: D43102692

Pulled By: anand1976

fbshipit-source-id: aa2648f4802c33991b76a3233c5a58d4cc9e77fd
2023-02-10 14:58:13 -08:00
anand76 1185bb75ca Return any errors returned by ReadAsync to the MultiGet caller (#11171)
Summary:
Currently, we incorrectly return a Status::Corruption to the MultiGet caller if the file system ReadAsync cannot issue a read and returns an error for some reason, such as IOStatus::NotSupported(). In this PR, we copy the ReadAsync error to the request status so it can be returned to the user.

Tests:
Update existing unit tests and add a new one for this scenario

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11171

Reviewed By: akankshamahajan15

Differential Revision: D42950057

Pulled By: anand1976

fbshipit-source-id: 85ffcb015fa6c064c311f8a28488fec78c487869
2023-02-02 17:28:08 -08:00
Andrew Kryczka d57ec3f896 update HISTORY.md and version.h for 7.10.1 2023-02-02 08:51:12 -08:00
Andrew Kryczka 8a354a1197 add release note for GetMergeOperands() fix 2023-02-01 17:05:04 -08:00
Andrew Kryczka fcb0580b08 Fix GetMergeOperands() returning MergeInProgress (#11136)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11136

Test Plan: the provided unit test used to fail due to `GetMergeOperands()` returning `Status::MergeInProgress()`; it passes now because the `GetMergeOperands()` call returns `Status::OK()`

Reviewed By: pdillinger

Differential Revision: D42759198

Pulled By: ajkr

fbshipit-source-id: 878f9f40ccc1d7e2fe7b1352814bae3a49c19939
2023-02-01 17:03:31 -08:00
Andrew Kryczka 06765b5131 Allow canceling manual compaction while waiting for conflicting compaction (#11165)
Summary:
This PR adds logic to the `RunManualCompaction()` loop to check for cancellation before waiting on any conflicting compactions to finish. In case of cancellation, `RunManualCompaction()` no longer waits on conflicting compactions

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11165

Test Plan: repro test case

Reviewed By: cbi42

Differential Revision: D42864058

Pulled By: ajkr

fbshipit-source-id: ea4dd1a8f294abe212905495a8fbe8f07fca3f5a
2023-02-01 17:00:51 -08:00
Hui Xiao fa13962e0c Fix data race on ColumnFamilyData::flush_reason by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)
Summary:
**Context:**
Concurrent flushes on the same CF can set on `ColumnFamilyData::flush_reason` before each other flush finishes. An symptom is one CF has different flush_reason with others though all of them are in an atomic flush  `db_stress: db/db_impl/db_impl_compaction_flush.cc:423: rocksdb::Status rocksdb::DBImpl::AtomicFlushMemTablesToOutputFiles(const rocksdb::autovector<rocksdb::DBImpl::BGFlushArg>&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority): Assertion cfd->GetFlushReason() == cfds[0]->GetFlushReason() failed. `

**Summary:**
Suggested by ltamasi, we now refactor and let FlushRequest/Job to own flush_reason as there is no good way to define `ColumnFamilyData::flush_reason` in face of concurrent flushes on the same CF (which wasn't the case a long time ago when `ColumnFamilyData::flush_reason ` first introduced`)

**Tets:**
- new unit test
- make check
- aggressive crash test rehearsal

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11111

Reviewed By: ajkr

Differential Revision: D42644600

Pulled By: hx235

fbshipit-source-id: 8589c8184869d3415e5b780c887f877818a5ebaf
2023-02-01 16:59:41 -08:00
Peter Dillinger e5dcebf756 Fix DelayWrite() calls for two_write_queues (#11130)
Summary:
PR https://github.com/facebook/rocksdb/issues/11020 fixed a case where it was easy to deadlock the DB with LockWAL() but introduced a bug showing up as a rare assertion failure in the stress test. Specifically, `assert(w->state == STATE_INIT)` in `WriteThread::LinkOne()` called from `BeginWriteStall()`, `DelayWrite()`, `WriteImplWALOnly()`. I haven't been about to generate a unit test that reproduces this failure but I believe the root cause is that DelayWrite() was never meant to be re-entrant, only called from the DB's write_thread_ leader. https://github.com/facebook/rocksdb/issues/11020 introduced a call to DelayWrite() from the nonmem_write_thread_ group leader.

This fix is to make DelayWrite() apply to the specific write queue that it is being called from (inject a dummy write stall entry to the head of the appropriate write queue). WriteController is re-entrant, based on polling and state changes signalled with bg_cv_, so can manage stalling two queues. The only anticipated complication (called out by Andrew in previous PR) is that we don't want timed write delays being injected in parallel for the two queues, because that dimishes the intended throttling effect. Thus, we only allow timed delays for the primary write queue.

HISTORY not updated because this is intended for the same release where the bug was introduced.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11130

Test Plan:
Although I was not able to reproduce the assertion failure, I was able to reproduce a distinct flaw with what I believe is the same root cause: a kind of deadlock if both write queues need to wake up from stopped writes. Only one will be waiting on bg_cv_ (the other waiting in `LinkOne()` for the write queue to open up), so a single SignalAll() will only unblock one of the queues, with the other re-instating the stop until another signal on bg_cv_. A simple unit test is added for this case.

Will also run crash_test_with_multiops_wc_txn for a while looking for issues.

Reviewed By: ajkr

Differential Revision: D42749330

Pulled By: pdillinger

fbshipit-source-id: 4317dd899a93d57c26fd5af7143038f82d4d4d1b
2023-01-25 14:25:33 -08:00
Hui Xiao ab389242fb Update history for 7.10.fb 2023-01-23 11:11:36 -08:00
510 changed files with 5730 additions and 1039 deletions

No files matched your search

+23
View File
@@ -302,6 +302,27 @@ jobs:
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
build-linux-lite:
executor: linux-docker
resource_class: large
steps:
- pre-steps
- run: LITE=1 make V=1 J=8 -j8 check
- post-steps
build-linux-lite-release:
executor: linux-docker
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: LITE=1 make V=1 -j8 release
- run: ./db_stress --version # ensure with gflags
- run: make clean
- run: apt-get remove -y libgflags-dev
- run: LITE=1 make V=1 -j8 release
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- post-steps
build-linux-clang-no_test_run:
executor: linux-docker
resource_class: xlarge
@@ -800,6 +821,7 @@ workflows:
- build-linux-cmake-with-folly-coroutines
- build-linux-cmake-with-benchmark
- build-linux-encrypted_env-no_compression
- build-linux-lite
jobs-linux-run-tests-san:
jobs:
- build-linux-clang10-asan
@@ -810,6 +832,7 @@ workflows:
jobs:
- build-linux-release
- build-linux-release-rtti
- build-linux-lite-release
- build-examples
- build-fuzzers
- build-linux-clang-no_test_run
+6
View File
@@ -496,6 +496,12 @@ if(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-builtin-memcmp")
endif()
option(ROCKSDB_LITE "Build RocksDBLite version" OFF)
if(ROCKSDB_LITE)
add_definitions(-DROCKSDB_LITE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-exceptions -Os")
endif()
if(CMAKE_SYSTEM_NAME MATCHES "Cygwin")
add_definitions(-fno-builtin-memcmp -DCYGWIN)
elseif(CMAKE_SYSTEM_NAME MATCHES "Darwin")
+5 -17
View File
@@ -1,26 +1,14 @@
# Rocksdb Change Log
## Unreleased
### Behavior changes
* `ReadOptions::verify_checksums=false` disables checksum verification for more reads of non-`CacheEntryRole::kDataBlock` blocks.
## 7.10.2 (02/10/2023)
### Bug Fixes
* Fixed a bug in DB open/recovery from a compressed WAL that was caused due to incorrect handling of certain record fragments with the same offset within a WAL block.
## 7.10.1 (02/01/2023)
### Bug Fixes
* Fixed a data race on `ColumnFamilyData::flush_reason` caused by concurrent flushes.
* Fixed an issue in `Get` and `MultiGet` when user-defined timestamps is enabled in combination with BlobDB.
* Fixed some atypical behaviors for `LockWAL()` such as allowing concurrent/recursive use and not expecting `UnlockWAL()` after non-OK result. See API comments.
* Fixed a feature interaction bug where for blobs `GetEntity` would expose the blob reference instead of the blob value.
* Fixed `DisableManualCompaction()` and `CompactRangeOptions::canceled` to cancel compactions even when they are waiting on conflicting compactions to finish
* Fixed a bug in which a successful `GetMergeOperands()` could transiently return `Status::MergeInProgress()`
### Feature Removal
* Remove RocksDB Lite.
* The feature block_cache_compressed is removed. Statistics related to it are removed too.
* Remove deprecated Env::LoadEnv(). Use Env::CreateFromString() instead.
* Remove deprecated FileSystem::Load(). Use FileSystem::CreateFromString() instead.
* Removed the deprecated version of these utility functions and the corresponding Java bindings: `LoadOptionsFromFile`, `LoadLatestOptions`, `CheckOptionsCompatibility`.
### Public API Changes
* Completely removed the following deprecated/obsolete statistics: the tickers `BLOCK_CACHE_INDEX_BYTES_EVICT`, `BLOCK_CACHE_FILTER_BYTES_EVICT`, `BLOOM_FILTER_MICROS`, `NO_FILE_CLOSES`, `STALL_L0_SLOWDOWN_MICROS`, `STALL_MEMTABLE_COMPACTION_MICROS`, `STALL_L0_NUM_FILES_MICROS`, `RATE_LIMIT_DELAY_MILLIS`, `NO_ITERATORS`, `NUMBER_FILTERED_DELETES`, `WRITE_TIMEDOUT`, `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`, `BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, `BLOCK_CACHE_COMPRESSION_DICT_BYTES_EVICT` as well as the histograms `STALL_L0_SLOWDOWN_COUNT`, `STALL_MEMTABLE_COMPACTION_COUNT`, `STALL_L0_NUM_FILES_COUNT`, `HARD_RATE_LIMIT_DELAY_COUNT`, `SOFT_RATE_LIMIT_DELAY_COUNT`, `BLOB_DB_GC_MICROS`, and `NUM_DATA_BLOCKS_READ_PER_LEVEL`. Note that as a result, the C++ enum values of the still supported statistics have changed. Developers are advised to not rely on the actual numeric values.
* Return the correct error (Status::NotSupported()) to MultiGet caller when ReadOptions::async_io flag is true and IO uring is not enabled. Previously, Status::Corruption() was being returned when the actual failure was lack of async IO support.
## 7.10.0 (01/23/2023)
### Behavior changes
+1 -1
View File
@@ -178,7 +178,7 @@ to build a portable binary, add `PORTABLE=1` before your make commands, like thi
gmake rocksdbjava
* **iOS**:
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define an important pre-processing macros: `IOS_CROSS_COMPILE`.
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define two important pre-processing macros: `ROCKSDB_LITE` and `IOS_CROSS_COMPILE`.
* **Windows** (Visual Studio 2017 to up):
* Read and follow the instructions at CMakeLists.txt
+39
View File
@@ -83,9 +83,27 @@ endif
$(info $$DEBUG_LEVEL is ${DEBUG_LEVEL})
# Lite build flag.
LITE ?= 0
ifeq ($(LITE), 0)
ifneq ($(filter -DROCKSDB_LITE,$(OPT)),)
# Be backward compatible and support older format where OPT=-DROCKSDB_LITE is
# specified instead of LITE=1 on the command line.
LITE=1
endif
else ifeq ($(LITE), 1)
ifeq ($(filter -DROCKSDB_LITE,$(OPT)),)
OPT += -DROCKSDB_LITE
endif
endif
# Figure out optimize level.
ifneq ($(DEBUG_LEVEL), 2)
ifeq ($(LITE), 0)
OPTIMIZE_LEVEL ?= -O2
else
OPTIMIZE_LEVEL ?= -Os
endif
endif
# `OPTIMIZE_LEVEL` is empty when the user does not set it and `DEBUG_LEVEL=2`.
# In that case, the compiler default (`-O0` for gcc and clang) will be used.
@@ -319,6 +337,13 @@ endif
ifeq ($(PLATFORM), OS_SOLARIS)
PLATFORM_CXXFLAGS += -D _GLIBCXX_USE_C99
endif
ifneq ($(filter -DROCKSDB_LITE,$(OPT)),)
# found
CFLAGS += -fno-exceptions
CXXFLAGS += -fno-exceptions
# LUA is not supported under ROCKSDB_LITE
LUA_PATH =
endif
ifeq ($(LIB_MODE),shared)
# So that binaries are executable from build location, in addition to install location
@@ -1057,11 +1082,13 @@ check: all
rm -rf $(TEST_TMPDIR)
ifneq ($(PLATFORM), OS_AIX)
$(PYTHON) tools/check_all_python.py
ifeq ($(filter -DROCKSDB_LITE,$(OPT)),)
ifndef ASSERT_STATUS_CHECKED # not yet working with these tests
$(PYTHON) tools/ldb_test.py
sh tools/rocksdb_dump_test.sh
endif
endif
endif
ifndef SKIP_FORMAT_BUCK_CHECKS
$(MAKE) check-format
$(MAKE) check-buck-targets
@@ -2462,6 +2489,18 @@ build_size:
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib $$(stat --printf="%s" `readlink -f librocksdb.so`)
strip `readlink -f librocksdb.so`
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_stripped $$(stat --printf="%s" `readlink -f librocksdb.so`)
# === lite build, static ===
$(MAKE) clean
$(MAKE) LITE=1 static_lib
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.static_lib_lite $$(stat --printf="%s" librocksdb.a)
strip librocksdb.a
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.static_lib_lite_stripped $$(stat --printf="%s" librocksdb.a)
# === lite build, shared ===
$(MAKE) clean
$(MAKE) LITE=1 shared_lib
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_lite $$(stat --printf="%s" `readlink -f librocksdb.so`)
strip `readlink -f librocksdb.so`
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_lite_stripped $$(stat --printf="%s" `readlink -f librocksdb.so`)
# ---------------------------------------------------------------------------
# Platform-specific compilation
+21
View File
@@ -0,0 +1,21 @@
# RocksDBLite
RocksDBLite is a project focused on mobile use cases, which don't need a lot of fancy things we've built for server workloads and they are very sensitive to binary size. For that reason, we added a compile flag ROCKSDB_LITE that comments out a lot of the nonessential code and keeps the binary lean.
Some examples of the features disabled by ROCKSDB_LITE:
* compiled-in support for LDB tool
* No backup engine
* No support for replication (which we provide in form of TransactionalIterator)
* No advanced monitoring tools
* No special-purpose memtables that are highly optimized for specific use cases
* No Transactions
When adding a new big feature to RocksDB, please add ROCKSDB_LITE compile guard if:
* Nobody from mobile really needs your feature,
* Your feature is adding a lot of weight to the binary.
Don't add ROCKSDB_LITE compile guard if:
* It would introduce a lot of code complexity. Compile guards make code harder to read. It's a trade-off.
* Your feature is not adding a lot of weight.
If unsure, ask. :)
+1 -1
View File
@@ -26,7 +26,7 @@ from util import ColorString
# $python3 buckifier/buckify_rocksdb.py \
# '{"fake": {
# "extra_deps": [":test_dep", "//fakes/module:mock1"],
# "extra_compiler_flags": ["-DFOO_BAR", "-Os"]
# "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"]
# }
# }'
# (Generated TARGETS file has test_dep and mock1 as dependencies for RocksDB
+1 -1
View File
@@ -154,7 +154,7 @@ case "$TARGET_OS" in
;;
IOS)
PLATFORM=IOS
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE "
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE -DROCKSDB_LITE"
PLATFORM_SHARED_EXT=dylib
PLATFORM_SHARED_LDFLAGS="-dynamiclib -install_name "
CROSS_COMPILE=true
+13
View File
@@ -16,6 +16,7 @@
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
static std::unordered_map<std::string, OptionTypeInfo>
lru_cache_options_type_info = {
{"capacity",
@@ -63,6 +64,7 @@ static std::unordered_map<std::string, OptionTypeInfo>
OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kMutable}},
};
#endif // ROCKSDB_LITE
Status SecondaryCache::CreateFromString(
const ConfigOptions& config_options, const std::string& value,
@@ -73,6 +75,7 @@ Status SecondaryCache::CreateFromString(
Status status;
std::shared_ptr<SecondaryCache> sec_cache;
#ifndef ROCKSDB_LITE
CompressedSecondaryCacheOptions sec_cache_opts;
status = OptionTypeInfo::ParseStruct(config_options, "",
&comp_sec_cache_options_type_info, "",
@@ -81,6 +84,11 @@ Status SecondaryCache::CreateFromString(
sec_cache = NewCompressedSecondaryCache(sec_cache_opts);
}
#else
(void)config_options;
status = Status::NotSupported(
"Cannot load compressed secondary cache in LITE mode ", args);
#endif //! ROCKSDB_LITE
if (status.ok()) {
result->swap(sec_cache);
@@ -100,6 +108,7 @@ Status Cache::CreateFromString(const ConfigOptions& config_options,
if (value.find('=') == std::string::npos) {
cache = NewLRUCache(ParseSizeT(value));
} else {
#ifndef ROCKSDB_LITE
LRUCacheOptions cache_opts;
status = OptionTypeInfo::ParseStruct(config_options, "",
&lru_cache_options_type_info, "",
@@ -107,6 +116,10 @@ Status Cache::CreateFromString(const ConfigOptions& config_options,
if (status.ok()) {
cache = NewLRUCache(cache_opts);
}
#else
(void)config_options;
status = Status::NotSupported("Cannot load cache in LITE mode ", value);
#endif //! ROCKSDB_LITE
}
if (status.ok()) {
result->swap(cache);
+4
View File
@@ -77,9 +77,11 @@ DEFINE_bool(lean, false,
"If true, no additional computation is performed besides cache "
"operations.");
#ifndef ROCKSDB_LITE
DEFINE_string(secondary_cache_uri, "",
"Full URI for creating a custom secondary cache object");
static class std::shared_ptr<ROCKSDB_NAMESPACE::SecondaryCache> secondary_cache;
#endif // ROCKSDB_LITE
DEFINE_string(cache_type, "lru_cache", "Type of block cache.");
@@ -303,6 +305,7 @@ class CacheBench {
LRUCacheOptions opts(FLAGS_cache_size, FLAGS_num_shard_bits,
false /* strict_capacity_limit */,
0.5 /* high_pri_pool_ratio */);
#ifndef ROCKSDB_LITE
if (!FLAGS_secondary_cache_uri.empty()) {
Status s = SecondaryCache::CreateFromString(
ConfigOptions(), FLAGS_secondary_cache_uri, &secondary_cache);
@@ -315,6 +318,7 @@ class CacheBench {
}
opts.secondary_cache = secondary_cache;
}
#endif // ROCKSDB_LITE
cache_ = NewLRUCache(opts);
} else {
+2
View File
@@ -817,6 +817,7 @@ class CompressedSecondaryCacheTestWithCompressionParam
bool sec_cache_is_compressed_;
};
#ifndef ROCKSDB_LITE
TEST_P(CompressedSecondaryCacheTestWithCompressionParam, BasicTestFromString) {
std::shared_ptr<SecondaryCache> sec_cache{nullptr};
@@ -881,6 +882,7 @@ TEST_P(CompressedSecondaryCacheTestWithCompressionParam,
BasicTestHelper(sec_cache, sec_cache_is_compressed_);
}
#endif // ROCKSDB_LITE
TEST_P(CompressedSecondaryCacheTestWithCompressionParam, FailsTest) {
FailsTest(sec_cache_is_compressed_);
+39
View File
@@ -1784,6 +1784,43 @@ TEST_F(DBSecondaryCacheTest, SecondaryCacheFailureTest) {
Destroy(options);
}
TEST_F(DBSecondaryCacheTest, TestSecondaryWithCompressedCache) {
if (!Snappy_Supported()) {
ROCKSDB_GTEST_SKIP("Compressed cache test requires snappy support");
return;
}
LRUCacheOptions opts(2000 /* capacity */, 0 /* num_shard_bits */,
false /* strict_capacity_limit */,
0.5 /* high_pri_pool_ratio */,
nullptr /* memory_allocator */, kDefaultToAdaptiveMutex,
kDontChargeCacheMetadata);
std::shared_ptr<TestSecondaryCache> secondary_cache(
new TestSecondaryCache(2048 * 1024));
opts.secondary_cache = secondary_cache;
std::shared_ptr<Cache> cache = NewLRUCache(opts);
BlockBasedTableOptions table_options;
table_options.block_cache_compressed = cache;
table_options.no_block_cache = true;
table_options.block_size = 1234;
Options options = GetDefaultOptions();
options.compression = kSnappyCompression;
options.create_if_missing = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
DestroyAndReopen(options);
Random rnd(301);
const int N = 6;
for (int i = 0; i < N; i++) {
// Partly compressible
std::string p_v = rnd.RandomString(507) + std::string(500, ' ');
ASSERT_OK(Put(Key(i), p_v));
}
ASSERT_OK(Flush());
for (int i = 0; i < 2 * N; i++) {
std::string v = Get(Key(i % N));
ASSERT_EQ(1007, v.size());
}
}
TEST_F(LRUCacheSecondaryCacheTest, BasicWaitAllTest) {
LRUCacheOptions opts(1024 /* capacity */, 2 /* num_shard_bits */,
false /* strict_capacity_limit */,
@@ -1975,6 +2012,7 @@ class LRUCacheWithStat : public LRUCache {
uint32_t lookup_count_;
};
#ifndef ROCKSDB_LITE
TEST_F(DBSecondaryCacheTest, LRUCacheDumpLoadBasic) {
LRUCacheOptions cache_opts(1024 * 1024 /* capacity */, 0 /* num_shard_bits */,
@@ -2589,6 +2627,7 @@ TEST_F(DBSecondaryCacheTest, TestSecondaryCacheOptionTwoDB) {
ASSERT_OK(DestroyDB(dbname2, options));
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+17
View File
@@ -23,19 +23,32 @@ class BlobFileCompletionCallback {
const std::vector<std::shared_ptr<EventListener>>& listeners,
const std::string& dbname)
: event_logger_(event_logger), listeners_(listeners), dbname_(dbname) {
#ifndef ROCKSDB_LITE
sst_file_manager_ = sst_file_manager;
mutex_ = mutex;
error_handler_ = error_handler;
#else
(void)sst_file_manager;
(void)mutex;
(void)error_handler;
#endif // ROCKSDB_LITE
}
void OnBlobFileCreationStarted(const std::string& file_name,
const std::string& column_family_name,
int job_id,
BlobFileCreationReason creation_reason) {
#ifndef ROCKSDB_LITE
// Notify the listeners.
EventHelpers::NotifyBlobFileCreationStarted(listeners_, dbname_,
column_family_name, file_name,
job_id, creation_reason);
#else
(void)file_name;
(void)column_family_name;
(void)job_id;
(void)creation_reason;
#endif
}
Status OnBlobFileCompleted(const std::string& file_name,
@@ -48,6 +61,7 @@ class BlobFileCompletionCallback {
uint64_t blob_count, uint64_t blob_bytes) {
Status s;
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(sst_file_manager_);
if (sfm) {
// Report new blob files to SstFileManagerImpl
@@ -60,6 +74,7 @@ class BlobFileCompletionCallback {
error_handler_->SetBGError(s, BackgroundErrorReason::kFlush);
}
}
#endif // !ROCKSDB_LITE
// Notify the listeners.
EventHelpers::LogAndNotifyBlobFileCreationFinished(
@@ -74,9 +89,11 @@ class BlobFileCompletionCallback {
}
private:
#ifndef ROCKSDB_LITE
SstFileManager* sst_file_manager_;
InstrumentedMutex* mutex_;
ErrorHandler* error_handler_;
#endif // ROCKSDB_LITE
EventLogger* event_logger_;
std::vector<std::shared_ptr<EventListener>> listeners_;
std::string dbname_;
+2
View File
@@ -30,6 +30,7 @@ BlobSource::BlobSource(const ImmutableOptions* immutable_options,
blob_file_cache_(blob_file_cache),
blob_cache_(immutable_options->blob_cache),
lowest_used_cache_tier_(immutable_options->lowest_used_cache_tier) {
#ifndef ROCKSDB_LITE
auto bbto =
immutable_options->table_factory->GetOptions<BlockBasedTableOptions>();
if (bbto &&
@@ -38,6 +39,7 @@ BlobSource::BlobSource(const ImmutableOptions* immutable_options,
blob_cache_ = SharedCacheInterface{std::make_shared<ChargedCache>(
immutable_options->blob_cache, bbto->block_cache)};
}
#endif // ROCKSDB_LITE
}
BlobSource::~BlobSource() = default;
+2
View File
@@ -1391,6 +1391,7 @@ class BlobSourceCacheReservationTest : public DBTestBase {
std::string db_session_id_;
};
#ifndef ROCKSDB_LITE
TEST_F(BlobSourceCacheReservationTest, SimpleCacheReservation) {
options_.cf_paths.emplace_back(
test::PerThreadDBPath(
@@ -1605,6 +1606,7 @@ TEST_F(BlobSourceCacheReservationTest, IncreaseCacheReservationOnFullCache) {
}
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+8 -207
View File
@@ -11,7 +11,6 @@
#include "db/blob/blob_index.h"
#include "db/blob/blob_log_format.h"
#include "db/db_test_util.h"
#include "db/db_with_timestamp_test_util.h"
#include "port/stack_trace.h"
#include "test_util/sync_point.h"
#include "utilities/fault_injection_env.h"
@@ -585,6 +584,7 @@ TEST_F(DBBlobBasicTest, MultiGetBlobsFromCache) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, MultiGetWithDirectIO) {
Options options = GetDefaultOptions();
@@ -773,6 +773,7 @@ TEST_F(DBBlobBasicTest, MultiGetWithDirectIO) {
ASSERT_EQ(values[2], second_blob);
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, MultiGetBlobsFromMultipleFiles) {
Options options = GetDefaultOptions();
@@ -1061,6 +1062,7 @@ TEST_F(DBBlobBasicTest, GetBlob_IndexWithInvalidFileNumber) {
.IsCorruption());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, GenerateIOTracing) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -1115,6 +1117,7 @@ TEST_F(DBBlobBasicTest, GenerateIOTracing) {
ASSERT_GT(blob_files_op_count, 2);
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, BestEffortsRecovery_MissingNewestBlobFile) {
Options options = GetDefaultOptions();
@@ -1216,6 +1219,7 @@ TEST_F(DBBlobBasicTest, MultiGetMergeBlobWithPut) {
ASSERT_EQ(values[2], "v2_0");
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, Properties) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -1378,6 +1382,7 @@ TEST_F(DBBlobBasicTest, PropertiesMultiVersion) {
BlobLogRecord::CalculateAdjustmentForRecordHeader(key_size) +
blob_size + BlobLogFooter::kSize));
}
#endif // !ROCKSDB_LITE
class DBBlobBasicIOErrorTest : public DBBlobBasicTest,
public testing::WithParamInterface<std::string> {
@@ -1627,6 +1632,7 @@ TEST_F(DBBlobBasicTest, WarmCacheWithBlobsDuringFlush) {
options.statistics->getTickerCount(BLOB_DB_CACHE_ADD));
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, DynamicallyWarmCacheDuringFlush) {
Options options = GetDefaultOptions();
@@ -1694,6 +1700,7 @@ TEST_F(DBBlobBasicTest, DynamicallyWarmCacheDuringFlush) {
/*end=*/nullptr));
EXPECT_EQ(0, options.statistics->getTickerCount(BLOB_DB_CACHE_ADD));
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, WarmCacheWithBlobsSecondary) {
CompressedSecondaryCacheOptions secondary_cache_opts;
@@ -1772,212 +1779,6 @@ TEST_F(DBBlobBasicTest, WarmCacheWithBlobsSecondary) {
1);
}
TEST_F(DBBlobBasicTest, GetEntityBlob) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
options.min_blob_size = 0;
Reopen(options);
constexpr char key[] = "key";
constexpr char blob_value[] = "blob_value";
ASSERT_OK(Put(key, blob_value));
ASSERT_OK(Flush());
PinnableWideColumns result;
ASSERT_OK(
db_->GetEntity(ReadOptions(), db_->DefaultColumnFamily(), key, &result));
WideColumns expected_columns{{kDefaultWideColumnName, blob_value}};
ASSERT_EQ(result.columns(), expected_columns);
}
class DBBlobWithTimestampTest : public DBBasicTestWithTimestampBase {
protected:
DBBlobWithTimestampTest()
: DBBasicTestWithTimestampBase("db_blob_with_timestamp_test") {}
};
TEST_F(DBBlobWithTimestampTest, GetBlob) {
Options options = GetDefaultOptions();
options.create_if_missing = true;
options.enable_blob_files = true;
options.min_blob_size = 0;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
constexpr char key[] = "key";
constexpr char blob_value[] = "blob_value";
ASSERT_OK(db_->Put(write_opts, key, ts, blob_value));
ASSERT_OK(Flush());
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
std::string value;
ASSERT_OK(db_->Get(read_opts, key, &value));
ASSERT_EQ(value, blob_value);
}
TEST_F(DBBlobWithTimestampTest, MultiGetBlobs) {
constexpr size_t min_blob_size = 6;
Options options = GetDefaultOptions();
options.enable_blob_files = true;
options.min_blob_size = min_blob_size;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
// Put then retrieve three key-values. The first value is below the size limit
// and is thus stored inline; the other two are stored separately as blobs.
constexpr size_t num_keys = 3;
constexpr char first_key[] = "first_key";
constexpr char first_value[] = "short";
static_assert(sizeof(first_value) - 1 < min_blob_size,
"first_value too long to be inlined");
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, first_key, ts, first_value));
constexpr char second_key[] = "second_key";
constexpr char second_value[] = "long_value";
static_assert(sizeof(second_value) - 1 >= min_blob_size,
"second_value too short to be stored as blob");
ASSERT_OK(db_->Put(write_opts, second_key, ts, second_value));
constexpr char third_key[] = "third_key";
constexpr char third_value[] = "other_long_value";
static_assert(sizeof(third_value) - 1 >= min_blob_size,
"third_value too short to be stored as blob");
ASSERT_OK(db_->Put(write_opts, third_key, ts, third_value));
ASSERT_OK(Flush());
ReadOptions read_options;
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
read_options.timestamp = &read_ts_slice;
std::array<Slice, num_keys> keys{{first_key, second_key, third_key}};
{
std::array<PinnableSlice, num_keys> values;
std::array<Status, num_keys> statuses;
db_->MultiGet(read_options, db_->DefaultColumnFamily(), num_keys, &keys[0],
&values[0], &statuses[0]);
ASSERT_OK(statuses[0]);
ASSERT_EQ(values[0], first_value);
ASSERT_OK(statuses[1]);
ASSERT_EQ(values[1], second_value);
ASSERT_OK(statuses[2]);
ASSERT_EQ(values[2], third_value);
}
}
TEST_F(DBBlobWithTimestampTest, GetMergeBlobWithPut) {
Options options = GetDefaultOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.enable_blob_files = true;
options.min_blob_size = 0;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, "Key1", ts, "v1"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v2"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v3"));
ASSERT_OK(Flush());
std::string value;
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
ASSERT_OK(db_->Get(read_opts, "Key1", &value));
ASSERT_EQ(value, "v1,v2,v3");
}
TEST_F(DBBlobWithTimestampTest, MultiGetMergeBlobWithPut) {
constexpr size_t num_keys = 3;
Options options = GetDefaultOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.enable_blob_files = true;
options.min_blob_size = 0;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, "Key0", ts, "v0_0"));
ASSERT_OK(db_->Put(write_opts, "Key1", ts, "v1_0"));
ASSERT_OK(db_->Put(write_opts, "Key2", ts, "v2_0"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key0", ts, "v0_1"));
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v1_1"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key0", ts, "v0_2"));
ASSERT_OK(Flush());
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
std::array<Slice, num_keys> keys{{"Key0", "Key1", "Key2"}};
std::array<PinnableSlice, num_keys> values;
std::array<Status, num_keys> statuses;
db_->MultiGet(read_opts, db_->DefaultColumnFamily(), num_keys, &keys[0],
&values[0], &statuses[0]);
ASSERT_OK(statuses[0]);
ASSERT_EQ(values[0], "v0_0,v0_1,v0_2");
ASSERT_OK(statuses[1]);
ASSERT_EQ(values[1], "v1_0,v1_1");
ASSERT_OK(statuses[2]);
ASSERT_EQ(values[2], "v2_0");
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+14
View File
@@ -16,6 +16,7 @@ class DBBlobCompactionTest : public DBTestBase {
explicit DBBlobCompactionTest()
: DBTestBase("db_blob_compaction_test", /*env_do_fsync=*/false) {}
#ifndef ROCKSDB_LITE
const std::vector<InternalStats::CompactionStats>& GetCompactionStats() {
VersionSet* const versions = dbfull()->GetVersionSet();
assert(versions);
@@ -29,6 +30,7 @@ class DBBlobCompactionTest : public DBTestBase {
return internal_stats->TEST_GetCompactionStats();
}
#endif // ROCKSDB_LITE
};
namespace {
@@ -248,6 +250,7 @@ TEST_F(DBBlobCompactionTest, FilterByKeyLength) {
ASSERT_OK(db_->Get(ReadOptions(), long_key, &value));
ASSERT_EQ("value", value);
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -255,6 +258,7 @@ TEST_F(DBBlobCompactionTest, FilterByKeyLength) {
// this involves neither reading nor writing blobs
ASSERT_EQ(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -295,6 +299,7 @@ TEST_F(DBBlobCompactionTest, FilterByValueLength) {
ASSERT_EQ(long_value, value);
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -302,10 +307,12 @@ TEST_F(DBBlobCompactionTest, FilterByValueLength) {
// this involves reading but not writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobCompactionTest, BlobCompactWithStartingLevel) {
Options options = GetDefaultOptions();
@@ -381,6 +388,7 @@ TEST_F(DBBlobCompactionTest, BlobCompactWithStartingLevel) {
Close();
}
#endif
TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
Options options = GetDefaultOptions();
@@ -405,6 +413,7 @@ TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
ASSERT_EQ(new_blob_value, Get(key));
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -412,6 +421,7 @@ TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
// this involves writing but not reading blobs
ASSERT_EQ(compaction_stats[1].bytes_read_blob, 0);
ASSERT_GT(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -530,6 +540,7 @@ TEST_F(DBBlobCompactionTest, CompactionFilter) {
ASSERT_EQ(kv.second + std::string(padding), Get(kv.first));
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -537,6 +548,7 @@ TEST_F(DBBlobCompactionTest, CompactionFilter) {
// this involves reading and writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_GT(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -594,6 +606,7 @@ TEST_F(DBBlobCompactionTest, CompactionFilterReadBlobAndKeep) {
/*end=*/nullptr));
ASSERT_EQ(blob_files, GetBlobFileNumbers());
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -601,6 +614,7 @@ TEST_F(DBBlobCompactionTest, CompactionFilterReadBlobAndKeep) {
// this involves reading but not writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
+2
View File
@@ -34,6 +34,7 @@ class DBBlobCorruptionTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBBlobCorruptionTest, VerifyWholeBlobFileChecksum) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -70,6 +71,7 @@ TEST_F(DBBlobCorruptionTest, VerifyWholeBlobFileChecksum) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+6
View File
@@ -131,7 +131,9 @@ class DBBlobIndexTest : public DBTestBase {
ASSERT_OK(Flush());
ASSERT_OK(
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
break;
}
}
@@ -457,6 +459,7 @@ TEST_F(DBBlobIndexTest, Iterate) {
verify(15, Status::kOk, get_value(16, 0), get_value(14, 0),
create_blob_iterator, check_is_blob(false));
#ifndef ROCKSDB_LITE
// Iterator with blob support and using seek.
ASSERT_OK(dbfull()->SetOptions(
cfh(), {{"max_sequential_skip_in_iterations", "0"}}));
@@ -481,6 +484,7 @@ TEST_F(DBBlobIndexTest, Iterate) {
create_blob_iterator, check_is_blob(false));
verify(15, Status::kOk, get_value(16, 0), get_value(14, 0),
create_blob_iterator, check_is_blob(false));
#endif // !ROCKSDB_LITE
for (auto* snapshot : snapshots) {
dbfull()->ReleaseSnapshot(snapshot);
@@ -580,10 +584,12 @@ TEST_F(DBBlobIndexTest, IntegratedBlobIterate) {
Status expected_status;
verify(1, expected_status, expected_value);
#ifndef ROCKSDB_LITE
// Test DBIter::FindValueForCurrentKeyUsingSeek flow.
ASSERT_OK(dbfull()->SetOptions(cfh(),
{{"max_sequential_skip_in_iterations", "0"}}));
verify(1, expected_status, expected_value);
#endif // !ROCKSDB_LITE
}
} // namespace ROCKSDB_NAMESPACE
+2
View File
@@ -107,9 +107,11 @@ Status BuildTable(
std::vector<std::string> blob_file_paths;
std::string file_checksum = kUnknownFileChecksum;
std::string file_checksum_func_name = kUnknownFileChecksumFuncName;
#ifndef ROCKSDB_LITE
EventHelpers::NotifyTableFileCreationStarted(ioptions.listeners, dbname,
tboptions.column_family_name,
fname, job_id, tboptions.reason);
#endif // !ROCKSDB_LITE
Env* env = db_options.env;
assert(env);
FileSystem* fs = db_options.fs.get();
+12 -7
View File
@@ -7,6 +7,7 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef ROCKSDB_LITE
#include "rocksdb/c.h"
@@ -68,7 +69,6 @@ using ROCKSDB_NAMESPACE::CompactionOptionsFIFO;
using ROCKSDB_NAMESPACE::CompactRangeOptions;
using ROCKSDB_NAMESPACE::Comparator;
using ROCKSDB_NAMESPACE::CompressionType;
using ROCKSDB_NAMESPACE::ConfigOptions;
using ROCKSDB_NAMESPACE::CuckooTableOptions;
using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::DBOptions;
@@ -2498,12 +2498,8 @@ void rocksdb_load_latest_options(
rocksdb_options_t*** list_column_family_options, char** errptr) {
DBOptions db_opt;
std::vector<ColumnFamilyDescriptor> cf_descs;
ConfigOptions config_opts;
config_opts.ignore_unknown_options = ignore_unknown_options;
config_opts.input_strings_escaped = true;
config_opts.env = env->rep;
Status s = LoadLatestOptions(config_opts, std::string(db_path), &db_opt,
&cf_descs, &cache->rep);
Status s = LoadLatestOptions(std::string(db_path), env->rep, &db_opt,
&cf_descs, ignore_unknown_options, &cache->rep);
if (s.ok()) {
char** cf_names = (char**)malloc(cf_descs.size() * sizeof(char*));
rocksdb_options_t** cf_options = (rocksdb_options_t**)malloc(
@@ -2624,6 +2620,14 @@ void rocksdb_block_based_options_set_block_cache(
}
}
void rocksdb_block_based_options_set_block_cache_compressed(
rocksdb_block_based_table_options_t* options,
rocksdb_cache_t* block_cache_compressed) {
if (block_cache_compressed) {
options->rep.block_cache_compressed = block_cache_compressed->rep;
}
}
void rocksdb_block_based_options_set_whole_key_filtering(
rocksdb_block_based_table_options_t* options, unsigned char v) {
options->rep.whole_key_filtering = v;
@@ -6398,3 +6402,4 @@ void rocksdb_enable_manual_compaction(rocksdb_t* db) {
} // end extern "C"
#endif // !ROCKSDB_LITE
+11
View File
@@ -5,6 +5,8 @@
#include <stdio.h>
#ifndef ROCKSDB_LITE // Lite does not support C API
#include <assert.h>
#include <stddef.h>
#include <stdlib.h>
@@ -3466,3 +3468,12 @@ int main(int argc, char** argv) {
fprintf(stderr, "PASS\n");
return 0;
}
#else
int main(void) {
fprintf(stderr, "SKIPPED\n");
return 0;
}
#endif // !ROCKSDB_LITE
+13
View File
@@ -53,9 +53,11 @@ ColumnFamilyHandleImpl::ColumnFamilyHandleImpl(
ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl() {
if (cfd_ != nullptr) {
#ifndef ROCKSDB_LITE
for (auto& listener : cfd_->ioptions()->listeners) {
listener->OnColumnFamilyHandleDeletionStarted(this);
}
#endif // ROCKSDB_LITE
// Job id == 0 means that this is not our background process, but rather
// user thread
// Need to hold some shared pointers owned by the initial_cf_options
@@ -86,10 +88,15 @@ const std::string& ColumnFamilyHandleImpl::GetName() const {
}
Status ColumnFamilyHandleImpl::GetDescriptor(ColumnFamilyDescriptor* desc) {
#ifndef ROCKSDB_LITE
// accessing mutable cf-options requires db mutex.
InstrumentedMutexLock l(mutex_);
*desc = ColumnFamilyDescriptor(cfd()->GetName(), cfd()->GetLatestCFOptions());
return Status::OK();
#else
(void)desc;
return Status::NotSupported();
#endif // !ROCKSDB_LITE
}
const Comparator* ColumnFamilyHandleImpl::GetComparator() const {
@@ -340,6 +347,7 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
result.hard_pending_compaction_bytes_limit;
}
#ifndef ROCKSDB_LITE
// When the DB is stopped, it's possible that there are some .trash files that
// were not deleted yet, when we open the DB we will find these .trash files
// and schedule them to be deleted (or delete immediately if SstFileManager
@@ -351,6 +359,7 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
result.cf_paths[i].path)
.PermitUncheckedError();
}
#endif
if (result.cf_paths.empty()) {
result.cf_paths = db_options.db_paths;
@@ -593,6 +602,7 @@ ColumnFamilyData::ColumnFamilyData(
if (ioptions_.compaction_style == kCompactionStyleLevel) {
compaction_picker_.reset(
new LevelCompactionPicker(ioptions_, &internal_comparator_));
#ifndef ROCKSDB_LITE
} else if (ioptions_.compaction_style == kCompactionStyleUniversal) {
compaction_picker_.reset(
new UniversalCompactionPicker(ioptions_, &internal_comparator_));
@@ -606,6 +616,7 @@ ColumnFamilyData::ColumnFamilyData(
"Column family %s does not use any background compaction. "
"Compactions can only be done via CompactFiles\n",
GetName().c_str());
#endif // !ROCKSDB_LITE
} else {
ROCKS_LOG_ERROR(ioptions_.logger,
"Unable to recognize the specified compaction style %d. "
@@ -1429,6 +1440,7 @@ Status ColumnFamilyData::ValidateOptions(
return s;
}
#ifndef ROCKSDB_LITE
Status ColumnFamilyData::SetOptions(
const DBOptions& db_opts,
const std::unordered_map<std::string, std::string>& options_map) {
@@ -1447,6 +1459,7 @@ Status ColumnFamilyData::SetOptions(
}
return s;
}
#endif // ROCKSDB_LITE
// REQUIRES: DB mutex held
Env::WriteLifeTimeHint ColumnFamilyData::CalculateSSTWriteHint(int level) {
+2
View File
@@ -335,10 +335,12 @@ class ColumnFamilyData {
// Validate CF options against DB options
static Status ValidateOptions(const DBOptions& db_options,
const ColumnFamilyOptions& cf_options);
#ifndef ROCKSDB_LITE
// REQUIRES: DB mutex held
Status SetOptions(
const DBOptions& db_options,
const std::unordered_map<std::string, std::string>& options_map);
#endif // ROCKSDB_LITE
InternalStats* internal_stats() { return internal_stats_.get(); }
+71
View File
@@ -71,7 +71,11 @@ class ColumnFamilyTestBase : public testing::Test {
for (auto h : handles_) {
ColumnFamilyDescriptor cfdescriptor;
Status s = h->GetDescriptor(&cfdescriptor);
#ifdef ROCKSDB_LITE
EXPECT_TRUE(s.IsNotSupported());
#else
EXPECT_OK(s);
#endif // ROCKSDB_LITE
column_families.push_back(cfdescriptor);
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
@@ -193,10 +197,12 @@ class ColumnFamilyTestBase : public testing::Test {
&db_);
}
#ifndef ROCKSDB_LITE // ReadOnlyDB is not supported
void AssertOpenReadOnly(std::vector<std::string> cf,
std::vector<ColumnFamilyOptions> options = {}) {
ASSERT_OK(OpenReadOnly(cf, options));
}
#endif // !ROCKSDB_LITE
void Open(std::vector<std::string> cf,
std::vector<ColumnFamilyOptions> options = {}) {
@@ -218,16 +224,27 @@ class ColumnFamilyTestBase : public testing::Test {
}
bool IsDbWriteStopped() {
#ifndef ROCKSDB_LITE
uint64_t v;
EXPECT_TRUE(dbfull()->GetIntProperty("rocksdb.is-write-stopped", &v));
return (v == 1);
#else
return dbfull()->TEST_write_controler().IsStopped();
#endif // !ROCKSDB_LITE
}
uint64_t GetDbDelayedWriteRate() {
#ifndef ROCKSDB_LITE
uint64_t v;
EXPECT_TRUE(
dbfull()->GetIntProperty("rocksdb.actual-delayed-write-rate", &v));
return v;
#else
if (!dbfull()->TEST_write_controler().NeedsDelay()) {
return 0;
}
return dbfull()->TEST_write_controler().delayed_write_rate();
#endif // !ROCKSDB_LITE
}
void Destroy(const std::vector<ColumnFamilyDescriptor>& column_families =
@@ -250,6 +267,7 @@ class ColumnFamilyTestBase : public testing::Test {
db_->CreateColumnFamily(current_cf_opt, cfs[i], &handles_[cfi]));
names_[cfi] = cfs[i];
#ifndef ROCKSDB_LITE // RocksDBLite does not support GetDescriptor
// Verify the CF options of the returned CF handle.
ColumnFamilyDescriptor desc;
ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));
@@ -258,6 +276,7 @@ class ColumnFamilyTestBase : public testing::Test {
ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(
ConfigOptions(), desc.options,
SanitizeOptions(dbfull()->immutable_db_options(), current_cf_opt)));
#endif // !ROCKSDB_LITE
cfi++;
}
}
@@ -306,6 +325,7 @@ class ColumnFamilyTestBase : public testing::Test {
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
}
#ifndef ROCKSDB_LITE // TEST functions in DB are not supported in lite
void WaitForFlush(int cf) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[cf]));
}
@@ -319,6 +339,7 @@ class ColumnFamilyTestBase : public testing::Test {
void AssertMaxTotalInMemoryState(uint64_t value) {
ASSERT_EQ(value, MaxTotalInMemoryState());
}
#endif // !ROCKSDB_LITE
Status Put(int cf, const std::string& key, const std::string& value) {
return db_->Put(WriteOptions(), handles_[cf], Slice(key), Slice(value));
@@ -356,6 +377,7 @@ class ColumnFamilyTestBase : public testing::Test {
"rocksdb.num-files-at-level" + std::to_string(level));
}
#ifndef ROCKSDB_LITE
// Return spread of files per level
std::string FilesPerLevel(int cf) {
std::string result;
@@ -372,19 +394,31 @@ class ColumnFamilyTestBase : public testing::Test {
result.resize(last_non_zero_offset);
return result;
}
#endif
void AssertFilesPerLevel(const std::string& value, int cf) {
#ifndef ROCKSDB_LITE
ASSERT_EQ(value, FilesPerLevel(cf));
#else
(void)value;
(void)cf;
#endif
}
#ifndef ROCKSDB_LITE // GetLiveFilesMetaData is not supported
int CountLiveFiles() {
std::vector<LiveFileMetaData> metadata;
db_->GetLiveFilesMetaData(&metadata);
return static_cast<int>(metadata.size());
}
#endif // !ROCKSDB_LITE
void AssertCountLiveFiles(int expected_value) {
#ifndef ROCKSDB_LITE
ASSERT_EQ(expected_value, CountLiveFiles());
#else
(void)expected_value;
#endif
}
// Do n memtable flushes, each of which produces an sstable
@@ -398,6 +432,7 @@ class ColumnFamilyTestBase : public testing::Test {
}
}
#ifndef ROCKSDB_LITE // GetSortedWalFiles is not supported
int CountLiveLogFiles() {
int micros_wait_for_log_deletion = 20000;
env_->SleepForMicroseconds(micros_wait_for_log_deletion);
@@ -426,18 +461,25 @@ class ColumnFamilyTestBase : public testing::Test {
return ret;
return 0;
}
#endif // !ROCKSDB_LITE
void AssertCountLiveLogFiles(int value) {
#ifndef ROCKSDB_LITE // GetSortedWalFiles is not supported
ASSERT_EQ(value, CountLiveLogFiles());
#else
(void)value;
#endif // !ROCKSDB_LITE
}
void AssertNumberOfImmutableMemtables(std::vector<int> num_per_cf) {
assert(num_per_cf.size() == handles_.size());
#ifndef ROCKSDB_LITE // GetProperty is not supported in lite
for (size_t i = 0; i < num_per_cf.size(); ++i) {
ASSERT_EQ(num_per_cf[i], GetProperty(static_cast<int>(i),
"rocksdb.num-immutable-mem-table"));
}
#endif // !ROCKSDB_LITE
}
void CopyFile(const std::string& source, const std::string& destination,
@@ -533,6 +575,7 @@ TEST_P(ColumnFamilyTest, DontReuseColumnFamilyID) {
}
}
#ifndef ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
Open();
@@ -555,6 +598,7 @@ TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // !ROCKSDB_LITE
class FlushEmptyCFTestWithParam
: public ColumnFamilyTestBase,
@@ -898,6 +942,7 @@ TEST_P(ColumnFamilyTest, IgnoreRecoveredLog) {
}
}
#ifndef ROCKSDB_LITE // TEST functions used are not supported
TEST_P(ColumnFamilyTest, FlushTest) {
Open();
CreateColumnFamiliesAndReopen({"one", "two"});
@@ -1012,6 +1057,7 @@ TEST_P(ColumnFamilyTest, LogDeletionTest) {
AssertCountLiveLogFiles(4);
Close();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CrashAfterFlush) {
std::unique_ptr<FaultInjectionTestEnv> fault_env(
@@ -1051,6 +1097,7 @@ TEST_P(ColumnFamilyTest, OpenNonexistentColumnFamily) {
ASSERT_TRUE(TryOpen({"default", "dne"}).IsInvalidArgument());
}
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
// Makes sure that obsolete log files get deleted
TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
// disable flushing stale column families
@@ -1158,12 +1205,14 @@ TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
AssertCountLiveLogFiles(7);
Close();
}
#endif // !ROCKSDB_LITE
// The test is commented out because we want to test that snapshot is
// not created for memtables not supported it, but There isn't a memtable
// that doesn't support snapshot right now. If we have one later, we can
// re-enable the test.
//
// #ifndef ROCKSDB_LITE // Cuckoo is not supported in lite
// TEST_P(ColumnFamilyTest, MemtableNotSupportSnapshot) {
// db_options_.allow_concurrent_memtable_write = false;
// Open();
@@ -1183,6 +1232,7 @@ TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
// {second}); auto* s3 = dbfull()->GetSnapshot(); ASSERT_TRUE(s3 == nullptr);
// Close();
// }
// #endif // !ROCKSDB_LITE
class TestComparator : public Comparator {
int Compare(const ROCKSDB_NAMESPACE::Slice& /*a*/,
@@ -1249,6 +1299,7 @@ TEST_P(ColumnFamilyTest, DifferentMergeOperators) {
Close();
}
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
TEST_P(ColumnFamilyTest, DifferentCompactionStyles) {
Open();
CreateColumnFamilies({"one", "two"});
@@ -1316,7 +1367,9 @@ TEST_P(ColumnFamilyTest, DifferentCompactionStyles) {
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE
// Sync points not supported in RocksDB Lite
TEST_P(ColumnFamilyTest, MultipleManualCompactions) {
@@ -1980,7 +2033,9 @@ TEST_P(ColumnFamilyTest, SameCFAutomaticManualCompactions) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // Tailing iterator not supported
namespace {
std::string IterStatus(Iterator* iter) {
std::string result;
@@ -2038,7 +2093,9 @@ TEST_P(ColumnFamilyTest, NewIteratorsTest) {
Destroy();
}
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // ReadOnlyDB is not supported
TEST_P(ColumnFamilyTest, ReadOnlyDBTest) {
Open();
CreateColumnFamiliesAndReopen({"one", "two", "three", "four"});
@@ -2087,7 +2144,9 @@ TEST_P(ColumnFamilyTest, ReadOnlyDBTest) {
s = OpenReadOnly({"one", "four"});
ASSERT_TRUE(!s.ok());
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported in lite
TEST_P(ColumnFamilyTest, DontRollEmptyLogs) {
Open();
CreateColumnFamiliesAndReopen({"one", "two", "three", "four"});
@@ -2109,7 +2168,9 @@ TEST_P(ColumnFamilyTest, DontRollEmptyLogs) {
ASSERT_EQ(static_cast<size_t>(total_new_writable_files), handles_.size() + 1);
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForCompaction() is not supported in lite
TEST_P(ColumnFamilyTest, FlushStaleColumnFamilies) {
Open();
CreateColumnFamilies({"one", "two"});
@@ -2156,6 +2217,7 @@ TEST_P(ColumnFamilyTest, FlushStaleColumnFamilies) {
ASSERT_EQ(0, dbfull()->TEST_total_log_size());
Close();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateMissingColumnFamilies) {
Status s = TryOpen({"one", "two"});
@@ -2395,6 +2457,8 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
Destroy();
}
#ifndef ROCKSDB_LITE
// skipped as persisting options is not supported in ROCKSDB_LITE
namespace {
std::atomic<int> test_stage(0);
std::atomic<bool> ordered_by_writethread(false);
@@ -2476,6 +2540,7 @@ TEST_P(ColumnFamilyTest, CreateAndDropRace) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, WriteStallSingleColumnFamily) {
const uint64_t kBaseRate = 800000u;
@@ -2885,6 +2950,7 @@ TEST_P(ColumnFamilyTest, CreateDropAndDestroy) {
ASSERT_OK(db_->DestroyColumnFamilyHandle(cfh));
}
#ifndef ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateDropAndDestroyWithoutFileDeletion) {
ColumnFamilyHandle* cfh;
Open();
@@ -2939,7 +3005,9 @@ TEST_P(ColumnFamilyTest, FlushCloseWALFiles) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
TEST_P(ColumnFamilyTest, IteratorCloseWALFile1) {
SpecialEnv env(Env::Default());
db_options_.env = &env;
@@ -3046,7 +3114,9 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // TEST functions are not supported in lite
TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
SpecialEnv env(Env::Default());
// Allow both of flush and purge job to schedule.
@@ -3122,6 +3192,7 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
// Disable on windows because SyncWAL requires env->IsSyncThreadSafe()
// to return true which is not so in unbuffered mode.
+11
View File
@@ -3,6 +3,7 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include <mutex>
#include <string>
@@ -489,3 +490,13 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as DBImpl::CompactFiles is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
+35 -44
View File
@@ -221,43 +221,39 @@ void CompactionIterator::Next() {
bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
Slice* skip_until) {
if (!compaction_filter_) {
return true;
}
// TODO: support compaction filter for wide-column entities
if (ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex) {
if (!compaction_filter_ ||
(ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex)) {
return true;
}
CompactionFilter::Decision decision =
CompactionFilter::Decision::kUndetermined;
bool error = false;
// If the user has specified a compaction filter and the sequence
// number is greater than any external snapshot, then invoke the
// filter. If the return value of the compaction filter is true,
// replace the entry with a deletion marker.
CompactionFilter::Decision filter = CompactionFilter::Decision::kUndetermined;
compaction_filter_value_.clear();
compaction_filter_skip_until_.Clear();
CompactionFilter::ValueType value_type =
ikey_.type == kTypeValue ? CompactionFilter::ValueType::kValue
: CompactionFilter::ValueType::kBlobIndex;
// Hack: pass internal key to BlobIndexCompactionFilter since it needs
// to get sequence number.
assert(compaction_filter_);
const Slice& filter_key =
(ikey_.type != kTypeBlobIndex ||
Slice& filter_key =
(ikey_.type == kTypeValue ||
!compaction_filter_->IsStackedBlobDbInternalCompactionFilter())
? ikey_.user_key
: key_;
compaction_filter_value_.clear();
compaction_filter_skip_until_.Clear();
{
StopWatchNano timer(clock_, report_detailed_time_);
if (ikey_.type == kTypeBlobIndex) {
decision = compaction_filter_->FilterBlobByKey(
if (kTypeBlobIndex == ikey_.type) {
filter = compaction_filter_->FilterBlobByKey(
level_, filter_key, &compaction_filter_value_,
compaction_filter_skip_until_.rep());
if (decision == CompactionFilter::Decision::kUndetermined &&
if (CompactionFilter::Decision::kUndetermined == filter &&
!compaction_filter_->IsStackedBlobDbInternalCompactionFilter()) {
if (!compaction_) {
if (compaction_ == nullptr) {
status_ =
Status::Corruption("Unexpected blob index outside of compaction");
validity_info_.Invalidate();
@@ -303,18 +299,17 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
value_type = CompactionFilter::ValueType::kValue;
}
}
if (decision == CompactionFilter::Decision::kUndetermined) {
decision = compaction_filter_->FilterV2(
if (CompactionFilter::Decision::kUndetermined == filter) {
filter = compaction_filter_->FilterV2(
level_, filter_key, value_type,
blob_value_.empty() ? value_ : blob_value_, &compaction_filter_value_,
compaction_filter_skip_until_.rep());
}
iter_stats_.total_filter_time +=
env_ != nullptr && report_detailed_time_ ? timer.ElapsedNanos() : 0;
}
if (decision == CompactionFilter::Decision::kUndetermined) {
if (CompactionFilter::Decision::kUndetermined == filter) {
// Should not reach here, since FilterV2 should never return kUndetermined.
status_ =
Status::NotSupported("FilterV2() should never return kUndetermined");
@@ -322,15 +317,15 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
return false;
}
if (decision == CompactionFilter::Decision::kRemoveAndSkipUntil &&
if (filter == CompactionFilter::Decision::kRemoveAndSkipUntil &&
cmp_->Compare(*compaction_filter_skip_until_.rep(), ikey_.user_key) <=
0) {
// Can't skip to a key smaller than the current one.
// Keep the key as per FilterV2 documentation.
decision = CompactionFilter::Decision::kKeep;
filter = CompactionFilter::Decision::kKeep;
}
if (decision == CompactionFilter::Decision::kRemove) {
if (filter == CompactionFilter::Decision::kRemove) {
// convert the current key to a delete; key_ is pointing into
// current_key_ at this point, so updating current_key_ updates key()
ikey_.type = kTypeDeletion;
@@ -338,7 +333,7 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
// no value associated with delete
value_.clear();
iter_stats_.num_record_drop_user++;
} else if (decision == CompactionFilter::Decision::kPurge) {
} else if (filter == CompactionFilter::Decision::kPurge) {
// convert the current key to a single delete; key_ is pointing into
// current_key_ at this point, so updating current_key_ updates key()
ikey_.type = kTypeSingleDeletion;
@@ -346,19 +341,19 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
// no value associated with single delete
value_.clear();
iter_stats_.num_record_drop_user++;
} else if (decision == CompactionFilter::Decision::kChangeValue) {
if (ikey_.type != kTypeValue) {
} else if (filter == CompactionFilter::Decision::kChangeValue) {
if (ikey_.type == kTypeBlobIndex) {
// value transfer from blob file to inlined data
ikey_.type = kTypeValue;
current_key_.UpdateInternalKey(ikey_.sequence, kTypeValue);
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
}
value_ = compaction_filter_value_;
} else if (decision == CompactionFilter::Decision::kRemoveAndSkipUntil) {
} else if (filter == CompactionFilter::Decision::kRemoveAndSkipUntil) {
*need_skip = true;
compaction_filter_skip_until_.ConvertFromUserKey(kMaxSequenceNumber,
kValueTypeForSeek);
*skip_until = compaction_filter_skip_until_.Encode();
} else if (decision == CompactionFilter::Decision::kChangeBlobIndex) {
} else if (filter == CompactionFilter::Decision::kChangeBlobIndex) {
// Only the StackableDB-based BlobDB impl's compaction filter should return
// kChangeBlobIndex. Decision about rewriting blob and changing blob index
// in the integrated BlobDB impl is made in subsequent call to
@@ -370,27 +365,23 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
validity_info_.Invalidate();
return false;
}
if (ikey_.type != kTypeBlobIndex) {
if (ikey_.type == kTypeValue) {
// value transfer from inlined data to blob file
ikey_.type = kTypeBlobIndex;
current_key_.UpdateInternalKey(ikey_.sequence, kTypeBlobIndex);
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
}
value_ = compaction_filter_value_;
} else if (decision == CompactionFilter::Decision::kIOError) {
} else if (filter == CompactionFilter::Decision::kIOError) {
if (!compaction_filter_->IsStackedBlobDbInternalCompactionFilter()) {
status_ = Status::NotSupported(
"CompactionFilter for integrated BlobDB should not return kIOError");
validity_info_.Invalidate();
return false;
}
status_ = Status::IOError("Failed to access blob during compaction filter");
validity_info_.Invalidate();
return false;
error = true;
}
return true;
return !error;
}
void CompactionIterator::NextFromInput() {
+20
View File
@@ -978,6 +978,7 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
void CompactionJob::NotifyOnSubcompactionBegin(
SubcompactionState* sub_compact) {
#ifndef ROCKSDB_LITE
Compaction* c = compact_->compaction;
if (db_options_.listeners.empty()) {
@@ -1003,10 +1004,14 @@ void CompactionJob::NotifyOnSubcompactionBegin(
}
info.status.PermitUncheckedError();
#else
(void)sub_compact;
#endif // ROCKSDB_LITE
}
void CompactionJob::NotifyOnSubcompactionCompleted(
SubcompactionState* sub_compact) {
#ifndef ROCKSDB_LITE
if (db_options_.listeners.empty()) {
return;
@@ -1027,12 +1032,16 @@ void CompactionJob::NotifyOnSubcompactionCompleted(
for (const auto& listener : db_options_.listeners) {
listener->OnSubcompactionCompleted(info);
}
#else
(void)sub_compact;
#endif // ROCKSDB_LITE
}
void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
assert(sub_compact);
assert(sub_compact->compaction);
#ifndef ROCKSDB_LITE
if (db_options_.compaction_service) {
CompactionServiceJobStatus comp_status =
ProcessKeyValueCompactionWithCompactionService(sub_compact);
@@ -1043,6 +1052,7 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
// fallback to local compaction
assert(comp_status == CompactionServiceJobStatus::kUseLocal);
}
#endif // !ROCKSDB_LITE
uint64_t prev_cpu_micros = db_options_.clock->CPUMicros();
@@ -1605,6 +1615,7 @@ Status CompactionJob::FinishCompactionOutputFile(
TableFileCreationReason::kCompaction, status_for_listener, file_checksum,
file_checksum_func_name);
#ifndef ROCKSDB_LITE
// Report new file to SstFileManagerImpl
auto sfm =
static_cast<SstFileManagerImpl*>(db_options_.sst_file_manager.get());
@@ -1623,6 +1634,7 @@ Status CompactionJob::FinishCompactionOutputFile(
db_error_handler_->SetBGError(s, BackgroundErrorReason::kCompaction);
}
}
#endif
outputs.ResetBuilder();
return s;
@@ -1747,9 +1759,11 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
std::string fname = GetTableFileName(file_number);
// Fire events.
ColumnFamilyData* cfd = sub_compact->compaction->column_family_data();
#ifndef ROCKSDB_LITE
EventHelpers::NotifyTableFileCreationStarted(
cfd->ioptions()->listeners, dbname_, cfd->GetName(), fname, job_id_,
TableFileCreationReason::kCompaction);
#endif // !ROCKSDB_LITE
// Make the output file
std::unique_ptr<FSWritableFile> writable_file;
#ifndef NDEBUG
@@ -1886,6 +1900,7 @@ void CompactionJob::CleanupCompaction() {
compact_ = nullptr;
}
#ifndef ROCKSDB_LITE
namespace {
void CopyPrefix(const Slice& src, size_t prefix_length, std::string* dst) {
assert(prefix_length > 0);
@@ -1894,6 +1909,7 @@ void CopyPrefix(const Slice& src, size_t prefix_length, std::string* dst) {
}
} // namespace
#endif // !ROCKSDB_LITE
void CompactionJob::UpdateCompactionStats() {
assert(compact_);
@@ -1940,6 +1956,7 @@ void CompactionJob::UpdateCompactionInputStatsHelper(int* num_files,
void CompactionJob::UpdateCompactionJobStats(
const InternalStats::CompactionStats& stats) const {
#ifndef ROCKSDB_LITE
compaction_job_stats_->elapsed_micros = stats.micros;
// input information
@@ -1966,6 +1983,9 @@ void CompactionJob::UpdateCompactionJobStats(
CopyPrefix(compact_->LargestUserKey(), CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->largest_output_key_prefix);
}
#else
(void)stats;
#endif // !ROCKSDB_LITE
}
void CompactionJob::LogCompaction() {
@@ -59,6 +59,7 @@
#include "utilities/merge_operators.h"
#if !defined(IOS_CROSS_COMPILE)
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
static std::string RandomString(Random* rnd, int len, double ratio) {
@@ -958,6 +959,15 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED, not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
#else
+11
View File
@@ -3,6 +3,7 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_job.h"
@@ -2441,3 +2442,13 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as CompactionJobStats is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+2
View File
@@ -877,6 +877,7 @@ Compaction* CompactionPicker::CompactRange(
return compaction;
}
#ifndef ROCKSDB_LITE
namespace {
// Test whether two files have overlapping key-ranges.
bool HaveOverlappingKeyRanges(const Comparator* c, const SstFileMetaData& a,
@@ -1115,6 +1116,7 @@ Status CompactionPicker::SanitizeCompactionInputFiles(
return Status::OK();
}
#endif // !ROCKSDB_LITE
void CompactionPicker::RegisterCompaction(Compaction* c) {
if (c == nullptr) {
+6
View File
@@ -93,9 +93,11 @@ class CompactionPicker {
// into a valid one by adding more files, the function will return a
// non-ok status with specific reason.
//
#ifndef ROCKSDB_LITE
Status SanitizeCompactionInputFiles(std::unordered_set<uint64_t>* input_files,
const ColumnFamilyMetaData& cf_meta,
const int output_level) const;
#endif // ROCKSDB_LITE
// Free up the files that participated in a compaction
//
@@ -227,9 +229,11 @@ class CompactionPicker {
// A helper function to SanitizeCompactionInputFiles() that
// sanitizes "input_files" by adding necessary files.
#ifndef ROCKSDB_LITE
virtual Status SanitizeCompactionInputFilesForAllLevels(
std::unordered_set<uint64_t>* input_files,
const ColumnFamilyMetaData& cf_meta, const int output_level) const;
#endif // ROCKSDB_LITE
// Keeps track of all compactions that are running on Level0.
// Protected by DB mutex
@@ -242,6 +246,7 @@ class CompactionPicker {
const InternalKeyComparator* const icmp_;
};
#ifndef ROCKSDB_LITE
// A dummy compaction that never triggers any automatic
// compaction.
class NullCompactionPicker : public CompactionPicker {
@@ -282,6 +287,7 @@ class NullCompactionPicker : public CompactionPicker {
return false;
}
};
#endif // !ROCKSDB_LITE
// Attempts to find an intra L0 compaction conforming to the given parameters.
//
+2
View File
@@ -8,6 +8,7 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction_picker_fifo.h"
#ifndef ROCKSDB_LITE
#include <cinttypes>
#include <string>
@@ -442,3 +443,4 @@ Compaction* FIFOCompactionPicker::CompactRange(
}
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+2
View File
@@ -8,6 +8,7 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#pragma once
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_picker.h"
@@ -59,3 +60,4 @@ class FIFOCompactionPicker : public CompactionPicker {
LogBuffer* log_buffer);
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+5
View File
@@ -482,6 +482,8 @@ TEST_F(CompactionPickerTest, LevelTriggerDynamic4) {
ASSERT_EQ(num_levels - 1, compaction->output_level());
}
// Universal and FIFO Compactions are not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(CompactionPickerTest, NeedsCompactionUniversal) {
NewVersionStorage(1, kCompactionStyleUniversal);
UniversalCompactionPicker universal_compaction_picker(ioptions_, &icmp_);
@@ -1243,6 +1245,7 @@ TEST_F(CompactionPickerTest, FIFOToWarmWithHotBetweenWarms) {
ASSERT_EQ(2U, compaction->input(0, 0)->fd.GetNumber());
}
#endif // ROCKSDB_LITE
TEST_F(CompactionPickerTest, CompactionPriMinOverlapping1) {
NewVersionStorage(6, kCompactionStyleLevel);
@@ -2870,6 +2873,7 @@ TEST_F(CompactionPickerTest, IntraL0MaxCompactionBytesHit) {
ASSERT_EQ(0, compaction->output_level());
}
#ifndef ROCKSDB_LITE
TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
const uint64_t kFileSize = 100000;
@@ -3978,6 +3982,7 @@ TEST_P(PerKeyPlacementCompactionPickerTest,
INSTANTIATE_TEST_CASE_P(PerKeyPlacementCompactionPickerTest,
PerKeyPlacementCompactionPickerTest, ::testing::Bool());
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
@@ -8,6 +8,7 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction_picker_universal.h"
#ifndef ROCKSDB_LITE
#include <cinttypes>
#include <limits>
@@ -1450,3 +1451,4 @@ uint64_t UniversalCompactionBuilder::GetMaxOverlappingBytes() const {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
@@ -8,6 +8,7 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#pragma once
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_picker.h"
@@ -28,3 +29,4 @@ class UniversalCompactionPicker : public CompactionPicker {
const VersionStorageInfo* vstorage) const override;
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+2
View File
@@ -16,6 +16,7 @@
#include "options/options_helper.h"
#include "rocksdb/utilities/options_type.h"
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
class SubcompactionState;
@@ -831,3 +832,4 @@ bool CompactionServiceInput::TEST_Equals(CompactionServiceInput* other,
#endif // NDEBUG
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+11
View File
@@ -3,6 +3,7 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_test_util.h"
#include "port/stack_trace.h"
@@ -953,3 +954,13 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as CompactionService is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+6
View File
@@ -15,9 +15,11 @@
namespace ROCKSDB_NAMESPACE {
static std::unordered_map<std::string, OptionTypeInfo>
sst_fixed_prefix_type_info = {
#ifndef ROCKSDB_LITE
{"length",
{0, OptionType::kSizeT, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
#endif // ROCKSDB_LITE
};
SstPartitionerFixedPrefixFactory::SstPartitionerFixedPrefixFactory(size_t len)
@@ -56,6 +58,7 @@ std::shared_ptr<SstPartitionerFactory> NewSstPartitionerFixedPrefixFactory(
return std::make_shared<SstPartitionerFixedPrefixFactory>(prefix_len);
}
#ifndef ROCKSDB_LITE
namespace {
static int RegisterSstPartitionerFactories(ObjectLibrary& library,
const std::string& /*arg*/) {
@@ -70,14 +73,17 @@ static int RegisterSstPartitionerFactories(ObjectLibrary& library,
return 1;
}
} // namespace
#endif // ROCKSDB_LITE
Status SstPartitionerFactory::CreateFromString(
const ConfigOptions& options, const std::string& value,
std::shared_ptr<SstPartitionerFactory>* result) {
#ifndef ROCKSDB_LITE
static std::once_flag once;
std::call_once(once, [&]() {
RegisterSstPartitionerFactories(*(ObjectLibrary::Default().get()), "");
});
#endif // ROCKSDB_LITE
return LoadSharedObject<SstPartitionerFactory>(options, value, nullptr,
result);
}
+2
View File
@@ -99,6 +99,7 @@ class SubcompactionState {
penultimate_level_outputs_.RemoveLastEmptyOutput();
}
#ifndef ROCKSDB_LITE
void BuildSubcompactionJobInfo(
SubcompactionJobInfo& subcompaction_job_info) const {
const Compaction* c = compaction;
@@ -112,6 +113,7 @@ class SubcompactionState {
subcompaction_job_info.output_level = c->output_level();
subcompaction_job_info.stats = compaction_job_stats;
}
#endif // !ROCKSDB_LITE
SubcompactionState() = delete;
SubcompactionState(const SubcompactionState&) = delete;
+8
View File
@@ -17,6 +17,7 @@
namespace ROCKSDB_NAMESPACE {
#if !defined(ROCKSDB_LITE)
class TieredCompactionTest : public DBTestBase,
public testing::WithParamInterface<bool> {
@@ -2138,11 +2139,18 @@ TEST_F(PrecludeLastLevelTest, RangeDelsCauseFileEndpointsToOverlap) {
Close();
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+2
View File
@@ -4,6 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
//
#ifndef ROCKSDB_LITE
#include "rocksdb/convenience.h"
@@ -77,3 +78,4 @@ Status VerifySstFileChecksum(const Options& options,
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+56 -53
View File
@@ -8,6 +8,7 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "rocksdb/options.h"
#ifndef ROCKSDB_LITE
#include <fcntl.h>
#include <sys/stat.h>
@@ -42,36 +43,33 @@ namespace ROCKSDB_NAMESPACE {
static constexpr int kValueSize = 1000;
namespace {
// A wrapper that allows injection of errors.
class ErrorFS : public FileSystemWrapper {
class ErrorEnv : public EnvWrapper {
public:
bool writable_file_error_;
int num_writable_file_errors_;
explicit ErrorFS(const std::shared_ptr<FileSystem>& _target)
: FileSystemWrapper(_target),
explicit ErrorEnv(Env* _target)
: EnvWrapper(_target),
writable_file_error_(false),
num_writable_file_errors_(0) {}
const char* Name() const override { return "ErrorEnv"; }
virtual IOStatus NewWritableFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override {
virtual Status NewWritableFile(const std::string& fname,
std::unique_ptr<WritableFile>* result,
const EnvOptions& soptions) override {
result->reset();
if (writable_file_error_) {
++num_writable_file_errors_;
return IOStatus::IOError(fname, "fake error");
return Status::IOError(fname, "fake error");
}
return target()->NewWritableFile(fname, opts, result, dbg);
return target()->NewWritableFile(fname, result, soptions);
}
};
} // anonymous namespace
class CorruptionTest : public testing::Test {
public:
std::shared_ptr<Env> env_guard_;
std::shared_ptr<ErrorFS> fs_;
std::unique_ptr<Env> env_;
Env* base_env_;
ErrorEnv* env_;
std::string dbname_;
std::shared_ptr<Cache> tiny_cache_;
Options options_;
@@ -82,15 +80,14 @@ class CorruptionTest : public testing::Test {
// set it to 0), test SequenceNumberRecovery will fail, likely because of a
// bug in recovery code. Keep it 4 for now to make the test passes.
tiny_cache_ = NewLRUCache(100, 4);
base_env_ = Env::Default();
Env* base_env = Env::Default();
EXPECT_OK(
test::CreateEnvFromSystem(ConfigOptions(), &base_env_, &env_guard_));
EXPECT_NE(base_env_, nullptr);
fs_.reset(new ErrorFS(base_env_->GetFileSystem()));
env_ = NewCompositeEnv(fs_);
test::CreateEnvFromSystem(ConfigOptions(), &base_env, &env_guard_));
EXPECT_NE(base_env, nullptr);
env_ = new ErrorEnv(base_env);
options_.wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
options_.env = env_.get();
dbname_ = test::PerThreadDBPath(env_.get(), "corruption_test");
options_.env = env_;
dbname_ = test::PerThreadDBPath(env_, "corruption_test");
Status s = DestroyDB(dbname_, options_);
EXPECT_OK(s);
@@ -113,9 +110,10 @@ class CorruptionTest : public testing::Test {
fprintf(stdout, "db is still at %s\n", dbname_.c_str());
} else {
Options opts;
opts.env = base_env_;
opts.env = env_->target();
EXPECT_OK(DestroyDB(dbname_, opts));
}
delete env_;
}
void CloseDb() {
@@ -130,7 +128,7 @@ class CorruptionTest : public testing::Test {
if (opt.env == Options().env) {
// If env is not overridden, replace it with ErrorEnv.
// Otherwise, the test already uses a non-default Env.
opt.env = env_.get();
opt.env = env_;
}
opt.arena_block_size = 4096;
BlockBasedTableOptions table_options;
@@ -167,10 +165,6 @@ class CorruptionTest : public testing::Test {
void Build(int n, int flush_every = 0) { Build(n, 0, flush_every); }
void Check(int min_expected, int max_expected) {
Check(min_expected, max_expected, ReadOptions(false, true));
}
void Check(int min_expected, int max_expected, ReadOptions read_options) {
uint64_t next_expected = 0;
uint64_t missed = 0;
int bad_keys = 0;
@@ -182,7 +176,7 @@ class CorruptionTest : public testing::Test {
// Instead, we want the reads to be successful and this test
// will detect whether the appropriate corruptions have
// occurred.
Iterator* iter = db_->NewIterator(read_options);
Iterator* iter = db_->NewIterator(ReadOptions(false, true));
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ASSERT_OK(iter->status());
uint64_t key;
@@ -229,7 +223,7 @@ class CorruptionTest : public testing::Test {
}
ASSERT_TRUE(!fname.empty()) << filetype;
ASSERT_OK(test::CorruptFile(env_.get(), fname, offset, bytes_to_corrupt,
ASSERT_OK(test::CorruptFile(env_, fname, offset, bytes_to_corrupt,
/*verify_checksum*/ filetype == kTableFile));
}
@@ -240,7 +234,7 @@ class CorruptionTest : public testing::Test {
db_->GetLiveFilesMetaData(&metadata);
for (const auto& m : metadata) {
if (m.level == level) {
ASSERT_OK(test::CorruptFile(env_.get(), dbname_ + "/" + m.name, offset,
ASSERT_OK(test::CorruptFile(env_, dbname_ + "/" + m.name, offset,
bytes_to_corrupt));
return;
}
@@ -314,7 +308,7 @@ class CorruptionTest : public testing::Test {
if (bytes_to_truncate == 0) {
new_size = 0;
}
ASSERT_OK(test::TruncateFile(env_.get(), path, new_size));
ASSERT_OK(test::TruncateFile(env_, path, new_size));
}
};
@@ -408,14 +402,14 @@ TEST_F(CorruptionTest, PostPITRCorruptionWALsRetained) {
}
TEST_F(CorruptionTest, RecoverWriteError) {
fs_->writable_file_error_ = true;
env_->writable_file_error_ = true;
Status s = TryReopen();
ASSERT_TRUE(!s.ok());
}
TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
// Do enough writing to force minor compaction
fs_->writable_file_error_ = true;
env_->writable_file_error_ = true;
const int num =
static_cast<int>(3 + (Options().write_buffer_size / kValueSize));
std::string value_storage;
@@ -431,8 +425,8 @@ TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
ASSERT_TRUE(!failed || !s.ok());
}
ASSERT_TRUE(!s.ok());
ASSERT_GE(fs_->num_writable_file_errors_, 1);
fs_->writable_file_error_ = false;
ASSERT_GE(env_->num_writable_file_errors_, 1);
env_->writable_file_error_ = false;
Reopen();
}
@@ -450,7 +444,7 @@ TEST_F(CorruptionTest, TableFile) {
TEST_F(CorruptionTest, VerifyChecksumReadahead) {
Options options;
SpecialEnv senv(base_env_);
SpecialEnv senv(env_->target());
options.env = &senv;
// Disable block cache as we are going to check checksum for
// the same file twice and measure number of reads.
@@ -518,7 +512,7 @@ TEST_F(CorruptionTest, TableFileIndexData) {
dbi = static_cast_with_check<DBImpl>(db_);
// one full file may be readable, since only one was corrupted
// the other file should be fully non-readable, since index was corrupted
Check(0, 5000, ReadOptions(true, true));
Check(0, 5000);
ASSERT_NOK(dbi->VerifyChecksum());
// In paranoid mode, the db cannot be opened due to the corrupted file.
@@ -593,8 +587,8 @@ TEST_F(CorruptionTest, TableFileWrongSize) {
// Make the file smaller with truncation.
// First leaving a partial footer, and then completely removing footer.
for (size_t bytes_lost : {8, 100}) {
ASSERT_OK(test::TruncateFile(env_.get(), filename,
metadata[0].size - bytes_lost));
ASSERT_OK(
test::TruncateFile(env_, filename, metadata[0].size - bytes_lost));
// Reported well with paranoid checks
options_.paranoid_checks = true;
@@ -659,7 +653,7 @@ TEST_F(CorruptionTest, CorruptedDescriptor) {
TEST_F(CorruptionTest, CompactionInputError) {
Options options;
options.env = env_.get();
options.env = env_;
Reopen(&options);
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
@@ -680,7 +674,7 @@ TEST_F(CorruptionTest, CompactionInputError) {
TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
Options options;
options.env = env_.get();
options.env = env_;
options.paranoid_checks = true;
options.write_buffer_size = 131072;
options.max_write_buffer_number = 2;
@@ -767,7 +761,7 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
ImmutableOptions(options_), kRangeDelBlockName, &range_del_handle));
ASSERT_OK(TryReopen());
ASSERT_OK(test::CorruptFile(env_.get(), filename,
ASSERT_OK(test::CorruptFile(env_, filename,
static_cast<int>(range_del_handle.offset()), 1));
ASSERT_TRUE(TryReopen().IsCorruption());
}
@@ -775,7 +769,7 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
TEST_F(CorruptionTest, FileSystemStateCorrupted) {
for (int iter = 0; iter < 2; ++iter) {
Options options;
options.env = env_.get();
options.env = env_;
options.paranoid_checks = true;
options.create_if_missing = true;
Reopen(&options);
@@ -814,7 +808,7 @@ static const auto& corruption_modes = {
TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
Options options;
options.env = env_.get();
options.env = env_;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -842,7 +836,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
Options options;
options.env = env_.get();
options.env = env_;
options.paranoid_file_checks = true;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
@@ -875,7 +869,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
Options options;
options.env = env_.get();
options.env = env_;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -911,7 +905,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
Options options;
options.env = env_.get();
options.env = env_;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -950,7 +944,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
Options options;
options.env = env_.get();
options.env = env_;
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -986,7 +980,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
Options options;
options.env = env_.get();
options.env = env_;
options.create_if_missing = true;
options.allow_data_in_errors = true;
auto mode = mock::MockTableFactory::kCorruptKey;
@@ -1015,7 +1009,7 @@ TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
Options options;
options.env = env_.get();
options.env = env_;
options.paranoid_file_checks = false;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
@@ -1042,7 +1036,7 @@ TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
TEST_F(CorruptionTest, FlushKeyOrderCheck) {
Options options;
options.env = env_.get();
options.env = env_;
options.paranoid_file_checks = false;
options.create_if_missing = true;
ASSERT_OK(db_->SetOptions({{"check_flush_compaction_key_order", "true"}}));
@@ -1095,7 +1089,7 @@ TEST_F(CorruptionTest, DisableKeyOrderCheck) {
TEST_F(CorruptionTest, VerifyWholeTableChecksum) {
CloseDb();
Options options;
options.env = env_.get();
options.env = env_;
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.file_checksum_gen_factory =
@@ -1184,7 +1178,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, CrashDuringRecovery) {
track_and_verify_wals_in_manifest_;
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.avoid_flush_during_recovery = false;
options.env = env_.get();
options.env = env_;
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.max_write_buffer_number = 8;
@@ -1356,7 +1350,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, TxnDbCrashDuringRecovery) {
options.track_and_verify_wals_in_manifest =
track_and_verify_wals_in_manifest_;
options.avoid_flush_during_recovery = false;
options.env = env_.get();
options.env = env_;
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.max_write_buffer_number = 3;
@@ -1551,7 +1545,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, CrashDuringRecoveryWithFlush) {
Options options;
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.avoid_flush_during_recovery = false;
options.env = env_.get();
options.env = env_;
options.create_if_missing = true;
ASSERT_OK(DestroyDB(dbname_, options));
@@ -1667,3 +1661,12 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED as RepairDB() is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
+10
View File
@@ -3,6 +3,7 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_impl/db_impl.h"
#include "db/db_test_util.h"
@@ -349,3 +350,12 @@ int main(int argc, char** argv) {
}
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED as Cuckoo table is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+136 -35
View File
@@ -20,7 +20,9 @@
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "util/file_checksum_helper.h"
#include "util/random.h"
#include "utilities/counted_fs.h"
@@ -30,6 +32,9 @@
namespace ROCKSDB_NAMESPACE {
static bool enable_io_uring = true;
extern "C" bool RocksDbIOUringEnable() { return enable_io_uring; }
class DBBasicTest : public DBTestBase {
public:
DBBasicTest() : DBTestBase("db_basic_test", /*env_do_fsync=*/false) {}
@@ -93,6 +98,7 @@ TEST_F(DBBasicTest, UniqueSession) {
EXPECT_MATCHES_REGEX(sid2, expected);
EXPECT_MATCHES_REGEX(sid3, expected);
#ifndef ROCKSDB_LITE
Close();
ASSERT_OK(ReadOnlyReopen(options));
ASSERT_OK(db_->GetDbSessionId(sid1));
@@ -107,6 +113,7 @@ TEST_F(DBBasicTest, UniqueSession) {
ASSERT_NE(sid1, sid2);
ASSERT_EQ(sid2, sid3);
#endif // ROCKSDB_LITE
CreateAndReopenWithCF({"goku"}, options);
ASSERT_OK(db_->GetDbSessionId(sid1));
@@ -123,6 +130,7 @@ TEST_F(DBBasicTest, UniqueSession) {
ASSERT_NE(sid1, sid4);
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, ReadOnlyDB) {
ASSERT_OK(Put("foo", "v1"));
ASSERT_OK(Put("bar", "v2"));
@@ -359,6 +367,7 @@ TEST_F(DBBasicTest, LevelLimitReopen) {
options.max_bytes_for_level_multiplier_additional.resize(10, 1);
ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#endif // ROCKSDB_LITE
TEST_F(DBBasicTest, PutDeleteGet) {
do {
@@ -420,6 +429,7 @@ TEST_F(DBBasicTest, GetFromVersions) {
} while (ChangeOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, GetSnapshot) {
anon::OptionsOverride options_override;
options_override.skip_policy = kSkipNoSnapshot;
@@ -440,6 +450,7 @@ TEST_F(DBBasicTest, GetSnapshot) {
}
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
TEST_F(DBBasicTest, CheckLock) {
do {
@@ -671,27 +682,7 @@ TEST_F(DBBasicTest, IdentityAcrossRestarts) {
} while (ChangeCompactOptions());
}
TEST_F(DBBasicTest, LockFileRecovery) {
Options options = CurrentOptions();
// Regardless of best_efforts_recovery
for (bool ber : {false, true}) {
options.best_efforts_recovery = ber;
DestroyAndReopen(options);
std::string id1, id2;
ASSERT_OK(db_->GetDbIdentity(id1));
Close();
// Should be OK to re-open DB after lock file deleted
std::string lockfilename = LockFileName(dbname_);
ASSERT_OK(env_->DeleteFile(lockfilename));
Reopen(options);
// Should be same DB as before
ASSERT_OK(db_->GetDbIdentity(id2));
ASSERT_EQ(id1, id2);
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, Snapshot) {
env_->SetMockSleep();
anon::OptionsOverride options_override;
@@ -763,6 +754,7 @@ TEST_F(DBBasicTest, Snapshot) {
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
class DBBasicMultiConfigs : public DBBasicTest,
public ::testing::WithParamInterface<int> {
@@ -2173,6 +2165,7 @@ class DBMultiGetAsyncIOTest : public DBBasicTest,
options_.disable_auto_compactions = true;
options_.statistics = statistics_;
options_.table_factory.reset(NewBlockBasedTableFactory(bbto));
options_.env = Env::Default();
Reopen(options_);
int num_keys = 0;
@@ -2239,6 +2232,20 @@ class DBMultiGetAsyncIOTest : public DBBasicTest,
const std::shared_ptr<Statistics>& statistics() { return statistics_; }
protected:
void PrepareDBForTest() {
#ifdef ROCKSDB_IOURING_PRESENT
Reopen(options_);
#else // ROCKSDB_IOURING_PRESENT
// Warm up the block cache so we don't need to use the IO uring
Iterator* iter = dbfull()->NewIterator(ReadOptions());
for (iter->SeekToFirst(); iter->Valid() && iter->status().ok();
iter->Next())
;
EXPECT_OK(iter->status());
delete iter;
#endif // ROCKSDB_IOURING_PRESENT
}
void ReopenDB() { Reopen(options_); }
private:
@@ -2253,6 +2260,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
std::vector<PinnableSlice> values(key_strs.size());
std::vector<Status> statuses(key_strs.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2271,6 +2280,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
// With async IO, lookups will happen in parallel for each key
#ifdef ROCKSDB_IOURING_PRESENT
if (GetParam()) {
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
@@ -2280,6 +2290,11 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
// L0 file
ASSERT_EQ(multiget_io_batch_size.count, 3);
}
#else // ROCKSDB_IOURING_PRESENT
if (GetParam()) {
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
@@ -2297,6 +2312,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2310,6 +2327,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
ASSERT_EQ(values[1], "val_l1_" + std::to_string(54));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2317,9 +2335,11 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
// A batch of 3 async IOs is expected, one for each overlapping file in L1
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
#endif // ROCKSDB_IOURING_PRESENT
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#ifdef ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
@@ -2335,9 +2355,9 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
values.resize(keys.size());
statuses.resize(keys.size());
int count = 0;
SyncPoint::GetInstance()->SetCallBack(
"TableCache::GetTableReader:BeforeOpenFile", [&](void* status) {
static int count = 0;
count++;
// Fail the last table reader open, which is the 6th SST file
// since 3 overlapping L0 files + 3 L1 files containing the keys
@@ -2360,7 +2380,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
});
SyncPoint::GetInstance()->EnableProcessing();
ReopenDB();
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
@@ -2382,6 +2402,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
ASSERT_EQ(multiget_io_batch_size.max, 2);
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 2);
}
#endif // ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
std::vector<std::string> key_strs;
@@ -2399,6 +2420,8 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2412,6 +2435,7 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
ASSERT_EQ(values[1], "val_l1_" + std::to_string(54));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2422,6 +2446,7 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
// will lookup 2 files in parallel and issue 2 async reads
ASSERT_EQ(multiget_io_batch_size.count, 2);
ASSERT_EQ(multiget_io_batch_size.max, 2);
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
@@ -2440,6 +2465,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2453,6 +2480,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
ASSERT_EQ(values[1], "val_l2_" + std::to_string(56));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2462,6 +2490,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
// Otherwise, the L2 lookup will happen after L1.
ASSERT_EQ(multiget_io_batch_size.count, GetParam() ? 1 : 2);
ASSERT_EQ(multiget_io_batch_size.max, GetParam() ? 3 : 2);
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
@@ -2478,6 +2507,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2493,6 +2524,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 2);
}
#ifdef ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeDelInL1) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
@@ -2507,6 +2539,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeDelInL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2536,6 +2570,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2WithRangeDelInL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2550,6 +2586,45 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2WithRangeDelInL1) {
// Bloom filters in L0/L1 will avoid the coroutine calls in those levels
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#endif // ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetNoIOUring) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
std::vector<PinnableSlice> values;
std::vector<Status> statuses;
key_strs.push_back(Key(33));
key_strs.push_back(Key(54));
key_strs.push_back(Key(102));
keys.push_back(key_strs[0]);
keys.push_back(key_strs[1]);
keys.push_back(key_strs[2]);
values.resize(keys.size());
statuses.resize(keys.size());
enable_io_uring = false;
ReopenDB();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
dbfull()->MultiGet(ro, dbfull()->DefaultColumnFamily(), keys.size(),
keys.data(), values.data(), statuses.data());
ASSERT_EQ(values.size(), 3);
ASSERT_EQ(statuses[0], Status::NotSupported());
ASSERT_EQ(statuses[1], Status::NotSupported());
ASSERT_EQ(statuses[2], Status::NotSupported());
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
// A batch of 3 async IOs is expected, one for each overlapping file in L1
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
INSTANTIATE_TEST_CASE_P(DBMultiGetAsyncIOTest, DBMultiGetAsyncIOTest,
testing::Bool());
@@ -2719,6 +2794,7 @@ TEST_P(MultiGetPrefixExtractorTest, Batched) {
INSTANTIATE_TEST_CASE_P(MultiGetPrefix, MultiGetPrefixExtractorTest,
::testing::Bool());
#ifndef ROCKSDB_LITE
class DBMultiGetRowCacheTest : public DBBasicTest,
public ::testing::WithParamInterface<bool> {};
@@ -2873,6 +2949,7 @@ TEST_F(DBBasicTest, ValueTypeString) {
ASSERT_TRUE(key_version.GetTypeName() != "Invalid");
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBasicTest, MultiGetIOBufferOverrun) {
Options options = CurrentOptions();
@@ -2973,6 +3050,7 @@ TEST_F(DBBasicTest, BestEffortsRecoveryWithVersionBuildingFailure) {
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
namespace {
class TableFileListener : public EventListener {
public:
@@ -3247,6 +3325,7 @@ TEST_F(DBBasicTest, DisableTrackWal) {
ASSERT_TRUE(dbfull()->GetVersionSet()->GetWalSet().GetWals().empty());
Close();
}
#endif // !ROCKSDB_LITE
TEST_F(DBBasicTest, ManifestChecksumMismatch) {
Options options = CurrentOptions();
@@ -3290,6 +3369,7 @@ TEST_F(DBBasicTest, ConcurrentlyCloseDB) {
}
}
#ifndef ROCKSDB_LITE
class DBBasicTestTrackWal : public DBTestBase,
public testing::WithParamInterface<bool> {
public:
@@ -3345,16 +3425,21 @@ TEST_P(DBBasicTestTrackWal, DoNotTrackObsoleteWal) {
INSTANTIATE_TEST_CASE_P(DBBasicTestTrackWal, DBBasicTestTrackWal,
testing::Bool());
#endif // ROCKSDB_LITE
class DBBasicTestMultiGet : public DBTestBase {
public:
DBBasicTestMultiGet(std::string test_dir, int num_cfs,
DBBasicTestMultiGet(std::string test_dir, int num_cfs, bool compressed_cache,
bool uncompressed_cache, bool _compression_enabled,
bool _fill_cache, uint32_t compression_parallel_threads)
: DBTestBase(test_dir, /*env_do_fsync=*/false) {
compression_enabled_ = _compression_enabled;
fill_cache_ = _fill_cache;
if (compressed_cache) {
std::shared_ptr<Cache> cache = NewLRUCache(1048576);
compressed_cache_ = std::make_shared<MyBlockCache>(cache);
}
if (uncompressed_cache) {
std::shared_ptr<Cache> cache = NewLRUCache(1048576);
uncompressed_cache_ = std::make_shared<MyBlockCache>(cache);
@@ -3366,6 +3451,7 @@ class DBBasicTestMultiGet : public DBTestBase {
Random rnd(301);
BlockBasedTableOptions table_options;
#ifndef ROCKSDB_LITE
if (compression_enabled_) {
std::vector<CompressionType> compression_types;
compression_types = GetSupportedCompressions();
@@ -3384,6 +3470,12 @@ class DBBasicTestMultiGet : public DBTestBase {
compression_enabled_ = false;
}
}
#else
// GetSupportedCompressions() is not available in LITE build
if (!Snappy_Supported()) {
compression_enabled_ = false;
}
#endif // ROCKSDB_LITE
table_options.block_cache = uncompressed_cache_;
if (table_options.block_cache == nullptr) {
@@ -3391,6 +3483,7 @@ class DBBasicTestMultiGet : public DBTestBase {
} else {
table_options.pin_l0_filter_and_index_blocks_in_cache = true;
}
table_options.block_cache_compressed = compressed_cache_;
table_options.flush_block_policy_factory.reset(
new MyFlushBlockPolicyFactory());
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@@ -3573,14 +3666,16 @@ class DBBasicTestMultiGet : public DBTestBase {
std::vector<std::string> cf_names_;
};
class DBBasicTestWithParallelIO : public DBBasicTestMultiGet,
public testing::WithParamInterface<
std::tuple<bool, bool, bool, uint32_t>> {
class DBBasicTestWithParallelIO
: public DBBasicTestMultiGet,
public testing::WithParamInterface<
std::tuple<bool, bool, bool, bool, uint32_t>> {
public:
DBBasicTestWithParallelIO()
: DBBasicTestMultiGet("/db_basic_test_with_parallel_io", 1,
std::get<0>(GetParam()), std::get<1>(GetParam()),
std::get<2>(GetParam()), std::get<3>(GetParam())) {}
std::get<2>(GetParam()), std::get<3>(GetParam()),
std::get<4>(GetParam())) {}
};
TEST_P(DBBasicTestWithParallelIO, MultiGet) {
@@ -3706,6 +3801,7 @@ TEST_P(DBBasicTestWithParallelIO, MultiGet) {
}
}
#ifndef ROCKSDB_LITE
TEST_P(DBBasicTestWithParallelIO, MultiGetDirectIO) {
class FakeDirectIOEnv : public EnvWrapper {
class FakeDirectIOSequentialFile;
@@ -3822,6 +3918,7 @@ TEST_P(DBBasicTestWithParallelIO, MultiGetDirectIO) {
}
Close();
}
#endif // ROCKSDB_LITE
TEST_P(DBBasicTestWithParallelIO, MultiGetWithChecksumMismatch) {
std::vector<std::string> key_data(10);
@@ -3909,12 +4006,13 @@ TEST_P(DBBasicTestWithParallelIO, MultiGetWithMissingFile) {
INSTANTIATE_TEST_CASE_P(ParallelIO, DBBasicTestWithParallelIO,
// Params are as follows -
// Param 0 - Uncompressed cache enabled
// Param 1 - Data compression enabled
// Param 2 - ReadOptions::fill_cache
// Param 3 - CompressionOptions::parallel_threads
// Param 0 - Compressed cache enabled
// Param 1 - Uncompressed cache enabled
// Param 2 - Data compression enabled
// Param 3 - ReadOptions::fill_cache
// Param 4 - CompressionOptions::parallel_threads
::testing::Combine(::testing::Bool(), ::testing::Bool(),
::testing::Bool(),
::testing::Bool(), ::testing::Bool(),
::testing::Values(1, 4)));
// Forward declaration
@@ -4125,8 +4223,9 @@ class DBBasicTestMultiGetDeadline : public DBBasicTestMultiGet,
DBBasicTestMultiGetDeadline()
: DBBasicTestMultiGet(
"db_basic_test_multiget_deadline" /*Test dir*/,
10 /*# of column families*/, true /*uncompressed cache enabled*/,
true /*compression enabled*/, true /*ReadOptions.fill_cache*/,
10 /*# of column families*/, false /*compressed cache enabled*/,
true /*uncompressed cache enabled*/, true /*compression enabled*/,
true /*ReadOptions.fill_cache*/,
1 /*# of parallel compression threads*/) {}
inline void CheckStatus(std::vector<Status>& statuses, size_t num_ok) {
@@ -4362,6 +4461,7 @@ TEST_F(DBBasicTest, FailOpenIfLoggerCreationFail) {
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, VerifyFileChecksums) {
Options options = GetDefaultOptions();
options.create_if_missing = true;
@@ -4427,6 +4527,7 @@ TEST_F(DBBasicTest, DISABLED_ManualWalSync) {
ASSERT_TRUE(TryReopen(options).IsCorruption());
}
#endif // !ROCKSDB_LITE
// A test class for intercepting random reads and injecting artificial
// delays. Used for testing the deadline/timeout feature
+356 -10
View File
@@ -44,6 +44,10 @@ class DBBlockCacheTest : public DBTestBase {
size_t compression_dict_miss_count_ = 0;
size_t compression_dict_hit_count_ = 0;
size_t compression_dict_insert_count_ = 0;
size_t compressed_miss_count_ = 0;
size_t compressed_hit_count_ = 0;
size_t compressed_insert_count_ = 0;
size_t compressed_failure_count_ = 0;
public:
const size_t kNumBlocks = 10;
@@ -81,6 +85,14 @@ class DBBlockCacheTest : public DBTestBase {
hit_count_ = TestGetTickerCount(options, BLOCK_CACHE_HIT);
insert_count_ = TestGetTickerCount(options, BLOCK_CACHE_ADD);
failure_count_ = TestGetTickerCount(options, BLOCK_CACHE_ADD_FAILURES);
compressed_miss_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS);
compressed_hit_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT);
compressed_insert_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD);
compressed_failure_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD_FAILURES);
}
void RecordCacheCountersForCompressionDict(const Options& options) {
@@ -132,6 +144,30 @@ class DBBlockCacheTest : public DBTestBase {
compression_dict_insert_count_ = new_compression_dict_insert_count;
}
void CheckCompressedCacheCounters(const Options& options,
size_t expected_misses,
size_t expected_hits,
size_t expected_inserts,
size_t expected_failures) {
size_t new_miss_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS);
size_t new_hit_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT);
size_t new_insert_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD);
size_t new_failure_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD_FAILURES);
ASSERT_EQ(compressed_miss_count_ + expected_misses, new_miss_count);
ASSERT_EQ(compressed_hit_count_ + expected_hits, new_hit_count);
ASSERT_EQ(compressed_insert_count_ + expected_inserts, new_insert_count);
ASSERT_EQ(compressed_failure_count_ + expected_failures, new_failure_count);
compressed_miss_count_ = new_miss_count;
compressed_hit_count_ = new_hit_count;
compressed_insert_count_ = new_insert_count;
compressed_failure_count_ = new_failure_count;
}
#ifndef ROCKSDB_LITE
const std::array<size_t, kNumCacheEntryRoles> GetCacheEntryRoleCountsBg() {
// Verify in cache entry role stats
std::array<size_t, kNumCacheEntryRoles> cache_entry_role_counts;
@@ -145,6 +181,7 @@ class DBBlockCacheTest : public DBTestBase {
}
return cache_entry_role_counts;
}
#endif // ROCKSDB_LITE
};
TEST_F(DBBlockCacheTest, IteratorBlockCacheUsage) {
@@ -237,6 +274,84 @@ TEST_F(DBBlockCacheTest, TestWithoutCompressedBlockCache) {
}
#ifdef SNAPPY
TEST_F(DBBlockCacheTest, TestWithCompressedBlockCache) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.no_block_cache = true;
table_options.block_cache_compressed = nullptr;
table_options.block_size = 1;
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
table_options.cache_index_and_filter_blocks = false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.compression = CompressionType::kSnappyCompression;
DestroyAndReopen(options);
std::string value(kValueSize, 'a');
for (size_t i = 0; i < kNumBlocks; i++) {
ASSERT_OK(Put(std::to_string(i), value));
ASSERT_OK(Flush());
}
ReadOptions read_options;
std::shared_ptr<Cache> compressed_cache = NewLRUCache(1 << 25, 0, false);
LRUCacheOptions co;
co.capacity = 0;
co.num_shard_bits = 0;
co.strict_capacity_limit = false;
// Needed not to count entry stats collector
co.metadata_charge_policy = kDontChargeCacheMetadata;
std::shared_ptr<Cache> cache = NewLRUCache(co);
table_options.block_cache = cache;
table_options.no_block_cache = false;
table_options.block_cache_compressed = compressed_cache;
table_options.max_auto_readahead_size = 0;
table_options.cache_index_and_filter_blocks = false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
Reopen(options);
RecordCacheCounters(options);
// Load blocks into cache.
for (size_t i = 0; i < kNumBlocks - 1; i++) {
ASSERT_EQ(value, Get(std::to_string(i)));
CheckCacheCounters(options, 1, 0, 1, 0);
CheckCompressedCacheCounters(options, 1, 0, 1, 0);
}
size_t usage = cache->GetUsage();
ASSERT_EQ(0, usage);
ASSERT_EQ(usage, cache->GetPinnedUsage());
size_t compressed_usage = compressed_cache->GetUsage();
ASSERT_LT(0, compressed_usage);
// Compressed block cache cannot be pinned.
ASSERT_EQ(0, compressed_cache->GetPinnedUsage());
// Set strict capacity limit flag. Now block will only load into compressed
// block cache.
cache->SetCapacity(usage);
cache->SetStrictCapacityLimit(true);
ASSERT_EQ(usage, cache->GetPinnedUsage());
// Load last key block.
ASSERT_EQ(
"Operation aborted: Memory limit reached: Insert failed due to LRU cache "
"being full.",
Get(std::to_string(kNumBlocks - 1)));
// Failure will also record the miss counter.
CheckCacheCounters(options, 1, 0, 0, 1);
CheckCompressedCacheCounters(options, 1, 0, 1, 0);
// Clear strict capacity limit flag. This time we shall hit compressed block
// cache and load into block cache.
cache->SetStrictCapacityLimit(false);
// Load last key block.
ASSERT_EQ(value, Get(std::to_string(kNumBlocks - 1)));
CheckCacheCounters(options, 1, 0, 1, 0);
CheckCompressedCacheCounters(options, 0, 1, 0, 0);
}
namespace {
class PersistentCacheFromCache : public PersistentCache {
@@ -298,8 +413,83 @@ class ReadOnlyCacheWrapper : public CacheWrapper {
};
} // anonymous namespace
TEST_F(DBBlockCacheTest, TestWithSameCompressed) {
auto table_options = GetTableOptions();
auto options = GetOptions(table_options);
InitTable(options);
std::shared_ptr<Cache> rw_cache{NewLRUCache(1000000)};
std::shared_ptr<PersistentCacheFromCache> rw_pcache{
new PersistentCacheFromCache(rw_cache, /*read_only*/ false)};
// Exercise some obscure behavior with read-only wrappers
std::shared_ptr<Cache> ro_cache{new ReadOnlyCacheWrapper(rw_cache)};
std::shared_ptr<PersistentCacheFromCache> ro_pcache{
new PersistentCacheFromCache(rw_cache, /*read_only*/ true)};
// Simple same pointer
table_options.block_cache = rw_cache;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache same as block_cache_compressed not "
"currently supported, and would be bad for performance anyway");
// Other cases
table_options.block_cache = ro_cache;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache and block_cache_compressed share "
"the same key space, which is not supported");
table_options.block_cache = rw_cache;
table_options.block_cache_compressed = ro_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache_compressed and block_cache share "
"the same key space, which is not supported");
table_options.block_cache = ro_cache;
table_options.block_cache_compressed.reset();
table_options.persistent_cache = rw_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache and persistent_cache share the same "
"key space, which is not supported");
table_options.block_cache = rw_cache;
table_options.block_cache_compressed.reset();
table_options.persistent_cache = ro_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: persistent_cache and block_cache share the same "
"key space, which is not supported");
table_options.block_cache.reset();
table_options.no_block_cache = true;
table_options.block_cache_compressed = ro_cache;
table_options.persistent_cache = rw_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache_compressed and persistent_cache "
"share the same key space, which is not supported");
table_options.block_cache.reset();
table_options.no_block_cache = true;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache = ro_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: persistent_cache and block_cache_compressed "
"share the same key space, which is not supported");
}
#endif // SNAPPY
#ifndef ROCKSDB_LITE
// Make sure that when options.block_cache is set, after a new table is
// created its index/filter blocks are added to block cache.
@@ -419,6 +609,11 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
ASSERT_EQ(cache->GetUsage(), index_bytes_insert + filter_bytes_insert);
// set the cache capacity to the current usage
cache->SetCapacity(index_bytes_insert + filter_bytes_insert);
// The index and filter eviction statistics were broken by the refactoring
// that moved the readers out of the block cache. Disabling these until we can
// bring the stats back.
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT), 0);
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT), 0);
// Note that the second key needs to be no longer than the first one.
// Otherwise the second index block may not fit in cache.
ASSERT_OK(Put(1, "key", "val"));
@@ -429,6 +624,13 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
index_bytes_insert);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_INSERT),
filter_bytes_insert);
// The index and filter eviction statistics were broken by the refactoring
// that moved the readers out of the block cache. Disabling these until we can
// bring the stats back.
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT),
// index_bytes_insert);
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT),
// filter_bytes_insert);
}
#if (defined OS_LINUX || defined OS_WIN)
@@ -884,6 +1086,124 @@ TEST_F(DBBlockCacheTest, ParanoidFileChecks) {
TestGetTickerCount(options, BLOCK_CACHE_ADD));
}
TEST_F(DBBlockCacheTest, CompressedCache) {
if (!Snappy_Supported()) {
return;
}
int num_iter = 80;
// Run this test three iterations.
// Iteration 1: only a uncompressed block cache
// Iteration 2: only a compressed block cache
// Iteration 3: both block cache and compressed cache
// Iteration 4: both block cache and compressed cache, but DB is not
// compressed
for (int iter = 0; iter < 4; iter++) {
Options options = CurrentOptions();
options.write_buffer_size = 64 * 1024; // small write buffer
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
switch (iter) {
case 0:
// only uncompressed block cache
table_options.block_cache = NewLRUCache(8 * 1024);
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 1:
// no block cache, only compressed cache
table_options.no_block_cache = true;
table_options.block_cache = nullptr;
table_options.block_cache_compressed = NewLRUCache(8 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 2:
// both compressed and uncompressed block cache
table_options.block_cache = NewLRUCache(1024);
table_options.block_cache_compressed = NewLRUCache(8 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 3:
// both block cache and compressed cache, but DB is not compressed
// also, make block cache sizes bigger, to trigger block cache hits
table_options.block_cache = NewLRUCache(1024 * 1024);
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.compression = kNoCompression;
break;
default:
FAIL();
}
CreateAndReopenWithCF({"pikachu"}, options);
// default column family doesn't have block cache
Options no_block_cache_opts;
no_block_cache_opts.statistics = options.statistics;
no_block_cache_opts = CurrentOptions(no_block_cache_opts);
BlockBasedTableOptions table_options_no_bc;
table_options_no_bc.no_block_cache = true;
no_block_cache_opts.table_factory.reset(
NewBlockBasedTableFactory(table_options_no_bc));
ReopenWithColumnFamilies(
{"default", "pikachu"},
std::vector<Options>({no_block_cache_opts, options}));
Random rnd(301);
// Write 8MB (80 values, each 100K)
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 0);
std::vector<std::string> values;
std::string str;
for (int i = 0; i < num_iter; i++) {
if (i % 4 == 0) { // high compression ratio
str = rnd.RandomString(1000);
}
values.push_back(str);
ASSERT_OK(Put(1, Key(i), values[i]));
}
// flush all data from memtable so that reads are from block cache
ASSERT_OK(Flush(1));
for (int i = 0; i < num_iter; i++) {
ASSERT_EQ(Get(1, Key(i)), values[i]);
}
// check that we triggered the appropriate code paths in the cache
switch (iter) {
case 0:
// only uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 1:
// no block cache, only compressed cache
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 2:
// both compressed and uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 3:
// both compressed and uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_HIT), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
// compressed doesn't have any hits since blocks are not compressed on
// storage
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT), 0);
break;
default:
FAIL();
}
options.create_if_missing = true;
DestroyAndReopen(options);
}
}
TEST_F(DBBlockCacheTest, CacheCompressionDict) {
const int kNumFiles = 4;
const int kNumEntriesPerFile = 128;
@@ -1332,6 +1652,7 @@ TEST_F(DBBlockCacheTest, HyperClockCacheReportProblems) {
EXPECT_EQ(logger->PopCounts(), (std::array<int, 3>{{0, 1, 0}}));
}
#endif // ROCKSDB_LITE
class DBBlockCacheKeyTest
: public DBTestBase,
@@ -1387,16 +1708,31 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
uint64_t expected_stat = 0;
std::function<void()> verify_stats;
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_DATA_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_INDEX_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_FILTER_ADD));
};
if (use_compressed_cache_) {
if (!Snappy_Supported()) {
ROCKSDB_GTEST_SKIP("Compressed cache test requires snappy support");
return;
}
options.compression = CompressionType::kSnappyCompression;
table_options.no_block_cache = true;
table_options.block_cache_compressed = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
// One for ordinary SST file and one for external SST file
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_ADD));
};
} else {
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_DATA_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_INDEX_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_FILTER_ADD));
};
}
table_options.filter_policy.reset(NewBloomFilterPolicy(10, false));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@@ -1438,6 +1774,7 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
++key_count;
}
#ifndef ROCKSDB_LITE
// Save an export of those ordinary SST files for later
std::string export_files_dir = dbname_ + "/exported";
ExportImportFilesMetaData* metadata_ptr_ = nullptr;
@@ -1466,6 +1803,13 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
ASSERT_OK(db_->IngestExternalFile(handles_[1], {f}, ingest_opts));
}
if (exclude_file_numbers_) {
// FIXME(peterd): figure out where these extra ADDs are coming from
options.statistics->recordTick(BLOCK_CACHE_COMPRESSED_ADD,
uint64_t{0} - uint64_t{2});
}
#endif
perform_gets();
verify_stats();
@@ -1478,6 +1822,7 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
// Make sure we can cache hit even on a full copy of the DB. Using
// StableCacheKeyTestFS, Checkpoint will resort to full copy not hard link.
// (Checkpoint not available in LITE mode to test this.)
#ifndef ROCKSDB_LITE
auto db_copy_name = dbname_ + "-copy";
ASSERT_OK(Checkpoint::Create(db_, &checkpoint));
ASSERT_OK(checkpoint->CreateCheckpoint(db_copy_name));
@@ -1516,6 +1861,7 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
perform_gets();
verify_stats();
#endif // !ROCKSDB_LITE
Close();
Destroy(options);
+24
View File
@@ -245,12 +245,14 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloomCustomPrefixExtractor) {
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
// No bloom on extractor changed
#ifndef ROCKSDB_LITE
ASSERT_OK(db_->SetOptions({{"prefix_extractor", "capped:10"}}));
ASSERT_EQ("NOT_FOUND", Get("foobarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 3);
ASSERT_EQ(
3,
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
#endif // ROCKSDB_LITE
// No bloom on extractor changed, after re-open
options.prefix_extractor.reset(NewCappedPrefixTransform(10));
@@ -315,12 +317,14 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloom) {
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
// No bloom on extractor changed
#ifndef ROCKSDB_LITE
ASSERT_OK(db_->SetOptions({{"prefix_extractor", "capped:10"}}));
ASSERT_EQ("NOT_FOUND", Get("foobarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 3);
ASSERT_EQ(
3,
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
#endif // ROCKSDB_LITE
get_perf_context()->Reset();
}
@@ -560,6 +564,7 @@ TEST_P(DBBloomFilterTestWithParam, BloomFilter) {
ASSERT_LE(reads, 3 * N / 100);
}
#ifndef ROCKSDB_LITE
// Sanity check some table properties
std::map<std::string, std::string> props;
ASSERT_TRUE(db_->GetMapProperty(
@@ -578,6 +583,7 @@ TEST_P(DBBloomFilterTestWithParam, BloomFilter) {
uint64_t num_filter_entries = ParseUint64(props["num_filter_entries"]);
EXPECT_EQ(num_filter_entries, nkeys);
#endif // ROCKSDB_LITE
env_->delay_sstable_sync_.store(false, std::memory_order_release);
Close();
@@ -643,8 +649,10 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
PutFn();
GetFn();
};
#ifndef ROCKSDB_LITE
std::map<std::string, std::string> props;
const auto& kAggTableProps = DB::Properties::kAggregatedTableProperties;
#endif // ROCKSDB_LITE
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -667,9 +675,11 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Test 2: use custom API to skip filters -> no filter constructed
// or read.
@@ -683,9 +693,11 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Control test: using an actual filter with 100% FP rate -> the filter
// is constructed and checked on read.
@@ -701,9 +713,11 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(
TestGetAndResetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE),
maxKey);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_NE(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Test 3 (options test): Able to read existing filters with longstanding
// generated options file entry `filter_policy=rocksdb.BuiltinBloomFilter`
@@ -729,9 +743,11 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
}
#if !defined(ROCKSDB_VALGRIND_RUN) || defined(ROCKSDB_FULL_VALGRIND_RUN)
@@ -1487,6 +1503,7 @@ TEST_P(DBFilterConstructionCorruptionTestWithParam, DetectCorruption) {
}
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
TEST_P(DBFilterConstructionCorruptionTestWithParam,
DynamicallyTurnOnAndOffDetectConstructCorruption) {
Options options = CurrentOptions();
@@ -1570,6 +1587,7 @@ TEST_P(DBFilterConstructionCorruptionTestWithParam,
db_->GetOptions().table_factory->GetOptions<BlockBasedTableOptions>();
EXPECT_FALSE(updated_table_options->detect_filter_construct_corruption);
}
#endif // ROCKSDB_LITE
namespace {
// NOTE: This class is referenced by HISTORY.md as a model for a wrapper
@@ -1738,6 +1756,7 @@ TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
EXPECT_LE(useful_count, maxKey * 0.91);
}
} else {
#ifndef ROCKSDB_LITE
// Also try external SST file
{
std::string file_path = dbname_ + "/external.sst";
@@ -1749,6 +1768,7 @@ TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
// Note: kCompactionStyleLevel is default, ignored if num_levels == -1
EXPECT_EQ(policy->DumpTestReport(),
"cf=abe,s=kCompactionStyleLevel,n=-1,l=-1,b=0,r=kMisc\n");
#endif
}
// Destroy
@@ -2016,7 +2036,9 @@ TEST_P(DBBloomFilterTestVaryPrefixAndFormatVer, PartitionedMultiGet) {
ASSERT_OK(Put(UKey(i), UKey(i)));
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ(TotalTableFiles(), 1);
#endif
constexpr uint32_t Q = 29;
// MultiGet In
@@ -2168,6 +2190,7 @@ INSTANTIATE_TEST_CASE_P(DBBloomFilterTestVaryPrefixAndFormatVer,
std::make_tuple(true, 3), std::make_tuple(true, 4),
std::make_tuple(true, 5)));
#ifndef ROCKSDB_LITE
namespace {
static const std::string kPlainTable = "test_PlainTableBloom";
} // anonymous namespace
@@ -3463,6 +3486,7 @@ TEST_F(DBBloomFilterTest, WeirdPrefixExtractorWithFilter3) {
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+6
View File
@@ -285,6 +285,7 @@ class ChangeFilterFactory : public CompactionFilterFactory {
const char* Name() const override { return "ChangeFilterFactory"; }
};
#ifndef ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilter) {
Options options = CurrentOptions();
options.max_open_files = -1;
@@ -468,6 +469,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterDeletesAll) {
delete itr;
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterFlush) {
// Tests a `CompactionFilterFactory` that filters when table file is created
@@ -653,6 +655,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterWithMergeOperator) {
ASSERT_EQ(newvalue, four);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterContextManual) {
KeepFilterFactory* filter = new KeepFilterFactory(true, true);
@@ -712,6 +715,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextManual) {
ASSERT_EQ(count, 0);
}
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterContextCfId) {
KeepFilterFactory* filter = new KeepFilterFactory(false, true);
@@ -742,6 +746,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextCfId) {
ASSERT_TRUE(filter->compaction_filter_created());
}
#ifndef ROCKSDB_LITE
// Compaction filters aplies to all records, regardless snapshots.
TEST_F(DBTestCompactionFilter, CompactionFilterIgnoreSnapshot) {
std::string five = std::to_string(5);
@@ -802,6 +807,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterIgnoreSnapshot) {
// removed.
db_->ReleaseSnapshot(snapshot);
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, SkipUntil) {
Options options = CurrentOptions();
+8
View File
@@ -30,6 +30,7 @@
namespace ROCKSDB_NAMESPACE {
// SYNC_POINT is not supported in released Windows mode.
#if !defined(ROCKSDB_LITE)
class CompactionStatsCollector : public EventListener {
public:
@@ -9108,11 +9109,18 @@ TEST_F(DBCompactionTest, BottommostFileCompactionAllowIngestBehind) {
// ASSERT_OK(dbfull()->TEST_WaitForCompact(true /* wait_unscheduled */));
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+8
View File
@@ -10,6 +10,7 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "port/port.h"
@@ -491,9 +492,16 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+4
View File
@@ -6,7 +6,9 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "rocksdb/perf_context.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include <iostream>
#include <string>
@@ -25,6 +27,7 @@ class DBEncryptionTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBEncryptionTest, CheckEncrypted) {
ASSERT_OK(Put("foo567", "v1.fetdq"));
@@ -116,6 +119,7 @@ TEST_F(DBEncryptionTest, ReadEmptyFile) {
ASSERT_TRUE(data.empty());
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+2
View File
@@ -4,6 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
//
#ifndef ROCKSDB_LITE
#include <algorithm>
#include <cstdint>
@@ -438,3 +439,4 @@ Status DBImpl::GetLiveFilesStorageInfo(
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+33
View File
@@ -73,7 +73,9 @@ TEST_F(DBFlushTest, FlushWhileWritingManifest) {
ASSERT_OK(dbfull()->Flush(no_wait));
// If the issue is hit we will wait here forever.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
ASSERT_EQ(2, TotalTableFiles());
#endif // ROCKSDB_LITE
}
// Disable this test temporarily on Travis as it fails intermittently.
@@ -103,7 +105,9 @@ TEST_F(DBFlushTest, SyncFail) {
// Now the background job will do the flush; wait for it.
// Returns the IO error happend during flush.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
ASSERT_EQ("", FilesPerLevel()); // flush failed.
#endif // ROCKSDB_LITE
Destroy(options);
}
@@ -660,6 +664,7 @@ TEST_F(DBFlushTest, StatisticsGarbageRangeDeletes) {
Close();
}
#ifndef ROCKSDB_LITE
// This simple Listener can only handle one flush at a time.
class TestFlushListener : public EventListener {
public:
@@ -739,8 +744,10 @@ class TestFlushListener : public EventListener {
Env* env_;
DBFlushTest* test_;
};
#endif // !ROCKSDB_LITE
// RocksDB lite does not support GetLiveFiles()
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, FixFlushReasonRaceFromConcurrentFlushes) {
Options options = CurrentOptions();
options.atomic_flush = true;
@@ -795,6 +802,7 @@ TEST_F(DBFlushTest, FixFlushReasonRaceFromConcurrentFlushes) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // !ROCKSDB_LITE
TEST_F(DBFlushTest, MemPurgeBasic) {
Options options = CurrentOptions();
@@ -827,16 +835,24 @@ TEST_F(DBFlushTest, MemPurgeBasic) {
// Enforce size of a single MemTable to 64MB (64MB = 67108864 bytes).
options.write_buffer_size = 1 << 20;
#ifndef ROCKSDB_LITE
// Initially deactivate the MemPurge prototype.
options.experimental_mempurge_threshold = 0.0;
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#else
// Activate directly the MemPurge prototype.
// (RocksDB lite does not support dynamic options)
options.experimental_mempurge_threshold = 1.0;
#endif // !ROCKSDB_LITE
ASSERT_OK(TryReopen(options));
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
// Dynamically activate the MemPurge prototype without restarting the DB.
ColumnFamilyHandle* cfh = db_->DefaultColumnFamily();
ASSERT_OK(db_->SetOptions(cfh, {{"experimental_mempurge_threshold", "1.0"}}));
#endif
std::atomic<uint32_t> mempurge_count{0};
std::atomic<uint32_t> sst_count{0};
@@ -969,6 +985,7 @@ TEST_F(DBFlushTest, MemPurgeBasic) {
}
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, MemPurgeBasicToggle) {
Options options = CurrentOptions();
@@ -1081,10 +1098,12 @@ TEST_F(DBFlushTest, MemPurgeBasicToggle) {
Close();
}
// Closes the "#ifndef ROCKSDB_LITE"
// End of MemPurgeBasicToggle, which is not
// supported with RocksDB LITE because it
// relies on dynamically changing the option
// flag experimental_mempurge_threshold.
#endif
// At the moment, MemPurge feature is deactivated
// when atomic_flush is enabled. This is because the level
@@ -1202,8 +1221,10 @@ TEST_F(DBFlushTest, MemPurgeDeleteAndDeleteRange) {
options.compression = kNoCompression;
options.inplace_update_support = false;
options.allow_concurrent_memtable_write = true;
#ifndef ROCKSDB_LITE
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#endif // !ROCKSDB_LITE
// Enforce size of a single MemTable to 64MB (64MB = 67108864 bytes).
options.write_buffer_size = 1 << 20;
// Activate the MemPurge prototype.
@@ -1401,8 +1422,10 @@ TEST_F(DBFlushTest, MemPurgeAndCompactionFilter) {
options.compression = kNoCompression;
options.inplace_update_support = false;
options.allow_concurrent_memtable_write = true;
#ifndef ROCKSDB_LITE
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#endif // !ROCKSDB_LITE
// Create a ConditionalUpdate compaction filter
// that will update all the values of the KV pairs
// where the keys are "lower" than KEY4.
@@ -1855,10 +1878,12 @@ TEST_F(DBFlushTest, ManualFlushFailsInReadOnlyMode) {
ASSERT_OK(db_->ContinueBackgroundWork());
// We ingested the error to env, so the returned status is not OK.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
uint64_t num_bg_errors;
ASSERT_TRUE(
db_->GetIntProperty(DB::Properties::kBackgroundErrors, &num_bg_errors));
ASSERT_GT(num_bg_errors, 0);
#endif // ROCKSDB_LITE
// In the bug scenario, triggering another flush would cause the second flush
// to hang forever. After the fix we expect it to return an error.
@@ -1900,6 +1925,7 @@ TEST_F(DBFlushTest, CFDropRaceWithWaitForFlushMemTables) {
SyncPoint::GetInstance()->DisableProcessing();
}
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
class TestListener : public EventListener {
public:
@@ -1990,6 +2016,7 @@ TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
TEST_F(DBFlushTest, FlushWithBlob) {
constexpr uint64_t min_blob_size = 10;
@@ -2051,6 +2078,7 @@ TEST_F(DBFlushTest, FlushWithBlob) {
ASSERT_EQ(blob_file->GetTotalBlobCount(), 1);
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
@@ -2066,6 +2094,7 @@ TEST_F(DBFlushTest, FlushWithBlob) {
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED],
compaction_stats[0].bytes_written +
compaction_stats[0].bytes_written_blob);
#endif // ROCKSDB_LITE
}
TEST_F(DBFlushTest, FlushWithChecksumHandoff1) {
@@ -2379,6 +2408,7 @@ TEST_P(DBFlushTestBlobError, FlushError) {
ASSERT_NE(type, kBlobFile);
}
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
@@ -2402,8 +2432,10 @@ TEST_P(DBFlushTestBlobError, FlushError) {
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED],
compaction_stats[0].bytes_written +
compaction_stats[0].bytes_written_blob);
#endif // ROCKSDB_LITE
}
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, TombstoneVisibleInSnapshot) {
class SimpleTestFlushListener : public EventListener {
public:
@@ -2584,6 +2616,7 @@ TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
}
}
#endif // ROCKSDB_LITE
TEST_P(DBAtomicFlushTest, PrecomputeMinLogNumberToKeepNon2PC) {
Options options = CurrentOptions();
+2
View File
@@ -3,6 +3,7 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_impl/compacted_db_impl.h"
#include "db/db_impl/db_impl.h"
@@ -253,3 +254,4 @@ Status CompactedDBImpl::Open(const Options& options, const std::string& dbname,
}
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+4
View File
@@ -4,6 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -132,10 +133,12 @@ class CompactedDBImpl : public DBImpl {
// Share with DBImplReadOnly?
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
private:
friend class DB;
@@ -148,3 +151,4 @@ class CompactedDBImpl : public DBImpl {
LevelFilesBrief files_;
};
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+95 -47
View File
@@ -213,13 +213,17 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
has_unpersisted_data_(false),
unable_to_release_oldest_log_(false),
num_running_ingest_file_(0),
#ifndef ROCKSDB_LITE
wal_manager_(immutable_db_options_, file_options_, io_tracer_,
seq_per_batch),
#endif // ROCKSDB_LITE
bg_work_paused_(0),
bg_compaction_paused_(0),
refitting_level_(false),
opened_successfully_(false),
#ifndef ROCKSDB_LITE
periodic_task_scheduler_(),
#endif // ROCKSDB_LITE
two_write_queues_(options.two_write_queues),
manual_wal_flush_(options.manual_wal_flush),
// last_sequencee_ is always maintained by the main queue that also writes
@@ -243,8 +247,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
atomic_flush_install_cv_(&mutex_),
blob_callback_(immutable_db_options_.sst_file_manager.get(), &mutex_,
&error_handler_, &event_logger_,
immutable_db_options_.listeners, dbname_),
lock_wal_count_(0) {
immutable_db_options_.listeners, dbname_) {
// !batch_per_trx_ implies seq_per_batch_ because it is only unset for
// WriteUnprepared, which should use seq_per_batch_.
assert(batch_per_txn_ || seq_per_batch_);
@@ -262,6 +265,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
SetDbSessionId();
assert(!db_session_id_.empty());
#ifndef ROCKSDB_LITE
periodic_task_functions_.emplace(PeriodicTaskType::kDumpStats,
[this]() { this->DumpStats(); });
periodic_task_functions_.emplace(PeriodicTaskType::kPersistStats,
@@ -271,6 +275,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
periodic_task_functions_.emplace(
PeriodicTaskType::kRecordSeqnoTime,
[this]() { this->RecordSeqnoToTimeMapping(); });
#endif // ROCKSDB_LITE
versions_.reset(new VersionSet(dbname_, &immutable_db_options_, file_options_,
table_cache_.get(), write_buffer_manager_,
@@ -491,6 +496,7 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Shutdown: canceling all background work");
#ifndef ROCKSDB_LITE
for (uint8_t task_type = 0;
task_type < static_cast<uint8_t>(PeriodicTaskType::kMax); task_type++) {
Status s = periodic_task_scheduler_.Unregister(
@@ -501,6 +507,7 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
task_type, s.ToString().c_str());
}
}
#endif // !ROCKSDB_LITE
InstrumentedMutexLock l(&mutex_);
if (!shutting_down_.load(std::memory_order_acquire) &&
@@ -697,6 +704,7 @@ Status DBImpl::CloseHelper() {
ROCKS_LOG_INFO(immutable_db_options_.info_log, "Shutdown complete");
LogFlush(immutable_db_options_.info_log);
#ifndef ROCKSDB_LITE
// If the sst_file_manager was allocated by us during DB::Open(), ccall
// Close() on it before closing the info_log. Otherwise, background thread
// in SstFileManagerImpl might try to log something
@@ -705,6 +713,7 @@ Status DBImpl::CloseHelper() {
immutable_db_options_.sst_file_manager.get());
sfm->Close();
}
#endif // ROCKSDB_LITE
if (immutable_db_options_.info_log && own_info_log_) {
Status s = immutable_db_options_.info_log->Close();
@@ -782,6 +791,7 @@ void DBImpl::PrintStatistics() {
}
Status DBImpl::StartPeriodicTaskScheduler() {
#ifndef ROCKSDB_LITE
#ifndef NDEBUG
// It only used by test to disable scheduler
@@ -824,9 +834,13 @@ Status DBImpl::StartPeriodicTaskScheduler() {
periodic_task_functions_.at(PeriodicTaskType::kFlushInfoLog));
return s;
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
Status DBImpl::RegisterRecordSeqnoTimeWorker() {
#ifndef ROCKSDB_LITE
uint64_t min_time_duration = std::numeric_limits<uint64_t>::max();
uint64_t max_time_duration = std::numeric_limits<uint64_t>::min();
{
@@ -869,6 +883,9 @@ Status DBImpl::RegisterRecordSeqnoTimeWorker() {
}
return s;
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
// esitmate the total size of stats_history_
@@ -889,6 +906,7 @@ size_t DBImpl::EstimateInMemoryStatsHistorySize() const {
void DBImpl::PersistStats() {
TEST_SYNC_POINT("DBImpl::PersistStats:Entry");
#ifndef ROCKSDB_LITE
if (shutdown_initiated_) {
return;
}
@@ -993,6 +1011,7 @@ void DBImpl::PersistStats() {
stats_history_size, stats_history_.size());
}
TEST_SYNC_POINT("DBImpl::PersistStats:End");
#endif // !ROCKSDB_LITE
}
bool DBImpl::FindStatsByTime(uint64_t start_time, uint64_t end_time,
@@ -1034,6 +1053,7 @@ Status DBImpl::GetStatsHistory(
void DBImpl::DumpStats() {
TEST_SYNC_POINT("DBImpl::DumpStats:1");
#ifndef ROCKSDB_LITE
std::string stats;
if (shutdown_initiated_) {
return;
@@ -1098,6 +1118,7 @@ void DBImpl::DumpStats() {
ROCKS_LOG_INFO(immutable_db_options_.info_log, "%s", stats.c_str());
}
}
#endif // !ROCKSDB_LITE
PrintStatistics();
}
@@ -1152,6 +1173,11 @@ FSDirectory* DBImpl::GetDataDir(ColumnFamilyData* cfd, size_t path_id) const {
Status DBImpl::SetOptions(
ColumnFamilyHandle* column_family,
const std::unordered_map<std::string, std::string>& options_map) {
#ifdef ROCKSDB_LITE
(void)column_family;
(void)options_map;
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
if (options_map.empty()) {
@@ -1209,10 +1235,15 @@ Status DBImpl::SetOptions(
}
LogFlush(immutable_db_options_.info_log);
return s;
#endif // ROCKSDB_LITE
}
Status DBImpl::SetDBOptions(
const std::unordered_map<std::string, std::string>& options_map) {
#ifdef ROCKSDB_LITE
(void)options_map;
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
if (options_map.empty()) {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"SetDBOptions(), empty input.");
@@ -1377,6 +1408,7 @@ Status DBImpl::SetDBOptions(
}
LogFlush(immutable_db_options_.info_log);
return s;
#endif // ROCKSDB_LITE
}
// return the same level if it cannot be moved
@@ -1430,10 +1462,15 @@ Status DBImpl::FlushWAL(bool sync) {
return SyncWAL();
}
bool DBImpl::WALBufferIsEmpty() {
InstrumentedMutexLock l(&log_write_mutex_);
bool DBImpl::WALBufferIsEmpty(bool lock) {
if (lock) {
log_write_mutex_.Lock();
}
log::Writer* cur_log_writer = logs_.back().writer;
auto res = cur_log_writer->BufferIsEmpty();
if (lock) {
log_write_mutex_.Unlock();
}
return res;
}
@@ -1535,57 +1572,29 @@ Status DBImpl::ApplyWALToManifest(VersionEdit* synced_wals) {
Status DBImpl::LockWAL() {
{
InstrumentedMutexLock lock(&mutex_);
if (lock_wal_count_ > 0) {
assert(lock_wal_write_token_);
++lock_wal_count_;
} else {
WriteThread::Writer w;
write_thread_.EnterUnbatched(&w, &mutex_);
WriteThread::Writer nonmem_w;
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
// NOTE: releasing mutex in EnterUnbatched might mean we are actually
// now lock_wal_count > 0
if (lock_wal_count_ == 0) {
assert(!lock_wal_write_token_);
lock_wal_write_token_ = write_controller_.GetStopToken();
}
++lock_wal_count_;
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
write_thread_.ExitUnbatched(&w);
WriteThread::Writer w;
write_thread_.EnterUnbatched(&w, &mutex_);
WriteThread::Writer nonmem_w;
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
lock_wal_write_token_ = write_controller_.GetStopToken();
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
write_thread_.ExitUnbatched(&w);
}
// NOTE: avoid I/O holding DB mutex
Status s = FlushWAL(/*sync=*/false);
if (!s.ok()) {
// Non-OK return should not be in locked state
UnlockWAL().PermitUncheckedError();
}
return s;
return FlushWAL(/*sync=*/false);
}
Status DBImpl::UnlockWAL() {
bool signal = false;
{
InstrumentedMutexLock lock(&mutex_);
if (lock_wal_count_ == 0) {
return Status::Aborted("No LockWAL() in effect");
}
--lock_wal_count_;
if (lock_wal_count_ == 0) {
lock_wal_write_token_.reset();
signal = true;
}
}
if (signal) {
// SignalAll outside of mutex for efficiency
bg_cv_.SignalAll();
lock_wal_write_token_.reset();
}
bg_cv_.SignalAll();
return Status::OK();
}
@@ -3286,6 +3295,11 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
assert(cfd != nullptr);
ReadCallback* read_callback = nullptr; // No read callback provided.
if (read_options.tailing) {
#ifdef ROCKSDB_LITE
// not supported in lite version
result = nullptr;
#else
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
auto iter = new ForwardIterator(this, read_options, cfd, sv,
/* allow_unprepared_value */ true);
@@ -3294,6 +3308,7 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
cfd->user_comparator(), iter, sv->current, kMaxSequenceNumber,
sv->mutable_cf_options.max_sequential_skip_in_iterations, read_callback,
this, cfd);
#endif
} else {
// Note: no need to consider the special case of
// last_seq_same_as_publish_seq_==false since NewIterator is overridden in
@@ -3424,6 +3439,10 @@ Status DBImpl::NewIterators(
iterators->clear();
iterators->reserve(column_families.size());
if (read_options.tailing) {
#ifdef ROCKSDB_LITE
return Status::InvalidArgument(
"Tailing iterator not supported in RocksDB lite");
#else
for (auto cfh : column_families) {
auto cfd = static_cast_with_check<ColumnFamilyHandleImpl>(cfh)->cfd();
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
@@ -3435,6 +3454,7 @@ Status DBImpl::NewIterators(
sv->mutable_cf_options.max_sequential_skip_in_iterations,
read_callback, this, cfd));
}
#endif
} else {
// Note: no need to consider the special case of
// last_seq_same_as_publish_seq_==false since NewIterators is overridden in
@@ -3456,9 +3476,11 @@ Status DBImpl::NewIterators(
const Snapshot* DBImpl::GetSnapshot() { return GetSnapshotImpl(false); }
#ifndef ROCKSDB_LITE
const Snapshot* DBImpl::GetSnapshotForWriteConflictBoundary() {
return GetSnapshotImpl(true);
}
#endif // ROCKSDB_LITE
std::pair<Status, std::shared_ptr<const Snapshot>>
DBImpl::CreateTimestampedSnapshot(SequenceNumber snapshot_seq, uint64_t ts) {
@@ -3706,6 +3728,7 @@ void DBImpl::ReleaseSnapshot(const Snapshot* s) {
delete casted_s;
}
#ifndef ROCKSDB_LITE
Status DBImpl::GetPropertiesOfAllTables(ColumnFamilyHandle* column_family,
TablePropertiesCollection* props) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
@@ -3749,6 +3772,7 @@ Status DBImpl::GetPropertiesOfTablesInRange(ColumnFamilyHandle* column_family,
return s;
}
#endif // ROCKSDB_LITE
const std::string& DBImpl::GetName() const { return dbname_; }
@@ -3767,6 +3791,7 @@ SystemClock* DBImpl::GetSystemClock() const {
return immutable_db_options_.clock;
}
#ifndef ROCKSDB_LITE
Status DBImpl::StartIOTrace(const TraceOptions& trace_options,
std::unique_ptr<TraceWriter>&& trace_writer) {
@@ -3780,6 +3805,7 @@ Status DBImpl::EndIOTrace() {
return Status::OK();
}
#endif // ROCKSDB_LITE
Options DBImpl::GetOptions(ColumnFamilyHandle* column_family) const {
InstrumentedMutexLock l(&mutex_);
@@ -3908,6 +3934,7 @@ bool DBImpl::GetPropertyHandleOptionsStatistics(std::string* value) {
return true;
}
#ifndef ROCKSDB_LITE
Status DBImpl::ResetStats() {
InstrumentedMutexLock l(&mutex_);
for (auto* cfd : *versions_->GetColumnFamilySet()) {
@@ -3917,6 +3944,7 @@ Status DBImpl::ResetStats() {
}
return Status::OK();
}
#endif // ROCKSDB_LITE
bool DBImpl::GetAggregatedIntProperty(const Slice& property,
uint64_t* aggregated_value) {
@@ -4125,6 +4153,7 @@ void DBImpl::ReleaseFileNumberFromPendingOutputs(
}
}
#ifndef ROCKSDB_LITE
Status DBImpl::GetUpdatesSince(
SequenceNumber seq, std::unique_ptr<TransactionLogIterator>* iter,
const TransactionLogIterator::ReadOptions& read_options) {
@@ -4373,6 +4402,7 @@ void DBImpl::GetAllColumnFamilyMetaData(
}
}
#endif // ROCKSDB_LITE
Status DBImpl::CheckConsistency() {
mutex_.AssertHeld();
@@ -4723,6 +4753,7 @@ Status DestroyDB(const std::string& dbname, const Options& options,
Status DBImpl::WriteOptionsFile(bool need_mutex_lock,
bool need_enter_write_thread) {
#ifndef ROCKSDB_LITE
WriteThread::Writer w;
if (need_mutex_lock) {
mutex_.Lock();
@@ -4779,9 +4810,14 @@ Status DBImpl::WriteOptionsFile(bool need_mutex_lock,
s.ToString().c_str());
}
}
#else
(void)need_mutex_lock;
(void)need_enter_write_thread;
#endif // !ROCKSDB_LITE
return Status::OK();
}
#ifndef ROCKSDB_LITE
namespace {
void DeleteOptionsFilesHelper(const std::map<uint64_t, std::string>& filenames,
const size_t num_files_to_keep,
@@ -4799,8 +4835,10 @@ void DeleteOptionsFilesHelper(const std::map<uint64_t, std::string>& filenames,
}
}
} // namespace
#endif // !ROCKSDB_LITE
Status DBImpl::DeleteObsoleteOptionsFiles() {
#ifndef ROCKSDB_LITE
std::vector<std::string> filenames;
// use ordered map to store keep the filenames sorted from the newest
// to the oldest.
@@ -4828,9 +4866,13 @@ Status DBImpl::DeleteObsoleteOptionsFiles() {
DeleteOptionsFilesHelper(options_filenames, kNumOptionsFilesKept,
immutable_db_options_.info_log, GetEnv());
return Status::OK();
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
Status DBImpl::RenameTempFileToOptionsFile(const std::string& file_name) {
#ifndef ROCKSDB_LITE
Status s;
uint64_t options_file_number = versions_->NewFileNumber();
@@ -4874,6 +4916,10 @@ Status DBImpl::RenameTempFileToOptionsFile(const std::string& file_name) {
DeleteObsoleteOptionsFiles().PermitUncheckedError();
}
return s;
#else
(void)file_name;
return Status::OK();
#endif // !ROCKSDB_LITE
}
#ifdef ROCKSDB_USING_THREAD_STATUS
@@ -4921,6 +4967,7 @@ void DumpRocksDBBuildVersion(Logger* log) {
}
}
#ifndef ROCKSDB_LITE
SequenceNumber DBImpl::GetEarliestMemTableSequenceNumber(SuperVersion* sv,
bool include_history) {
// Find the earliest sequence number that we know we can rely on reading
@@ -5877,5 +5924,6 @@ void DBImpl::RecordSeqnoToTimeMapping() {
seqno, unix_time);
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+25 -8
View File
@@ -54,9 +54,13 @@
#include "rocksdb/env.h"
#include "rocksdb/memtablerep.h"
#include "rocksdb/status.h"
#ifndef ROCKSDB_LITE
#include "rocksdb/trace_reader_writer.h"
#endif // ROCKSDB_LITE
#include "rocksdb/transaction_log.h"
#ifndef ROCKSDB_LITE
#include "rocksdb/utilities/replayer.h"
#endif // ROCKSDB_LITE
#include "rocksdb/write_buffer_manager.h"
#include "table/merging_iterator.h"
#include "table/scoped_arena_iterator.h"
@@ -426,7 +430,7 @@ class DBImpl : public DB {
const FlushOptions& options,
const std::vector<ColumnFamilyHandle*>& column_families) override;
virtual Status FlushWAL(bool sync) override;
bool WALBufferIsEmpty();
bool WALBufferIsEmpty(bool lock = true);
virtual Status SyncWAL() override;
virtual Status LockWAL() override;
virtual Status UnlockWAL() override;
@@ -465,6 +469,7 @@ class DBImpl : public DB {
uint64_t start_time, uint64_t end_time,
std::unique_ptr<StatsHistoryIterator>* stats_iterator) override;
#ifndef ROCKSDB_LITE
using DB::ResetStats;
virtual Status ResetStats() override;
// All the returned filenames start with "/"
@@ -592,6 +597,7 @@ class DBImpl : public DB {
ColumnFamilyHandle* column_family, const Range* range, std::size_t n,
TablePropertiesCollection* props) override;
#endif // ROCKSDB_LITE
// ---- End of implementations of the DB interface ----
SystemClock* GetSystemClock() const;
@@ -649,6 +655,7 @@ class DBImpl : public DB {
// depends also on data written to the WAL but not to the memtable.
SequenceNumber TEST_GetLastVisibleSequence() const;
#ifndef ROCKSDB_LITE
// Similar to Write() but will call the callback once on the single write
// thread to determine whether it is safe to perform the write.
virtual Status WriteWithCallback(const WriteOptions& write_options,
@@ -713,6 +720,7 @@ class DBImpl : public DB {
Status TraceIteratorSeekForPrev(const uint32_t& cf_id, const Slice& key,
const Slice& lower_bound,
const Slice upper_bound);
#endif // ROCKSDB_LITE
// Similar to GetSnapshot(), but also lets the db know that this snapshot
// will be used for transaction write-conflict checking. The DB can then
@@ -1168,7 +1176,9 @@ class DBImpl : public DB {
return files_grabbed_for_purge_;
}
#ifndef ROCKSDB_LITE
const PeriodicTaskScheduler& TEST_GetPeriodicTaskScheduler() const;
#endif // !ROCKSDB_LITE
#endif // NDEBUG
@@ -1391,10 +1401,12 @@ class DBImpl : public DB {
void NotifyOnMemTableSealed(ColumnFamilyData* cfd,
const MemTableInfo& mem_table_info);
#ifndef ROCKSDB_LITE
void NotifyOnExternalFileIngested(
ColumnFamilyData* cfd, const ExternalSstFileIngestionJob& ingestion_job);
virtual Status FlushForGetLiveFiles();
#endif // !ROCKSDB_LITE
void NewThreadStatusCfInfo(ColumnFamilyData* cfd) const;
@@ -1542,7 +1554,9 @@ class DBImpl : public DB {
friend class WriteUnpreparedTxnDB;
friend class WriteUnpreparedTxn;
#ifndef ROCKSDB_LITE
friend class ForwardIterator;
#endif
friend struct SuperVersion;
friend class CompactedDBImpl;
friend class DBTest_ConcurrentFlushWAL_Test;
@@ -2009,6 +2023,7 @@ class DBImpl : public DB {
// Used by WriteImpl to update bg_error_ in case of memtable insert error.
void MemTableInsertStatusCheck(const Status& memtable_insert_status);
#ifndef ROCKSDB_LITE
Status CompactFilesImpl(const CompactionOptions& compact_options,
ColumnFamilyData* cfd, Version* version,
const std::vector<std::string>& input_file_names,
@@ -2016,6 +2031,7 @@ class DBImpl : public DB {
const int output_level, int output_path_id,
JobContext* job_context, LogBuffer* log_buffer,
CompactionJobInfo* compaction_job_info);
#endif // ROCKSDB_LITE
ColumnFamilyData* GetColumnFamilyDataByName(const std::string& cf_name);
@@ -2154,6 +2170,7 @@ class DBImpl : public DB {
bool ShouldntRunManualCompaction(ManualCompactionState* m);
bool HaveManualCompaction(ColumnFamilyData* cfd);
bool MCOverlap(ManualCompactionState* m, ManualCompactionState* m1);
#ifndef ROCKSDB_LITE
void BuildCompactionJobInfo(const ColumnFamilyData* cfd, Compaction* c,
const Status& st,
const CompactionJobStats& compaction_job_stats,
@@ -2166,6 +2183,7 @@ class DBImpl : public DB {
ColumnFamilyData* cfd, uint64_t num,
std::unique_ptr<std::list<uint64_t>::iterator>& pending_output_elem,
uint64_t* next_file_number);
#endif //! ROCKSDB_LITE
bool ShouldPurge(uint64_t file_number) const;
void MarkAsGrabbedForPurge(uint64_t file_number);
@@ -2575,7 +2593,9 @@ class DBImpl : public DB {
// REQUIRES: mutex held
int num_running_ingest_file_;
#ifndef ROCKSDB_LITE
WalManager wal_manager_;
#endif // ROCKSDB_LITE
// A value of > 0 temporarily disables scheduling of background work
int bg_work_paused_;
@@ -2603,12 +2623,14 @@ class DBImpl : public DB {
// Only to be set during initialization
std::unique_ptr<PreReleaseCallback> recoverable_state_pre_release_callback_;
#ifndef ROCKSDB_LITE
// Scheduler to run DumpStats(), PersistStats(), and FlushInfoLog().
// Currently, internally it has a global timer instance for running the tasks.
PeriodicTaskScheduler periodic_task_scheduler_;
// It contains the implementations for each periodic task.
std::map<PeriodicTaskType, const PeriodicTaskFunc> periodic_task_functions_;
#endif
// When set, we use a separate queue for writes that don't write to memtable.
// In 2PC these are the writes at Prepare phase.
@@ -2663,14 +2685,9 @@ class DBImpl : public DB {
// thread safe, both read and write need db mutex hold.
SeqnoToTimeMapping seqno_time_mapping_;
// Stop write token that is acquired when first LockWAL() is called.
// Destroyed when last UnlockWAL() is called. Controlled by DB mutex.
// See lock_wal_count_
// stop write token that is acquired when LockWal() is called. Destructed
// when UnlockWal() is called.
std::unique_ptr<WriteControllerToken> lock_wal_write_token_;
// The number of LockWAL called without matching UnlockWAL call.
// See also lock_wal_write_token_
uint32_t lock_wal_count_;
};
class GetWithTimestampReadCallback : public ReadCallback {
+66
View File
@@ -30,6 +30,7 @@ bool DBImpl::EnoughRoomForCompaction(
bool* sfm_reserved_compact_space, LogBuffer* log_buffer) {
// Check if we have enough room to do the compaction
bool enough_room = true;
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm) {
@@ -44,6 +45,11 @@ bool DBImpl::EnoughRoomForCompaction(
*sfm_reserved_compact_space = true;
}
}
#else
(void)cfd;
(void)inputs;
(void)sfm_reserved_compact_space;
#endif // ROCKSDB_LITE
if (!enough_room) {
// Just in case tests want to change the value of enough_room
TEST_SYNC_POINT_CALLBACK(
@@ -253,9 +259,11 @@ Status DBImpl::FlushMemTableToOutputFile(
TEST_SYNC_POINT_CALLBACK(
"DBImpl::FlushMemTableToOutputFile:AfterPickMemtables", &flush_job);
#ifndef ROCKSDB_LITE
// may temporarily unlock and lock the mutex.
NotifyOnFlushBegin(cfd, &file_meta, mutable_cf_options, job_context->job_id,
flush_reason);
#endif // ROCKSDB_LITE
bool switched_to_mempurge = false;
// Within flush_job.Run, rocksdb may call event listener to notify
@@ -336,6 +344,7 @@ Status DBImpl::FlushMemTableToOutputFile(
// If flush ran smoothly and no mempurge happened
// install new SST file path.
if (s.ok() && (!switched_to_mempurge)) {
#ifndef ROCKSDB_LITE
// may temporarily unlock and lock the mutex.
NotifyOnFlushCompleted(cfd, mutable_cf_options,
flush_job.GetCommittedFlushJobsInfo());
@@ -358,6 +367,7 @@ Status DBImpl::FlushMemTableToOutputFile(
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
}
#endif // ROCKSDB_LITE
}
TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:Finish");
return s;
@@ -476,6 +486,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
IOStatus log_io_s = IOStatus::OK();
assert(num_cfs == static_cast<int>(jobs.size()));
#ifndef ROCKSDB_LITE
for (int i = 0; i != num_cfs; ++i) {
const MutableCFOptions& mutable_cf_options = all_mutable_cf_options.at(i);
// may temporarily unlock and lock the mutex.
@@ -483,6 +494,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
NotifyOnFlushBegin(cfds[i], &file_meta[i], mutable_cf_options,
job_context->job_id, flush_reason);
}
#endif /* !ROCKSDB_LITE */
if (logfile_number_ > 0) {
// TODO (yanqin) investigate whether we should sync the closed logs for
@@ -691,8 +703,10 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
mems_list.emplace_back(&mems);
mutable_cf_options_list.emplace_back(&all_mutable_cf_options[i]);
tmp_file_meta.emplace_back(&file_meta[i]);
#ifndef ROCKSDB_LITE
committed_flush_jobs_info.emplace_back(
jobs[i]->GetCommittedFlushJobsInfo());
#endif //! ROCKSDB_LITE
}
}
@@ -744,6 +758,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
if (made_progress) {
*made_progress = true;
}
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
assert(all_mutable_cf_options.size() == static_cast<size_t>(num_cfs));
@@ -774,6 +789,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
}
}
}
#endif // ROCKSDB_LITE
}
// Need to undo atomic flush if something went wrong, i.e. s is not OK and
@@ -811,6 +827,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
void DBImpl::NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
const MutableCFOptions& mutable_cf_options,
int job_id, FlushReason flush_reason) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.listeners.size() == 0U) {
return;
}
@@ -850,11 +867,19 @@ void DBImpl::NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
mutex_.Lock();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)file_meta;
(void)mutable_cf_options;
(void)job_id;
(void)flush_reason;
#endif // ROCKSDB_LITE
}
void DBImpl::NotifyOnFlushCompleted(
ColumnFamilyData* cfd, const MutableCFOptions& mutable_cf_options,
std::list<std::unique_ptr<FlushJobInfo>>* flush_jobs_info) {
#ifndef ROCKSDB_LITE
assert(flush_jobs_info != nullptr);
if (immutable_db_options_.listeners.size() == 0U) {
return;
@@ -886,6 +911,11 @@ void DBImpl::NotifyOnFlushCompleted(
mutex_.Lock();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)mutable_cf_options;
(void)flush_jobs_info;
#endif // ROCKSDB_LITE
}
Status DBImpl::CompactRange(const CompactRangeOptions& options,
@@ -1268,6 +1298,17 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
const int output_level, const int output_path_id,
std::vector<std::string>* const output_file_names,
CompactionJobInfo* compaction_job_info) {
#ifdef ROCKSDB_LITE
(void)compact_options;
(void)column_family;
(void)input_file_names;
(void)output_level;
(void)output_path_id;
(void)output_file_names;
(void)compaction_job_info;
// not supported in lite version
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
if (column_family == nullptr) {
return Status::InvalidArgument("ColumnFamilyHandle must be non-null.");
}
@@ -1326,8 +1367,10 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
}
return s;
#endif // ROCKSDB_LITE
}
#ifndef ROCKSDB_LITE
Status DBImpl::CompactFilesImpl(
const CompactionOptions& compact_options, ColumnFamilyData* cfd,
Version* version, const std::vector<std::string>& input_file_names,
@@ -1469,12 +1512,14 @@ Status DBImpl::CompactFilesImpl(
// SetBGError
compaction_job.io_status().PermitUncheckedError();
c->ReleaseCompactionFiles(s);
#ifndef ROCKSDB_LITE
// Need to make sure SstFileManager does its bookkeeping
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm && sfm_reserved_compact_space) {
sfm->OnCompactionCompletion(c.get());
}
#endif // ROCKSDB_LITE
ReleaseFileNumberFromPendingOutputs(pending_outputs_inserted_elem);
@@ -1531,6 +1576,7 @@ Status DBImpl::CompactFilesImpl(
return status;
}
#endif // ROCKSDB_LITE
Status DBImpl::PauseBackgroundWork() {
InstrumentedMutexLock guard_lock(&mutex_);
@@ -1564,6 +1610,7 @@ void DBImpl::NotifyOnCompactionBegin(ColumnFamilyData* cfd, Compaction* c,
const Status& st,
const CompactionJobStats& job_stats,
int job_id) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.listeners.empty()) {
return;
}
@@ -1592,11 +1639,19 @@ void DBImpl::NotifyOnCompactionBegin(ColumnFamilyData* cfd, Compaction* c,
}
mutex_.Lock();
current->Unref();
#else
(void)cfd;
(void)c;
(void)st;
(void)job_stats;
(void)job_id;
#endif // ROCKSDB_LITE
}
void DBImpl::NotifyOnCompactionCompleted(
ColumnFamilyData* cfd, Compaction* c, const Status& st,
const CompactionJobStats& compaction_job_stats, const int job_id) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.listeners.size() == 0U) {
return;
}
@@ -1626,6 +1681,13 @@ void DBImpl::NotifyOnCompactionCompleted(
current->Unref();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)c;
(void)st;
(void)compaction_job_stats;
(void)job_id;
#endif // ROCKSDB_LITE
}
// REQUIREMENT: block all background work by calling PauseBackgroundWork()
@@ -3539,12 +3601,14 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
c->ReleaseCompactionFiles(status);
*made_progress = true;
#ifndef ROCKSDB_LITE
// Need to make sure SstFileManager does its bookkeeping
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm && sfm_reserved_compact_space) {
sfm->OnCompactionCompletion(c.get());
}
#endif // ROCKSDB_LITE
NotifyOnCompactionCompleted(c->column_family_data(), c.get(), status,
compaction_job_stats, job_context->job_id);
@@ -3721,6 +3785,7 @@ bool DBImpl::MCOverlap(ManualCompactionState* m, ManualCompactionState* m1) {
return false;
}
#ifndef ROCKSDB_LITE
void DBImpl::BuildCompactionJobInfo(
const ColumnFamilyData* cfd, Compaction* c, const Status& st,
const CompactionJobStats& compaction_job_stats, const int job_id,
@@ -3789,6 +3854,7 @@ void DBImpl::BuildCompactionJobInfo(
std::move(blob_file_garbage_info));
}
}
#endif
// SuperVersionContext gets created and destructed outside of the lock --
// we use this conveniently to:
+2
View File
@@ -289,6 +289,7 @@ size_t DBImpl::TEST_GetWalPreallocateBlockSize(
return GetWalPreallocateBlockSize(write_buffer_size);
}
#ifndef ROCKSDB_LITE
void DBImpl::TEST_WaitForPeriodicTaskRun(std::function<void()> callback) const {
periodic_task_scheduler_.TEST_WaitForRun(callback);
}
@@ -302,6 +303,7 @@ SeqnoToTimeMapping DBImpl::TEST_GetSeqnoToTimeMapping() const {
return seqno_time_mapping_;
}
#endif // !ROCKSDB_LITE
size_t DBImpl::TEST_EstimateInMemoryStatsHistorySize() const {
return EstimateInMemoryStatsHistorySize();
+2
View File
@@ -20,6 +20,7 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
Status DBImpl::SuggestCompactRange(ColumnFamilyHandle* column_family,
const Slice* begin, const Slice* end) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
@@ -153,5 +154,6 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
return status;
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+4
View File
@@ -599,11 +599,13 @@ void DBImpl::PurgeObsoleteFiles(JobContext& state, bool schedule_only) {
to_delete;
}
#ifndef ROCKSDB_LITE
if (type == kWalFile && (immutable_db_options_.WAL_ttl_seconds > 0 ||
immutable_db_options_.WAL_size_limit_MB > 0)) {
wal_manager_.ArchiveWALFile(fname, number);
continue;
}
#endif // !ROCKSDB_LITE
// If I do not own these files, e.g. secondary instance with max_open_files
// = -1, then no need to delete or schedule delete these files since they
@@ -667,7 +669,9 @@ void DBImpl::PurgeObsoleteFiles(JobContext& state, bool schedule_only) {
}
}
}
#ifndef ROCKSDB_LITE
wal_manager_.PurgeObsoleteWALFiles();
#endif // ROCKSDB_LITE
LogFlush(immutable_db_options_.info_log);
InstrumentedMutexLock l(&mutex_);
--pending_purge_obsolete_files_;
+16
View File
@@ -153,6 +153,7 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
result.avoid_flush_during_recovery = false;
}
#ifndef ROCKSDB_LITE
ImmutableDBOptions immutable_db_options(result);
if (!immutable_db_options.IsWalDirSameAsDBPath()) {
// Either the WAL dir and db_paths[0]/db_name are not the same, or we
@@ -194,6 +195,7 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
NewSstFileManager(result.env, result.info_log));
result.sst_file_manager = sst_file_manager;
}
#endif // !ROCKSDB_LITE
// Supported wal compression types
if (!StreamingCompressionTypeSupported(result.wal_compression)) {
@@ -843,6 +845,7 @@ Status DBImpl::LogAndApplyForRecovery(const RecoveryContext& recovery_ctx) {
}
void DBImpl::InvokeWalFilterIfNeededOnColumnFamilyToWalNumberMap() {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.wal_filter == nullptr) {
return;
}
@@ -860,6 +863,7 @@ void DBImpl::InvokeWalFilterIfNeededOnColumnFamilyToWalNumberMap() {
}
wal_filter.ColumnFamilyLogNumberMap(cf_lognumber_map, cf_name_id_map);
#endif // !ROCKSDB_LITE
}
bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
@@ -868,6 +872,7 @@ bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
Status& status,
bool& stop_replay,
WriteBatch& batch) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.wal_filter == nullptr) {
return true;
}
@@ -953,6 +958,15 @@ bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
batch = new_batch;
}
return true;
#else // !ROCKSDB_LITE
(void)wal_number;
(void)wal_fname;
(void)reporter;
(void)status;
(void)stop_replay;
(void)batch;
return true;
#endif // ROCKSDB_LITE
}
// REQUIRES: wal_numbers are sorted in ascending order
@@ -1967,6 +1981,7 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
}
impl->mutex_.Unlock();
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
impl->immutable_db_options_.sst_file_manager.get());
if (s.ok() && sfm) {
@@ -2050,6 +2065,7 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
impl->immutable_db_options_.db_paths[0].path);
}
#endif // !ROCKSDB_LITE
if (s.ok()) {
ROCKS_LOG_HEADER(impl->immutable_db_options_.info_log, "DB pointer %p",
+17
View File
@@ -16,6 +16,7 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options,
const std::string& dbname)
@@ -320,5 +321,21 @@ Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
return s;
}
#else // !ROCKSDB_LITE
Status DB::OpenForReadOnly(const Options& /*options*/,
const std::string& /*dbname*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
Status DB::OpenForReadOnly(
const DBOptions& /*db_options*/, const std::string& /*dbname*/,
const std::vector<ColumnFamilyDescriptor>& /*column_families*/,
std::vector<ColumnFamilyHandle*>* /*handles*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+4
View File
@@ -5,6 +5,7 @@
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -145,10 +146,12 @@ class DBImplReadOnly : public DBImpl {
// FIXME: some missing overrides for more "write" functions
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
private:
// A "helper" function for DB::OpenForReadOnly without column families
@@ -164,3 +167,4 @@ class DBImplReadOnly : public DBImpl {
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+18
View File
@@ -17,6 +17,7 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
DBImplSecondary::DBImplSecondary(const DBOptions& db_options,
const std::string& dbname,
std::string secondary_path)
@@ -945,5 +946,22 @@ Status DB::OpenAndCompact(
output, override_options);
}
#else // !ROCKSDB_LITE
Status DB::OpenAsSecondary(const Options& /*options*/,
const std::string& /*name*/,
const std::string& /*secondary_path*/,
DB** /*dbptr*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
Status DB::OpenAsSecondary(
const DBOptions& /*db_options*/, const std::string& /*dbname*/,
const std::string& /*secondary_path*/,
const std::vector<ColumnFamilyDescriptor>& /*column_families*/,
std::vector<ColumnFamilyHandle*>* /*handles*/, DB** /*dbptr*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+4
View File
@@ -5,6 +5,7 @@
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -268,10 +269,12 @@ class DBImplSecondary : public DBImpl {
#endif // NDEBUG
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
// ColumnFamilyCollector is a write batch handler which does nothing
// except recording unique column family IDs
@@ -404,3 +407,4 @@ class DBImplSecondary : public DBImpl {
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+8
View File
@@ -149,6 +149,7 @@ Status DBImpl::Write(const WriteOptions& write_options, WriteBatch* my_batch) {
return s;
}
#ifndef ROCKSDB_LITE
Status DBImpl::WriteWithCallback(const WriteOptions& write_options,
WriteBatch* my_batch,
WriteCallback* callback) {
@@ -162,6 +163,7 @@ Status DBImpl::WriteWithCallback(const WriteOptions& write_options,
}
return s;
}
#endif // ROCKSDB_LITE
// The main write queue. This is the only write queue that updates LastSequence.
// When using one write queue, the same sequence also indicates the last
@@ -2030,6 +2032,7 @@ Status DBImpl::ScheduleFlushes(WriteContext* context) {
return status;
}
#ifndef ROCKSDB_LITE
void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
const MemTableInfo& mem_table_info) {
if (immutable_db_options_.listeners.size() == 0U) {
@@ -2045,6 +2048,7 @@ void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
}
mutex_.Lock();
}
#endif // ROCKSDB_LITE
// REQUIRES: mutex_ is held
// REQUIRES: this thread is currently at the front of the writer queue
@@ -2083,12 +2087,14 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
const MutableCFOptions mutable_cf_options = *cfd->GetLatestMutableCFOptions();
// Set memtable_info for memtable sealed callback
#ifndef ROCKSDB_LITE
MemTableInfo memtable_info;
memtable_info.cf_name = cfd->GetName();
memtable_info.first_seqno = cfd->mem()->GetFirstSequenceNumber();
memtable_info.earliest_seqno = cfd->mem()->GetEarliestSequenceNumber();
memtable_info.num_entries = cfd->mem()->num_entries();
memtable_info.num_deletes = cfd->mem()->num_deletes();
#endif // ROCKSDB_LITE
// Log this later after lock release. It may be outdated, e.g., if background
// flush happens before logging, but that should be ok.
int num_imm_unflushed = cfd->imm()->NumNotFlushed();
@@ -2247,9 +2253,11 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
InstallSuperVersionAndScheduleWork(cfd, &context->superversion_context,
mutable_cf_options);
#ifndef ROCKSDB_LITE
// Notify client that memtable is sealed, now that we have successfully
// installed a new memtable
NotifyOnMemTableSealed(cfd, memtable_info);
#endif // ROCKSDB_LITE
// It is possible that we got here without checking the value of i_os, but
// that is okay. If we did, it most likely means that s was already an error.
// In any case, ignore any unchecked error for i_os here.
+4
View File
@@ -19,6 +19,7 @@ class DBIOFailureTest : public DBTestBase {
DBIOFailureTest() : DBTestBase("db_io_failure_test", /*env_do_fsync=*/true) {}
};
#ifndef ROCKSDB_LITE
// Check that number of files does not grow when writes are dropped
TEST_F(DBIOFailureTest, DropWrites) {
do {
@@ -122,6 +123,7 @@ TEST_F(DBIOFailureTest, NoSpaceCompactRange) {
env_->no_space_.store(false, std::memory_order_release);
} while (ChangeCompactOptions());
}
#endif // ROCKSDB_LITE
TEST_F(DBIOFailureTest, NonWritableFileSystem) {
do {
@@ -145,6 +147,7 @@ TEST_F(DBIOFailureTest, NonWritableFileSystem) {
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBIOFailureTest, ManifestWriteError) {
// Test for the following problem:
// (a) Compaction produces file F
@@ -579,6 +582,7 @@ TEST_F(DBIOFailureTest, CompactionSstSyncError) {
ASSERT_EQ("bar3", Get(1, "foo"));
}
#endif // !(defined NDEBUG) || !defined(OS_WIN)
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+4
View File
@@ -1439,6 +1439,7 @@ void DBIter::Seek(const Slice& target) {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
StopWatch sw(clock_, statistics_, DB_SEEK);
#ifndef ROCKSDB_LITE
if (db_impl_ != nullptr && cfd_ != nullptr) {
// TODO: What do we do if this returns an error?
Slice lower_bound, upper_bound;
@@ -1455,6 +1456,7 @@ void DBIter::Seek(const Slice& target) {
db_impl_->TraceIteratorSeek(cfd_->GetID(), target, lower_bound, upper_bound)
.PermitUncheckedError();
}
#endif // ROCKSDB_LITE
status_ = Status::OK();
ReleaseTempPinnedData();
@@ -1512,6 +1514,7 @@ void DBIter::SeekForPrev(const Slice& target) {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
StopWatch sw(clock_, statistics_, DB_SEEK);
#ifndef ROCKSDB_LITE
if (db_impl_ != nullptr && cfd_ != nullptr) {
// TODO: What do we do if this returns an error?
Slice lower_bound, upper_bound;
@@ -1530,6 +1533,7 @@ void DBIter::SeekForPrev(const Slice& target) {
upper_bound)
.PermitUncheckedError();
}
#endif // ROCKSDB_LITE
status_ = Status::OK();
ReleaseTempPinnedData();
+6
View File
@@ -388,7 +388,13 @@ class DBIter final : public Iterator {
MergeContext merge_context_;
LocalStatistics local_stats_;
PinnedIteratorsManager pinned_iters_mgr_;
#ifdef ROCKSDB_LITE
ROCKSDB_FIELD_UNUSED
#endif
DBImpl* db_impl_;
#ifdef ROCKSDB_LITE
ROCKSDB_FIELD_UNUSED
#endif
ColumnFamilyData* cfd_;
const Slice* const timestamp_ub_;
const Slice* const timestamp_lb_;
+12
View File
@@ -891,6 +891,7 @@ TEST_P(DBIteratorTest, IteratorDeleteAfterCfDrop) {
}
// SetOptions not defined in ROCKSDB LITE
#ifndef ROCKSDB_LITE
TEST_P(DBIteratorTest, DBIteratorBoundTest) {
Options options = CurrentOptions();
options.env = env_;
@@ -1118,6 +1119,7 @@ TEST_P(DBIteratorTest, DBIteratorBoundMultiSeek) {
TestGetTickerCount(options, BLOCK_CACHE_MISS));
}
}
#endif
TEST_P(DBIteratorTest, DBIteratorBoundOptimizationTest) {
for (auto format_version : {2, 3, 4}) {
@@ -1562,6 +1564,7 @@ INSTANTIATE_TEST_CASE_P(DBIteratorTestForPinnedDataInstance,
DBIteratorTestForPinnedData,
testing::Values(true, false));
#ifndef ROCKSDB_LITE
TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
Options options = CurrentOptions();
BlockBasedTableOptions table_options;
@@ -1631,6 +1634,7 @@ TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
delete iter;
}
#endif
TEST_P(DBIteratorTest, PinnedDataIteratorMergeOperator) {
Options options = CurrentOptions();
@@ -2212,7 +2216,9 @@ TEST_P(DBIteratorTest, ReadAhead) {
ASSERT_OK(Put(Key(i), value));
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ("1,1,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
env_->random_read_bytes_counter_ = 0;
options.statistics->setTickerCount(NO_FILE_OPENS, 0);
@@ -2223,6 +2229,7 @@ TEST_P(DBIteratorTest, ReadAhead) {
size_t bytes_read = env_->random_read_bytes_counter_;
delete iter;
int64_t num_file_closes = TestGetTickerCount(options, NO_FILE_CLOSES);
env_->random_read_bytes_counter_ = 0;
options.statistics->setTickerCount(NO_FILE_OPENS, 0);
read_options.readahead_size = 1024 * 10;
@@ -2231,7 +2238,10 @@ TEST_P(DBIteratorTest, ReadAhead) {
int64_t num_file_opens_readahead = TestGetTickerCount(options, NO_FILE_OPENS);
size_t bytes_read_readahead = env_->random_read_bytes_counter_;
delete iter;
int64_t num_file_closes_readahead =
TestGetTickerCount(options, NO_FILE_CLOSES);
ASSERT_EQ(num_file_opens, num_file_opens_readahead);
ASSERT_EQ(num_file_closes, num_file_closes_readahead);
ASSERT_GT(bytes_read_readahead, bytes_read);
ASSERT_GT(bytes_read_readahead, read_options.readahead_size * 3);
@@ -2275,10 +2285,12 @@ TEST_P(DBIteratorTest, DBIteratorSkipRecentDuplicatesTest) {
ASSERT_OK(Put("b", std::to_string(i + 1).c_str()));
}
#ifndef ROCKSDB_LITE
// Check that memtable wasn't flushed.
std::string val;
ASSERT_TRUE(db_->GetProperty("rocksdb.num-files-at-level0", &val));
EXPECT_EQ("0", val);
#endif
// Seek iterator to a smaller key.
get_perf_context()->Reset();
+8
View File
@@ -10,6 +10,7 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "env/mock_env.h"
@@ -289,9 +290,16 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBlobs) {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+16
View File
@@ -72,8 +72,12 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenClose) {
printf("Open\n");
ASSERT_OK(DB::Open(options, dbname_, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db));
#endif
}
ASSERT_EQ(2, cache_->Size());
ASSERT_TRUE(cache_->Contains(data_path_0_));
@@ -100,8 +104,12 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenDelete) {
printf("Open\n");
ASSERT_OK(DB::Open(options, dbname_, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db));
#endif
}
ASSERT_EQ(1, cache_->Size());
ASSERT_TRUE(cache_->Contains(dbname_));
@@ -253,12 +261,16 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenWithColumnFamilies) {
{"default", ColumnFamilyOptions()}},
&cfs, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_,
{{"cf1", cf_options},
{"cf2", cf_options},
{"default", ColumnFamilyOptions()}},
&cfs, &db));
#endif
}
// Logical block sizes of dbname_ and cf_path_0_ are cached during Open.
@@ -348,10 +360,14 @@ TEST_F(DBLogicalBlockSizeCacheTest, DestroyColumnFamilyHandle) {
options, dbname_,
{{"cf", cf_options}, {"default", ColumnFamilyOptions()}}, &cfs, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(
options, dbname_,
{{"cf", cf_options}, {"default", ColumnFamilyOptions()}}, &cfs, &db));
#endif
}
// cf_path_0_ and dbname_ are cached.
ASSERT_EQ(2, cache_->Size());
+2
View File
@@ -8,7 +8,9 @@
#include "rocksdb/perf_context.h"
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_builder.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "rocksdb/merge_operator.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators.h"
+4
View File
@@ -202,6 +202,7 @@ TEST_F(DBMergeOperatorTest, MergeErrorOnIteration) {
VerifyDBInternal({{"k1", "v1"}, {"k2", "corrupted"}, {"k2", "v2"}});
}
#ifndef ROCKSDB_LITE
TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
// This is like a mini-stress test dedicated to `OpFailureScope::kMustMerge`.
@@ -354,6 +355,7 @@ TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
}
}
#endif // ROCKSDB_LITE
class MergeOperatorPinningTest : public DBMergeOperatorTest,
public testing::WithParamInterface<bool> {
@@ -366,6 +368,7 @@ class MergeOperatorPinningTest : public DBMergeOperatorTest,
INSTANTIATE_TEST_CASE_P(MergeOperatorPinningTest, MergeOperatorPinningTest,
::testing::Bool());
#ifndef ROCKSDB_LITE
TEST_P(MergeOperatorPinningTest, OperandsMultiBlocks) {
Options options = CurrentOptions();
BlockBasedTableOptions table_options;
@@ -636,6 +639,7 @@ TEST_F(DBMergeOperatorTest, TailingIteratorMemtableUnrefedBySomeoneElse) {
EXPECT_TRUE(pushed_first_operand);
EXPECT_TRUE(stepped_to_next_operand);
}
#endif // ROCKSDB_LITE
TEST_F(DBMergeOperatorTest, SnapshotCheckerAndReadCallback) {
Options options = CurrentOptions();
+4
View File
@@ -29,6 +29,7 @@ class DBOptionsTest : public DBTestBase {
public:
DBOptionsTest() : DBTestBase("db_options_test", /*env_do_fsync=*/true) {}
#ifndef ROCKSDB_LITE
std::unordered_map<std::string, std::string> GetMutableDBOptionsMap(
const DBOptions& options) {
std::string options_str;
@@ -75,6 +76,7 @@ class DBOptionsTest : public DBTestBase {
auto sanitized_options = SanitizeOptions(dbname_, db_options);
return GetMutableDBOptionsMap(sanitized_options);
}
#endif // ROCKSDB_LITE
};
TEST_F(DBOptionsTest, ImmutableTrackAndVerifyWalsInManifest) {
@@ -110,6 +112,7 @@ TEST_F(DBOptionsTest, ImmutableVerifySstUniqueIdInManifest) {
}
// RocksDB lite don't support dynamic options.
#ifndef ROCKSDB_LITE
TEST_F(DBOptionsTest, AvoidUpdatingOptions) {
Options options;
@@ -1145,6 +1148,7 @@ TEST_F(DBOptionsTest, ChangeCompression) {
SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
TEST_F(DBOptionsTest, BottommostCompressionOptsWithFallbackType) {
// Verify the bottommost compression options still take effect even when the
+6
View File
@@ -55,6 +55,7 @@ class DBPropertiesTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, Empty) {
do {
Options options;
@@ -1111,6 +1112,7 @@ TEST_F(DBPropertiesTest, EstimateCompressionRatio) {
ASSERT_GT(CompressionRatioAtLevel(1), 10.0);
}
#endif // ROCKSDB_LITE
class CountingUserTblPropCollector : public TablePropertiesCollector {
public:
@@ -1261,6 +1263,7 @@ class BlockCountingTablePropertiesCollectorFactory
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
Options options = CurrentOptions();
options.level0_file_num_compaction_trigger = (1 << 30);
@@ -1300,6 +1303,7 @@ TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_GT(collector_factory->num_created_, 0U);
}
#endif // ROCKSDB_LITE
TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
Options options = CurrentOptions();
@@ -1361,6 +1365,7 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
ASSERT_GT(collector_factory->num_created_, 0U);
}
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
Random rnd(301);
@@ -2190,6 +2195,7 @@ TEST_F(DBPropertiesTest, TableMetaIndexKeys) {
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+10
View File
@@ -27,6 +27,9 @@ class DBRangeDelTest : public DBTestBase {
}
};
// PlainTableFactory, WriteBatchWithIndex, and NumTableFilesAtLevel() are not
// supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, NonBlockBasedTableNotSupported) {
// TODO: figure out why MmapReads trips the iterator pinning assertion in
// RangeDelAggregator. Ideally it would be supported; otherwise it should at
@@ -295,6 +298,7 @@ TEST_F(DBRangeDelTest, CompactRangeDelsSameStartKey) {
ASSERT_TRUE(db_->Get(ReadOptions(), "b1", &value).IsNotFound());
}
}
#endif // ROCKSDB_LITE
TEST_F(DBRangeDelTest, FlushRemovesCoveredKeys) {
const int kNum = 300, kRangeBegin = 50, kRangeEnd = 250;
@@ -331,6 +335,8 @@ TEST_F(DBRangeDelTest, FlushRemovesCoveredKeys) {
db_->ReleaseSnapshot(snapshot);
}
// NumTableFilesAtLevel() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, CompactionRemovesCoveredKeys) {
const int kNumPerFile = 100, kNumFiles = 4;
Options opts = CurrentOptions();
@@ -511,6 +517,7 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
std::numeric_limits<uint64_t>::max() /* max_file_num_to_ignore */,
"" /*trim_ts*/));
}
#endif // ROCKSDB_LITE
TEST_F(DBRangeDelTest, CompactionRemovesCoveredMergeOperands) {
const int kNumPerFile = 3, kNumFiles = 3;
@@ -582,6 +589,8 @@ TEST_F(DBRangeDelTest, PutDeleteRangeMergeFlush) {
ASSERT_EQ(expected, actual);
}
// NumTableFilesAtLevel() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, ObsoleteTombstoneCleanup) {
// During compaction to bottommost level, verify range tombstones older than
// the oldest snapshot are removed, while others are preserved.
@@ -3015,6 +3024,7 @@ TEST_F(DBRangeDelTest, DoubleCountRangeTombstoneCompensatedSize) {
db_->ReleaseSnapshot(snapshot);
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+15
View File
@@ -96,10 +96,19 @@ std::string GetTestNameSuffix(
return oss.str();
}
#ifndef ROCKSDB_LITE
INSTANTIATE_TEST_CASE_P(DBRateLimiterOnReadTest, DBRateLimiterOnReadTest,
::testing::Combine(::testing::Bool(), ::testing::Bool(),
::testing::Bool()),
GetTestNameSuffix);
#else // ROCKSDB_LITE
// Cannot use direct I/O in lite mode.
INSTANTIATE_TEST_CASE_P(DBRateLimiterOnReadTest, DBRateLimiterOnReadTest,
::testing::Combine(::testing::Values(false),
::testing::Bool(),
::testing::Bool()),
GetTestNameSuffix);
#endif // ROCKSDB_LITE
TEST_P(DBRateLimiterOnReadTest, Get) {
if (use_direct_io_ && !IsDirectIOSupported()) {
@@ -225,6 +234,7 @@ TEST_P(DBRateLimiterOnReadTest, Iterator) {
ASSERT_EQ(expected, options_.rate_limiter->GetTotalRequests(Env::IO_USER));
}
#if !defined(ROCKSDB_LITE)
TEST_P(DBRateLimiterOnReadTest, VerifyChecksum) {
if (use_direct_io_ && !IsDirectIOSupported()) {
@@ -254,6 +264,7 @@ TEST_P(DBRateLimiterOnReadTest, VerifyFileChecksums) {
ASSERT_EQ(expected, options_.rate_limiter->GetTotalRequests(Env::IO_USER));
}
#endif // !defined(ROCKSDB_LITE)
class DBRateLimiterOnWriteTest : public DBTestBase {
public:
@@ -308,8 +319,10 @@ TEST_F(DBRateLimiterOnWriteTest, Compact) {
// Pre-comaction:
// level-0 : `kNumFiles` SST files overlapping on [kStartKey, kEndKey]
#ifndef ROCKSDB_LITE
std::string files_per_level_pre_compaction = std::to_string(kNumFiles);
ASSERT_EQ(files_per_level_pre_compaction, FilesPerLevel(0 /* cf */));
#endif // !ROCKSDB_LITE
std::int64_t prev_total_request =
options_.rate_limiter->GetTotalRequests(Env::IO_TOTAL);
@@ -324,8 +337,10 @@ TEST_F(DBRateLimiterOnWriteTest, Compact) {
// Post-comaction:
// level-0 : 0 SST file
// level-1 : 1 SST file
#ifndef ROCKSDB_LITE
std::string files_per_level_post_compaction = "0,1";
ASSERT_EQ(files_per_level_post_compaction, FilesPerLevel(0 /* cf */));
#endif // !ROCKSDB_LITE
std::int64_t exepcted_compaction_request = 1;
EXPECT_EQ(actual_compaction_request, exepcted_compaction_request);
+4
View File
@@ -17,6 +17,7 @@ class DBReadOnlyTestWithTimestamp : public DBBasicTestWithTimestampBase {
: DBBasicTestWithTimestampBase("db_readonly_test_with_timestamp") {}
protected:
#ifndef ROCKSDB_LITE
void CheckDBOpenedAsCompactedDBWithOneLevel0File() {
VersionSet* const versions = dbfull()->GetVersionSet();
ASSERT_NE(versions, nullptr);
@@ -62,8 +63,10 @@ class DBReadOnlyTestWithTimestamp : public DBBasicTestWithTimestampBase {
ASSERT_TRUE(
storage_info->LevelFilesBrief(highest_non_empty_level).num_files > 0);
}
#endif // !ROCKSDB_LITE
};
#ifndef ROCKSDB_LITE
TEST_F(DBReadOnlyTestWithTimestamp, IteratorAndGetReadTimestampSizeMismatch) {
const int kNumKeysPerFile = 128;
const uint64_t kMaxKey = 1024;
@@ -946,6 +949,7 @@ TEST_F(DBReadOnlyTestWithTimestamp,
Close();
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+2
View File
@@ -18,6 +18,7 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
class DBSecondaryTestBase : public DBBasicTestWithTimestampBase {
public:
explicit DBSecondaryTestBase(const std::string& dbname)
@@ -1681,6 +1682,7 @@ TEST_F(DBSecondaryTestWithTimestamp, Iterators) {
Close();
}
#endif //! ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+4
View File
@@ -24,6 +24,7 @@ class DBSSTTest : public DBTestBase {
DBSSTTest() : DBTestBase("db_sst_test", /*env_do_fsync=*/true) {}
};
#ifndef ROCKSDB_LITE
// A class which remembers the name of each flushed file.
class FlushedFileCollector : public EventListener {
public:
@@ -52,6 +53,7 @@ class FlushedFileCollector : public EventListener {
std::vector<std::string> flushed_files_;
std::mutex mutex_;
};
#endif // ROCKSDB_LITE
TEST_F(DBSSTTest, DontDeletePendingOutputs) {
Options options;
@@ -149,6 +151,7 @@ TEST_F(DBSSTTest, SkipCheckingSSTFileSizesOnDBOpen) {
ASSERT_EQ("choo", Get("pika"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBSSTTest, DontDeleteMovedFile) {
// This test triggers move compaction and verifies that the file is not
// deleted when it's part of move compaction
@@ -1853,6 +1856,7 @@ TEST_F(DBSSTTest, DBWithSFMForBlobFilesAtomicFlush) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+2
View File
@@ -157,6 +157,7 @@ TEST_F(DBStatisticsTest, ExcludeTickers) {
ASSERT_GT(options.statistics->getTickerCount(BYTES_READ), 0);
}
#ifndef ROCKSDB_LITE
TEST_F(DBStatisticsTest, VerifyChecksumReadStat) {
Options options = CurrentOptions();
@@ -203,6 +204,7 @@ TEST_F(DBStatisticsTest, VerifyChecksumReadStat) {
}
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+2
View File
@@ -24,6 +24,7 @@
#include "test_util/testutil.h"
#include "util/random.h"
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
@@ -615,6 +616,7 @@ INSTANTIATE_TEST_CASE_P(DBTablePropertiesTest, DBTablePropertiesTest,
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
int main(int argc, char** argv) {
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
+9
View File
@@ -10,6 +10,7 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "db/forward_iterator.h"
@@ -240,6 +241,7 @@ TEST_P(DBTestTailingIterator, TailingIteratorTrimSeekToNext) {
iterh = nullptr;
BlockBasedTableOptions table_options;
table_options.no_block_cache = true;
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ReopenWithColumnFamilies({"default", "pikachu"}, options);
read_options.read_tier = kBlockCacheTier;
@@ -587,9 +589,16 @@ TEST_P(DBTestTailingIterator, SeekToFirstWithUpperBoundBug) {
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+64 -2
View File
@@ -129,6 +129,8 @@ TEST_F(DBTest, MockEnvTest) {
ASSERT_TRUE(!iterator->Valid());
delete iterator;
// TEST_FlushMemTable() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
DBImpl* dbi = static_cast_with_check<DBImpl>(db);
ASSERT_OK(dbi->TEST_FlushMemTable());
@@ -137,10 +139,14 @@ TEST_F(DBTest, MockEnvTest) {
ASSERT_OK(db->Get(ReadOptions(), keys[i], &res));
ASSERT_TRUE(res == vals[i]);
}
#endif // ROCKSDB_LITE
delete db;
}
// NewMemEnv returns nullptr in ROCKSDB_LITE since class InMemoryEnv isn't
// defined.
#ifndef ROCKSDB_LITE
TEST_F(DBTest, MemEnvTest) {
std::unique_ptr<Env> env{NewMemEnv(Env::Default())};
Options options;
@@ -193,6 +199,7 @@ TEST_F(DBTest, MemEnvTest) {
}
delete db;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, WriteEmptyBatch) {
Options options = CurrentOptions();
@@ -451,6 +458,7 @@ TEST_F(DBTest, MixedSlowdownOptionsStop) {
wo.no_slowdown = true;
ASSERT_OK(dbfull()->Put(wo, "foo3", "bar"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, LevelLimitReopen) {
Options options = CurrentOptions();
@@ -473,7 +481,9 @@ TEST_F(DBTest, LevelLimitReopen) {
options.max_bytes_for_level_multiplier_additional.resize(10, 1);
ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, LevelReopenWithFIFO) {
const int kLevelCount = 4;
const int kKeyCount = 5;
@@ -600,6 +610,7 @@ TEST_F(DBTest, LevelReopenWithFIFO) {
ASSERT_EQ(expected_files_per_level_after_fifo[i], FilesPerLevel(kCF));
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBTest, PutSingleDeleteGet) {
do {
@@ -846,7 +857,9 @@ TEST_F(DBTest, DISABLED_VeryLargeValue) {
ASSERT_OK(Put(key2, raw));
dbfull()->TEST_WaitForFlushMemTable();
#ifndef ROCKSDB_LITE
ASSERT_EQ(1, NumTableFilesAtLevel(0));
#endif // !ROCKSDB_LITE
std::string value;
Status s = db_->Get(ReadOptions(), key1, &value);
@@ -923,6 +936,7 @@ TEST_F(DBTest, WrongLevel0Config) {
ASSERT_OK(DB::Open(options, dbname_, &db_));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, GetOrderedByLevels) {
do {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
@@ -993,6 +1007,7 @@ TEST_F(DBTest, GetEncountersEmptyLevel) {
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 1); // XXX
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FlushMultipleMemtable) {
do {
@@ -1012,6 +1027,7 @@ TEST_F(DBTest, FlushMultipleMemtable) {
ASSERT_OK(Flush(1));
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, FlushSchedule) {
Options options = CurrentOptions();
options.disable_auto_compactions = true;
@@ -1056,6 +1072,7 @@ TEST_F(DBTest, FlushSchedule) {
ASSERT_LE(pikachu_tables, static_cast<uint64_t>(10));
ASSERT_GT(pikachu_tables, static_cast<uint64_t>(0));
}
#endif // ROCKSDB_LITE
namespace {
class KeepFilter : public CompactionFilter {
@@ -1120,12 +1137,14 @@ class DelayFilterFactory : public CompactionFilterFactory {
};
} // anonymous namespace
#ifndef ROCKSDB_LITE
static std::string CompressibleString(Random* rnd, int len) {
std::string r;
test::CompressibleString(rnd, 0.8, len, &r);
return r;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FailMoreDbPaths) {
Options options = CurrentOptions();
@@ -1247,6 +1266,7 @@ void CheckLiveFilesMeta(
}
}
#ifndef ROCKSDB_LITE
void AddBlobFile(const ColumnFamilyHandle* cfh, uint64_t blob_file_number,
uint64_t total_blob_count, uint64_t total_blob_bytes,
const std::string& checksum_method,
@@ -1538,7 +1558,9 @@ TEST_F(DBTest, DISABLED_RepeatedWritesToSameKey) {
}
} while (ChangeCompactOptions());
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
static bool Between(uint64_t val, uint64_t low, uint64_t high) {
bool result = (val >= low) && (val <= high);
if (!result) {
@@ -1935,7 +1957,9 @@ TEST_F(DBTest, ApproximateSizes_MixOfSmallAndLarge) {
// ApproximateOffsetOf() is not yet implemented in plain table format.
} while (ChangeOptions(kSkipPlainTable));
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, Snapshot) {
env_->SetMockSleep();
anon::OptionsOverride options_override;
@@ -2057,6 +2081,7 @@ TEST_F(DBTest, HiddenValuesAreRemoved) {
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction |
kSkipPlainTable));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, UnremovableSingleDelete) {
// If we compact:
@@ -2107,6 +2132,7 @@ TEST_F(DBTest, UnremovableSingleDelete) {
kSkipMergePut));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DeletionMarkers1) {
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
@@ -2226,6 +2252,7 @@ TEST_F(DBTest, OverlapInLevel0) {
ASSERT_EQ("NOT_FOUND", Get(1, "600"));
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, ComparatorCheck) {
class NewComparator : public Comparator {
@@ -2413,6 +2440,7 @@ TEST_F(DBTest, DestroyDBMetaDatabase) {
ASSERT_TRUE(!(DB::Open(options, metametadbname, &db)).ok());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, SnapshotFiles) {
do {
Options options = CurrentOptions();
@@ -2639,6 +2667,7 @@ TEST_F(DBTest, GetLiveBlobFiles) {
ASSERT_EQ(cfmd.blob_file_count, 1U);
ASSERT_EQ(cfmd.blob_file_size, bmd.blob_file_size);
}
#endif
TEST_F(DBTest, PurgeInfoLogs) {
Options options = CurrentOptions();
@@ -2686,6 +2715,7 @@ TEST_F(DBTest, PurgeInfoLogs) {
}
}
#ifndef ROCKSDB_LITE
// Multi-threaded test:
namespace {
@@ -2872,6 +2902,7 @@ INSTANTIATE_TEST_CASE_P(
::testing::Combine(
::testing::ValuesIn(MultiThreadedDBTest::GenerateOptionConfigs()),
::testing::Bool()));
#endif // ROCKSDB_LITE
// Group commit test:
#if !defined(OS_WIN)
@@ -3078,6 +3109,7 @@ class ModelDB : public DB {
return s;
}
#ifndef ROCKSDB_LITE
using DB::IngestExternalFile;
Status IngestExternalFile(
ColumnFamilyHandle* /*column_family*/,
@@ -3119,6 +3151,7 @@ class ModelDB : public DB {
std::size_t /*n*/, TablePropertiesCollection* /*props*/) override {
return Status();
}
#endif // ROCKSDB_LITE
using DB::KeyMayExist;
bool KeyMayExist(const ReadOptions& /*options*/,
@@ -3299,6 +3332,7 @@ class ModelDB : public DB {
Status DisableFileDeletions() override { return Status::OK(); }
Status EnableFileDeletions(bool /*force*/) override { return Status::OK(); }
#ifndef ROCKSDB_LITE
Status GetLiveFiles(std::vector<std::string>&, uint64_t* /*size*/,
bool /*flush_memtable*/ = true) override {
@@ -3342,6 +3376,7 @@ class ModelDB : public DB {
void GetColumnFamilyMetaData(ColumnFamilyHandle* /*column_family*/,
ColumnFamilyMetaData* /*metadata*/) override {}
#endif // ROCKSDB_LITE
Status GetDbIdentity(std::string& /*identity*/) const override {
return Status::OK();
@@ -3590,10 +3625,12 @@ TEST_F(DBTest, BlockBasedTablePrefixIndexTest) {
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#ifndef ROCKSDB_LITE
// Back to original
ASSERT_OK(dbfull()->SetOptions({{"prefix_extractor", "fixed:1"}}));
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#endif // !ROCKSDB_LITE
// Same if there's a problem initally loading prefix transform
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
@@ -3605,10 +3642,12 @@ TEST_F(DBTest, BlockBasedTablePrefixIndexTest) {
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#ifndef ROCKSDB_LITE
// Change again
ASSERT_OK(dbfull()->SetOptions({{"prefix_extractor", "fixed:2"}}));
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#endif // !ROCKSDB_LITE
SyncPoint::GetInstance()->DisableProcessing();
// Reopen with no prefix extractor, make sure everything still works.
@@ -3730,6 +3769,7 @@ TEST_F(DBTest, ChecksumTest) {
ASSERT_EQ("h", Get("g"));
}
#ifndef ROCKSDB_LITE
TEST_P(DBTestWithParam, FIFOCompactionTest) {
for (int iter = 0; iter < 2; ++iter) {
// first iteration -- auto compaction
@@ -4101,7 +4141,9 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
options.compaction_options_fifo.max_table_files_size);
}
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
/*
* This test is not reliable enough as it heavily depends on disk behavior.
* Disable as it is flaky.
@@ -4282,6 +4324,7 @@ TEST_F(DBTest, ConcurrentMemtableNotSupported) {
ASSERT_NOK(db_->CreateColumnFamily(cf_options, "name", &handle));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, SanitizeNumThreads) {
for (int attempt = 0; attempt < 2; attempt++) {
@@ -4435,6 +4478,7 @@ TEST_F(DBTest, ManualFlushWalAndWriteRace) {
ASSERT_EQ("value2", Get("foo2"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DynamicMemtableOptions) {
const uint64_t k64KB = 1 << 16;
const uint64_t k128KB = 1 << 17;
@@ -4588,6 +4632,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
#ifdef ROCKSDB_USING_THREAD_STATUS
namespace {
@@ -5037,6 +5082,7 @@ TEST_P(DBTestWithParam, PreShutdownCompactionMiddle) {
#endif // ROCKSDB_USING_THREAD_STATUS
#ifndef ROCKSDB_LITE
TEST_F(DBTest, FlushOnDestroy) {
WriteOptions wo;
wo.disableWAL = true;
@@ -5589,6 +5635,7 @@ TEST_F(DBTest, DynamicUniversalCompactionOptions) {
dbfull()->GetOptions().compaction_options_universal.allow_trivial_move,
false);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FileCreationRandomFailure) {
Options options;
@@ -5651,6 +5698,7 @@ TEST_F(DBTest, FileCreationRandomFailure) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DynamicMiscOptions) {
// Test max_sequential_skip_in_iterations
@@ -5745,6 +5793,7 @@ TEST_F(DBTest, DynamicMiscOptions) {
&mutable_cf_options));
ASSERT_FALSE(mutable_cf_options.check_flush_compaction_key_order);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, L0L1L2AndUpHitCounter) {
const int kNumLevels = 3;
@@ -5963,6 +6012,7 @@ TEST_F(DBTest, MergeTestTime) {
#endif // ROCKSDB_USING_THREAD_STATUS
}
#ifndef ROCKSDB_LITE
TEST_P(DBTestWithParam, MergeCompactionTimeTest) {
SetPerfLevel(kEnableTime);
Options options = CurrentOptions();
@@ -6025,6 +6075,7 @@ TEST_P(DBTestWithParam, FilterCompactionTimeTest) {
TestGetTickerCount(options, FILTER_OPERATION_TOTAL_TIME));
delete itr;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, TestLogCleanup) {
Options options = CurrentOptions();
@@ -6041,6 +6092,7 @@ TEST_F(DBTest, TestLogCleanup) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, EmptyCompactedDB) {
Options options = CurrentOptions();
options.max_open_files = -1;
@@ -6050,7 +6102,9 @@ TEST_F(DBTest, EmptyCompactedDB) {
ASSERT_TRUE(s.IsNotSupported());
Close();
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, SuggestCompactRangeTest) {
class CompactionFilterFactoryGetContext : public CompactionFilterFactory {
public:
@@ -6279,6 +6333,7 @@ TEST_F(DBTest, CompactRangeWithEmptyBottomLevel) {
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
ASSERT_EQ(NumTableFilesAtLevel(1), kNumL0Files);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
const int kNumL0Files = 50;
@@ -6336,6 +6391,7 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
Options options = CurrentOptions();
options.max_background_compactions = 1;
@@ -6395,6 +6451,7 @@ TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
ASSERT_LE(cf_meta_data.levels[0].files.size(),
options.level0_file_num_compaction_trigger);
}
#endif // ROCKSDB_LITE
// Github issue #595
// Large write batch with column families
@@ -6605,7 +6662,7 @@ TEST_F(DBTest, HardLimit) {
sleeping_task_low.WaitUntilDone();
}
#if !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
#if !defined(ROCKSDB_LITE) && !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
class WriteStallListener : public EventListener {
public:
WriteStallListener() : condition_(WriteStallCondition::kNormal) {}
@@ -6847,7 +6904,8 @@ TEST_F(DBTest, LastWriteBufferDelay) {
sleeping_task.WakeUp();
sleeping_task.WaitUntilDone();
}
#endif // !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
#endif // !defined(ROCKSDB_LITE) &&
// !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
TEST_F(DBTest, FailWhenCompressionNotSupportedTest) {
CompressionType compressions[] = {kZlibCompression, kBZip2Compression,
@@ -6883,6 +6941,7 @@ TEST_F(DBTest, CreateColumnFamilyShouldFailOnIncompatibleOptions) {
delete handle;
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, RowCache) {
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -7014,6 +7073,7 @@ TEST_F(DBTest, ReusePinnableSlice) {
1);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, DeletingOldWalAfterDrop) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
@@ -7139,6 +7199,7 @@ TEST_F(DBTest, LargeBlockSizeTest) {
ASSERT_NOK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, CreationTimeOfOldestFile) {
const int kNumKeysPerFile = 32;
@@ -7327,6 +7388,7 @@ TEST_F(DBTest, ShuttingDownNotBlockStalledWrites) {
thd.join();
}
#endif
} // namespace ROCKSDB_NAMESPACE
+138
View File
@@ -48,6 +48,7 @@ class DBTest2 : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, OpenForReadOnly) {
DB* db_ptr = nullptr;
std::string dbname = test::PerThreadDBPath("db_readonly");
@@ -105,6 +106,66 @@ TEST_F(DBTest2, OpenForReadOnlyWithColumnFamilies) {
ASSERT_NOK(env_->FileExists(dbname));
}
class TestReadOnlyWithCompressedCache
: public DBTestBase,
public testing::WithParamInterface<std::tuple<int, bool>> {
public:
TestReadOnlyWithCompressedCache()
: DBTestBase("test_readonly_with_compressed_cache",
/*env_do_fsync=*/true) {
max_open_files_ = std::get<0>(GetParam());
use_mmap_ = std::get<1>(GetParam());
}
int max_open_files_;
bool use_mmap_;
};
TEST_P(TestReadOnlyWithCompressedCache, ReadOnlyWithCompressedCache) {
if (use_mmap_ && !IsMemoryMappedAccessSupported()) {
ROCKSDB_GTEST_SKIP("Test requires MMAP support");
return;
}
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Put("foo2", "barbarbarbarbarbarbarbar"));
ASSERT_OK(Flush());
DB* db_ptr = nullptr;
Options options = CurrentOptions();
options.allow_mmap_reads = use_mmap_;
options.max_open_files = max_open_files_;
options.compression = kSnappyCompression;
BlockBasedTableOptions table_options;
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
table_options.no_block_cache = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.statistics = CreateDBStatistics();
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db_ptr));
std::string v;
ASSERT_OK(db_ptr->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("bar", v);
ASSERT_EQ(0, options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
ASSERT_OK(db_ptr->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("bar", v);
if (Snappy_Supported()) {
if (use_mmap_) {
ASSERT_EQ(0,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
} else {
ASSERT_EQ(1,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
}
}
delete db_ptr;
}
INSTANTIATE_TEST_CASE_P(TestReadOnlyWithCompressedCache,
TestReadOnlyWithCompressedCache,
::testing::Combine(::testing::Values(-1, 100),
::testing::Bool()));
class PartitionedIndexTestListener : public EventListener {
public:
void OnFlushCompleted(DB* /*db*/, const FlushJobInfo& info) override {
@@ -145,6 +206,7 @@ TEST_F(DBTest2, PartitionedIndexUserToInternalKey) {
}
}
#endif // ROCKSDB_LITE
class PrefixFullBloomWithReverseComparator
: public DBTestBase,
@@ -288,6 +350,7 @@ TEST_F(DBTest2, MaxSuccessiveMergesChangeWithDBRecovery) {
Reopen(options);
}
#ifndef ROCKSDB_LITE
class DBTestSharedWriteBufferAcrossCFs
: public DBTestBase,
public testing::WithParamInterface<std::tuple<bool, bool>> {
@@ -1541,7 +1604,9 @@ TEST_P(PresetCompressionDictTest, CompactNonBottommost) {
}
ASSERT_OK(Flush());
}
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
uint64_t prev_compression_dict_bytes_inserted =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT);
@@ -1549,7 +1614,9 @@ TEST_P(PresetCompressionDictTest, CompactNonBottommost) {
// file is not bottommost due to the existing L2 file covering the same key-
// range.
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// We can use `BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT` to detect whether a
// compression dictionary exists since dictionaries would be preloaded when
// the compaction finishes.
@@ -1613,13 +1680,17 @@ TEST_P(PresetCompressionDictTest, CompactBottommost) {
}
ASSERT_OK(Flush());
}
#ifndef ROCKSDB_LITE
ASSERT_EQ("2", FilesPerLevel(0));
#endif // ROCKSDB_LITE
uint64_t prev_compression_dict_bytes_inserted =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT);
CompactRangeOptions cro;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
ASSERT_GT(
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT),
prev_compression_dict_bytes_inserted);
@@ -1986,6 +2057,7 @@ TEST_F(DBTest2, CompactionStall) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, FirstSnapshotTest) {
Options options;
@@ -2002,6 +2074,7 @@ TEST_F(DBTest2, FirstSnapshotTest) {
db_->ReleaseSnapshot(s1);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, DuplicateSnapshot) {
Options options;
options = CurrentOptions(options);
@@ -2033,6 +2106,7 @@ TEST_F(DBTest2, DuplicateSnapshot) {
db_->ReleaseSnapshot(s);
}
}
#endif // ROCKSDB_LITE
class PinL0IndexAndFilterBlocksTest
: public DBTestBase,
@@ -2277,6 +2351,7 @@ INSTANTIATE_TEST_CASE_P(PinL0IndexAndFilterBlocksTest,
std::make_tuple(false, false),
std::make_tuple(false, true)));
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, MaxCompactionBytesTest) {
Options options = CurrentOptions();
options.memtable_factory.reset(test::NewSpecialSkipListFactory(
@@ -2550,6 +2625,7 @@ TEST_F(DBTest2, PersistentCache) {
new MockPersistentCache(type, 10 * 1024));
table_options.no_block_cache = true;
table_options.block_cache = bsize ? NewLRUCache(bsize) : nullptr;
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
DestroyAndReopen(options);
@@ -2642,6 +2718,7 @@ TEST_F(DBTest2, SyncPointMarker) {
ASSERT_EQ(sync_point_called.load(), 1);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif
size_t GetEncodedEntrySize(size_t key_size, size_t value_size) {
std::string buffer;
@@ -2848,6 +2925,7 @@ TEST_F(DBTest2, ReadAmpBitmapLiveInCacheAfterDBClose) {
}
#endif // !OS_SOLARIS
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, AutomaticCompactionOverlapManualCompaction) {
Options options = CurrentOptions();
options.num_levels = 3;
@@ -3128,7 +3206,9 @@ TEST_F(DBTest2, PausingManualCompaction3) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::Run():PausingManualCompaction:1",
@@ -3142,14 +3222,18 @@ TEST_F(DBTest2, PausingManualCompaction3) {
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
// As manual compaction disabled, not even reach sync point
ASSERT_EQ(run_manual_compactions, 0);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"CompactionJob::Run():PausingManualCompaction:1");
dbfull()->EnableManualCompaction();
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3178,7 +3262,9 @@ TEST_F(DBTest2, PausingManualCompaction4) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::Run():PausingManualCompaction:2", [&](void* arg) {
@@ -3205,13 +3291,17 @@ TEST_F(DBTest2, PausingManualCompaction4) {
.IsManualCompactionPaused());
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
ASSERT_EQ(run_manual_compactions, 1);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"CompactionJob::Run():PausingManualCompaction:2");
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3244,7 +3334,9 @@ TEST_F(DBTest2, CancelManualCompaction1) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
@@ -3268,7 +3360,9 @@ TEST_F(DBTest2, CancelManualCompaction1) {
// E.g. we should call the compaction function exactly one time.
ASSERT_EQ(compactions_run, 0);
ASSERT_EQ(run_manual_compactions, 0);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
compactions_run = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
@@ -3299,7 +3393,9 @@ TEST_F(DBTest2, CancelManualCompaction1) {
compact_options.canceled->store(false, std::memory_order_relaxed);
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3333,7 +3429,9 @@ TEST_F(DBTest2, CancelManualCompaction2) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -3381,7 +3479,9 @@ TEST_F(DBTest2, CancelManualCompaction2) {
compact_options.canceled->store(false, std::memory_order_relaxed);
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3609,6 +3709,7 @@ TEST_F(DBTest2, OptimizeForSmallDB) {
value.Reset();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, IterRaceFlush1) {
ASSERT_OK(Put("foo", "v1"));
@@ -3891,6 +3992,7 @@ TEST_F(DBTest2, LowPriWrite) {
ASSERT_EQ(1, rate_limit_count.load());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, RateLimitedCompactionReads) {
// compaction input has 512KB data
const int kNumKeysPerFile = 128;
@@ -3971,6 +4073,7 @@ TEST_F(DBTest2, RateLimitedCompactionReads) {
}
}
}
#endif // ROCKSDB_LITE
// Make sure DB can be reopen with reduced number of levels, given no file
// is on levels higher than the new num_levels.
@@ -3983,15 +4086,21 @@ TEST_F(DBTest2, ReduceLevel) {
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Flush());
MoveFilesToLevel(6);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
CompactRangeOptions compact_options;
compact_options.change_level = true;
compact_options.target_level = 1;
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
options.num_levels = 3;
Reopen(options);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
}
// Test that ReadCallback is actually used in both memtbale and sst tables
@@ -4026,14 +4135,18 @@ TEST_F(DBTest2, ReadCallbackTest) {
}
ASSERT_OK(Flush());
MoveFilesToLevel(6);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
for (; i < 30; i++) {
ASSERT_OK(Put(key, value + std::to_string(i)));
auto snapshot = dbfull()->GetSnapshot();
snapshots.push_back(snapshot);
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ("1,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
// And also add some values to the memtable
for (; i < 40; i++) {
ASSERT_OK(Put(key, value + std::to_string(i)));
@@ -4075,6 +4188,7 @@ TEST_F(DBTest2, ReadCallbackTest) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, LiveFilesOmitObsoleteFiles) {
// Regression test for race condition where an obsolete file is returned to
@@ -5143,6 +5257,7 @@ TEST_F(DBTest2, TraceWithFilter) {
ASSERT_EQ(count, 6);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, PinnableSliceAndMmapReads) {
Options options = CurrentOptions();
@@ -5173,6 +5288,7 @@ TEST_F(DBTest2, PinnableSliceAndMmapReads) {
// compaction. It crashes if it does.
ASSERT_EQ(pinned_value.ToString(), "bar");
#ifndef ROCKSDB_LITE
pinned_value.Reset();
// Unsafe to pin mmap files when they could be kicked out of table cache
Close();
@@ -5190,6 +5306,7 @@ TEST_F(DBTest2, PinnableSliceAndMmapReads) {
ASSERT_EQ(Get("foo", &pinned_value), Status::OK());
ASSERT_TRUE(pinned_value.IsPinned());
ASSERT_EQ(pinned_value.ToString(), "bar");
#endif
}
TEST_F(DBTest2, DISABLED_IteratorPinnedMemory) {
@@ -5422,6 +5539,7 @@ TEST_F(DBTest2, TestGetColumnFamilyHandleUnlocked) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, TestCompactFiles) {
// Setup sync point dependency to reproduce the race condition of
// DBImpl::GetColumnFamilyHandleUnlocked
@@ -5489,6 +5607,7 @@ TEST_F(DBTest2, TestCompactFiles) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, MultiDBParallelOpenTest) {
const int kNumDbs = 2;
@@ -5674,6 +5793,7 @@ TEST_F(DBTest2, PrefixBloomFilteredOut) {
delete iter;
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, RowCacheSnapshot) {
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -5717,6 +5837,7 @@ TEST_F(DBTest2, RowCacheSnapshot) {
db_->ReleaseSnapshot(s2);
db_->ReleaseSnapshot(s3);
}
#endif // ROCKSDB_LITE
// When DB is reopened with multiple column families, the manifest file
// is written after the first CF is flushed, and it is written again
@@ -5906,7 +6027,9 @@ TEST_F(DBTest2, SameSmallestInSameLevel) {
ASSERT_OK(db_->Merge(WriteOptions(), "key", "8"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,4,1", FilesPerLevel());
#endif // ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", Get("key"));
}
@@ -6039,7 +6162,11 @@ TEST_F(DBTest2, ChangePrefixExtractor) {
// Sometimes filter is checked based on upper bound. Assert counters
// for that case. Otherwise, only check data correctness.
#ifndef ROCKSDB_LITE
bool expect_filter_check = !use_partitioned_filter;
#else
bool expect_filter_check = false;
#endif
table_options.partition_filters = use_partitioned_filter;
if (use_partitioned_filter) {
table_options.index_type =
@@ -6237,6 +6364,7 @@ TEST_F(DBTest2, BlockBasedTablePrefixGetIndexNotFound) {
ASSERT_EQ("ok", Get("b1"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, AutoPrefixMode1) {
do {
// create a DB with block prefix index
@@ -7114,6 +7242,7 @@ TEST_F(DBTest2, FileTemperatureManifestFixup) {
std::vector<ColumnFamilyDescriptor> column_families;
for (size_t i = 0; i < handles_.size(); ++i) {
ColumnFamilyDescriptor cfdescriptor;
// GetDescriptor is not implemented for ROCKSDB_LITE
handles_[i]->GetDescriptor(&cfdescriptor).PermitUncheckedError();
column_families.push_back(cfdescriptor);
}
@@ -7153,6 +7282,7 @@ TEST_F(DBTest2, FileTemperatureManifestFixup) {
Close();
}
#endif // ROCKSDB_LITE
// WAL recovery mode is WALRecoveryMode::kPointInTimeRecovery.
TEST_F(DBTest2, PointInTimeRecoveryWithIOErrorWhileReadingWal) {
@@ -7208,6 +7338,7 @@ TEST_F(DBTest2, PointInTimeRecoveryWithSyncFailureInCFCreation) {
ReopenWithColumnFamilies({"default", "test1", "test2"}, options);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, SortL0FilesByEpochNumber) {
Options options = CurrentOptions();
options.num_levels = 1;
@@ -7417,6 +7548,7 @@ TEST_F(DBTest2, RecoverEpochNumber) {
}
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, RenameDirectory) {
Options options = CurrentOptions();
@@ -7494,7 +7626,9 @@ TEST_F(DBTest2, SstUniqueIdVerifyBackwardCompatible) {
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// Reopen (with verification)
ASSERT_TRUE(options.verify_sst_unique_id_in_manifest);
@@ -7549,7 +7683,9 @@ TEST_F(DBTest2, SstUniqueIdVerify) {
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// Reopen with verification should fail
options.verify_sst_unique_id_in_manifest = true;
@@ -7672,6 +7808,7 @@ TEST_F(DBTest2, BestEffortsRecoveryWithSstUniqueIdVerification) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, GetLatestSeqAndTsForKey) {
Destroy(last_options_);
@@ -7728,6 +7865,7 @@ TEST_F(DBTest2, GetLatestSeqAndTsForKey) {
// Verify that no read to SST files.
ASSERT_EQ(0, options.statistics->getTickerCount(GET_HIT_L0));
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+33
View File
@@ -70,6 +70,7 @@ DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
if (getenv("MEM_ENV")) {
mem_env_ = MockEnv::Create(base_env, base_env->GetSystemClock());
}
#ifndef ROCKSDB_LITE
if (getenv("ENCRYPTED_ENV")) {
std::shared_ptr<EncryptionProvider> provider;
std::string provider_id = getenv("ENCRYPTED_ENV");
@@ -81,6 +82,7 @@ DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
&provider));
encrypted_env_ = NewEncryptedEnv(mem_env_ ? mem_env_ : base_env, provider);
}
#endif // !ROCKSDB_LITE
env_ = new SpecialEnv(encrypted_env_ ? encrypted_env_
: (mem_env_ ? mem_env_ : base_env));
env_->SetBackgroundThreads(1, Env::LOW);
@@ -122,6 +124,22 @@ DBTestBase::~DBTestBase() {
}
bool DBTestBase::ShouldSkipOptions(int option_config, int skip_mask) {
#ifdef ROCKSDB_LITE
// These options are not supported in ROCKSDB_LITE
if (option_config == kHashSkipList ||
option_config == kPlainTableFirstBytePrefix ||
option_config == kPlainTableCappedPrefix ||
option_config == kPlainTableCappedPrefixNonMmap ||
option_config == kPlainTableAllBytesPrefix ||
option_config == kVectorRep || option_config == kHashLinkList ||
option_config == kUniversalCompaction ||
option_config == kUniversalCompactionMultiLevel ||
option_config == kUniversalSubcompactions ||
option_config == kFIFOCompaction ||
option_config == kConcurrentSkipList) {
return true;
}
#endif
if ((skip_mask & kSkipUniversalCompaction) &&
(option_config == kUniversalCompaction ||
@@ -359,6 +377,7 @@ Options DBTestBase::GetOptions(
bool can_allow_mmap = IsMemoryMappedAccessSupported();
switch (option_config) {
#ifndef ROCKSDB_LITE
case kHashSkipList:
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
options.memtable_factory.reset(NewHashSkipListRepFactory(16));
@@ -412,6 +431,7 @@ Options DBTestBase::GetOptions(
SetupSyncPointsToMockDirectIO();
break;
}
#endif // ROCKSDB_LITE
case kMergePut:
options.merge_operator = MergeOperators::CreatePutOperator();
break;
@@ -460,6 +480,10 @@ Options DBTestBase::GetOptions(
options.compaction_style = kCompactionStyleUniversal;
options.num_levels = 8;
break;
case kCompressedBlockCache:
options.allow_mmap_writes = can_allow_mmap;
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
break;
case kInfiniteMaxOpenFiles:
options.max_open_files = -1;
break;
@@ -682,6 +706,7 @@ void DBTestBase::Destroy(const Options& options, bool delete_cf_paths) {
if (delete_cf_paths) {
for (size_t i = 0; i < handles_.size(); ++i) {
ColumnFamilyDescriptor cfdescriptor;
// GetDescriptor is not implemented for ROCKSDB_LITE
handles_[i]->GetDescriptor(&cfdescriptor).PermitUncheckedError();
column_families.push_back(cfdescriptor);
}
@@ -1022,6 +1047,7 @@ std::string DBTestBase::AllEntriesFor(const Slice& user_key, int cf) {
return result;
}
#ifndef ROCKSDB_LITE
int DBTestBase::NumSortedRuns(int cf) {
ColumnFamilyMetaData cf_meta;
if (cf == 0) {
@@ -1140,6 +1166,7 @@ std::string DBTestBase::FilesPerLevel(int cf) {
return result;
}
#endif // !ROCKSDB_LITE
std::vector<uint64_t> DBTestBase::GetBlobFileNumbers() {
VersionSet* const versions = dbfull()->GetVersionSet();
@@ -1257,6 +1284,7 @@ void DBTestBase::MoveFilesToLevel(int level, int cf) {
}
}
#ifndef ROCKSDB_LITE
void DBTestBase::DumpFileCounts(const char* label) {
fprintf(stderr, "---\n%s:\n", label);
fprintf(stderr, "maxoverlap: %" PRIu64 "\n",
@@ -1268,6 +1296,7 @@ void DBTestBase::DumpFileCounts(const char* label) {
}
}
}
#endif // !ROCKSDB_LITE
std::string DBTestBase::DumpSSTableList() {
std::string property;
@@ -1593,6 +1622,7 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
}
if (tailing_iter) {
#ifndef ROCKSDB_LITE
// Tailing iterator
int iter_cnt = 0;
ReadOptions ro;
@@ -1621,6 +1651,7 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
}
delete iter;
#endif // ROCKSDB_LITE
}
if (total_reads_res) {
@@ -1648,6 +1679,7 @@ void DBTestBase::VerifyDBInternal(
iter->~InternalIterator();
}
#ifndef ROCKSDB_LITE
uint64_t DBTestBase::GetNumberOfSstFilesForColumnFamily(
DB* db, std::string column_family_name) {
@@ -1668,6 +1700,7 @@ uint64_t DBTestBase::GetSstSizeHelper(Temperature temperature) {
&prop));
return static_cast<uint64_t>(std::atoi(prop.c_str()));
}
#endif // ROCKSDB_LITE
void VerifySstUniqueIds(const TablePropertiesCollection& props) {
ASSERT_FALSE(props.empty()); // suspicious test if empty
+18 -9
View File
@@ -695,6 +695,7 @@ class SpecialEnv : public EnvWrapper {
bool no_slowdown_;
};
#ifndef ROCKSDB_LITE
class FileTemperatureTestFS : public FileSystemWrapper {
public:
explicit FileTemperatureTestFS(const std::shared_ptr<FileSystem>& fs)
@@ -869,6 +870,7 @@ class FlushCounterListener : public EventListener {
ASSERT_EQ(expected_flush_reason.load(), flush_job_info.flush_reason);
}
};
#endif
// A test merge operator mimics put but also fails if one of merge operands is
// "corrupted", "corrupted_try_merge", or "corrupted_must_merge".
@@ -1054,15 +1056,16 @@ class DBTestBase : public testing::Test {
kHashSkipList = 18,
kUniversalCompaction = 19,
kUniversalCompactionMultiLevel = 20,
kInfiniteMaxOpenFiles = 21,
kCRC32cChecksum = 22,
kFIFOCompaction = 23,
kOptimizeFiltersForHits = 24,
kRowCache = 25,
kRecycleLogFiles = 26,
kConcurrentSkipList = 27,
kPipelinedWrite = 28,
kConcurrentWALWrites = 29,
kCompressedBlockCache = 21,
kInfiniteMaxOpenFiles = 22,
kCRC32cChecksum = 23,
kFIFOCompaction = 24,
kOptimizeFiltersForHits = 25,
kRowCache = 26,
kRecycleLogFiles = 27,
kConcurrentSkipList = 28,
kPipelinedWrite = 29,
kConcurrentWALWrites = 30,
kDirectIO,
kLevelSubcompactions,
kBlockBasedTableWithIndexRestartInterval,
@@ -1258,6 +1261,7 @@ class DBTestBase : public testing::Test {
const std::vector<std::string>& cfs,
const Options& options);
#ifndef ROCKSDB_LITE
int NumSortedRuns(int cf = 0);
uint64_t TotalSize(int cf = 0);
@@ -1273,6 +1277,7 @@ class DBTestBase : public testing::Test {
double CompressionRatioAtLevel(int level, int cf = 0);
int TotalTableFiles(int cf = 0, int levels = -1);
#endif // ROCKSDB_LITE
std::vector<uint64_t> GetBlobFileNumbers();
@@ -1308,7 +1313,9 @@ class DBTestBase : public testing::Test {
void MoveFilesToLevel(int level, int cf = 0);
#ifndef ROCKSDB_LITE
void DumpFileCounts(const char* label);
#endif // ROCKSDB_LITE
std::string DumpSSTableList();
@@ -1377,10 +1384,12 @@ class DBTestBase : public testing::Test {
void VerifyDBInternal(
std::vector<std::pair<std::string, std::string>> true_data);
#ifndef ROCKSDB_LITE
uint64_t GetNumberOfSstFilesForColumnFamily(DB* db,
std::string column_family_name);
uint64_t GetSstSizeHelper(Temperature temperature);
#endif // ROCKSDB_LITE
uint64_t TestGetTickerCount(const Options& options, Tickers ticker_type) {
return options.statistics->getTickerCount(ticker_type);
+8
View File
@@ -9,6 +9,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#if !defined(ROCKSDB_LITE)
#include "rocksdb/utilities/table_properties_collectors.h"
#include "test_util/sync_point.h"
#include "test_util/testutil.h"
@@ -2219,9 +2220,16 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+21 -4
View File
@@ -444,6 +444,7 @@ TEST_F(DBWALTest, RecoverWithBlob) {
ASSERT_EQ(blob_file->GetTotalBlobCount(), 1);
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
ASSERT_NE(internal_stats, nullptr);
@@ -459,6 +460,7 @@ TEST_F(DBWALTest, RecoverWithBlob) {
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED],
compaction_stats[0].bytes_written +
compaction_stats[0].bytes_written_blob);
#endif // ROCKSDB_LITE
}
TEST_F(DBWALTest, RecoverWithBlobMultiSST) {
@@ -608,14 +610,25 @@ TEST_F(DBWALTest, WALWithChecksumHandoff) {
#endif // ROCKSDB_ASSERT_STATUS_CHECKED
}
#ifndef ROCKSDB_LITE
TEST_F(DBWALTest, LockWal) {
do {
Options options = CurrentOptions();
options.create_if_missing = true;
DestroyAndReopen(options);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->LoadDependency(
{{"DBWALTest::LockWal:AfterGetSortedWal",
"DBWALTest::LockWal:BeforeFlush:1"}});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put("foo", "v"));
ASSERT_OK(Put("bar", "v"));
port::Thread worker([&]() {
TEST_SYNC_POINT("DBWALTest::LockWal:BeforeFlush:1");
Status tmp_s = db_->Flush(FlushOptions());
ASSERT_OK(tmp_s);
});
ASSERT_OK(db_->LockWAL());
// Verify writes are stopped
@@ -628,10 +641,7 @@ TEST_F(DBWALTest, LockWal) {
ASSERT_OK(db_->GetSortedWalFiles(wals));
ASSERT_FALSE(wals.empty());
}
port::Thread worker([&]() {
Status tmp_s = db_->Flush(FlushOptions());
ASSERT_OK(tmp_s);
});
TEST_SYNC_POINT("DBWALTest::LockWal:AfterGetSortedWal");
FlushOptions flush_opts;
flush_opts.wait = false;
s = db_->Flush(flush_opts);
@@ -640,8 +650,11 @@ TEST_F(DBWALTest, LockWal) {
ASSERT_OK(db_->Put(WriteOptions(), "foo", "dontcare"));
worker.join();
SyncPoint::GetInstance()->DisableProcessing();
} while (ChangeWalOptions());
}
#endif //! ROCKSDB_LITE
class DBRecoveryTestBlobError
: public DBWALTest,
@@ -879,6 +892,7 @@ TEST_F(DBWALTest, PreallocateBlock) {
}
#endif // !(defined NDEBUG) || !defined(OS_WIN)
#ifndef ROCKSDB_LITE
TEST_F(DBWALTest, DISABLED_FullPurgePreservesRecycledLog) {
// TODO(ajkr): Disabled until WAL recycling is fixed for
// `kPointInTimeRecovery`.
@@ -2358,6 +2372,7 @@ TEST_F(DBWALTest, WalInManifestButNotInSortedWals) {
Close();
}
#endif // ROCKSDB_LITE
TEST_F(DBWALTest, WalTermTest) {
Options options = CurrentOptions();
@@ -2383,6 +2398,7 @@ TEST_F(DBWALTest, WalTermTest) {
ASSERT_EQ("NOT_FOUND", Get(1, "foo2"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBWALTest, GetCompressedWalsAfterSync) {
if (db_->GetOptions().wal_compression == kNoCompression) {
ROCKSDB_GTEST_BYPASS("stream compression not present");
@@ -2417,6 +2433,7 @@ TEST_F(DBWALTest, GetCompressedWalsAfterSync) {
Status s = dbfull()->GetSortedWalFiles(wals);
ASSERT_OK(s);
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+6
View File
@@ -13,7 +13,9 @@
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "test_util/testutil.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators/string_append/stringappend2.h"
@@ -643,6 +645,7 @@ TEST_F(DBBasicTestWithTimestamp, OpenAndTrimHistoryInvalidOptionTest) {
.IsInvalidArgument());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTestWithTimestamp, GetTimestampTableProperties) {
Options options = CurrentOptions();
const size_t kTimestampSize = Timestamp(0, 0).size();
@@ -672,6 +675,7 @@ TEST_F(DBBasicTestWithTimestamp, GetTimestampTableProperties) {
}
Close();
}
#endif // !ROCKSDB_LITE
class DBBasicTestWithTimestampTableOptions
: public DBBasicTestWithTimestampBase,
@@ -2673,6 +2677,7 @@ TEST_P(DBBasicTestWithTimestampCompressionSettings, PutDeleteGet) {
}
}
#ifndef ROCKSDB_LITE
// A class which remembers the name of each flushed file.
class FlushedFileCollector : public EventListener {
public:
@@ -2965,6 +2970,7 @@ TEST_F(DBBasicTestWithTimestamp, MultiGetNoReturnTs) {
Close();
}
#endif // !ROCKSDB_LITE
INSTANTIATE_TEST_CASE_P(
Timestamp, DBBasicTestWithTimestampCompressionSettings,
+2
View File
@@ -198,6 +198,7 @@ class TestFilePartitionerFactory : public SstPartitionerFactory {
const char* Name() const override { return "TestFilePartitionerFactory"; }
};
#ifndef ROCKSDB_LITE
TEST_F(TimestampCompatibleCompactionTest, CompactFilesRangeCheckL0) {
Options options = CurrentOptions();
options.env = env_;
@@ -343,6 +344,7 @@ TEST_F(TimestampCompatibleCompactionTest, EmptyCompactionOutput) {
cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+2
View File
@@ -780,6 +780,7 @@ TEST_P(DBWriteBufferManagerTest, MixedSlowDownOptionsMultipleDB) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#ifndef ROCKSDB_LITE
// Tests a `WriteBufferManager` constructed with `allow_stall == false` does not
// thrash memtable switching when full and a CF receives multiple writes.
@@ -846,6 +847,7 @@ TEST_P(DBWriteBufferManagerTest, StopSwitchingMemTablesOnceFlushing) {
delete shared_wbm_db;
}
#endif // ROCKSDB_LITE
INSTANTIATE_TEST_CASE_P(DBWriteBufferManagerTest, DBWriteBufferManagerTest,
testing::Bool());
+12 -114
View File
@@ -4,7 +4,6 @@
// (found in the LICENSE.Apache file in the root directory).
#include <atomic>
#include <cstdint>
#include <fstream>
#include <memory>
#include <thread>
@@ -606,124 +605,23 @@ TEST_P(DBWriteTest, IOErrorOnSwitchMemtable) {
Close();
}
// Test that db->LockWAL() flushes the WAL after locking, which can fail
TEST_P(DBWriteTest, LockWALInEffect) {
// Test that db->LockWAL() flushes the WAL after locking.
TEST_P(DBWriteTest, LockWalInEffect) {
Options options = GetOptions();
std::unique_ptr<FaultInjectionTestEnv> mock_env(
new FaultInjectionTestEnv(env_));
options.env = mock_env.get();
options.paranoid_checks = false;
Reopen(options);
// try the 1st WAL created during open
ASSERT_OK(Put("key0", "value"));
ASSERT_NE(options.manual_wal_flush, dbfull()->WALBufferIsEmpty());
ASSERT_OK(db_->LockWAL());
ASSERT_TRUE(dbfull()->WALBufferIsEmpty());
ASSERT_OK(db_->UnlockWAL());
ASSERT_OK(Put("key" + std::to_string(0), "value"));
ASSERT_TRUE(options.manual_wal_flush != dbfull()->WALBufferIsEmpty());
ASSERT_OK(dbfull()->LockWAL());
ASSERT_TRUE(dbfull()->WALBufferIsEmpty(false));
ASSERT_OK(dbfull()->UnlockWAL());
// try the 2nd wal created during SwitchWAL
ASSERT_OK(dbfull()->TEST_SwitchWAL());
ASSERT_OK(Put("key1", "value"));
ASSERT_NE(options.manual_wal_flush, dbfull()->WALBufferIsEmpty());
ASSERT_OK(db_->LockWAL());
ASSERT_TRUE(dbfull()->WALBufferIsEmpty());
ASSERT_OK(db_->UnlockWAL());
// Fail the WAL flush if applicable
mock_env->SetFilesystemActive(false);
Status s = Put("key2", "value");
if (options.manual_wal_flush) {
ASSERT_OK(s);
// I/O failure
ASSERT_NOK(db_->LockWAL());
// Should not need UnlockWAL after LockWAL fails
} else {
ASSERT_NOK(s);
ASSERT_OK(db_->LockWAL());
ASSERT_OK(db_->UnlockWAL());
}
mock_env->SetFilesystemActive(true);
// Writes should work again
ASSERT_OK(Put("key3", "value"));
ASSERT_EQ(Get("key3"), "value");
// Should be extraneous, but allowed
ASSERT_NOK(db_->UnlockWAL());
// Close before mock_env destruct.
Close();
}
TEST_P(DBWriteTest, LockWALConcurrentRecursive) {
Options options = GetOptions();
Reopen(options);
ASSERT_OK(Put("k1", "val"));
ASSERT_OK(db_->LockWAL()); // 0 -> 1
auto frozen_seqno = db_->GetLatestSequenceNumber();
std::atomic<bool> t1_completed{false};
port::Thread t1{[&]() {
// Won't finish until WAL unlocked
ASSERT_OK(Put("k1", "val2"));
t1_completed = true;
}};
ASSERT_OK(db_->LockWAL()); // 1 -> 2
// Read-only ops are OK
ASSERT_EQ(Get("k1"), "val");
{
std::vector<LiveFileStorageInfo> files;
LiveFilesStorageInfoOptions lf_opts;
// A DB flush could deadlock
lf_opts.wal_size_for_flush = UINT64_MAX;
ASSERT_OK(db_->GetLiveFilesStorageInfo({lf_opts}, &files));
}
port::Thread t2{[&]() {
ASSERT_OK(db_->LockWAL()); // 2 -> 3 or 1 -> 2
}};
ASSERT_OK(db_->UnlockWAL()); // 2 -> 1 or 3 -> 2
// Give t1 an extra chance to jump in case of bug
std::this_thread::yield();
t2.join();
ASSERT_FALSE(t1_completed.load());
// Should now have 2 outstanding LockWAL
ASSERT_EQ(Get("k1"), "val");
ASSERT_OK(db_->UnlockWAL()); // 2 -> 1
ASSERT_FALSE(t1_completed.load());
ASSERT_EQ(Get("k1"), "val");
ASSERT_EQ(frozen_seqno, db_->GetLatestSequenceNumber());
// Ensure final Unlock is concurrency safe and extra Unlock is safe but
// non-OK
std::atomic<int> unlock_ok{0};
port::Thread t3{[&]() {
if (db_->UnlockWAL().ok()) {
unlock_ok++;
}
ASSERT_OK(db_->LockWAL());
if (db_->UnlockWAL().ok()) {
unlock_ok++;
}
}};
if (db_->UnlockWAL().ok()) {
unlock_ok++;
}
t3.join();
// There was one extra unlock, so just one non-ok
ASSERT_EQ(unlock_ok.load(), 2);
// Write can proceed
t1.join();
ASSERT_TRUE(t1_completed.load());
ASSERT_EQ(Get("k1"), "val2");
// And new writes
ASSERT_OK(Put("k2", "val"));
ASSERT_EQ(Get("k2"), "val");
ASSERT_OK(Put("key" + std::to_string(0), "value"));
ASSERT_TRUE(options.manual_wal_flush != dbfull()->WALBufferIsEmpty());
ASSERT_OK(dbfull()->LockWAL());
ASSERT_TRUE(dbfull()->WALBufferIsEmpty(false));
ASSERT_OK(dbfull()->UnlockWAL());
}
TEST_P(DBWriteTest, ConcurrentlyDisabledWAL) {
+11
View File
@@ -7,6 +7,7 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef ROCKSDB_LITE
#include <stdlib.h>
@@ -601,3 +602,13 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as DBImpl::DeleteFile is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
+27
View File
@@ -228,6 +228,7 @@ std::map<std::tuple<BackgroundErrorReason, bool>, Status::Severity>
};
void ErrorHandler::CancelErrorRecovery() {
#ifndef ROCKSDB_LITE
db_mutex_->AssertHeld();
// We'll release the lock before calling sfm, so make sure no new
@@ -248,6 +249,7 @@ void ErrorHandler::CancelErrorRecovery() {
// If auto recovery is also runing to resume from the retryable error,
// we should wait and end the auto recovery.
EndAutoRecovery();
#endif
}
STATIC_AVOID_DESTRUCTION(const Status, kOkStatus){Status::OK()};
@@ -497,6 +499,7 @@ const Status& ErrorHandler::SetBGError(const Status& bg_status,
Status ErrorHandler::OverrideNoSpaceError(const Status& bg_error,
bool* auto_recovery) {
#ifndef ROCKSDB_LITE
if (bg_error.severity() >= Status::Severity::kFatalError) {
return bg_error;
}
@@ -525,9 +528,14 @@ Status ErrorHandler::OverrideNoSpaceError(const Status& bg_error,
}
return bg_error;
#else
(void)auto_recovery;
return Status(bg_error, Status::Severity::kFatalError);
#endif
}
void ErrorHandler::RecoverFromNoSpace() {
#ifndef ROCKSDB_LITE
SstFileManagerImpl* sfm =
reinterpret_cast<SstFileManagerImpl*>(db_options_.sst_file_manager.get());
@@ -535,9 +543,11 @@ void ErrorHandler::RecoverFromNoSpace() {
if (sfm) {
sfm->StartErrorRecovery(this, bg_error_);
}
#endif
}
Status ErrorHandler::ClearBGError() {
#ifndef ROCKSDB_LITE
db_mutex_->AssertHeld();
// Signal that recovery succeeded
@@ -556,9 +566,13 @@ Status ErrorHandler::ClearBGError() {
bg_error_, db_mutex_);
}
return recovery_error_;
#else
return bg_error_;
#endif
}
Status ErrorHandler::RecoverFromBGError(bool is_manual) {
#ifndef ROCKSDB_LITE
InstrumentedMutexLock l(db_mutex_);
bool no_bg_work_original_flag = soft_error_no_bg_work_;
if (is_manual) {
@@ -611,10 +625,15 @@ Status ErrorHandler::RecoverFromBGError(bool is_manual) {
recovery_in_prog_ = false;
}
return s;
#else
(void)is_manual;
return bg_error_;
#endif
}
const Status& ErrorHandler::StartRecoverFromRetryableBGIOError(
const IOStatus& io_error) {
#ifndef ROCKSDB_LITE
db_mutex_->AssertHeld();
if (bg_error_.ok()) {
return bg_error_;
@@ -648,11 +667,16 @@ const Status& ErrorHandler::StartRecoverFromRetryableBGIOError(
} else {
return bg_error_;
}
#else
(void)io_error;
return bg_error_;
#endif
}
// Automatic recover from Retryable BG IO error. Must be called after db
// mutex is released.
void ErrorHandler::RecoverFromRetryableBGIOError() {
#ifndef ROCKSDB_LITE
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeStart");
InstrumentedMutexLock l(db_mutex_);
if (end_recovery_) {
@@ -760,6 +784,9 @@ void ErrorHandler::RecoverFromRetryableBGIOError() {
ERROR_HANDLER_AUTORESUME_RETRY_COUNT, retry_count);
}
return;
#else
return;
#endif
}
void ErrorHandler::CheckAndSetRecoveryAndBGError(const Status& bg_err) {
+12
View File
@@ -6,13 +6,16 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef ROCKSDB_LITE
#include "db/db_test_util.h"
#include "file/sst_file_manager_impl.h"
#include "port/stack_trace.h"
#include "rocksdb/io_status.h"
#include "rocksdb/sst_file_manager.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "util/random.h"
#include "utilities/fault_injection_env.h"
#include "utilities/fault_injection_fs.h"
@@ -2861,3 +2864,12 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED as Cuckoo table is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+48
View File
@@ -10,11 +10,13 @@
#include "rocksdb/utilities/customizable_util.h"
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
Status EventListener::CreateFromString(const ConfigOptions& config_options,
const std::string& id,
std::shared_ptr<EventListener>* result) {
return LoadSharedObject<EventListener>(config_options, id, nullptr, result);
}
#endif // ROCKSDB_LITE
namespace {
template <class T>
@@ -30,6 +32,7 @@ void EventHelpers::AppendCurrentTime(JSONWriter* jwriter) {
.count();
}
#ifndef ROCKSDB_LITE
void EventHelpers::NotifyTableFileCreationStarted(
const std::vector<std::shared_ptr<EventListener>>& listeners,
const std::string& db_name, const std::string& cf_name,
@@ -47,11 +50,13 @@ void EventHelpers::NotifyTableFileCreationStarted(
listener->OnTableFileCreationStarted(info);
}
}
#endif // !ROCKSDB_LITE
void EventHelpers::NotifyOnBackgroundError(
const std::vector<std::shared_ptr<EventListener>>& listeners,
BackgroundErrorReason reason, Status* bg_error, InstrumentedMutex* db_mutex,
bool* auto_recovery) {
#ifndef ROCKSDB_LITE
if (listeners.empty()) {
return;
}
@@ -66,6 +71,13 @@ void EventHelpers::NotifyOnBackgroundError(
}
}
db_mutex->Lock();
#else
(void)listeners;
(void)reason;
(void)bg_error;
(void)db_mutex;
(void)auto_recovery;
#endif // ROCKSDB_LITE
}
void EventHelpers::LogAndNotifyTableFileCreationFinished(
@@ -167,6 +179,7 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
event_logger->Log(jwriter);
}
#ifndef ROCKSDB_LITE
if (listeners.empty()) {
return;
}
@@ -185,6 +198,13 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
listener->OnTableFileCreated(info);
}
info.status.PermitUncheckedError();
#else
(void)listeners;
(void)db_name;
(void)cf_name;
(void)file_path;
(void)reason;
#endif // !ROCKSDB_LITE
}
void EventHelpers::LogAndNotifyTableFileDeletion(
@@ -206,6 +226,7 @@ void EventHelpers::LogAndNotifyTableFileDeletion(
event_logger->Log(jwriter);
#ifndef ROCKSDB_LITE
if (listeners.empty()) {
return;
}
@@ -218,12 +239,18 @@ void EventHelpers::LogAndNotifyTableFileDeletion(
listener->OnTableFileDeleted(info);
}
info.status.PermitUncheckedError();
#else
(void)file_path;
(void)dbname;
(void)listeners;
#endif // !ROCKSDB_LITE
}
void EventHelpers::NotifyOnErrorRecoveryEnd(
const std::vector<std::shared_ptr<EventListener>>& listeners,
const Status& old_bg_error, const Status& new_bg_error,
InstrumentedMutex* db_mutex) {
#ifndef ROCKSDB_LITE
if (!listeners.empty()) {
db_mutex->AssertHeld();
// release lock while notifying events
@@ -239,8 +266,15 @@ void EventHelpers::NotifyOnErrorRecoveryEnd(
}
db_mutex->Lock();
}
#else
(void)listeners;
(void)old_bg_error;
(void)new_bg_error;
(void)db_mutex;
#endif // ROCKSDB_LITE
}
#ifndef ROCKSDB_LITE
void EventHelpers::NotifyBlobFileCreationStarted(
const std::vector<std::shared_ptr<EventListener>>& listeners,
const std::string& db_name, const std::string& cf_name,
@@ -255,6 +289,7 @@ void EventHelpers::NotifyBlobFileCreationStarted(
listener->OnBlobFileCreationStarted(info);
}
}
#endif // !ROCKSDB_LITE
void EventHelpers::LogAndNotifyBlobFileCreationFinished(
EventLogger* event_logger,
@@ -279,6 +314,7 @@ void EventHelpers::LogAndNotifyBlobFileCreationFinished(
event_logger->Log(jwriter);
}
#ifndef ROCKSDB_LITE
if (listeners.empty()) {
return;
}
@@ -289,6 +325,12 @@ void EventHelpers::LogAndNotifyBlobFileCreationFinished(
listener->OnBlobFileCreated(info);
}
info.status.PermitUncheckedError();
#else
(void)listeners;
(void)db_name;
(void)file_path;
(void)creation_reason;
#endif
}
void EventHelpers::LogAndNotifyBlobFileDeletion(
@@ -310,6 +352,7 @@ void EventHelpers::LogAndNotifyBlobFileDeletion(
jwriter.EndObject();
event_logger->Log(jwriter);
}
#ifndef ROCKSDB_LITE
if (listeners.empty()) {
return;
}
@@ -318,6 +361,11 @@ void EventHelpers::LogAndNotifyBlobFileDeletion(
listener->OnBlobFileDeleted(info);
}
info.status.PermitUncheckedError();
#else
(void)listeners;
(void)dbname;
(void)file_path;
#endif // !ROCKSDB_LITE
}
} // namespace ROCKSDB_NAMESPACE
Loaded 100 of 510 files, more files were not shown because too many files have changed in this diff. Show more