Compare commits

..
28 Commits
Author SHA1 Message Date
Yanqin Jin c509dd3249 Fix the logic of setting read_amp_bytes_per_bit from OPTIONS file (#7680)
Summary:
Instead of using `EncodeFixed32` which always serialize a integer to
little endian, we should use the local machine's endianness when
populating a native data structure during options parsing.
Without this fix, `read_amp_bytes_per_bit` may be populated incorrectly
on big-endian machines.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7680

Test Plan: make check

Reviewed By: pdillinger

Differential Revision: D24999166

Pulled By: riversand963

fbshipit-source-id: dc603cff6e17f8fa32479ce6df93b93082e6b0c4
2020-11-17 10:18:19 -08:00
Yanqin Jin 19c8025386 Bump version and update HISTORY 2020-11-15 15:04:28 -08:00
Yanqin Jin 63462c3a94 Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659)
Summary:
A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and
6.14. The bug will write out 8 bytes to OPTIONS file from the starting
address of BlockBasedTableOptions.read_amp_bytes_per_bit which is
actually a uint32. Consequently, the value of read_amp_bytes_per_bit
written in the OPTIONS file is wrong. From 6.15, RocksDB will
try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32.
To be able to load OPTIONS file generated by affected releases before
the fix, we need to manually parse read_amp_bytes_per_bit with this hack.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7659

Test Plan:
Generate a db with current 6.14.fb (head at https://github.com/facebook/rocksdb/commit/b6db05dbb5364c658c5401a8078d73697bb5f31d). Maybe use db_stress.

Checkout this PR, run
```
 ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only
```
Expect success, and should not see
```
Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184
```

Also
make check

Reviewed By: anand1976

Differential Revision: D24954752

Pulled By: riversand963

fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394
2020-11-15 14:52:54 -08:00
Huisheng Liu 62f110f527 fix read_amp_bytes_per_bit field size (#7651)
Summary:
The field in BlockBasedTableOptions is 4 bytes:
  // Default: 0 (disabled)
  uint32_t read_amp_bytes_per_bit = 0;

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7651

Reviewed By: ltamasi

Differential Revision: D24844994

Pulled By: riversand963

fbshipit-source-id: e2695e55532256ef8996dd6939cad06987a80293
2020-11-11 12:21:58 -08:00
Andrew Kryczka 878b3a41de update HISTORY.md and version.h for 6.11.6 2020-10-12 13:15:15 -07:00
Andrew Kryczka f36f44c5e6 Fix bug in pinned partitioned indexes with some reads bypassing block cache
Backports part of 75d3b6fdf0.
2020-10-12 13:14:45 -07:00
Andrew Kryczka 8d70e53114 Fix bug in pinned partitioned user key indexes
Backports part of 75d3b6fdf0.
2020-10-12 13:14:45 -07:00
Yanqin Jin 5a4b8005a7 Update HISTORY and version 2020-07-23 14:42:25 -07:00
Yanqin Jin 4cd964946a Report corruption on unrecognized value type (#7121)
Summary:
During memtable lookup, an unrecognized value type should be reported as
Status::Corruption.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7121

Test Plan: make check

Reviewed By: cheng-chang

Differential Revision: D22512124

Pulled By: riversand963

fbshipit-source-id: 9b97be7d9b230c5aae9205f96054420e5ea09066
2020-07-23 14:37:30 -07:00
Andrew Kryczka 48bfca38f6 update HISTORY.md and version.h for 6.11.4 2020-07-15 17:54:15 -07:00
Adam Retter c8b9556cb6 Only check for python location once (#7123)
Summary:
This fixes an issue introduced in 0c56fc4 whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6.

The location is now only checked once.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123

Reviewed By: zhichao-cao

Differential Revision: D22532274

Pulled By: ajkr

fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970
2020-07-15 17:51:52 -07:00
Yanqin Jin 30bfa2a44e Report corrupted keys during compaction (#7124)
Summary:
Currently, RocksDB lets compaction to go through even in case of
corrupted keys, the number of which is reported in CompactionJobStats.
However, RocksDB does not check this value. We should let compaction run
in a stricter mode.

Temporarily disable two tests that allow corrupted keys in compaction.
With this PR, the two tests will assert(false) and terminate. Still need
to investigate what is the recommended google-test way of doing it.
Death test (EXPECT_DEATH) in gtest has warnings now.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7124

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D22530722

Pulled By: riversand963

fbshipit-source-id: 6a5a6a992028c6d4f92cb74693c92db462ae4ad6
2020-07-15 17:51:08 -07:00
Andrew Kryczka 3e1eb99ab7 bump version to 6.11.3 2020-07-09 15:51:37 -07:00
Yanqin Jin 3514cf51fa Fix data race to VersionSet::io_status_ (#7034)
Summary:
After https://github.com/facebook/rocksdb/issues/6949 , VersionSet::io_status_ can be concurrently accessed by multiple
threads without lock, causing tsan test to fail. For example, a bg flush thread
resets io_status_ before calling LogAndApply(), while another thread already in
the process of LogAndApply() reads io_status_. This is a bug.

We do not have to reset io_status_ each time we call LogAndApply(). io_status_
is part of the state of VersionSet, and it indicates the outcome of preceding
MANIFEST/CURRENT files IO operations. Its value should be updated only when:

1. MANIFEST/CURRENT files IO fail for the first time.
2. MANIFEST/CURRENT files IO succeed as part of recovering from a prior
   failure without process restart, e.g. calling Resume().

Test Plan (devserver):
COMPILE_WITH_TSAN=1 make check
COMPILE_WITH_TSAN=1 make db_test2
./db_test2 --gtest_filter=DBTest2.CompactionStall
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7034

Reviewed By: zhichao-cao

Differential Revision: D22247137

Pulled By: riversand963

fbshipit-source-id: 77b83e05390f3ee3cd2d96d3fdd6fe4f225e3216
2020-07-09 15:51:01 -07:00
Yanqin Jin dd63f04c83 First step towards handling MANIFEST write error (#6949)
Summary:
This PR provides preliminary support for handling IO error during MANIFEST write.
File write/sync is not guaranteed to be atomic. If we encounter an IOError while writing/syncing to the MANIFEST file, we cannot be sure about the state of the MANIFEST file. The version edits may or may not have reached the file. During cleanup, if we delete the newly-generated SST files referenced by the pending version edit(s), but the version edit(s) actually are persistent in the MANIFEST, then next recovery attempt will process the version edits(s) and then fail since the SST files have already been deleted.
One approach is to truncate the MANIFEST after write/sync error, so that it is safe to delete the SST files. However, file truncation may not be supported on certain file systems. Therefore, we take the following approach.
If an IOError is detected during MANIFEST write/sync, we disable file deletions for the faulty database. Depending on whether the IOError is retryable (set by underlying file system), either RocksDB or application can call `DB::Resume()`, or simply shutdown and restart. During `Resume()`, RocksDB will try to switch to a new MANIFEST and write all existing in-memory version storage in the new file. If this succeeds, then RocksDB may proceed. If all recovery is completed, then file deletions will be re-enabled.
Note that multiple threads can call `LogAndApply()` at the same time, though only one of them will be going through the process MANIFEST write, possibly batching the version edits of other threads. When the leading MANIFEST writer finishes, all of the MANIFEST writing threads in this batch will have the same IOError. They will all call `ErrorHandler::SetBGError()` in which file deletion will be disabled.

Possible future directions:
- Add an `ErrorContext` structure so that it is easier to pass more info to `ErrorHandler`. Currently, as in this example, a new `BackgroundErrorReason` has to be added.

Test plan (dev server):
make check
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6949

Reviewed By: anand1976

Differential Revision: D22026020

Pulled By: riversand963

fbshipit-source-id: f3c68a2ef45d9b505d0d625c7c5e0c88495b91c8
2020-07-09 15:50:33 -07:00
Akanksha Mahajan 115c9113ca Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7096)
Summary:
When format_version is high enough to support user-key and
there are index entries for same user key that spans multiple data
blocks then it changes from user-key mode to internal-key mode. But the
flush policy is not reset to point to Block Builder of internal-keys.
After this switch, no entries are added to user key index partition
result, thus it never triggers flushing the block.

Fix: 1. After adding the entry in sub_builder_index_, if there is a switch
from user-key to internal-key, then flush policy is updated to point to
Block Builder of internal-keys index partition.
2. Set sub_builder_index_->seperator_is_key_plus_seq_ = true if
seperator_is_key_plus_seq_  is set to true so that subsequent partitions
can also use internal key mode.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7096

Test Plan: make check -j64

Reviewed By: ajkr

Differential Revision: D22416598

Pulled By: akankshamahajan15

fbshipit-source-id: 01fc2dc07ea1b32f8fb803995ebe6e9a3fbe67ac
2020-07-09 15:48:54 -07:00
Andrew Kryczka 07e2ca100a bump patch version to 6.11.2 2020-07-07 09:34:16 -07:00
Peter Dillinger 9912ebe1b5 Exclude c_test from buck build opt mode
Summary: Fix a Facebook internal build

Test Plan: buck build @mode/opt :c_test :c_test_bin (was compilation
failure, now "not found")
buck build @mode/dev :c_test :c_test_bin (still passes)
2020-07-07 09:31:44 -07:00
sdong 0fdd019f62 Fix test in buck test (#7076)
Summary:
This is to fix special logic to run tests inside FB.
Buck test is broken after moving to cpp_unittest(). Move c_test back to the previous approach.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7076

Test Plan: Watch the Sandcastle run

Reviewed By: ajkr

Differential Revision: D22370096

fbshipit-source-id: 4a464d0903f2c76ae2de3a8ad373ffc9bedec64c
2020-07-07 09:12:18 -07:00
Levi Tamasi 43107e0a9a Move kNoExpiration to blob_db.h (#7018)
Summary:
The constant `kNoExpiration` is currently defined in an
internal/implementation header (`blob_log_format.h`); the patch moves it
to the public header `blob_db.h` so it is accessible to users.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7018

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D22191354

Pulled By: ltamasi

fbshipit-source-id: 98c8012a83b999a3f1a30e955ce6bb71ba29dc5c
2020-06-23 14:13:48 -07:00
Andrew Kryczka c0a8d89c27 add missing release note 2020-06-23 13:45:02 -07:00
Andrew Kryczka 85189fd64f skip 6.11.0 and go straight to 6.11.1 for clarity of HISTORY.md 2020-06-23 13:04:41 -07:00
sdong 1910560c2c Fix a bug that causes iterator to return wrong result in a rare data race (#6973)
Summary:
The bug fixed in https://github.com/facebook/rocksdb/pull/1816/ is now applicable to iterator too. This was not an issue but https://github.com/facebook/rocksdb/pull/2886 caused the regression. If a put and DB flush happens just between iterator to get latest sequence number and getting super version, empty result for the key or an older value can be returned, which is wrong.
Fix it in the same way as the fix in https://github.com/facebook/rocksdb/issues/1816, that is to get the sequence number after referencing the super version.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6973

Test Plan: Will run stress tests for a while to make sure there is no general regression.

Reviewed By: ajkr

Differential Revision: D22029348

fbshipit-source-id: 94390f93630906796d6e2fec321f44a920953fd1
2020-06-18 10:43:00 -07:00
Yanqin Jin 61cc9ef76f Fail recovery when MANIFEST record checksum mismatch (#6996)
Summary:
https://github.com/facebook/rocksdb/issues/5411 refactored `VersionSet::Recover` but introduced a bug, explained as follows.
Before, once a checksum mismatch happens, `reporter` will set `s` to be non-ok. Therefore, Recover will stop processing the MANIFEST any further.
```
// Correct
// Inside Recover
LogReporter reporter;
reporter.status = &s;
log::Reader reader(..., reporter);
while (reader.ReadRecord() && s.ok()) {
...
}
```
The bug is that, the local variable `s` in `ReadAndRecover` won't be updated by `reporter` while reading the MANIFEST. It is possible that the reader sees a checksum mismatch in a record, but `ReadRecord` retries internally read and finds the next valid record. The mismatched record will be ignored and no error is reported.
```
// Incorrect
// Inside Recover
LogReporter reporter;
reporter.status = &s;
log::Reader reader(..., reporter);
s = ReadAndRecover(reader, ...);

// Inside ReadAndRecover
  Status s;  // Shadows the s in Recover.
  while (reader.ReadRecord() && s.ok()) {
   ...
  }
```
`LogReporter` can use a separate `log_read_status` to track the errors while reading the MANIFEST. RocksDB can process more MANIFEST entries only if `log_read_status.ok()`.
Test plan (devserver):
make check
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6996

Reviewed By: ajkr

Differential Revision: D22105746

Pulled By: riversand963

fbshipit-source-id: b22f717a423457a41ca152a242abbb64cf91fc38
2020-06-18 10:38:16 -07:00
sdong f5d4dbbeef Fix the bug that compressed cache is disabled in read-only DBs (#6990)
Summary:
Compressed block cache is disabled in https://github.com/facebook/rocksdb/pull/4650 for no good reason. Re-enable it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6990

Test Plan: Add a unit test to make sure a general function works with read-only DB + compressed block cache.

Reviewed By: ltamasi

Differential Revision: D22072755

fbshipit-source-id: 2a55df6363de23a78979cf6c747526359e5dc7a1
2020-06-18 10:30:53 -07:00
Andrew Kryczka 84dcfe1a5f update minor version for 6.11 release (#6994)
Summary:
The 6.11.fb branch is already cut so I will also backport this PR to
that branch.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6994

Reviewed By: riversand963

Differential Revision: D22084532

Pulled By: ajkr

fbshipit-source-id: 0b025f738cc31c65c673cbf89302359e88a34d19
2020-06-17 09:41:11 -07:00
Yanqin Jin a818699f2f Fix a bug of overwriting return code (#6989)
Summary:
In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6989

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D22071418

Pulled By: riversand963

fbshipit-source-id: 5a4ea5dfb1a41f41c7a3fdaf62b163007b42f04b
2020-06-16 14:00:15 -07:00
Yanqin Jin 97a69f4372 Let best-efforts recovery ignore CURRENT file (#6970)
Summary:
Best-efforts recovery does not check the content of CURRENT file to determine which MANIFEST to recover from. However, it still checks the presence of CURRENT file to determine whether to create a new DB during `open()`. Therefore, we can tweak the logic in `open()` a little bit so that best-efforts recovery does not rely on CURRENT file at all.

Test plan (dev server):
make check
./db_basic_test --gtest_filter=DBBasicTest.RecoverWithNoCurrentFile
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6970

Reviewed By: anand1976

Differential Revision: D22013990

Pulled By: riversand963

fbshipit-source-id: db552a1868c60ed70e1f7cd252a3a076eb8ea58f
2020-06-16 09:35:52 -07:00
339 changed files with 5478 additions and 15073 deletions

No files matched your search

-54
View File
@@ -1,54 +0,0 @@
#! /bin/bash
# Work around issue with parallel make output causing random error, as in
# make[1]: write error: stdout
# Probably due to a kernel bug:
# https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393
# Seems to affect image ubuntu-1604:201903-01 and ubuntu-1604:202004-01
cd "$(dirname $0)"
if [ ! -x cat_ignore_eagain.out ]; then
cc -x c -o cat_ignore_eagain.out - << EOF
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
int main() {
int n, m, p;
char buf[1024];
for (;;) {
n = read(STDIN_FILENO, buf, 1024);
if (n > 0 && n <= 1024) {
for (m = 0; m < n;) {
p = write(STDOUT_FILENO, buf + m, n - m);
if (p < 0) {
if (errno == EAGAIN) {
// ignore but pause a bit
usleep(100);
} else {
perror("write failed");
return 42;
}
} else {
m += p;
}
}
} else if (n < 0) {
if (errno == EAGAIN) {
// ignore but pause a bit
usleep(100);
} else {
// Some non-ignorable error
perror("read failed");
return 43;
}
} else {
// EOF
return 0;
}
}
}
EOF
fi
exec ./cat_ignore_eagain.out
+13 -199
View File
@@ -18,22 +18,9 @@ jobs:
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
build-linux-shared_lib-alt_namespace-status_checked:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 ASSERT_STATUS_CHECKED=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 all check_some | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
- run: sudo apt-get update -y
- run: sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 make J=32 all check -j32
build-linux-release:
machine:
@@ -41,23 +28,16 @@ jobs:
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- run: make release -j32
build-linux-lite:
machine:
image: ubuntu-1604:201903-01
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 LITE=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
- store_test_results:
path: /tmp/test-results
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 LITE=1 make J=32 all check -j32
build-linux-lite-release:
machine:
@@ -65,11 +45,7 @@ jobs:
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- run: make release -j32
build-linux-clang-no-test:
machine:
@@ -77,71 +53,7 @@ jobs:
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: sudo apt-get update -y && sudo apt-get install -y clang libgflags-dev
- run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make V=1 -j32 all | .circleci/cat_ignore_eagain
build-linux-clang10-no-test:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 all | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
build-linux-clang10-asan:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_ASAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
- store_test_results:
path: /tmp/test-results
build-linux-clang10-mini-tsan:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_TSAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 EXCLUDE_TESTS_REGEX="TransactionStressTest|SnapshotConcurrentAccess|SeqAdvanceConcurrent|DeadlockStress|MultiThreadedDBTest.MultiThreaded|WriteUnpreparedStressTest.ReadYourOwnWriteStress|DBAsBaseDB/TransactionStressTest|FlushCloseWALFiles|BackgroundPurgeCFDropTest" GTEST_THROW_ON_FAILURE=0 GTEST_OUTPUT="xml:/tmp/test-results/" make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. Exclude FlushCloseWALFiles and BackgroundPurgeCFDropTest for occasional failures.
- store_test_results:
path: /tmp/test-results
build-linux-clang10-ubsan:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_UBSAN=1 OPT="-fsanitize-blacklist=.circleci/ubsan_suppression_list.txt" CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 -j32 ubsan_check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
build-linux-clang10-clang-analyze:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev clang-tools-10
- run: SKIP_FORMAT_BUCK_CHECKS=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 CLANG_ANALYZER="/usr/bin/clang++-10" CLANG_SCAN_BUILD=scan-build-10 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 -j32 analyze | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. For unknown, reason passing "clang++-10" as CLANG_ANALYZER doesn't work, and we need a full path.
- run: USE_CLANG=1 make all -j32
build-linux-cmake:
machine:
@@ -149,48 +61,22 @@ jobs:
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: (mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make V=1 -j32) | .circleci/cat_ignore_eagain
build-linux-unity:
docker: # executor type
- image: gcc:latest
resource_class: xlarge
steps:
- checkout # check out the code in the project directory
- run: apt-get update -y && apt-get install -y libgflags-dev
- run: TEST_TMPDIR=/dev/shm && make V=1 -j16 unity_test | .circleci/cat_ignore_eagain
- run: mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make -j32
build-windows:
executor: windows-2xlarge
parameters:
extra_cmake_opt:
default: ""
type: string
vs_year:
default: "2019"
type: string
cmake_generator:
default: "Visual Studio 16 2019"
type: string
environment:
THIRDPARTY_HOME: C:/Users/circleci/thirdparty
CMAKE_HOME: C:/Users/circleci/thirdparty/cmake-3.16.4-win64-x64
CMAKE_BIN: C:/Users/circleci/thirdparty/cmake-3.16.4-win64-x64/bin/cmake.exe
CMAKE_GENERATOR: Visual Studio 16 2019
SNAPPY_HOME: C:/Users/circleci/thirdparty/snappy-1.1.7
SNAPPY_INCLUDE: C:/Users/circleci/thirdparty/snappy-1.1.7;C:/Users/circleci/thirdparty/snappy-1.1.7/build
SNAPPY_LIB_DEBUG: C:/Users/circleci/thirdparty/snappy-1.1.7/build/Debug/snappy.lib
VS_YEAR: <<parameters.vs_year>>
CMAKE_GENERATOR: <<parameters.cmake_generator>>
steps:
- checkout
- run:
name: "Setup VS"
command: |
if [[ "${VS_YEAR}" == "2017" ]]; then
powershell .circleci/vs2017_install.ps1
elif [[ "${VS_YEAR}" == "2015" ]]; then
powershell .circleci/vs2015_install.ps1
fi
- run:
name: "Install thirdparty dependencies"
command: |
@@ -212,7 +98,7 @@ jobs:
command: |
mkdir build
cd build
${CMAKE_BIN} -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DJNI=1 << parameters.extra_cmake_opt >> ..
${CMAKE_BIN} -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DJNI=1 ..
cd ..
msbuild.exe build/rocksdb.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
- run:
@@ -221,44 +107,10 @@ jobs:
command: |
build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test,db_test2,env_basic_test,env_test,db_merge_operand_test -Concurrency 16
build-linux-java:
machine:
image: ubuntu-1604:201903-01
resource_class: 2xlarge
steps:
- checkout
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run:
name: "Build RocksDBJava"
command: |
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0
export PATH=$JAVA_HOME/bin:$PATH
echo "JAVA_HOME=${JAVA_HOME}"
which java && java -version
which javac && javac -version
SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 J=32 -j32 rocksdbjava jtest | .circleci/cat_ignore_eagain
build-examples:
machine:
image: ubuntu-1604:201903-01
resource_class: medium
steps:
- checkout # check out the code in the project directory
- run: pyenv global 3.5.2
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run:
name: "Build examples"
command: |
OPT=-DTRAVIS V=1 make -j4 static_lib && cd examples && make -j4 | ../.circleci/cat_ignore_eagain
workflows:
build-linux:
jobs:
- build-linux
build-linux-shared_lib-alt_namespace-status_checked:
jobs:
- build-linux-shared_lib-alt_namespace-status_checked
build-linux-lite:
jobs:
- build-linux-lite
@@ -271,47 +123,9 @@ workflows:
build-linux-clang-no-test:
jobs:
- build-linux-clang-no-test
build-linux-clang10-no-test:
jobs:
- build-linux-clang10-no-test
build-linux-clang10-asan:
jobs:
- build-linux-clang10-asan
build-linux-clang10-mini-tsan:
jobs:
- build-linux-clang10-mini-tsan
build-linux-clang10-ubsan:
jobs:
- build-linux-clang10-ubsan
build-linux-clang10-clang-analyze:
jobs:
- build-linux-clang10-clang-analyze
build-linux-cmake:
jobs:
- build-linux-cmake
build-linux-unity:
jobs:
- build-linux-unity
build-windows:
jobs:
- build-windows
build-windows-vs2017:
jobs:
- build-windows:
vs_year: "2017"
cmake_generator: "Visual Studio 15 Win64"
build-windows-vs2015:
jobs:
- build-windows:
vs_year: "2015"
cmake_generator: "Visual Studio 14 Win64"
build-windows-cxx20:
jobs:
- build-windows:
extra_cmake_opt: -DCMAKE_CXX_STANDARD=20
build-java:
jobs:
- build-linux-java
build-examples:
jobs:
- build-examples
-6
View File
@@ -1,6 +0,0 @@
# Supress UBSAN warnings related to stl_tree.h, e.g.
# UndefinedBehaviorSanitizer: undefined-behavior /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:1505:43 in
# /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_tree.h:1505:43:
# runtime error: upcast of address 0x000001fa8820 with insufficient space for an object of type
# 'std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, rocksdb::(anonymous namespace)::LockHoldingInfo> >'
src:*bits/stl_tree.h
-23
View File
@@ -1,23 +0,0 @@
$VS_DOWNLOAD_LINK = "https://go.microsoft.com/fwlink/?LinkId=691126"
$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe"
curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe
if ($LASTEXITCODE -ne 0) {
echo "Download of the VS 2015 installer failed"
exit 1
}
$VS_INSTALL_ARGS = @("/Quiet", "/NoRestart")
$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru
Remove-Item -Path vs_installer.exe -Force
$exitCode = $process.ExitCode
if (($exitCode -ne 0) -and ($exitCode -ne 3010)) {
echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]."
curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe
if ($LASTEXITCODE -ne 0) {
echo "Download of the VS Collect tool failed."
exit 1
}
Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru
New-Item -Path "C:\w\build-results" -ItemType "directory" -Force
Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\"
exit 1
}
-34
View File
@@ -1,34 +0,0 @@
$VS_DOWNLOAD_LINK = "https://aka.ms/vs/15/release/vs_buildtools.exe"
$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe"
$VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools",
"--add Microsoft.VisualStudio.Component.VC.Tools.14.13",
"--add Microsoft.Component.MSBuild",
"--add Microsoft.VisualStudio.Component.Roslyn.Compiler",
"--add Microsoft.VisualStudio.Component.TextTemplating",
"--add Microsoft.VisualStudio.Component.VC.CoreIde",
"--add Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
"--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
"--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
"--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81")
curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe
if ($LASTEXITCODE -ne 0) {
echo "Download of the VS 2017 installer failed"
exit 1
}
$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru
Remove-Item -Path vs_installer.exe -Force
$exitCode = $process.ExitCode
if (($exitCode -ne 0) -and ($exitCode -ne 3010)) {
echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]."
curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe
if ($LASTEXITCODE -ne 0) {
echo "Download of the VS Collect tool failed."
exit 1
}
Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru
New-Item -Path "C:\w\build-results" -ItemType "directory" -Force
Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\"
exit 1
}
+8 -3
View File
@@ -239,8 +239,13 @@ install:
CC=gcc-4.8 && CXX=g++-4.8;
fi
- if [[ "${JOB_NAME}" == cmake* ]] && [ "${TRAVIS_OS_NAME}" == linux ]; then
sudo snap install cmake --beta --classic;
export PATH=/snap/bin:$PATH;
CMAKE_DIST_URL="https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-$(uname -m).tar.bz2";
TAR_OPT="--strip-components=1 -xj";
if [ "aarch64" == "$(uname -m)" ]; then
sudo apt-get install -y libuv1 librhash0;
sudo apt-get upgrade -y libstdc++6;
fi;
mkdir cmake-dist && curl --silent --fail --show-error --location "${CMAKE_DIST_URL}" | tar -C cmake-dist ${TAR_OPT} && export PATH=$PWD/cmake-dist/bin:$PATH;
fi
- |
if [[ "${JOB_NAME}" == java_test || "${JOB_NAME}" == cmake* ]]; then
@@ -278,7 +283,7 @@ script:
OPT="-DTRAVIS -DROCKSDB_NAMESPACE=alternative_rocksdb_ns" V=1 make -j4 tools && OPT="-DTRAVIS -DROCKSDB_NAMESPACE=alternative_rocksdb_ns" V=1 ROCKSDBTESTS_START=db_iter_test ROCKSDBTESTS_END=options_file_test make -j4 check_some
;;
3)
OPT=-DTRAVIS LIB_MODE=shared V=1 ROCKSDBTESTS_START=options_file_test ROCKSDBTESTS_END=write_prepared_transaction_test make -j4 check_some
OPT=-DTRAVIS V=1 ROCKSDBTESTS_START=options_file_test ROCKSDBTESTS_END=write_prepared_transaction_test make -j4 check_some
;;
4)
OPT=-DTRAVIS V=1 ROCKSDBTESTS_START=write_prepared_transaction_test make -j4 check_some
+19 -44
View File
@@ -95,7 +95,7 @@ if(MSVC)
option(WITH_XPRESS "build with windows built in compression" OFF)
include(${CMAKE_CURRENT_SOURCE_DIR}/thirdparty.inc)
else()
if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD" AND NOT CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
# FreeBSD has jemalloc as default malloc
# but it does not have all the jemalloc files in include/...
set(WITH_JEMALLOC ON)
@@ -107,7 +107,6 @@ else()
endif()
endif()
set(GFLAGS_LIB)
if(WITH_GFLAGS)
# Config with namespace available since gflags 2.2.2
option(GFLAGS_USE_TARGET_NAMESPACE "Use gflags import target with namespace." ON)
@@ -115,17 +114,15 @@ else()
if(gflags_FOUND)
if(TARGET ${GFLAGS_TARGET})
# Config with GFLAGS_TARGET available since gflags 2.2.0
set(GFLAGS_LIB ${GFLAGS_TARGET})
list(APPEND THIRDPARTY_LIBS ${GFLAGS_TARGET})
else()
# Config with GFLAGS_LIBRARIES available since gflags 2.1.0
set(GFLAGS_LIB ${GFLAGS_LIBRARIES})
list(APPEND THIRDPARTY_LIBS ${GFLAGS_LIBRARIES})
endif()
else()
find_package(gflags REQUIRED)
set(GFLAGS_LIB gflags::gflags)
list(APPEND THIRDPARTY_LIBS gflags::gflags)
endif()
include_directories(${GFLAGS_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS ${GFLAGS_LIB})
add_definitions(-DGFLAGS=1)
endif()
@@ -451,8 +448,6 @@ elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
add_definitions(-DOS_LINUX)
elseif(CMAKE_SYSTEM_NAME MATCHES "SunOS")
add_definitions(-DOS_SOLARIS)
elseif(CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
add_definitions(-DOS_GNU_KFREEBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
add_definitions(-DOS_FREEBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "NetBSD")
@@ -511,11 +506,7 @@ if(HAVE_PTHREAD_MUTEX_ADAPTIVE_NP)
endif()
include(CheckCXXSymbolExists)
if(CMAKE_SYSTEM_NAME MATCHES "^FreeBSD")
check_cxx_symbol_exists(malloc_usable_size malloc_np.h HAVE_MALLOC_USABLE_SIZE)
else()
check_cxx_symbol_exists(malloc_usable_size malloc.h HAVE_MALLOC_USABLE_SIZE)
endif()
check_cxx_symbol_exists(malloc_usable_size malloc.h HAVE_MALLOC_USABLE_SIZE)
if(HAVE_MALLOC_USABLE_SIZE)
add_definitions(-DROCKSDB_MALLOC_USABLE_SIZE)
endif()
@@ -562,7 +553,6 @@ set(SOURCES
db/compaction/compaction_picker_fifo.cc
db/compaction/compaction_picker_level.cc
db/compaction/compaction_picker_universal.cc
db/compaction/sst_partitioner.cc
db/convenience.cc
db/db_filesnapshot.cc
db/db_impl/db_impl.cc
@@ -702,7 +692,6 @@ set(SOURCES
table/plain/plain_table_index.cc
table/plain/plain_table_key_coding.cc
table/plain/plain_table_reader.cc
table/sst_file_dumper.cc
table/sst_file_reader.cc
table/sst_file_writer.cc
table/table_properties.cc
@@ -752,8 +741,6 @@ set(SOURCES
utilities/debug.cc
utilities/env_mirror.cc
utilities/env_timed.cc
utilities/fault_injection_env.cc
utilities/fault_injection_fs.cc
utilities/leveldb_options/leveldb_options.cc
utilities/memory/memory_util.cc
utilities/merge_operators/bytesxor.cc
@@ -987,22 +974,12 @@ if(NOT WIN32 OR ROCKSDB_INSTALL_ON_WINDOWS)
)
endif()
option(WITH_ALL_TESTS "Build all test, rather than a small subset" ON)
if(WITH_TESTS OR WITH_BENCHMARK_TOOLS)
if(WITH_TESTS)
add_subdirectory(third-party/gtest-1.8.1/fused-src/gtest)
add_library(testharness STATIC
test_util/testharness.cc)
target_link_libraries(testharness gtest)
endif()
if(WITH_TESTS)
set(TESTS
db/db_basic_test.cc
env/env_basic_test.cc
)
if(WITH_ALL_TESTS)
list(APPEND TESTS
cache/cache_test.cc
cache/lru_cache_test.cc
db/blob/blob_file_addition_test.cc
@@ -1017,6 +994,7 @@ if(WITH_TESTS)
db/comparator_db_test.cc
db/corruption_test.cc
db/cuckoo_table_db_test.cc
db/db_basic_test.cc
db/db_with_timestamp_basic_test.cc
db/db_block_cache_test.cc
db/db_bloom_filter_test.cc
@@ -1079,6 +1057,7 @@ if(WITH_TESTS)
db/write_batch_test.cc
db/write_callback_test.cc
db/write_controller_test.cc
env/env_basic_test.cc
env/env_test.cc
env/io_posix_test.cc
env/mock_env_test.cc
@@ -1161,8 +1140,7 @@ if(WITH_TESTS)
utilities/transactions/write_unprepared_transaction_test.cc
utilities/ttl/ttl_test.cc
utilities/write_batch_with_index/write_batch_with_index_test.cc
)
endif()
)
if(WITH_LIBRADOS)
list(APPEND TESTS utilities/env_librados_test.cc)
endif()
@@ -1175,6 +1153,8 @@ if(WITH_TESTS)
db/db_test_util.cc
monitoring/thread_status_updater_debug.cc
table/mock_table.cc
test_util/fault_injection_test_env.cc
test_util/fault_injection_test_fs.cc
utilities/cassandra/test_utils.cc
)
enable_testing()
@@ -1200,7 +1180,7 @@ if(WITH_TESTS)
EXCLUDE_FROM_DEFAULT_BUILD_RELWITHDEBINFO 1
OUTPUT_NAME ${exename}${ARTIFACT_SUFFIX}
)
target_link_libraries(${CMAKE_PROJECT_NAME}_${exename}${ARTIFACT_SUFFIX} testutillib${ARTIFACT_SUFFIX} testharness gtest ${GFLAGS_LIB} ${ROCKSDB_LIB})
target_link_libraries(${CMAKE_PROJECT_NAME}_${exename}${ARTIFACT_SUFFIX} testutillib${ARTIFACT_SUFFIX} testharness gtest ${ROCKSDB_LIB})
if(NOT "${exename}" MATCHES "db_sanity_test")
add_test(NAME ${exename} COMMAND ${exename}${ARTIFACT_SUFFIX})
add_dependencies(check ${CMAKE_PROJECT_NAME}_${exename}${ARTIFACT_SUFFIX})
@@ -1237,37 +1217,37 @@ if(WITH_BENCHMARK_TOOLS)
tools/db_bench.cc
tools/db_bench_tool.cc)
target_link_libraries(db_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
add_executable(cache_bench
cache/cache_bench.cc)
target_link_libraries(cache_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
add_executable(memtablerep_bench
memtable/memtablerep_bench.cc)
target_link_libraries(memtablerep_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
add_executable(range_del_aggregator_bench
db/range_del_aggregator_bench.cc)
target_link_libraries(range_del_aggregator_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
add_executable(table_reader_bench
table/table_reader_bench.cc)
target_link_libraries(table_reader_bench
${ROCKSDB_LIB} testharness ${GFLAGS_LIB})
${ROCKSDB_LIB} testharness)
add_executable(filter_bench
util/filter_bench.cc)
target_link_libraries(filter_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
add_executable(hash_table_bench
utilities/persistent_cache/hash_table_bench.cc)
target_link_libraries(hash_table_bench
${ROCKSDB_LIB} ${GFLAGS_LIB})
${ROCKSDB_LIB})
endif()
if(WITH_CORE_TOOLS OR WITH_TOOLS)
@@ -1281,8 +1261,3 @@ if(WITH_TOOLS)
add_custom_target(tools
DEPENDS ${tool_deps})
endif()
option(WITH_EXAMPLES "build with examples" OFF)
if(WITH_EXAMPLES)
add_subdirectory(examples)
endif()
+17 -58
View File
@@ -1,77 +1,37 @@
# Rocksdb Change Log
## 6.12.8 (2020-11-15)
## 6.11.7 (11/15/2020)
### Bug Fixes
* Fix a bug of encoding and parsing BlockBasedTableOptions::read_amp_bytes_per_bit as a 64-bit integer.
* Fixed the logic of populating native data structure for `read_amp_bytes_per_bit` during OPTIONS file parsing on big-endian architecture. Without this fix, original code introduced in PR7659, when running on big-endian machine, can mistakenly store read_amp_bytes_per_bit (an uint32) in little endian format. Future access to `read_amp_bytes_per_bit` will give wrong values. Little endian architecture is not affected.
## 6.12.7 (2020-10-14)
### Other
Fix build issue to enable RocksJava release for ppc64le
## 6.12.6 (2020-10-13)
## 6.11.6 (10/12/2020)
### Bug Fixes
* Fix false positive flush/compaction `Status::Corruption` failure when `paranoid_file_checks == true` and range tombstones were written to the compaction output files.
## 6.12.5 (2020-10-12)
### Bug Fixes
* Since 6.12, memtable lookup should report unrecognized value_type as corruption (#7121).
* Fixed a bug in the following combination of features: indexes with user keys (`format_version >= 3`), indexes are partitioned (`index_type == kTwoLevelIndexSearch`), and some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`). The bug could cause keys to be truncated when read from the index leading to wrong read results or other unexpected behavior.
* Fixed a bug when indexes are partitioned (`index_type == kTwoLevelIndexSearch`), some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`), and partitions reads could be mixed between block cache and directly from the file (e.g., with `enable_index_compression == 1` and `mmap_read == 1`, partitions that were stored uncompressed due to poor compression ratio would be read directly from the file via mmap, while partitions that were stored compressed would be read from block cache). The bug could cause index partitions to be mistakenly considered empty during reads leading to wrong read results.
## 6.12.4 (2020-09-18)
### Public API Change
* Reworked `BackupableDBOptions::share_files_with_checksum_naming` (new in 6.12) with some minor improvements and to better support those who were extracting files sizes from backup file names.
## 6.11.5 (7/23/2020)
### Bug Fixes
* Memtable lookup should report unrecognized value_type as corruption (#7121).
## 6.12.3 (2020-09-16)
### Bug fixes
* Fixed a bug in size-amp-triggered and periodic-triggered universal compaction, where the compression settings for the first input level were used rather than the compression settings for the output (bottom) level.
## 6.11.4 (7/15/2020)
### Bug Fixes
* Make compaction report InternalKey corruption while iterating over the input.
## 6.12.2 (2020-09-14)
### Public API Change
* BlobDB now exposes the start of the expiration range of TTL blob files via the `GetLiveFilesMetaData` API.
## 6.11.3 (7/9/2020)
### Bug Fixes
* Fix a bug when index_type == kTwoLevelIndexSearch in PartitionedIndexBuilder to update FlushPolicy to point to internal key partitioner when it changes from user-key mode to internal-key mode in index partition.
* Disable file deletion after MANIFEST write/sync failure until db re-open or Resume() so that subsequent re-open will not see MANIFEST referencing deleted SSTs.
## 6.12.1 (2020-08-20)
### Bug fixes
* BackupEngine::CreateNewBackup could fail intermittently with non-OK status when backing up a read-write DB configured with a DBOptions::file_checksum_gen_factory. This issue has been worked-around such that CreateNewBackup should succeed, but (until fully fixed) BackupEngine might not see all checksums available in the DB.
## 6.12 (2020-07-28)
### Public API Change
* Encryption file classes now exposed for inheritance in env_encryption.h
* File I/O listener is extended to cover more I/O operations. Now class `EventListener` in listener.h contains new callback functions: `OnFileFlushFinish()`, `OnFileSyncFinish()`, `OnFileRangeSyncFinish()`, `OnFileTruncateFinish()`, and ``OnFileCloseFinish()``.
* `FileOperationInfo` now reports `duration` measured by `std::chrono::steady_clock` and `start_ts` measured by `std::chrono::system_clock` instead of start and finish timestamps measured by `system_clock`. Note that `system_clock` is called before `steady_clock` in program order at operation starts.
* `DB::GetDbSessionId(std::string& session_id)` is added. `session_id` stores a unique identifier that gets reset every time the DB is opened. This DB session ID should be unique among all open DB instances on all hosts, and should be unique among re-openings of the same or other DBs. This identifier is recorded in the LOG file on the line starting with "DB Session ID:".
* `DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`. This change is available in all 6.11 releases as well.
* A parameter `verify_with_checksum` is added to `BackupEngine::VerifyBackup`, which is false by default. If it is ture, `BackupEngine::VerifyBackup` verifies checksums and file sizes of backup files. Pass `false` for `verify_with_checksum` to maintain the previous behavior and performance of `BackupEngine::VerifyBackup`, by only verifying sizes of backup files.
### Behavior Changes
## 6.11.1 (6/23/2020)
### Bug Fixes
* Best-efforts recovery ignores CURRENT file completely. If CURRENT file is missing during recovery, best-efforts recovery still proceeds with MANIFEST file(s).
* In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early.
* When `file_checksum_gen_factory` is set to `GetFileChecksumGenCrc32cFactory()`, BackupEngine will compare the crc32c checksums of table files computed when creating a backup to the expected checksums stored in the DB manifest, and will fail `CreateNewBackup()` on mismatch (corruption). If the `file_checksum_gen_factory` is not set or set to any other customized factory, there is no checksum verification to detect if SST files in a DB are corrupt when read, copied, and independently checksummed by BackupEngine.
* When a DB sets `stats_dump_period_sec > 0`, either as the initial value for DB open or as a dynamic option change, the first stats dump is staggered in the following X seconds, where X is an integer in `[0, stats_dump_period_sec)`. Subsequent stats dumps are still spaced `stats_dump_period_sec` seconds apart.
* When the paranoid_file_checks option is true, a hash is generated of all keys and values are generated when the SST file is written, and then the values are read back in to validate the file. A corruption is signaled if the two hashes do not match.
### Bug fixes
* Compressed block cache was automatically disabled with read-only DBs by mistake. Now it is fixed: compressed block cache will be in effective with read-only DB too.
* Fix a bug of wrong iterator result if another thread finishes an update and a DB flush between two statement.
* Disable file deletion after MANIFEST write/sync failure until db re-open or Resume() so that subsequent re-open will not see MANIFEST referencing deleted SSTs.
* Fix a bug when index_type == kTwoLevelIndexSearch in PartitionedIndexBuilder to update FlushPolicy to point to internal key partitioner when it changes from user-key mode to internal-key mode in index partition.
* Make compaction report InternalKey corruption while iterating over the input.
* Fix a bug which may cause MultiGet to be slow because it may read more data than requested, but this won't affect correctness. The bug was introduced in 6.10 release.
* Fail recovery and report once hitting a physical log record checksum mismatch, while reading MANIFEST. RocksDB should not continue processing the MANIFEST any further.
* Fix a bug of wrong iterator result if another thread finishes an update and a DB flush between two statement.
### New Features
* DB identity (`db_id`) and DB session identity (`db_session_id`) are added to table properties and stored in SST files. SST files generated from SstFileWriter and Repairer have DB identity “SST Writer” and “DB Repairer”, respectively. Their DB session IDs are generated in the same way as `DB::GetDbSessionId`. The session ID for SstFileWriter (resp., Repairer) resets every time `SstFileWriter::Open` (resp., `Repairer::Run`) is called.
* Added experimental option BlockBasedTableOptions::optimize_filters_for_memory for reducing allocated memory size of Bloom filters (~10% savings with Jemalloc) while preserving the same general accuracy. To have an effect, the option requires format_version=5 and malloc_usable_size. Enabling this option is forward and backward compatible with existing format_version=5.
* `BackupableDBOptions::share_files_with_checksum_naming` is added with new default behavior for naming backup files with `share_files_with_checksum`, to address performance and backup integrity issues. See API comments for details.
* Added auto resume function to automatically recover the DB from background Retryable IO Error. When retryable IOError happens during flush and WAL write, the error is mapped to Hard Error and DB will be in read mode. When retryable IO Error happens during compaction, the error will be mapped to Soft Error. DB is still in write/read mode. Autoresume function will create a thread for a DB to call DB->ResumeImpl() to try the recover for Retryable IO Error during flush and WAL write. Compaction will be rescheduled by itself if retryable IO Error happens. Auto resume may also cause other Retryable IO Error during the recovery, so the recovery will fail. Retry the auto resume may solve the issue, so we use max_bgerror_resume_count to decide how many resume cycles will be tried in total. If it is <=0, auto resume retryable IO Error is disabled. Default is INT_MAX, which will lead to a infinit auto resume. bgerror_resume_retry_interval decides the time interval between two auto resumes.
* Option `max_subcompactions` can be set dynamically using DB::SetDBOptions().
* Added experimental ColumnFamilyOptions::sst_partitioner_factory to define determine the partitioning of sst files. This helps compaction to split the files on interesting boundaries (key prefixes) to make propagation of sst files less write amplifying (covering the whole key space).
### Performance Improvements
* Eliminate key copies for internal comparisons while accessing ingested block-based tables.
* Reduce key comparisons during random access in all block-based tables.
* BackupEngine avoids unnecessary repeated checksum computation for backing up a table file to the `shared_checksum` directory when using `share_files_with_checksum_naming = kUseDbSessionId` (new default), except on SST files generated before this version of RocksDB, which fall back on using `kLegacyCrc32cAndFileSize`.
### Public API Change
* `DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`.
## 6.11 (6/12/2020)
### Bug Fixes
@@ -97,7 +57,6 @@ Fix build issue to enable RocksJava release for ppc64le
* Disable delete triggered compaction (NewCompactOnDeletionCollectorFactory) in universal compaction mode and num_levels = 1 in order to avoid a corruption bug.
* `pin_l0_filter_and_index_blocks_in_cache` no longer applies to L0 files larger than `1.5 * write_buffer_size` to give more predictable memory usage. Such L0 files may exist due to intra-L0 compaction, external file ingestion, or user dynamically changing `write_buffer_size` (note, however, that files that are already pinned will continue being pinned, even after such a dynamic change).
* In point-in-time wal recovery mode, fail database recovery in case of IOError while reading the WAL to avoid data loss.
* A new method `Env::LowerThreadPoolCPUPriority(Priority, CpuPriority)` is added to `Env` to be able to lower to a specific priority such as `CpuPriority::kIdle`.
### New Features
* sst_dump to add a new --readahead_size argument. Users can specify read size when scanning the data. Sst_dump also tries to prefetch tail part of the SST files so usually some number of I/Os are saved there too.
+1 -3
View File
@@ -10,9 +10,7 @@ This is the list of all known third-party language bindings for RocksDB. If some
* Ruby - http://rubygems.org/gems/rocksdb-ruby
* Haskell - https://hackage.haskell.org/package/rocksdb-haskell
* PHP - https://github.com/Photonios/rocksdb-php
* C#
* https://github.com/warrenfalk/rocksdb-sharp
* https://github.com/curiosity-ai/rocksdb-sharp
* C# - https://github.com/warrenfalk/rocksdb-sharp
* Rust
* https://github.com/pingcap/rust-rocksdb (used in production fork of https://github.com/spacejam/rust-rocksdb)
* https://github.com/spacejam/rust-rocksdb
+589 -467
View File
File diff suppressed because it is too large. Load diff
+2 -3
View File
@@ -1,8 +1,7 @@
## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
[![CircleCI Status](https://circleci.com/gh/facebook/rocksdb.svg?style=svg)](https://circleci.com/gh/facebook/rocksdb)
[![TravisCI Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
[![Appveyor Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/master?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/master)
[![Linux/Mac Build Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
[![Windows Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/master?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/master)
[![PPC64le Build Status](http://140.211.168.68:8080/buildStatus/icon?job=Rocksdb)](http://140.211.168.68:8080/job/Rocksdb)
RocksDB is developed and maintained by Facebook Database Engineering Team.
+4 -15
View File
@@ -138,7 +138,6 @@ cpp_library(
"db/compaction/compaction_picker_fifo.cc",
"db/compaction/compaction_picker_level.cc",
"db/compaction/compaction_picker_universal.cc",
"db/compaction/sst_partitioner.cc",
"db/convenience.cc",
"db/db_filesnapshot.cc",
"db/db_impl/db_impl.cc",
@@ -194,7 +193,6 @@ cpp_library(
"env/env_hdfs.cc",
"env/env_posix.cc",
"env/file_system.cc",
"env/file_system_tracer.cc",
"env/fs_posix.cc",
"env/io_posix.cc",
"env/mock_env.cc",
@@ -284,7 +282,6 @@ cpp_library(
"table/plain/plain_table_index.cc",
"table/plain/plain_table_key_coding.cc",
"table/plain/plain_table_reader.cc",
"table/sst_file_dumper.cc",
"table/sst_file_reader.cc",
"table/sst_file_writer.cc",
"table/table_properties.cc",
@@ -297,7 +294,6 @@ cpp_library(
"tools/ldb_tool.cc",
"tools/sst_dump_tool.cc",
"trace_replay/block_cache_tracer.cc",
"trace_replay/io_tracer.cc",
"trace_replay/trace_replay.cc",
"util/build_version.cc",
"util/coding.cc",
@@ -334,8 +330,6 @@ cpp_library(
"utilities/debug.cc",
"utilities/env_mirror.cc",
"utilities/env_timed.cc",
"utilities/fault_injection_env.cc",
"utilities/fault_injection_fs.cc",
"utilities/leveldb_options/leveldb_options.cc",
"utilities/memory/memory_util.cc",
"utilities/merge_operators/bytesxor.cc",
@@ -389,6 +383,8 @@ cpp_library(
srcs = [
"db/db_test_util.cc",
"table/mock_table.cc",
"test_util/fault_injection_test_env.cc",
"test_util/fault_injection_test_fs.cc",
"test_util/testharness.cc",
"test_util/testutil.cc",
"tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
@@ -651,7 +647,7 @@ ROCKS_TESTS = [
[
"column_family_test",
"db/column_family_test.cc",
"parallel",
"serial",
[],
[],
],
@@ -945,7 +941,7 @@ ROCKS_TESTS = [
[
"db_test2",
"db/db_test2.cc",
"parallel",
"serial",
[],
[],
],
@@ -1173,13 +1169,6 @@ ROCKS_TESTS = [
[],
[],
],
[
"io_tracer_test",
"trace_replay/io_tracer_test.cc",
"serial",
[],
[],
],
[
"iostats_context_test",
"monitoring/iostats_context_test.cc",
+11 -2
View File
@@ -24,6 +24,15 @@ environment:
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
CMAKE_GENERATOR: Visual Studio 15 Win64
DEV_ENV: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\devenv.com
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
CMAKE_GENERATOR: Visual Studio 16
CMAKE_PLATEFORM_NAME: x64
DEV_ENV: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.com
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
CMAKE_GENERATOR: Visual Studio 16
CMAKE_PLATEFORM_NAME: x64
CMAKE_OPT: -DCMAKE_CXX_STANDARD=20
DEV_ENV: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.com
install:
- md %THIRDPARTY_HOME%
@@ -59,7 +68,7 @@ before_build:
- md %APPVEYOR_BUILD_FOLDER%\build
- cd %APPVEYOR_BUILD_FOLDER%\build
- if DEFINED CMAKE_PLATEFORM_NAME (set "PLATEFORM_OPT=-A %CMAKE_PLATEFORM_NAME%")
- cmake .. -G "%CMAKE_GENERATOR%" %PLATEFORM_OPT% %CMAKE_OPT% -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DLZ4=1 -DZSTD=1 -DXPRESS=1 -DJNI=1 -DWITH_ALL_TESTS=0
- cmake .. -G "%CMAKE_GENERATOR%" %PLATEFORM_OPT% %CMAKE_OPT% -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DLZ4=1 -DZSTD=1 -DXPRESS=1 -DJNI=1
- cd ..
build:
@@ -70,7 +79,7 @@ build:
test:
test_script:
- ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,env_basic_test -Concurrency 8
- ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_with_timestamp_basic_test,db_test2,db_test,env_basic_test,env_test,db_merge_operand_test -Concurrency 8
on_failure:
- cmd: 7z a build-failed.zip %APPVEYOR_BUILD_FOLDER%\build\ && appveyor PushArtifact build-failed.zip
+42 -31
View File
@@ -48,8 +48,8 @@ def parse_src_mk(repo_path):
if '=' in line:
current_src = line.split('=')[0].strip()
src_files[current_src] = []
elif '.c' in line:
src_path = line.split('\\')[0].strip()
elif '.cc' in line:
src_path = line.split('.cc')[0].strip() + '.cc'
src_files[current_src].append(src_path)
return src_files
@@ -64,16 +64,30 @@ def get_cc_files(repo_path):
continue
for filename in fnmatch.filter(filenames, '*.cc'):
cc_files.append(os.path.join(root, filename))
for filename in fnmatch.filter(filenames, '*.c'):
cc_files.append(os.path.join(root, filename))
return cc_files
# Get parallel tests from Makefile
def get_parallel_tests(repo_path):
# Get tests from Makefile
def get_tests(repo_path):
Makefile = repo_path + "/Makefile"
s = set({})
# Dictionary TEST_NAME => IS_PARALLEL
tests = {}
found_tests = False
for line in open(Makefile):
line = line.strip()
if line.startswith("TESTS ="):
found_tests = True
elif found_tests:
if line.endswith("\\"):
# remove the trailing \
line = line[:-1]
line = line.strip()
tests[line] = False
else:
# we consumed all the tests
break
found_parallel_tests = False
for line in open(Makefile):
@@ -85,12 +99,12 @@ def get_parallel_tests(repo_path):
# remove the trailing \
line = line[:-1]
line = line.strip()
s.add(line)
tests[line] = True
else:
# we consumed all the parallel tests
break
return s
return tests
# Parse extra dependencies passed by user from command line
def get_dependencies():
@@ -123,10 +137,10 @@ def generate_targets(repo_path, deps_map):
src_mk = parse_src_mk(repo_path)
# get all .cc files
cc_files = get_cc_files(repo_path)
# get parallel tests from Makefile
parallel_tests = get_parallel_tests(repo_path)
# get tests from Makefile
tests = get_tests(repo_path)
if src_mk is None or cc_files is None or parallel_tests is None:
if src_mk is None or cc_files is None or tests is None:
return False
TARGETS = TARGETSBuilder("%s/TARGETS" % repo_path)
@@ -163,42 +177,39 @@ def generate_targets(repo_path, deps_map):
print("Extra dependencies:\n{0}".format(json.dumps(deps_map)))
# Dictionary test executable name -> relative source file path
test_source_map = {}
print(src_mk)
# c_test.c is added through TARGETS.add_c_test(). If there
# are more than one .c test file, we need to extend
# TARGETS.add_c_test() to include other C tests too.
for test_src in src_mk.get("TEST_MAIN_SOURCES_C", []):
if test_src != 'db/c_test.c':
print("Don't know how to deal with " + test_src)
return False
TARGETS.add_c_test()
for test_src in src_mk.get("TEST_MAIN_SOURCES", []):
test = test_src.split('.c')[0].strip().split('/')[-1].strip()
test_source_map[test] = test_src
print("" + test + " " + test_src)
# test for every .cc test we found in the Makefile
for target_alias, deps in deps_map.items():
for test, test_src in sorted(test_source_map.items()):
if len(test) == 0:
print(ColorString.warning("Failed to get test name for %s" % test_src))
for test in sorted(tests):
if test == 'c_test':
continue
match_src = [src for src in cc_files if ("/%s.cc" % test) in src]
if len(match_src) == 0:
print(ColorString.warning("Cannot find .cc file for %s" % test))
continue
elif len(match_src) > 1:
print(ColorString.warning("Found more than one .cc for %s" % test))
print(match_src)
continue
assert(len(match_src) == 1)
is_parallel = tests[test]
test_target_name = \
test if not target_alias else test + "_" + target_alias
TARGETS.register_test(
test_target_name,
test_src,
test in parallel_tests,
match_src[0],
is_parallel,
json.dumps(deps['extra_deps']),
json.dumps(deps['extra_compiler_flags']))
if test in _EXPORTED_TEST_LIBS:
test_library = "%s_lib" % test_target_name
TARGETS.add_library(test_library, [test_src], [":rocksdb_test_lib"])
TARGETS.add_library(test_library, match_src, [":rocksdb_test_lib"])
TARGETS.flush_tests()
print(ColorString.info("Generated TARGETS Summary:"))
+14 -51
View File
@@ -89,16 +89,6 @@ if test -z "$CXX"; then
fi
fi
if test -z "$AR"; then
if [ -x "$(command -v gcc-ar)" ]; then
AR=gcc-ar
elif [ -x "$(command -v llvm-ar)" ]; then
AR=llvm-ar
else
AR=ar
fi
fi
# Detect OS
if test -z "$TARGET_OS"; then
TARGET_OS=`uname -s`
@@ -161,7 +151,7 @@ case "$TARGET_OS" in
else
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt -ldl"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
if test $ROCKSDB_USE_IO_URING; then
# check for liburing
$CXX $CFLAGS -x c++ - -luring -o /dev/null 2>/dev/null <<EOF
@@ -202,17 +192,6 @@ EOF
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread"
# PORT_FILES=port/freebsd/freebsd_specific.cc
;;
GNU/kFreeBSD)
PLATFORM=OS_GNU_KFREEBSD
COMMON_FLAGS="$COMMON_FLAGS -DOS_GNU_KFREEBSD"
if [ -z "$USE_CLANG" ]; then
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp"
else
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# PORT_FILES=port/gnu_kfreebsd/gnu_kfreebsd_specific.cc
;;
NetBSD)
PLATFORM=OS_NETBSD
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_NETBSD"
@@ -306,32 +285,24 @@ EOF
# Test whether gflags library is installed
# http://gflags.github.io/gflags/
# check if the namespace is gflags
if $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace GFLAGS_NAMESPACE;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
# check if namespace is gflags
elif $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace gflags;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1 -DGFLAGS_NAMESPACE=gflags"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
# check if namespace is google
elif $CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
else
# check if namespace is google
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null << EOF
#include <gflags/gflags.h>
using namespace google;
int main() {}
EOF
then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=1 -DGFLAGS_NAMESPACE=google"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DGFLAGS=google"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lgflags"
fi
fi
fi
@@ -459,7 +430,7 @@ EOF
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_MALLOC_USABLE_SIZE"
fi
fi
if ! test $ROCKSDB_DISABLE_MEMKIND; then
# Test whether memkind library is installed
$CXX $CFLAGS $COMMON_FLAGS -lmemkind -x c++ - -o /dev/null 2>/dev/null <<EOF
@@ -475,7 +446,7 @@ EOF
JAVA_LDFLAGS="$JAVA_LDFLAGS -lmemkind"
fi
fi
if ! test $ROCKSDB_DISABLE_PTHREAD_MUTEX_ADAPTIVE_NP; then
# Test whether PTHREAD_MUTEX_ADAPTIVE_NP mutex type is available
$CXX $CFLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
@@ -641,13 +612,6 @@ else
if test "$USE_SSE"; then
TRY_SSE_ETC="1"
fi
if [[ "${PLATFORM}" == "OS_MACOSX" ]]; then
# For portability compile for macOS 10.12 (2016) or newer
COMMON_FLAGS="$COMMON_FLAGS -mmacosx-version-min=10.12"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -mmacosx-version-min=10.12"
PLATFORM_SHARED_LDFLAGS="$PLATFORM_SHARED_LDFLAGS -mmacosx-version-min=10.12"
fi
fi
if test "$TRY_SSE_ETC"; then
@@ -766,7 +730,6 @@ ROCKSDB_PATCH=`build_tools/version.sh patch`
echo "CC=$CC" >> "$OUTPUT"
echo "CXX=$CXX" >> "$OUTPUT"
echo "AR=$AR" >> "$OUTPUT"
echo "PLATFORM=$PLATFORM" >> "$OUTPUT"
echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> "$OUTPUT"
echo "JAVA_LDFLAGS=$JAVA_LDFLAGS" >> "$OUTPUT"
-2
View File
@@ -109,7 +109,6 @@ if [ -z "$USE_CLANG" ]; then
# gcc
CC="$GCC_BASE/bin/gcc"
CXX="$GCC_BASE/bin/g++"
AR="$GCC_BASE/bin/gcc-ar"
CFLAGS+=" -B$BINUTILS/gold"
CFLAGS+=" -isystem $GLIBC_INCLUDE"
@@ -120,7 +119,6 @@ else
CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
CC="$CLANG_BIN/clang"
CXX="$CLANG_BIN/clang++"
AR="$CLANG_BIN/llvm-ar"
KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
-2
View File
@@ -69,7 +69,6 @@ if [ -z "$USE_CLANG" ]; then
# gcc
CC="$GCC_BASE/bin/gcc"
CXX="$GCC_BASE/bin/g++"
CXX="$GCC_BASE/bin/gcc-ar"
CFLAGS="-B$BINUTILS/gold -m64 -mtune=generic"
CFLAGS+=" -isystem $GLIBC_INCLUDE"
@@ -82,7 +81,6 @@ else
CLANG_INCLUDE="$CLANG_LIB/clang/*/include"
CC="$CLANG_BIN/clang"
CXX="$CLANG_BIN/clang++"
AR="$CLANG_BIN/llvm-ar"
KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include/"
-2
View File
@@ -118,7 +118,6 @@ if [ -z "$USE_CLANG" ]; then
# gcc
CC="$GCC_BASE/bin/gcc"
CXX="$GCC_BASE/bin/g++"
AR="$GCC_BASE/bin/gcc-ar"
CFLAGS+=" -B$BINUTILS/gold"
CFLAGS+=" -isystem $LIBGCC_INCLUDE"
@@ -129,7 +128,6 @@ else
CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
CC="$CLANG_BIN/clang"
CXX="$CLANG_BIN/clang++"
AR="$CLANG_BIN/llvm-ar"
KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
+1 -229
View File
@@ -421,66 +421,6 @@ STRESS_CRASH_TEST_COMMANDS="[
}
]"
#
# RocksDB blackbox stress/crash test
#
BLACKBOX_STRESS_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb Blackbox Stress and Crash Test',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug stress tests',
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 db_stress || $CONTRUN_NAME=db_stress $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
{
'name':'Build and run RocksDB debug blackbox crash tests',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 blackbox_crash_test || $CONTRUN_NAME=blackbox_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB whitebox stress/crash test
#
WHITEBOX_STRESS_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb Whitebox Stress and Crash Test',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug stress tests',
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 db_stress || $CONTRUN_NAME=db_stress $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
{
'name':'Build and run RocksDB debug whitebox crash tests',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 whitebox_crash_test || $CONTRUN_NAME=whitebox_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB stress/crash test with atomic flush
#
@@ -609,54 +549,6 @@ ASAN_CRASH_TEST_COMMANDS="[
}
]"
#
# RocksDB blackbox crash testing under address sanitizer
#
ASAN_BLACKBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb blackbox crash test under ASAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug blackbox asan_crash_test',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 blackbox_asan_crash_test || $CONTRUN_NAME=blackbox_asan_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB whitebox crash testing under address sanitizer
#
ASAN_WHITEBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb whitebox crash test under ASAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug whitebox asan_crash_test',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 whitebox_asan_crash_test || $CONTRUN_NAME=whitebox_asan_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB crash testing with atomic flush under address sanitizer
#
@@ -727,7 +619,7 @@ UBSAN_TEST_COMMANDS="[
]"
#
# RocksDB crash testing under undefined behavior sanitizer
# RocksDB crash testing under udnefined behavior sanitizer
#
UBSAN_CRASH_TEST_COMMANDS="[
{
@@ -750,54 +642,6 @@ UBSAN_CRASH_TEST_COMMANDS="[
}
]"
#
# RocksDB crash testing under undefined behavior sanitizer
#
UBSAN_BLACKBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb blackbox crash test under UBSAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug blackbox ubsan_crash_test',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH $CLANG make J=1 blackbox_ubsan_crash_test || $CONTRUN_NAME=blackbox_ubsan_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB crash testing under undefined behavior sanitizer
#
UBSAN_WHITEBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb whitebox crash test under UBSAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Build and run RocksDB debug whitebox ubsan_crash_test',
'timeout': 86400,
'shell':'$SHM $DEBUG $NON_TSAN_CRASH $CLANG make J=1 whitebox_ubsan_crash_test || $CONTRUN_NAME=whitebox_ubsan_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB crash testing with atomic flush under undefined behavior sanitizer
#
@@ -916,54 +760,6 @@ TSAN_CRASH_TEST_COMMANDS="[
}
]"
#
# RocksDB blackbox crash test under TSAN
#
TSAN_BLACKBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb Blackbox Crash Test under TSAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Compile and run',
'timeout': 86400,
'shell':'set -o pipefail && $SHM $DEBUG $TSAN $TSAN_CRASH CRASH_TEST_KILL_ODD=1887 make J=1 blackbox_crash_test || $CONTRUN_NAME=tsan_blackbox_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB whitebox crash test under TSAN
#
TSAN_WHITEBOX_CRASH_TEST_COMMANDS="[
{
'name':'Rocksdb Whitebox Crash Test under TSAN',
'oncall':'$ONCALL',
'executeLocal': 'true',
'timeout': 86400,
'steps': [
$CLEANUP_ENV,
{
'name':'Compile and run',
'timeout': 86400,
'shell':'set -o pipefail && $SHM $DEBUG $TSAN $TSAN_CRASH CRASH_TEST_KILL_ODD=1887 make J=1 whitebox_crash_test || $CONTRUN_NAME=tsan_whitebox_crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
]"
#
# RocksDB crash test with atomic flush under TSAN
#
@@ -1198,12 +994,6 @@ case $1 in
stress_crash)
echo $STRESS_CRASH_TEST_COMMANDS
;;
blackbox_stress_crash)
echo $BLACKBOX_STRESS_CRASH_TEST_COMMANDS
;;
whitebox_stress_crash)
echo $WHITEBOX_STRESS_CRASH_TEST_COMMANDS
;;
stress_crash_with_atomic_flush)
echo $STRESS_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS
;;
@@ -1219,12 +1009,6 @@ case $1 in
asan_crash)
echo $ASAN_CRASH_TEST_COMMANDS
;;
blackbox_asan_crash)
echo $ASAN_BLACKBOX_CRASH_TEST_COMMANDS
;;
whitebox_asan_crash)
echo $ASAN_WHITEBOX_CRASH_TEST_COMMANDS
;;
asan_crash_with_atomic_flush)
echo $ASAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS
;;
@@ -1237,12 +1021,6 @@ case $1 in
ubsan_crash)
echo $UBSAN_CRASH_TEST_COMMANDS
;;
blackbox_ubsan_crash)
echo $UBSAN_BLACKBOX_CRASH_TEST_COMMANDS
;;
whitebox_ubsan_crash)
echo $UBSAN_WHITEBOX_CRASH_TEST_COMMANDS
;;
ubsan_crash_with_atomic_flush)
echo $UBSAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS
;;
@@ -1258,12 +1036,6 @@ case $1 in
tsan_crash)
echo $TSAN_CRASH_TEST_COMMANDS
;;
blackbox_tsan_crash)
echo $TSAN_BLACKBOX_CRASH_TEST_COMMANDS
;;
whitebox_tsan_crash)
echo $TSAN_WHITEBOX_CRASH_TEST_COMMANDS
;;
tsan_crash_with_atomic_flush)
echo $TSAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS
;;
+2
View File
@@ -11,6 +11,7 @@
#include "util/crc32c.h"
namespace ROCKSDB_NAMESPACE {
namespace blob_db {
void BlobLogHeader::EncodeTo(std::string* dst) {
assert(dst != nullptr);
@@ -143,5 +144,6 @@ Status BlobLogRecord::CheckBlobCRC() const {
return Status::OK();
}
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+2
View File
@@ -18,6 +18,7 @@
#include "rocksdb/types.h"
namespace ROCKSDB_NAMESPACE {
namespace blob_db {
constexpr uint32_t kMagicNumber = 2395959; // 0x00248f37
constexpr uint32_t kVersion1 = 1;
@@ -125,5 +126,6 @@ struct BlobLogRecord {
Status CheckBlobCRC() const;
};
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+8 -7
View File
@@ -14,17 +14,17 @@
#include "util/stop_watch.h"
namespace ROCKSDB_NAMESPACE {
namespace blob_db {
BlobLogReader::BlobLogReader(
std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics)
Reader::Reader(std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics)
: file_(std::move(file_reader)),
env_(env),
statistics_(statistics),
buffer_(),
next_byte_(0) {}
Status BlobLogReader::ReadSlice(uint64_t size, Slice* slice, char* buf) {
Status Reader::ReadSlice(uint64_t size, Slice* slice, char* buf) {
StopWatch read_sw(env_, statistics_, BLOB_DB_BLOB_FILE_READ_MICROS);
Status s = file_->Read(IOOptions(), next_byte_, static_cast<size_t>(size),
slice, buf, nullptr);
@@ -39,7 +39,7 @@ Status BlobLogReader::ReadSlice(uint64_t size, Slice* slice, char* buf) {
return s;
}
Status BlobLogReader::ReadHeader(BlobLogHeader* header) {
Status Reader::ReadHeader(BlobLogHeader* header) {
assert(file_.get() != nullptr);
assert(next_byte_ == 0);
Status s = ReadSlice(BlobLogHeader::kSize, &buffer_, header_buf_);
@@ -54,8 +54,8 @@ Status BlobLogReader::ReadHeader(BlobLogHeader* header) {
return header->DecodeFrom(buffer_);
}
Status BlobLogReader::ReadRecord(BlobLogRecord* record, ReadLevel level,
uint64_t* blob_offset) {
Status Reader::ReadRecord(BlobLogRecord* record, ReadLevel level,
uint64_t* blob_offset) {
Status s = ReadSlice(BlobLogRecord::kHeaderSize, &buffer_, header_buf_);
if (!s.ok()) {
return s;
@@ -101,5 +101,6 @@ Status BlobLogReader::ReadRecord(BlobLogRecord* record, ReadLevel level,
return s;
}
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+12 -10
View File
@@ -22,14 +22,15 @@ namespace ROCKSDB_NAMESPACE {
class SequentialFileReader;
class Logger;
namespace blob_db {
/**
* BlobLogReader is a general purpose log stream reader implementation. The
* actual job of reading from the device is implemented by the SequentialFile
* interface.
* Reader is a general purpose log stream reader implementation. The actual job
* of reading from the device is implemented by the SequentialFile interface.
*
* Please see Writer for details on the file and record layout.
*/
class BlobLogReader {
class Reader {
public:
enum ReadLevel {
kReadHeader,
@@ -38,14 +39,14 @@ class BlobLogReader {
};
// Create a reader that will return log records from "*file".
// "*file" must remain live while this BlobLogReader is in use.
BlobLogReader(std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics);
// "*file" must remain live while this Reader is in use.
Reader(std::unique_ptr<RandomAccessFileReader>&& file_reader, Env* env,
Statistics* statistics);
// No copying allowed
BlobLogReader(const BlobLogReader&) = delete;
BlobLogReader& operator=(const BlobLogReader&) = delete;
Reader(const Reader&) = delete;
Reader& operator=(const Reader&) = delete;
~BlobLogReader() = default;
~Reader() = default;
Status ReadHeader(BlobLogHeader* header);
@@ -76,5 +77,6 @@ class BlobLogReader {
uint64_t next_byte_;
};
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+21 -24
View File
@@ -13,33 +13,33 @@
#include "file/writable_file_writer.h"
#include "monitoring/statistics.h"
#include "rocksdb/env.h"
#include "test_util/sync_point.h"
#include "util/coding.h"
#include "util/stop_watch.h"
namespace ROCKSDB_NAMESPACE {
namespace blob_db {
BlobLogWriter::BlobLogWriter(std::unique_ptr<WritableFileWriter>&& dest,
Env* env, Statistics* statistics,
uint64_t log_number, bool use_fs, uint64_t boffset)
Writer::Writer(std::unique_ptr<WritableFileWriter>&& dest, Env* env,
Statistics* statistics, uint64_t log_number, uint64_t bpsync,
bool use_fs, uint64_t boffset)
: dest_(std::move(dest)),
env_(env),
statistics_(statistics),
log_number_(log_number),
block_offset_(boffset),
bytes_per_sync_(bpsync),
next_sync_offset_(0),
use_fsync_(use_fs),
last_elem_type_(kEtNone) {}
Status BlobLogWriter::Sync() {
TEST_SYNC_POINT("BlobLogWriter::Sync");
Status Writer::Sync() {
StopWatch sync_sw(env_, statistics_, BLOB_DB_BLOB_FILE_SYNC_MICROS);
Status s = dest_->Sync(use_fsync_);
RecordTick(statistics_, BLOB_DB_BLOB_FILE_SYNCED);
return s;
}
Status BlobLogWriter::WriteHeader(BlobLogHeader& header) {
Status Writer::WriteHeader(BlobLogHeader& header) {
assert(block_offset_ == 0);
assert(last_elem_type_ == kEtNone);
std::string str;
@@ -56,7 +56,7 @@ Status BlobLogWriter::WriteHeader(BlobLogHeader& header) {
return s;
}
Status BlobLogWriter::AppendFooter(BlobLogFooter& footer) {
Status Writer::AppendFooter(BlobLogFooter& footer) {
assert(block_offset_ != 0);
assert(last_elem_type_ == kEtFileHdr || last_elem_type_ == kEtRecord);
@@ -66,10 +66,7 @@ Status BlobLogWriter::AppendFooter(BlobLogFooter& footer) {
Status s = dest_->Append(Slice(str));
if (s.ok()) {
block_offset_ += str.size();
s = Sync();
if (s.ok()) {
s = dest_->Close();
}
s = dest_->Close();
dest_.reset();
}
@@ -79,9 +76,9 @@ Status BlobLogWriter::AppendFooter(BlobLogFooter& footer) {
return s;
}
Status BlobLogWriter::AddRecord(const Slice& key, const Slice& val,
uint64_t expiration, uint64_t* key_offset,
uint64_t* blob_offset) {
Status Writer::AddRecord(const Slice& key, const Slice& val,
uint64_t expiration, uint64_t* key_offset,
uint64_t* blob_offset) {
assert(block_offset_ != 0);
assert(last_elem_type_ == kEtFileHdr || last_elem_type_ == kEtRecord);
@@ -92,8 +89,8 @@ Status BlobLogWriter::AddRecord(const Slice& key, const Slice& val,
return s;
}
Status BlobLogWriter::AddRecord(const Slice& key, const Slice& val,
uint64_t* key_offset, uint64_t* blob_offset) {
Status Writer::AddRecord(const Slice& key, const Slice& val,
uint64_t* key_offset, uint64_t* blob_offset) {
assert(block_offset_ != 0);
assert(last_elem_type_ == kEtFileHdr || last_elem_type_ == kEtRecord);
@@ -104,8 +101,8 @@ Status BlobLogWriter::AddRecord(const Slice& key, const Slice& val,
return s;
}
void BlobLogWriter::ConstructBlobHeader(std::string* buf, const Slice& key,
const Slice& val, uint64_t expiration) {
void Writer::ConstructBlobHeader(std::string* buf, const Slice& key,
const Slice& val, uint64_t expiration) {
BlobLogRecord record;
record.key = key;
record.value = val;
@@ -113,10 +110,9 @@ void BlobLogWriter::ConstructBlobHeader(std::string* buf, const Slice& key,
record.EncodeHeaderTo(buf);
}
Status BlobLogWriter::EmitPhysicalRecord(const std::string& headerbuf,
const Slice& key, const Slice& val,
uint64_t* key_offset,
uint64_t* blob_offset) {
Status Writer::EmitPhysicalRecord(const std::string& headerbuf,
const Slice& key, const Slice& val,
uint64_t* key_offset, uint64_t* blob_offset) {
StopWatch write_sw(env_, statistics_, BLOB_DB_BLOB_FILE_WRITE_MICROS);
Status s = dest_->Append(Slice(headerbuf));
if (s.ok()) {
@@ -138,5 +134,6 @@ Status BlobLogWriter::EmitPhysicalRecord(const std::string& headerbuf,
return s;
}
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+18 -9
View File
@@ -21,27 +21,29 @@ namespace ROCKSDB_NAMESPACE {
class WritableFileWriter;
namespace blob_db {
/**
* BlobLogWriter is the blob log stream writer. It provides an append-only
* Writer is the blob log stream writer. It provides an append-only
* abstraction for writing blob data.
*
*
* Look at blob_db_format.h to see the details of the record formats.
*/
class BlobLogWriter {
class Writer {
public:
// Create a writer that will append data to "*dest".
// "*dest" must be initially empty.
// "*dest" must remain live while this BlobLogWriter is in use.
BlobLogWriter(std::unique_ptr<WritableFileWriter>&& dest, Env* env,
Statistics* statistics, uint64_t log_number, bool use_fsync,
uint64_t boffset = 0);
// "*dest" must remain live while this Writer is in use.
Writer(std::unique_ptr<WritableFileWriter>&& dest, Env* env,
Statistics* statistics, uint64_t log_number, uint64_t bpsync,
bool use_fsync, uint64_t boffset = 0);
// No copying allowed
BlobLogWriter(const BlobLogWriter&) = delete;
BlobLogWriter& operator=(const BlobLogWriter&) = delete;
Writer(const Writer&) = delete;
Writer& operator=(const Writer&) = delete;
~BlobLogWriter() = default;
~Writer() = default;
static void ConstructBlobHeader(std::string* buf, const Slice& key,
const Slice& val, uint64_t expiration);
@@ -66,14 +68,20 @@ class BlobLogWriter {
uint64_t get_log_number() const { return log_number_; }
bool ShouldSync() const { return block_offset_ > next_sync_offset_; }
Status Sync();
void ResetSyncPointer() { next_sync_offset_ += bytes_per_sync_; }
private:
std::unique_ptr<WritableFileWriter> dest_;
Env* env_;
Statistics* statistics_;
uint64_t log_number_;
uint64_t block_offset_; // Current offset in block
uint64_t bytes_per_sync_;
uint64_t next_sync_offset_;
bool use_fsync_;
public:
@@ -81,5 +89,6 @@ class BlobLogWriter {
ElemType last_elem_type_;
};
} // namespace blob_db
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+1 -1
View File
@@ -46,7 +46,7 @@ class DBBlobIndexTest : public DBTestBase {
ColumnFamilyHandle* cfh() { return dbfull()->DefaultColumnFamily(); }
ColumnFamilyData* cfd() {
return static_cast_with_check<ColumnFamilyHandleImpl>(cfh())->cfd();
return reinterpret_cast<ColumnFamilyHandleImpl*>(cfh())->cfd();
}
Status PutBlobIndex(WriteBatch* batch, const Slice& key,
+10 -33
View File
@@ -51,8 +51,7 @@ TableBuilder* NewTableBuilder(
uint64_t sample_for_compression, const CompressionOptions& compression_opts,
int level, const bool skip_filters, const uint64_t creation_time,
const uint64_t oldest_key_time, const uint64_t target_file_size,
const uint64_t file_creation_time, const std::string& db_id,
const std::string& db_session_id) {
const uint64_t file_creation_time) {
assert((column_family_id ==
TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
column_family_name.empty());
@@ -62,7 +61,7 @@ TableBuilder* NewTableBuilder(
sample_for_compression, compression_opts,
skip_filters, column_family_name, level,
creation_time, oldest_key_time, target_file_size,
file_creation_time, db_id, db_session_id),
file_creation_time),
column_family_id, file);
}
@@ -86,16 +85,13 @@ Status BuildTable(
EventLogger* event_logger, int job_id, const Env::IOPriority io_priority,
TableProperties* table_properties, int level, const uint64_t creation_time,
const uint64_t oldest_key_time, Env::WriteLifeTimeHint write_hint,
const uint64_t file_creation_time, const std::string& db_id,
const std::string& db_session_id) {
const uint64_t file_creation_time) {
assert((column_family_id ==
TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
column_family_name.empty());
// Reports the IOStats for flush for every following bytes.
const size_t kReportFlushIOStatsEvery = 1048576;
uint64_t paranoid_hash = 0;
Status s;
IOStatus io_s;
meta->fd.file_size = 0;
iter->SeekToFirst();
std::unique_ptr<CompactionRangeDelAggregator> range_del_agg(
@@ -111,6 +107,7 @@ Status BuildTable(
ioptions.listeners, dbname, column_family_name, fname, job_id, reason);
#endif // !ROCKSDB_LITE
TableProperties tp;
if (iter->Valid() || !range_del_agg->IsEmpty()) {
TableBuilder* builder;
std::unique_ptr<WritableFileWriter> file_writer;
@@ -125,11 +122,7 @@ Status BuildTable(
bool use_direct_writes = file_options.use_direct_writes;
TEST_SYNC_POINT_CALLBACK("BuildTable:create_file", &use_direct_writes);
#endif // !NDEBUG
io_s = NewWritableFile(fs, fname, &file, file_options);
s = io_s;
if (io_status->ok()) {
*io_status = io_s;
}
s = NewWritableFile(fs, fname, &file, file_options);
if (!s.ok()) {
EventHelpers::LogAndNotifyTableFileCreationFinished(
event_logger, ioptions.listeners, dbname, column_family_name, fname,
@@ -149,7 +142,7 @@ Status BuildTable(
column_family_name, file_writer.get(), compression,
sample_for_compression, compression_opts_for_flush, level,
false /* skip_filters */, creation_time, oldest_key_time,
0 /*target_file_size*/, file_creation_time, db_id, db_session_id);
0 /*target_file_size*/, file_creation_time);
}
MergeHelper merge(env, internal_comparator.user_comparator(),
@@ -168,11 +161,6 @@ Status BuildTable(
const Slice& key = c_iter.key();
const Slice& value = c_iter.value();
const ParsedInternalKey& ikey = c_iter.ikey();
if (paranoid_file_checks) {
// Generate a rolling 64-bit hash of the key and values
paranoid_hash = Hash64(key.data(), key.size(), paranoid_hash);
paranoid_hash = Hash64(value.data(), value.size(), paranoid_hash);
}
builder->Add(key, value);
meta->UpdateBoundaries(key, value, ikey.sequence, ikey.type);
@@ -203,10 +191,7 @@ Status BuildTable(
} else {
s = builder->Finish();
}
io_s = builder->io_status();
if (io_status->ok()) {
*io_status = io_s;
}
*io_status = builder->io_status();
if (s.ok() && !empty) {
uint64_t file_size = builder->FileSize();
@@ -225,16 +210,16 @@ Status BuildTable(
StopWatch sw(env, ioptions.statistics, TABLE_SYNC_MICROS);
*io_status = file_writer->Sync(ioptions.use_fsync);
}
if (s.ok() && io_status->ok() && !empty) {
if (io_status->ok() && !empty) {
*io_status = file_writer->Close();
}
if (s.ok() && io_status->ok() && !empty) {
if (io_status->ok() && !empty) {
// Add the checksum information to file metadata.
meta->file_checksum = file_writer->GetFileChecksum();
meta->file_checksum_func_name = file_writer->GetFileChecksumFuncName();
}
if (s.ok()) {
if (!io_status->ok()) {
s = *io_status;
}
@@ -261,17 +246,9 @@ Status BuildTable(
/*allow_unprepared_value*/ false));
s = it->status();
if (s.ok() && paranoid_file_checks) {
uint64_t check_hash = 0;
for (it->SeekToFirst(); it->Valid(); it->Next()) {
// Generate a rolling 64-bit hash of the key and values
check_hash = Hash64(it->key().data(), it->key().size(), check_hash);
check_hash =
Hash64(it->value().data(), it->value().size(), check_hash);
}
s = it->status();
if (s.ok() && check_hash != paranoid_hash) {
s = Status::Corruption("Paraniod checksums do not match");
}
}
}
}
+2 -4
View File
@@ -51,8 +51,7 @@ TableBuilder* NewTableBuilder(
const CompressionOptions& compression_opts, int level,
const bool skip_filters = false, const uint64_t creation_time = 0,
const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0,
const uint64_t file_creation_time = 0, const std::string& db_id = "",
const std::string& db_session_id = "");
const uint64_t file_creation_time = 0);
// Build a Table file from the contents of *iter. The generated file
// will be named according to number specified in meta. On success, the rest of
@@ -84,7 +83,6 @@ extern Status BuildTable(
TableProperties* table_properties = nullptr, int level = -1,
const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET,
const uint64_t file_creation_time = 0, const std::string& db_id = "",
const std::string& db_session_id = "");
const uint64_t file_creation_time = 0);
} // namespace ROCKSDB_NAMESPACE
-292
View File
@@ -595,15 +595,6 @@ void rocksdb_backup_engine_restore_db_from_latest_backup(
restore_options->rep));
}
void rocksdb_backup_engine_restore_db_from_backup(
rocksdb_backup_engine_t* be, const char* db_dir, const char* wal_dir,
const rocksdb_restore_options_t* restore_options, const uint32_t backup_id,
char** errptr) {
SaveError(errptr, be->rep->RestoreDBFromBackup(backup_id, std::string(db_dir),
std::string(wal_dir),
restore_options->rep));
}
const rocksdb_backup_engine_info_t* rocksdb_backup_engine_get_backup_info(
rocksdb_backup_engine_t* be) {
rocksdb_backup_engine_info_t* result = new rocksdb_backup_engine_info_t;
@@ -1005,55 +996,6 @@ void rocksdb_multi_get_cf(
}
}
unsigned char rocksdb_key_may_exist(rocksdb_t* db,
const rocksdb_readoptions_t* options,
const char* key, size_t key_len,
char** value, size_t* val_len,
const char* timestamp, size_t timestamp_len,
unsigned char* value_found) {
std::string tmp;
std::string time;
if (timestamp) {
time.assign(timestamp, timestamp_len);
}
bool found = false;
const bool result = db->rep->KeyMayExist(options->rep, Slice(key, key_len),
&tmp, timestamp ? &time : nullptr,
value_found ? &found : nullptr);
if (value_found) {
*value_found = found;
if (found) {
*val_len = tmp.size();
*value = CopyString(tmp);
}
}
return result;
}
unsigned char rocksdb_key_may_exist_cf(
rocksdb_t* db, const rocksdb_readoptions_t* options,
rocksdb_column_family_handle_t* column_family, const char* key,
size_t key_len, char** value, size_t* val_len, const char* timestamp,
size_t timestamp_len, unsigned char* value_found) {
std::string tmp;
std::string time;
if (timestamp) {
time.assign(timestamp, timestamp_len);
}
bool found = false;
const bool result = db->rep->KeyMayExist(
options->rep, column_family->rep, Slice(key, key_len), &tmp,
timestamp ? &time : nullptr, value_found ? &found : nullptr);
if (value_found) {
*value_found = found;
if (found) {
*val_len = tmp.size();
*value = CopyString(tmp);
}
}
return result;
}
rocksdb_iterator_t* rocksdb_create_iterator(
rocksdb_t* db,
const rocksdb_readoptions_t* options) {
@@ -2459,10 +2401,6 @@ void rocksdb_options_set_max_compaction_bytes(rocksdb_options_t* opt,
opt->rep.max_compaction_bytes = n;
}
uint64_t rocksdb_options_get_max_compaction_bytes(rocksdb_options_t* opt) {
return opt->rep.max_compaction_bytes;
}
void rocksdb_options_set_max_bytes_for_level_multiplier_additional(
rocksdb_options_t* opt, int* level_values, size_t num_levels) {
opt->rep.max_bytes_for_level_multiplier_additional.resize(num_levels);
@@ -2538,26 +2476,14 @@ void rocksdb_options_set_wal_recovery_mode(rocksdb_options_t* opt,int mode) {
opt->rep.wal_recovery_mode = static_cast<WALRecoveryMode>(mode);
}
int rocksdb_options_get_wal_recovery_mode(rocksdb_options_t* opt) {
return static_cast<int>(opt->rep.wal_recovery_mode);
}
void rocksdb_options_set_compression(rocksdb_options_t* opt, int t) {
opt->rep.compression = static_cast<CompressionType>(t);
}
int rocksdb_options_get_compression(rocksdb_options_t* opt) {
return opt->rep.compression;
}
void rocksdb_options_set_bottommost_compression(rocksdb_options_t* opt, int t) {
opt->rep.bottommost_compression = static_cast<CompressionType>(t);
}
int rocksdb_options_get_bottommost_compression(rocksdb_options_t* opt) {
return opt->rep.bottommost_compression;
}
void rocksdb_options_set_compression_per_level(rocksdb_options_t* opt,
int* level_values,
size_t num_levels) {
@@ -2611,10 +2537,6 @@ void rocksdb_options_set_use_fsync(
opt->rep.use_fsync = use_fsync;
}
int rocksdb_options_get_use_fsync(rocksdb_options_t* opt) {
return opt->rep.use_fsync;
}
void rocksdb_options_set_db_log_dir(
rocksdb_options_t* opt, const char* db_log_dir) {
opt->rep.db_log_dir = db_log_dir;
@@ -2629,28 +2551,16 @@ void rocksdb_options_set_WAL_ttl_seconds(rocksdb_options_t* opt, uint64_t ttl) {
opt->rep.WAL_ttl_seconds = ttl;
}
uint64_t rocksdb_options_get_WAL_ttl_seconds(rocksdb_options_t* opt) {
return opt->rep.WAL_ttl_seconds;
}
void rocksdb_options_set_WAL_size_limit_MB(
rocksdb_options_t* opt, uint64_t limit) {
opt->rep.WAL_size_limit_MB = limit;
}
uint64_t rocksdb_options_get_WAL_size_limit_MB(rocksdb_options_t* opt) {
return opt->rep.WAL_size_limit_MB;
}
void rocksdb_options_set_manifest_preallocation_size(
rocksdb_options_t* opt, size_t v) {
opt->rep.manifest_preallocation_size = v;
}
size_t rocksdb_options_get_manifest_preallocation_size(rocksdb_options_t* opt) {
return opt->rep.manifest_preallocation_size;
}
// noop
void rocksdb_options_set_purge_redundant_kvs_while_flush(
rocksdb_options_t* /*opt*/, unsigned char /*v*/) {}
@@ -2660,86 +2570,41 @@ void rocksdb_options_set_use_direct_reads(rocksdb_options_t* opt,
opt->rep.use_direct_reads = v;
}
unsigned char rocksdb_options_get_use_direct_reads(rocksdb_options_t* opt) {
return opt->rep.use_direct_reads;
}
void rocksdb_options_set_use_direct_io_for_flush_and_compaction(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.use_direct_io_for_flush_and_compaction = v;
}
unsigned char rocksdb_options_get_use_direct_io_for_flush_and_compaction(
rocksdb_options_t* opt) {
return opt->rep.use_direct_io_for_flush_and_compaction;
}
void rocksdb_options_set_allow_mmap_reads(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.allow_mmap_reads = v;
}
unsigned char rocksdb_options_get_allow_mmap_reads(rocksdb_options_t* opt) {
return opt->rep.allow_mmap_reads;
}
void rocksdb_options_set_allow_mmap_writes(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.allow_mmap_writes = v;
}
unsigned char rocksdb_options_get_allow_mmap_writes(rocksdb_options_t* opt) {
return opt->rep.allow_mmap_writes;
}
void rocksdb_options_set_is_fd_close_on_exec(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.is_fd_close_on_exec = v;
}
unsigned char rocksdb_options_get_is_fd_close_on_exec(rocksdb_options_t* opt) {
return opt->rep.is_fd_close_on_exec;
}
void rocksdb_options_set_skip_log_error_on_recovery(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.skip_log_error_on_recovery = v;
}
unsigned char rocksdb_options_get_skip_log_error_on_recovery(
rocksdb_options_t* opt) {
return opt->rep.skip_log_error_on_recovery;
}
void rocksdb_options_set_stats_dump_period_sec(
rocksdb_options_t* opt, unsigned int v) {
opt->rep.stats_dump_period_sec = v;
}
unsigned int rocksdb_options_get_stats_dump_period_sec(rocksdb_options_t* opt) {
return opt->rep.stats_dump_period_sec;
}
void rocksdb_options_set_stats_persist_period_sec(rocksdb_options_t* opt,
unsigned int v) {
opt->rep.stats_persist_period_sec = v;
}
unsigned int rocksdb_options_get_stats_persist_period_sec(
rocksdb_options_t* opt) {
return opt->rep.stats_persist_period_sec;
}
void rocksdb_options_set_advise_random_on_open(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.advise_random_on_open = v;
}
unsigned char rocksdb_options_get_advise_random_on_open(
rocksdb_options_t* opt) {
return opt->rep.advise_random_on_open;
}
void rocksdb_options_set_access_hint_on_compaction_start(
rocksdb_options_t* opt, int v) {
switch(v) {
@@ -2762,78 +2627,41 @@ void rocksdb_options_set_access_hint_on_compaction_start(
}
}
int rocksdb_options_get_access_hint_on_compaction_start(
rocksdb_options_t* opt) {
return opt->rep.access_hint_on_compaction_start;
}
void rocksdb_options_set_use_adaptive_mutex(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.use_adaptive_mutex = v;
}
unsigned char rocksdb_options_get_use_adaptive_mutex(rocksdb_options_t* opt) {
return opt->rep.use_adaptive_mutex;
}
void rocksdb_options_set_wal_bytes_per_sync(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.wal_bytes_per_sync = v;
}
uint64_t rocksdb_options_get_wal_bytes_per_sync(rocksdb_options_t* opt) {
return opt->rep.wal_bytes_per_sync;
}
void rocksdb_options_set_bytes_per_sync(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.bytes_per_sync = v;
}
uint64_t rocksdb_options_get_bytes_per_sync(rocksdb_options_t* opt) {
return opt->rep.bytes_per_sync;
}
void rocksdb_options_set_writable_file_max_buffer_size(rocksdb_options_t* opt,
uint64_t v) {
opt->rep.writable_file_max_buffer_size = static_cast<size_t>(v);
}
uint64_t rocksdb_options_get_writable_file_max_buffer_size(
rocksdb_options_t* opt) {
return opt->rep.writable_file_max_buffer_size;
}
void rocksdb_options_set_allow_concurrent_memtable_write(rocksdb_options_t* opt,
unsigned char v) {
opt->rep.allow_concurrent_memtable_write = v;
}
unsigned char rocksdb_options_get_allow_concurrent_memtable_write(
rocksdb_options_t* opt) {
return opt->rep.allow_concurrent_memtable_write;
}
void rocksdb_options_set_enable_write_thread_adaptive_yield(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.enable_write_thread_adaptive_yield = v;
}
unsigned char rocksdb_options_get_enable_write_thread_adaptive_yield(
rocksdb_options_t* opt) {
return opt->rep.enable_write_thread_adaptive_yield;
}
void rocksdb_options_set_max_sequential_skip_in_iterations(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.max_sequential_skip_in_iterations = v;
}
uint64_t rocksdb_options_get_max_sequential_skip_in_iterations(
rocksdb_options_t* opt) {
return opt->rep.max_sequential_skip_in_iterations;
}
void rocksdb_options_set_max_write_buffer_number(rocksdb_options_t* opt, int n) {
opt->rep.max_write_buffer_number = n;
}
@@ -2903,130 +2731,67 @@ void rocksdb_options_set_max_background_jobs(rocksdb_options_t* opt, int n) {
opt->rep.max_background_jobs = n;
}
int rocksdb_options_get_max_background_jobs(rocksdb_options_t* opt) {
return opt->rep.max_background_jobs;
}
void rocksdb_options_set_max_background_compactions(rocksdb_options_t* opt, int n) {
opt->rep.max_background_compactions = n;
}
int rocksdb_options_get_max_background_compactions(rocksdb_options_t* opt) {
return opt->rep.max_background_compactions;
}
void rocksdb_options_set_base_background_compactions(rocksdb_options_t* opt,
int n) {
opt->rep.base_background_compactions = n;
}
int rocksdb_options_get_base_background_compactions(rocksdb_options_t* opt) {
return opt->rep.base_background_compactions;
}
void rocksdb_options_set_max_background_flushes(rocksdb_options_t* opt, int n) {
opt->rep.max_background_flushes = n;
}
int rocksdb_options_get_max_background_flushes(rocksdb_options_t* opt) {
return opt->rep.max_background_flushes;
}
void rocksdb_options_set_max_log_file_size(rocksdb_options_t* opt, size_t v) {
opt->rep.max_log_file_size = v;
}
size_t rocksdb_options_get_max_log_file_size(rocksdb_options_t* opt) {
return opt->rep.max_log_file_size;
}
void rocksdb_options_set_log_file_time_to_roll(rocksdb_options_t* opt, size_t v) {
opt->rep.log_file_time_to_roll = v;
}
size_t rocksdb_options_get_log_file_time_to_roll(rocksdb_options_t* opt) {
return opt->rep.log_file_time_to_roll;
}
void rocksdb_options_set_keep_log_file_num(rocksdb_options_t* opt, size_t v) {
opt->rep.keep_log_file_num = v;
}
size_t rocksdb_options_get_keep_log_file_num(rocksdb_options_t* opt) {
return opt->rep.keep_log_file_num;
}
void rocksdb_options_set_recycle_log_file_num(rocksdb_options_t* opt,
size_t v) {
opt->rep.recycle_log_file_num = v;
}
size_t rocksdb_options_get_recycle_log_file_num(rocksdb_options_t* opt) {
return opt->rep.recycle_log_file_num;
}
void rocksdb_options_set_soft_rate_limit(rocksdb_options_t* opt, double v) {
opt->rep.soft_rate_limit = v;
}
double rocksdb_options_get_soft_rate_limit(rocksdb_options_t* opt) {
return opt->rep.soft_rate_limit;
}
void rocksdb_options_set_hard_rate_limit(rocksdb_options_t* opt, double v) {
opt->rep.hard_rate_limit = v;
}
double rocksdb_options_get_hard_rate_limit(rocksdb_options_t* opt) {
return opt->rep.hard_rate_limit;
}
void rocksdb_options_set_soft_pending_compaction_bytes_limit(rocksdb_options_t* opt, size_t v) {
opt->rep.soft_pending_compaction_bytes_limit = v;
}
size_t rocksdb_options_get_soft_pending_compaction_bytes_limit(
rocksdb_options_t* opt) {
return opt->rep.soft_pending_compaction_bytes_limit;
}
void rocksdb_options_set_hard_pending_compaction_bytes_limit(rocksdb_options_t* opt, size_t v) {
opt->rep.hard_pending_compaction_bytes_limit = v;
}
size_t rocksdb_options_get_hard_pending_compaction_bytes_limit(
rocksdb_options_t* opt) {
return opt->rep.hard_pending_compaction_bytes_limit;
}
void rocksdb_options_set_rate_limit_delay_max_milliseconds(
rocksdb_options_t* opt, unsigned int v) {
opt->rep.rate_limit_delay_max_milliseconds = v;
}
unsigned int rocksdb_options_get_rate_limit_delay_max_milliseconds(
rocksdb_options_t* opt) {
return opt->rep.rate_limit_delay_max_milliseconds;
}
void rocksdb_options_set_max_manifest_file_size(
rocksdb_options_t* opt, size_t v) {
opt->rep.max_manifest_file_size = v;
}
size_t rocksdb_options_get_max_manifest_file_size(rocksdb_options_t* opt) {
return opt->rep.max_manifest_file_size;
}
void rocksdb_options_set_table_cache_numshardbits(
rocksdb_options_t* opt, int v) {
opt->rep.table_cache_numshardbits = v;
}
int rocksdb_options_get_table_cache_numshardbits(rocksdb_options_t* opt) {
return opt->rep.table_cache_numshardbits;
}
void rocksdb_options_set_table_cache_remove_scan_count_limit(
rocksdb_options_t* /*opt*/, int /*v*/) {
// this option is deprecated
@@ -3037,38 +2802,19 @@ void rocksdb_options_set_arena_block_size(
opt->rep.arena_block_size = v;
}
size_t rocksdb_options_get_arena_block_size(rocksdb_options_t* opt) {
return opt->rep.arena_block_size;
}
void rocksdb_options_set_disable_auto_compactions(rocksdb_options_t* opt, int disable) {
opt->rep.disable_auto_compactions = disable;
}
unsigned char rocksdb_options_get_disable_auto_compactions(
rocksdb_options_t* opt) {
return opt->rep.disable_auto_compactions;
}
void rocksdb_options_set_optimize_filters_for_hits(rocksdb_options_t* opt, int v) {
opt->rep.optimize_filters_for_hits = v;
}
unsigned char rocksdb_options_get_optimize_filters_for_hits(
rocksdb_options_t* opt) {
return opt->rep.optimize_filters_for_hits;
}
void rocksdb_options_set_delete_obsolete_files_period_micros(
rocksdb_options_t* opt, uint64_t v) {
opt->rep.delete_obsolete_files_period_micros = v;
}
uint64_t rocksdb_options_get_delete_obsolete_files_period_micros(
rocksdb_options_t* opt) {
return opt->rep.delete_obsolete_files_period_micros;
}
void rocksdb_options_prepare_for_bulk_load(rocksdb_options_t* opt) {
opt->rep.PrepareForBulkLoad();
}
@@ -3082,20 +2828,11 @@ void rocksdb_options_set_memtable_prefix_bloom_size_ratio(
opt->rep.memtable_prefix_bloom_size_ratio = v;
}
double rocksdb_options_get_memtable_prefix_bloom_size_ratio(
rocksdb_options_t* opt) {
return opt->rep.memtable_prefix_bloom_size_ratio;
}
void rocksdb_options_set_memtable_huge_page_size(rocksdb_options_t* opt,
size_t v) {
opt->rep.memtable_huge_page_size = v;
}
size_t rocksdb_options_get_memtable_huge_page_size(rocksdb_options_t* opt) {
return opt->rep.memtable_huge_page_size;
}
void rocksdb_options_set_hash_skip_list_rep(
rocksdb_options_t *opt, size_t bucket_count,
int32_t skiplist_height, int32_t skiplist_branching_factor) {
@@ -3130,56 +2867,31 @@ void rocksdb_options_set_max_successive_merges(
opt->rep.max_successive_merges = v;
}
size_t rocksdb_options_get_max_successive_merges(rocksdb_options_t* opt) {
return opt->rep.max_successive_merges;
}
void rocksdb_options_set_bloom_locality(
rocksdb_options_t* opt, uint32_t v) {
opt->rep.bloom_locality = v;
}
uint32_t rocksdb_options_get_bloom_locality(rocksdb_options_t* opt) {
return opt->rep.bloom_locality;
}
void rocksdb_options_set_inplace_update_support(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.inplace_update_support = v;
}
unsigned char rocksdb_options_get_inplace_update_support(
rocksdb_options_t* opt) {
return opt->rep.inplace_update_support;
}
void rocksdb_options_set_inplace_update_num_locks(
rocksdb_options_t* opt, size_t v) {
opt->rep.inplace_update_num_locks = v;
}
size_t rocksdb_options_get_inplace_update_num_locks(rocksdb_options_t* opt) {
return opt->rep.inplace_update_num_locks;
}
void rocksdb_options_set_report_bg_io_stats(
rocksdb_options_t* opt, int v) {
opt->rep.report_bg_io_stats = v;
}
unsigned char rocksdb_options_get_report_bg_io_stats(rocksdb_options_t* opt) {
return opt->rep.report_bg_io_stats;
}
void rocksdb_options_set_compaction_style(rocksdb_options_t *opt, int style) {
opt->rep.compaction_style =
static_cast<ROCKSDB_NAMESPACE::CompactionStyle>(style);
}
int rocksdb_options_get_compaction_style(rocksdb_options_t* opt) {
return opt->rep.compaction_style;
}
void rocksdb_options_set_universal_compaction_options(rocksdb_options_t *opt, rocksdb_universal_compaction_options_t *uco) {
opt->rep.compaction_options_universal = *(uco->rep);
}
@@ -3209,10 +2921,6 @@ void rocksdb_options_set_atomic_flush(rocksdb_options_t* opt,
opt->rep.atomic_flush = atomic_flush;
}
unsigned char rocksdb_options_get_atomic_flush(rocksdb_options_t* opt) {
return opt->rep.atomic_flush;
}
rocksdb_ratelimiter_t* rocksdb_ratelimiter_create(
int64_t rate_bytes_per_sec,
int64_t refill_period_us,
-486
View File
@@ -1296,29 +1296,6 @@ int main(int argc, char** argv) {
Free(&vals[i]);
}
{
unsigned char value_found = 0;
CheckCondition(!rocksdb_key_may_exist(db, roptions, "invalid_key", 11,
NULL, NULL, NULL, 0, NULL));
CheckCondition(!rocksdb_key_may_exist(db, roptions, "invalid_key", 11,
&vals[0], &vals_sizes[0], NULL, 0,
&value_found));
if (value_found) {
Free(&vals[0]);
}
CheckCondition(!rocksdb_key_may_exist_cf(db, roptions, handles[1],
"invalid_key", 11, NULL, NULL,
NULL, 0, NULL));
CheckCondition(!rocksdb_key_may_exist_cf(db, roptions, handles[1],
"invalid_key", 11, &vals[0],
&vals_sizes[0], NULL, 0, NULL));
if (value_found) {
Free(&vals[0]);
}
}
rocksdb_iterator_t* iter = rocksdb_create_iterator_cf(db, roptions, handles[1]);
CheckCondition(!rocksdb_iter_valid(iter));
rocksdb_iter_seek_to_first(iter);
@@ -1587,173 +1564,6 @@ int main(int argc, char** argv) {
rocksdb_options_set_max_subcompactions(o, 123456);
CheckCondition(123456 == rocksdb_options_get_max_subcompactions(o));
rocksdb_options_set_max_background_jobs(o, 2);
CheckCondition(2 == rocksdb_options_get_max_background_jobs(o));
rocksdb_options_set_max_background_compactions(o, 3);
CheckCondition(3 == rocksdb_options_get_max_background_compactions(o));
rocksdb_options_set_base_background_compactions(o, 4);
CheckCondition(4 == rocksdb_options_get_base_background_compactions(o));
rocksdb_options_set_max_background_flushes(o, 5);
CheckCondition(5 == rocksdb_options_get_max_background_flushes(o));
rocksdb_options_set_max_log_file_size(o, 6);
CheckCondition(6 == rocksdb_options_get_max_log_file_size(o));
rocksdb_options_set_log_file_time_to_roll(o, 7);
CheckCondition(7 == rocksdb_options_get_log_file_time_to_roll(o));
rocksdb_options_set_keep_log_file_num(o, 8);
CheckCondition(8 == rocksdb_options_get_keep_log_file_num(o));
rocksdb_options_set_recycle_log_file_num(o, 9);
CheckCondition(9 == rocksdb_options_get_recycle_log_file_num(o));
rocksdb_options_set_soft_rate_limit(o, 2.0);
CheckCondition(2.0 == rocksdb_options_get_soft_rate_limit(o));
rocksdb_options_set_hard_rate_limit(o, 4.0);
CheckCondition(4.0 == rocksdb_options_get_hard_rate_limit(o));
rocksdb_options_set_soft_pending_compaction_bytes_limit(o, 10);
CheckCondition(10 ==
rocksdb_options_get_soft_pending_compaction_bytes_limit(o));
rocksdb_options_set_hard_pending_compaction_bytes_limit(o, 11);
CheckCondition(11 ==
rocksdb_options_get_hard_pending_compaction_bytes_limit(o));
rocksdb_options_set_rate_limit_delay_max_milliseconds(o, 1);
CheckCondition(1 ==
rocksdb_options_get_rate_limit_delay_max_milliseconds(o));
rocksdb_options_set_max_manifest_file_size(o, 12);
CheckCondition(12 == rocksdb_options_get_max_manifest_file_size(o));
rocksdb_options_set_table_cache_numshardbits(o, 13);
CheckCondition(13 == rocksdb_options_get_table_cache_numshardbits(o));
rocksdb_options_set_arena_block_size(o, 14);
CheckCondition(14 == rocksdb_options_get_arena_block_size(o));
rocksdb_options_set_use_fsync(o, 1);
CheckCondition(1 == rocksdb_options_get_use_fsync(o));
rocksdb_options_set_WAL_ttl_seconds(o, 15);
CheckCondition(15 == rocksdb_options_get_WAL_ttl_seconds(o));
rocksdb_options_set_WAL_size_limit_MB(o, 16);
CheckCondition(16 == rocksdb_options_get_WAL_size_limit_MB(o));
rocksdb_options_set_manifest_preallocation_size(o, 17);
CheckCondition(17 == rocksdb_options_get_manifest_preallocation_size(o));
rocksdb_options_set_allow_mmap_reads(o, 1);
CheckCondition(1 == rocksdb_options_get_allow_mmap_reads(o));
rocksdb_options_set_allow_mmap_writes(o, 1);
CheckCondition(1 == rocksdb_options_get_allow_mmap_writes(o));
rocksdb_options_set_use_direct_reads(o, 1);
CheckCondition(1 == rocksdb_options_get_use_direct_reads(o));
rocksdb_options_set_use_direct_io_for_flush_and_compaction(o, 1);
CheckCondition(
1 == rocksdb_options_get_use_direct_io_for_flush_and_compaction(o));
rocksdb_options_set_is_fd_close_on_exec(o, 1);
CheckCondition(1 == rocksdb_options_get_is_fd_close_on_exec(o));
rocksdb_options_set_skip_log_error_on_recovery(o, 1);
CheckCondition(1 == rocksdb_options_get_skip_log_error_on_recovery(o));
rocksdb_options_set_stats_dump_period_sec(o, 18);
CheckCondition(18 == rocksdb_options_get_stats_dump_period_sec(o));
rocksdb_options_set_stats_persist_period_sec(o, 5);
CheckCondition(5 == rocksdb_options_get_stats_persist_period_sec(o));
rocksdb_options_set_advise_random_on_open(o, 1);
CheckCondition(1 == rocksdb_options_get_advise_random_on_open(o));
rocksdb_options_set_access_hint_on_compaction_start(o, 3);
CheckCondition(3 == rocksdb_options_get_access_hint_on_compaction_start(o));
rocksdb_options_set_use_adaptive_mutex(o, 1);
CheckCondition(1 == rocksdb_options_get_use_adaptive_mutex(o));
rocksdb_options_set_bytes_per_sync(o, 19);
CheckCondition(19 == rocksdb_options_get_bytes_per_sync(o));
rocksdb_options_set_wal_bytes_per_sync(o, 20);
CheckCondition(20 == rocksdb_options_get_wal_bytes_per_sync(o));
rocksdb_options_set_writable_file_max_buffer_size(o, 21);
CheckCondition(21 == rocksdb_options_get_writable_file_max_buffer_size(o));
rocksdb_options_set_allow_concurrent_memtable_write(o, 1);
CheckCondition(1 == rocksdb_options_get_allow_concurrent_memtable_write(o));
rocksdb_options_set_enable_write_thread_adaptive_yield(o, 1);
CheckCondition(1 ==
rocksdb_options_get_enable_write_thread_adaptive_yield(o));
rocksdb_options_set_max_sequential_skip_in_iterations(o, 22);
CheckCondition(22 ==
rocksdb_options_get_max_sequential_skip_in_iterations(o));
rocksdb_options_set_disable_auto_compactions(o, 1);
CheckCondition(1 == rocksdb_options_get_disable_auto_compactions(o));
rocksdb_options_set_optimize_filters_for_hits(o, 1);
CheckCondition(1 == rocksdb_options_get_optimize_filters_for_hits(o));
rocksdb_options_set_delete_obsolete_files_period_micros(o, 23);
CheckCondition(23 ==
rocksdb_options_get_delete_obsolete_files_period_micros(o));
rocksdb_options_set_memtable_prefix_bloom_size_ratio(o, 2.0);
CheckCondition(2.0 ==
rocksdb_options_get_memtable_prefix_bloom_size_ratio(o));
rocksdb_options_set_max_compaction_bytes(o, 24);
CheckCondition(24 == rocksdb_options_get_max_compaction_bytes(o));
rocksdb_options_set_memtable_huge_page_size(o, 25);
CheckCondition(25 == rocksdb_options_get_memtable_huge_page_size(o));
rocksdb_options_set_max_successive_merges(o, 26);
CheckCondition(26 == rocksdb_options_get_max_successive_merges(o));
rocksdb_options_set_bloom_locality(o, 27);
CheckCondition(27 == rocksdb_options_get_bloom_locality(o));
rocksdb_options_set_inplace_update_support(o, 1);
CheckCondition(1 == rocksdb_options_get_inplace_update_support(o));
rocksdb_options_set_inplace_update_num_locks(o, 28);
CheckCondition(28 == rocksdb_options_get_inplace_update_num_locks(o));
rocksdb_options_set_report_bg_io_stats(o, 1);
CheckCondition(1 == rocksdb_options_get_report_bg_io_stats(o));
rocksdb_options_set_wal_recovery_mode(o, 2);
CheckCondition(2 == rocksdb_options_get_wal_recovery_mode(o));
rocksdb_options_set_compression(o, 5);
CheckCondition(5 == rocksdb_options_get_compression(o));
rocksdb_options_set_bottommost_compression(o, 4);
CheckCondition(4 == rocksdb_options_get_bottommost_compression(o));
rocksdb_options_set_compaction_style(o, 2);
CheckCondition(2 == rocksdb_options_get_compaction_style(o));
rocksdb_options_set_atomic_flush(o, 1);
CheckCondition(1 == rocksdb_options_get_atomic_flush(o));
// Create a copy that should be equal to the original.
rocksdb_options_t* copy;
copy = rocksdb_options_create_copy(o);
@@ -1797,70 +1607,6 @@ int main(int argc, char** argv) {
CheckCondition(1 == rocksdb_options_get_enable_pipelined_write(copy));
CheckCondition(1 == rocksdb_options_get_unordered_write(copy));
CheckCondition(123456 == rocksdb_options_get_max_subcompactions(copy));
CheckCondition(2 == rocksdb_options_get_max_background_jobs(copy));
CheckCondition(3 == rocksdb_options_get_max_background_compactions(copy));
CheckCondition(4 == rocksdb_options_get_base_background_compactions(copy));
CheckCondition(5 == rocksdb_options_get_max_background_flushes(copy));
CheckCondition(6 == rocksdb_options_get_max_log_file_size(copy));
CheckCondition(7 == rocksdb_options_get_log_file_time_to_roll(copy));
CheckCondition(8 == rocksdb_options_get_keep_log_file_num(copy));
CheckCondition(9 == rocksdb_options_get_recycle_log_file_num(copy));
CheckCondition(2.0 == rocksdb_options_get_soft_rate_limit(copy));
CheckCondition(4.0 == rocksdb_options_get_hard_rate_limit(copy));
CheckCondition(
10 == rocksdb_options_get_soft_pending_compaction_bytes_limit(copy));
CheckCondition(
11 == rocksdb_options_get_hard_pending_compaction_bytes_limit(copy));
CheckCondition(1 ==
rocksdb_options_get_rate_limit_delay_max_milliseconds(copy));
CheckCondition(12 == rocksdb_options_get_max_manifest_file_size(copy));
CheckCondition(13 == rocksdb_options_get_table_cache_numshardbits(copy));
CheckCondition(14 == rocksdb_options_get_arena_block_size(copy));
CheckCondition(1 == rocksdb_options_get_use_fsync(copy));
CheckCondition(15 == rocksdb_options_get_WAL_ttl_seconds(copy));
CheckCondition(16 == rocksdb_options_get_WAL_size_limit_MB(copy));
CheckCondition(17 == rocksdb_options_get_manifest_preallocation_size(copy));
CheckCondition(1 == rocksdb_options_get_allow_mmap_reads(copy));
CheckCondition(1 == rocksdb_options_get_allow_mmap_writes(copy));
CheckCondition(1 == rocksdb_options_get_use_direct_reads(copy));
CheckCondition(
1 == rocksdb_options_get_use_direct_io_for_flush_and_compaction(copy));
CheckCondition(1 == rocksdb_options_get_is_fd_close_on_exec(copy));
CheckCondition(1 == rocksdb_options_get_skip_log_error_on_recovery(copy));
CheckCondition(18 == rocksdb_options_get_stats_dump_period_sec(copy));
CheckCondition(5 == rocksdb_options_get_stats_persist_period_sec(copy));
CheckCondition(1 == rocksdb_options_get_advise_random_on_open(copy));
CheckCondition(3 ==
rocksdb_options_get_access_hint_on_compaction_start(copy));
CheckCondition(1 == rocksdb_options_get_use_adaptive_mutex(copy));
CheckCondition(19 == rocksdb_options_get_bytes_per_sync(copy));
CheckCondition(20 == rocksdb_options_get_wal_bytes_per_sync(copy));
CheckCondition(21 ==
rocksdb_options_get_writable_file_max_buffer_size(copy));
CheckCondition(1 ==
rocksdb_options_get_allow_concurrent_memtable_write(copy));
CheckCondition(
1 == rocksdb_options_get_enable_write_thread_adaptive_yield(copy));
CheckCondition(22 ==
rocksdb_options_get_max_sequential_skip_in_iterations(copy));
CheckCondition(1 == rocksdb_options_get_disable_auto_compactions(copy));
CheckCondition(1 == rocksdb_options_get_optimize_filters_for_hits(copy));
CheckCondition(
23 == rocksdb_options_get_delete_obsolete_files_period_micros(copy));
CheckCondition(2.0 ==
rocksdb_options_get_memtable_prefix_bloom_size_ratio(copy));
CheckCondition(24 == rocksdb_options_get_max_compaction_bytes(copy));
CheckCondition(25 == rocksdb_options_get_memtable_huge_page_size(copy));
CheckCondition(26 == rocksdb_options_get_max_successive_merges(copy));
CheckCondition(27 == rocksdb_options_get_bloom_locality(copy));
CheckCondition(1 == rocksdb_options_get_inplace_update_support(copy));
CheckCondition(28 == rocksdb_options_get_inplace_update_num_locks(copy));
CheckCondition(1 == rocksdb_options_get_report_bg_io_stats(copy));
CheckCondition(2 == rocksdb_options_get_wal_recovery_mode(copy));
CheckCondition(5 == rocksdb_options_get_compression(copy));
CheckCondition(4 == rocksdb_options_get_bottommost_compression(copy));
CheckCondition(2 == rocksdb_options_get_compaction_style(copy));
CheckCondition(1 == rocksdb_options_get_atomic_flush(copy));
// Copies should be independent.
rocksdb_options_set_allow_ingest_behind(copy, 0);
@@ -1999,238 +1745,6 @@ int main(int argc, char** argv) {
CheckCondition(90001 == rocksdb_options_get_max_subcompactions(copy));
CheckCondition(123456 == rocksdb_options_get_max_subcompactions(o));
rocksdb_options_set_max_background_jobs(copy, 12);
CheckCondition(12 == rocksdb_options_get_max_background_jobs(copy));
CheckCondition(2 == rocksdb_options_get_max_background_jobs(o));
rocksdb_options_set_max_background_compactions(copy, 13);
CheckCondition(13 == rocksdb_options_get_max_background_compactions(copy));
CheckCondition(3 == rocksdb_options_get_max_background_compactions(o));
rocksdb_options_set_base_background_compactions(copy, 14);
CheckCondition(14 == rocksdb_options_get_base_background_compactions(copy));
CheckCondition(4 == rocksdb_options_get_base_background_compactions(o));
rocksdb_options_set_max_background_flushes(copy, 15);
CheckCondition(15 == rocksdb_options_get_max_background_flushes(copy));
CheckCondition(5 == rocksdb_options_get_max_background_flushes(o));
rocksdb_options_set_max_log_file_size(copy, 16);
CheckCondition(16 == rocksdb_options_get_max_log_file_size(copy));
CheckCondition(6 == rocksdb_options_get_max_log_file_size(o));
rocksdb_options_set_log_file_time_to_roll(copy, 17);
CheckCondition(17 == rocksdb_options_get_log_file_time_to_roll(copy));
CheckCondition(7 == rocksdb_options_get_log_file_time_to_roll(o));
rocksdb_options_set_keep_log_file_num(copy, 18);
CheckCondition(18 == rocksdb_options_get_keep_log_file_num(copy));
CheckCondition(8 == rocksdb_options_get_keep_log_file_num(o));
rocksdb_options_set_recycle_log_file_num(copy, 19);
CheckCondition(19 == rocksdb_options_get_recycle_log_file_num(copy));
CheckCondition(9 == rocksdb_options_get_recycle_log_file_num(o));
rocksdb_options_set_soft_rate_limit(copy, 4.0);
CheckCondition(4.0 == rocksdb_options_get_soft_rate_limit(copy));
CheckCondition(2.0 == rocksdb_options_get_soft_rate_limit(o));
rocksdb_options_set_hard_rate_limit(copy, 2.0);
CheckCondition(2.0 == rocksdb_options_get_hard_rate_limit(copy));
CheckCondition(4.0 == rocksdb_options_get_hard_rate_limit(o));
rocksdb_options_set_soft_pending_compaction_bytes_limit(copy, 110);
CheckCondition(
110 == rocksdb_options_get_soft_pending_compaction_bytes_limit(copy));
CheckCondition(10 ==
rocksdb_options_get_soft_pending_compaction_bytes_limit(o));
rocksdb_options_set_hard_pending_compaction_bytes_limit(copy, 111);
CheckCondition(
111 == rocksdb_options_get_hard_pending_compaction_bytes_limit(copy));
CheckCondition(11 ==
rocksdb_options_get_hard_pending_compaction_bytes_limit(o));
rocksdb_options_set_rate_limit_delay_max_milliseconds(copy, 0);
CheckCondition(0 ==
rocksdb_options_get_rate_limit_delay_max_milliseconds(copy));
CheckCondition(1 ==
rocksdb_options_get_rate_limit_delay_max_milliseconds(o));
rocksdb_options_set_max_manifest_file_size(copy, 112);
CheckCondition(112 == rocksdb_options_get_max_manifest_file_size(copy));
CheckCondition(12 == rocksdb_options_get_max_manifest_file_size(o));
rocksdb_options_set_table_cache_numshardbits(copy, 113);
CheckCondition(113 == rocksdb_options_get_table_cache_numshardbits(copy));
CheckCondition(13 == rocksdb_options_get_table_cache_numshardbits(o));
rocksdb_options_set_arena_block_size(copy, 114);
CheckCondition(114 == rocksdb_options_get_arena_block_size(copy));
CheckCondition(14 == rocksdb_options_get_arena_block_size(o));
rocksdb_options_set_use_fsync(copy, 0);
CheckCondition(0 == rocksdb_options_get_use_fsync(copy));
CheckCondition(1 == rocksdb_options_get_use_fsync(o));
rocksdb_options_set_WAL_ttl_seconds(copy, 115);
CheckCondition(115 == rocksdb_options_get_WAL_ttl_seconds(copy));
CheckCondition(15 == rocksdb_options_get_WAL_ttl_seconds(o));
rocksdb_options_set_WAL_size_limit_MB(copy, 116);
CheckCondition(116 == rocksdb_options_get_WAL_size_limit_MB(copy));
CheckCondition(16 == rocksdb_options_get_WAL_size_limit_MB(o));
rocksdb_options_set_manifest_preallocation_size(copy, 117);
CheckCondition(117 ==
rocksdb_options_get_manifest_preallocation_size(copy));
CheckCondition(17 == rocksdb_options_get_manifest_preallocation_size(o));
rocksdb_options_set_allow_mmap_reads(copy, 0);
CheckCondition(0 == rocksdb_options_get_allow_mmap_reads(copy));
CheckCondition(1 == rocksdb_options_get_allow_mmap_reads(o));
rocksdb_options_set_allow_mmap_writes(copy, 0);
CheckCondition(0 == rocksdb_options_get_allow_mmap_writes(copy));
CheckCondition(1 == rocksdb_options_get_allow_mmap_writes(o));
rocksdb_options_set_use_direct_reads(copy, 0);
CheckCondition(0 == rocksdb_options_get_use_direct_reads(copy));
CheckCondition(1 == rocksdb_options_get_use_direct_reads(o));
rocksdb_options_set_use_direct_io_for_flush_and_compaction(copy, 0);
CheckCondition(
0 == rocksdb_options_get_use_direct_io_for_flush_and_compaction(copy));
CheckCondition(
1 == rocksdb_options_get_use_direct_io_for_flush_and_compaction(o));
rocksdb_options_set_is_fd_close_on_exec(copy, 0);
CheckCondition(0 == rocksdb_options_get_is_fd_close_on_exec(copy));
CheckCondition(1 == rocksdb_options_get_is_fd_close_on_exec(o));
rocksdb_options_set_skip_log_error_on_recovery(copy, 0);
CheckCondition(0 == rocksdb_options_get_skip_log_error_on_recovery(copy));
CheckCondition(1 == rocksdb_options_get_skip_log_error_on_recovery(o));
rocksdb_options_set_stats_dump_period_sec(copy, 218);
CheckCondition(218 == rocksdb_options_get_stats_dump_period_sec(copy));
CheckCondition(18 == rocksdb_options_get_stats_dump_period_sec(o));
rocksdb_options_set_stats_persist_period_sec(copy, 600);
CheckCondition(600 == rocksdb_options_get_stats_persist_period_sec(copy));
CheckCondition(5 == rocksdb_options_get_stats_persist_period_sec(o));
rocksdb_options_set_advise_random_on_open(copy, 0);
CheckCondition(0 == rocksdb_options_get_advise_random_on_open(copy));
CheckCondition(1 == rocksdb_options_get_advise_random_on_open(o));
rocksdb_options_set_access_hint_on_compaction_start(copy, 2);
CheckCondition(2 ==
rocksdb_options_get_access_hint_on_compaction_start(copy));
CheckCondition(3 == rocksdb_options_get_access_hint_on_compaction_start(o));
rocksdb_options_set_use_adaptive_mutex(copy, 0);
CheckCondition(0 == rocksdb_options_get_use_adaptive_mutex(copy));
CheckCondition(1 == rocksdb_options_get_use_adaptive_mutex(o));
rocksdb_options_set_bytes_per_sync(copy, 219);
CheckCondition(219 == rocksdb_options_get_bytes_per_sync(copy));
CheckCondition(19 == rocksdb_options_get_bytes_per_sync(o));
rocksdb_options_set_wal_bytes_per_sync(copy, 120);
CheckCondition(120 == rocksdb_options_get_wal_bytes_per_sync(copy));
CheckCondition(20 == rocksdb_options_get_wal_bytes_per_sync(o));
rocksdb_options_set_writable_file_max_buffer_size(copy, 121);
CheckCondition(121 ==
rocksdb_options_get_writable_file_max_buffer_size(copy));
CheckCondition(21 == rocksdb_options_get_writable_file_max_buffer_size(o));
rocksdb_options_set_allow_concurrent_memtable_write(copy, 0);
CheckCondition(0 ==
rocksdb_options_get_allow_concurrent_memtable_write(copy));
CheckCondition(1 == rocksdb_options_get_allow_concurrent_memtable_write(o));
rocksdb_options_set_enable_write_thread_adaptive_yield(copy, 0);
CheckCondition(
0 == rocksdb_options_get_enable_write_thread_adaptive_yield(copy));
CheckCondition(1 ==
rocksdb_options_get_enable_write_thread_adaptive_yield(o));
rocksdb_options_set_max_sequential_skip_in_iterations(copy, 122);
CheckCondition(122 ==
rocksdb_options_get_max_sequential_skip_in_iterations(copy));
CheckCondition(22 ==
rocksdb_options_get_max_sequential_skip_in_iterations(o));
rocksdb_options_set_disable_auto_compactions(copy, 0);
CheckCondition(0 == rocksdb_options_get_disable_auto_compactions(copy));
CheckCondition(1 == rocksdb_options_get_disable_auto_compactions(o));
rocksdb_options_set_optimize_filters_for_hits(copy, 0);
CheckCondition(0 == rocksdb_options_get_optimize_filters_for_hits(copy));
CheckCondition(1 == rocksdb_options_get_optimize_filters_for_hits(o));
rocksdb_options_set_delete_obsolete_files_period_micros(copy, 123);
CheckCondition(
123 == rocksdb_options_get_delete_obsolete_files_period_micros(copy));
CheckCondition(23 ==
rocksdb_options_get_delete_obsolete_files_period_micros(o));
rocksdb_options_set_memtable_prefix_bloom_size_ratio(copy, 4.0);
CheckCondition(4.0 ==
rocksdb_options_get_memtable_prefix_bloom_size_ratio(copy));
CheckCondition(2.0 ==
rocksdb_options_get_memtable_prefix_bloom_size_ratio(o));
rocksdb_options_set_max_compaction_bytes(copy, 124);
CheckCondition(124 == rocksdb_options_get_max_compaction_bytes(copy));
CheckCondition(24 == rocksdb_options_get_max_compaction_bytes(o));
rocksdb_options_set_memtable_huge_page_size(copy, 125);
CheckCondition(125 == rocksdb_options_get_memtable_huge_page_size(copy));
CheckCondition(25 == rocksdb_options_get_memtable_huge_page_size(o));
rocksdb_options_set_max_successive_merges(copy, 126);
CheckCondition(126 == rocksdb_options_get_max_successive_merges(copy));
CheckCondition(26 == rocksdb_options_get_max_successive_merges(o));
rocksdb_options_set_bloom_locality(copy, 127);
CheckCondition(127 == rocksdb_options_get_bloom_locality(copy));
CheckCondition(27 == rocksdb_options_get_bloom_locality(o));
rocksdb_options_set_inplace_update_support(copy, 0);
CheckCondition(0 == rocksdb_options_get_inplace_update_support(copy));
CheckCondition(1 == rocksdb_options_get_inplace_update_support(o));
rocksdb_options_set_inplace_update_num_locks(copy, 128);
CheckCondition(128 == rocksdb_options_get_inplace_update_num_locks(copy));
CheckCondition(28 == rocksdb_options_get_inplace_update_num_locks(o));
rocksdb_options_set_report_bg_io_stats(copy, 0);
CheckCondition(0 == rocksdb_options_get_report_bg_io_stats(copy));
CheckCondition(1 == rocksdb_options_get_report_bg_io_stats(o));
rocksdb_options_set_wal_recovery_mode(copy, 1);
CheckCondition(1 == rocksdb_options_get_wal_recovery_mode(copy));
CheckCondition(2 == rocksdb_options_get_wal_recovery_mode(o));
rocksdb_options_set_compression(copy, 4);
CheckCondition(4 == rocksdb_options_get_compression(copy));
CheckCondition(5 == rocksdb_options_get_compression(o));
rocksdb_options_set_bottommost_compression(copy, 3);
CheckCondition(3 == rocksdb_options_get_bottommost_compression(copy));
CheckCondition(4 == rocksdb_options_get_bottommost_compression(o));
rocksdb_options_set_compaction_style(copy, 1);
CheckCondition(1 == rocksdb_options_get_compaction_style(copy));
CheckCondition(2 == rocksdb_options_get_compaction_style(o));
rocksdb_options_set_atomic_flush(copy, 0);
CheckCondition(0 == rocksdb_options_get_atomic_flush(copy));
CheckCondition(1 == rocksdb_options_get_atomic_flush(o));
rocksdb_options_destroy(copy);
rocksdb_options_destroy(o);
}
+10 -14
View File
@@ -34,7 +34,6 @@
#include "table/block_based/block_based_table_factory.h"
#include "table/merging_iterator.h"
#include "util/autovector.h"
#include "util/cast_util.h"
#include "util/compression.h"
namespace ROCKSDB_NAMESPACE {
@@ -347,7 +346,7 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
}
bool is_block_based_table =
(result.table_factory->Name() == BlockBasedTableFactory::kName);
(result.table_factory->Name() == BlockBasedTableFactory().Name());
const uint64_t kAdjustedTtl = 30 * 24 * 60 * 60;
if (result.ttl == kDefaultTtl) {
@@ -1047,14 +1046,13 @@ bool ColumnFamilyData::NeedsCompaction() const {
}
Compaction* ColumnFamilyData::PickCompaction(
const MutableCFOptions& mutable_options,
const MutableDBOptions& mutable_db_options, LogBuffer* log_buffer) {
const MutableCFOptions& mutable_options, LogBuffer* log_buffer) {
SequenceNumber earliest_mem_seqno =
std::min(mem_->GetEarliestSequenceNumber(),
imm_.current()->GetEarliestSequenceNumber(false));
auto* result = compaction_picker_->PickCompaction(
GetName(), mutable_options, mutable_db_options, current_->storage_info(),
log_buffer, earliest_mem_seqno);
GetName(), mutable_options, current_->storage_info(), log_buffer,
earliest_mem_seqno);
if (result != nullptr) {
result->SetInputVersion(current_);
}
@@ -1124,16 +1122,14 @@ const int ColumnFamilyData::kCompactAllLevels = -1;
const int ColumnFamilyData::kCompactToBaseLevel = -2;
Compaction* ColumnFamilyData::CompactRange(
const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, int input_level,
const MutableCFOptions& mutable_cf_options, int input_level,
int output_level, const CompactRangeOptions& compact_range_options,
const InternalKey* begin, const InternalKey* end,
InternalKey** compaction_end, bool* conflict,
uint64_t max_file_num_to_ignore) {
auto* result = compaction_picker_->CompactRange(
GetName(), mutable_cf_options, mutable_db_options,
current_->storage_info(), input_level, output_level,
compact_range_options, begin, end, compaction_end, conflict,
GetName(), mutable_cf_options, current_->storage_info(), input_level,
output_level, compact_range_options, begin, end, compaction_end, conflict,
max_file_num_to_ignore);
if (result != nullptr) {
result->SetInputVersion(current_);
@@ -1303,7 +1299,7 @@ Status ColumnFamilyData::ValidateOptions(
}
if (cf_options.ttl > 0 && cf_options.ttl != kDefaultTtl) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory::kName) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory().Name()) {
return Status::NotSupported(
"TTL is only supported in Block-Based Table format. ");
}
@@ -1311,7 +1307,7 @@ Status ColumnFamilyData::ValidateOptions(
if (cf_options.periodic_compaction_seconds > 0 &&
cf_options.periodic_compaction_seconds != kDefaultPeriodicCompSecs) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory::kName) {
if (cf_options.table_factory->Name() != BlockBasedTableFactory().Name()) {
return Status::NotSupported(
"Periodic Compaction is only supported in "
"Block-Based Table format. ");
@@ -1550,7 +1546,7 @@ ColumnFamilyHandle* ColumnFamilyMemTablesImpl::GetColumnFamilyHandle() {
uint32_t GetColumnFamilyID(ColumnFamilyHandle* column_family) {
uint32_t column_family_id = 0;
if (column_family != nullptr) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
column_family_id = cfh->GetID();
}
return column_family_id;
-2
View File
@@ -387,7 +387,6 @@ class ColumnFamilyData {
bool NeedsCompaction() const;
// REQUIRES: DB mutex held
Compaction* PickCompaction(const MutableCFOptions& mutable_options,
const MutableDBOptions& mutable_db_options,
LogBuffer* log_buffer);
// Check if the passed range overlap with any running compactions.
@@ -413,7 +412,6 @@ class ColumnFamilyData {
static const int kCompactToBaseLevel;
// REQUIRES: DB mutex held
Compaction* CompactRange(const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options,
int input_level, int output_level,
const CompactRangeOptions& compact_range_options,
const InternalKey* begin, const InternalKey* end,
+19 -23
View File
@@ -22,18 +22,26 @@
#include "rocksdb/env.h"
#include "rocksdb/iterator.h"
#include "rocksdb/utilities/object_registry.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/coding.h"
#include "util/string_util.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators.h"
namespace ROCKSDB_NAMESPACE {
static const int kValueSize = 1000;
namespace {
std::string RandomString(Random* rnd, int len) {
std::string r;
test::RandomString(rnd, len, &r);
return r;
}
} // anonymous namespace
// counts how many operations were performed
class EnvCounter : public EnvWrapper {
public:
@@ -101,11 +109,11 @@ class ColumnFamilyTestBase : public testing::Test {
// preserves the implementation that was in place when all of the
// magic values in this file were picked.
*storage = std::string(kValueSize, ' ');
return Slice(*storage);
} else {
Random r(k);
*storage = r.RandomString(kValueSize);
return test::RandomString(&r, kValueSize, storage);
}
return Slice(*storage);
}
void Build(int base, int n, int flush_every = 0) {
@@ -114,7 +122,7 @@ class ColumnFamilyTestBase : public testing::Test {
for (int i = 0; i < n; i++) {
if (flush_every != 0 && i != 0 && i % flush_every == 0) {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
}
@@ -219,7 +227,7 @@ class ColumnFamilyTestBase : public testing::Test {
Open({"default"});
}
DBImpl* dbfull() { return static_cast_with_check<DBImpl>(db_); }
DBImpl* dbfull() { return reinterpret_cast<DBImpl*>(db_); }
int GetProperty(int cf, std::string property) {
std::string value;
@@ -321,11 +329,11 @@ class ColumnFamilyTestBase : public testing::Test {
// 10 bytes for key, rest is value
if (!save) {
ASSERT_OK(Put(cf, test::RandomKey(&rnd_, 11),
rnd_.RandomString(key_value_size - 10)));
RandomString(&rnd_, key_value_size - 10)));
} else {
std::string key = test::RandomKey(&rnd_, 11);
keys_[cf].insert(key);
ASSERT_OK(Put(cf, key, rnd_.RandomString(key_value_size - 10)));
ASSERT_OK(Put(cf, key, RandomString(&rnd_, key_value_size - 10)));
}
}
db_->FlushWAL(false);
@@ -562,7 +570,7 @@ TEST_P(ColumnFamilyTest, DontReuseColumnFamilyID) {
Open();
CreateColumnFamilies({"one", "two", "three"});
for (size_t i = 0; i < handles_.size(); ++i) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(handles_[i]);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(handles_[i]);
ASSERT_EQ(i, cfh->GetID());
}
if (iter == 1) {
@@ -578,7 +586,7 @@ TEST_P(ColumnFamilyTest, DontReuseColumnFamilyID) {
CreateColumnFamilies({"three2"});
// ID 3 that was used for dropped column family "three" should not be
// reused
auto cfh3 = static_cast_with_check<ColumnFamilyHandleImpl>(handles_[3]);
auto cfh3 = reinterpret_cast<ColumnFamilyHandleImpl*>(handles_[3]);
ASSERT_EQ(4U, cfh3->GetID());
Close();
Destroy();
@@ -2424,10 +2432,7 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task,
Env::Priority::HIGH);
// Make sure the task is sleeping. Otherwise, it might start to execute
// after sleeping_task.WaitUntilDone() and cause TSAN warning.
sleeping_task.WaitUntilSleeping();
// 1MB should create ~10 files for each CF
int kKeysNum = 10000;
PutRandomData(1, kKeysNum, 100);
@@ -2441,9 +2446,6 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
// now we sleep again. this is just so we're certain that flush job finished
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task,
Env::Priority::HIGH);
// Make sure the task is sleeping. Otherwise, it might start to execute
// after sleeping_task.WaitUntilDone() and cause TSAN warning.
sleeping_task.WaitUntilSleeping();
sleeping_task.WakeUp();
sleeping_task.WaitUntilDone();
@@ -2993,9 +2995,6 @@ TEST_P(ColumnFamilyTest, FlushCloseWALFiles) {
test::SleepingBackgroundTask sleeping_task;
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task,
Env::Priority::HIGH);
// Make sure the task is sleeping. Otherwise, it might start to execute
// after sleeping_task.WaitUntilDone() and cause TSAN warning.
sleeping_task.WaitUntilSleeping();
WriteOptions wo;
wo.sync = true;
@@ -3041,9 +3040,6 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile1) {
test::SleepingBackgroundTask sleeping_task;
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task,
Env::Priority::HIGH);
// Make sure the task is sleeping. Otherwise, it might start to execute
// after sleeping_task.WaitUntilDone() and cause TSAN warning.
sleeping_task.WaitUntilSleeping();
WriteOptions wo;
wo.sync = true;
@@ -3358,7 +3354,7 @@ TEST_P(ColumnFamilyTest, MultipleCFPathsTest) {
// Re-open and verify the keys.
Reopen({ColumnFamilyOptions(), cf_opt1, cf_opt2});
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
for (int cf = 1; cf != 3; ++cf) {
ReadOptions read_options;
read_options.readahead_size = 0;
+4 -5
View File
@@ -16,7 +16,6 @@
#include "rocksdb/env.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "util/cast_util.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -149,7 +148,7 @@ TEST_F(CompactFilesTest, ObsoleteFiles) {
auto l0_files = collector->GetFlushedFiles();
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files, 1));
static_cast_with_check<DBImpl>(db)->TEST_WaitForCompact();
reinterpret_cast<DBImpl*>(db)->TEST_WaitForCompact();
// verify all compaction input files are deleted
for (auto fname : l0_files) {
@@ -184,13 +183,13 @@ TEST_F(CompactFilesTest, NotCutOutputOnLevel0) {
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
reinterpret_cast<DBImpl*>(db)->TEST_WaitForFlushMemTable();
auto l0_files_1 = collector->GetFlushedFiles();
collector->ClearFlushedFiles();
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
reinterpret_cast<DBImpl*>(db)->TEST_WaitForFlushMemTable();
auto l0_files_2 = collector->GetFlushedFiles();
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files_1, 0));
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files_2, 0));
@@ -384,7 +383,7 @@ TEST_F(CompactFilesTest, GetCompactionJobInfo) {
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
reinterpret_cast<DBImpl*>(db)->TEST_WaitForFlushMemTable();
auto l0_files_1 = collector->GetFlushedFiles();
CompactionOptions co;
co.compression = CompressionType::kLZ4Compression;
+2 -4
View File
@@ -5,11 +5,9 @@
#ifndef ROCKSDB_LITE
#include "db/compacted_db_impl.h"
#include "db/db_impl/db_impl.h"
#include "db/version_set.h"
#include "table/get_context.h"
#include "util/cast_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -92,8 +90,8 @@ Status CompactedDBImpl::Init(const Options& options) {
ColumnFamilyOptions(options));
Status s = Recover({cf}, true /* read only */, false, true);
if (s.ok()) {
cfd_ = static_cast_with_check<ColumnFamilyHandleImpl>(DefaultColumnFamily())
->cfd();
cfd_ = reinterpret_cast<ColumnFamilyHandleImpl*>(
DefaultColumnFamily())->cfd();
cfd_->InstallSuperVersion(&sv_context, &mutex_);
}
mutex_.Unlock();
+2 -29
View File
@@ -7,14 +7,12 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction.h"
#include <cinttypes>
#include <vector>
#include "db/column_family.h"
#include "db/compaction/compaction.h"
#include "rocksdb/compaction_filter.h"
#include "rocksdb/sst_partitioner.h"
#include "test_util/sync_point.h"
#include "util/string_util.h"
@@ -207,7 +205,6 @@ bool Compaction::IsFullCompaction(
Compaction::Compaction(VersionStorageInfo* vstorage,
const ImmutableCFOptions& _immutable_cf_options,
const MutableCFOptions& _mutable_cf_options,
const MutableDBOptions& _mutable_db_options,
std::vector<CompactionInputFiles> _inputs,
int _output_level, uint64_t _target_file_size,
uint64_t _max_compaction_bytes, uint32_t _output_path_id,
@@ -246,7 +243,7 @@ Compaction::Compaction(VersionStorageInfo* vstorage,
compaction_reason_ = CompactionReason::kManualCompaction;
}
if (max_subcompactions_ == 0) {
max_subcompactions_ = _mutable_db_options.max_subcompactions;
max_subcompactions_ = immutable_cf_options_.max_subcompactions;
}
if (!bottommost_level_) {
// Currently we only enable dictionary compression during compaction to the
@@ -331,8 +328,6 @@ bool Compaction::IsTrivialMove() const {
// assert inputs_.size() == 1
std::unique_ptr<SstPartitioner> partitioner = CreateSstPartitioner();
for (const auto& file : inputs_.front().files) {
std::vector<FileMetaData*> file_grand_parents;
if (output_level_ + 1 >= number_levels_) {
@@ -345,13 +340,6 @@ bool Compaction::IsTrivialMove() const {
if (compaction_size > max_compaction_bytes_) {
return false;
}
if (partitioner.get() != nullptr) {
if (!partitioner->CanDoTrivialMove(file->smallest.user_key(),
file->largest.user_key())) {
return false;
}
}
}
return true;
@@ -537,21 +525,6 @@ std::unique_ptr<CompactionFilter> Compaction::CreateCompactionFilter() const {
context);
}
std::unique_ptr<SstPartitioner> Compaction::CreateSstPartitioner() const {
if (!immutable_cf_options_.sst_partitioner_factory) {
return nullptr;
}
SstPartitioner::Context context;
context.is_full_compaction = is_full_compaction_;
context.is_manual_compaction = is_manual_compaction_;
context.output_level = output_level_;
context.smallest_user_key = smallest_user_key_;
context.largest_user_key = largest_user_key_;
return immutable_cf_options_.sst_partitioner_factory->CreatePartitioner(
context);
}
bool Compaction::IsOutputLevelEmpty() const {
return inputs_.back().level != output_level_ || inputs_.back().empty();
}
-5
View File
@@ -11,7 +11,6 @@
#include "db/version_set.h"
#include "memory/arena.h"
#include "options/cf_options.h"
#include "rocksdb/sst_partitioner.h"
#include "util/autovector.h"
namespace ROCKSDB_NAMESPACE {
@@ -72,7 +71,6 @@ class Compaction {
Compaction(VersionStorageInfo* input_version,
const ImmutableCFOptions& immutable_cf_options,
const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options,
std::vector<CompactionInputFiles> inputs, int output_level,
uint64_t target_file_size, uint64_t max_compaction_bytes,
uint32_t output_path_id, CompressionType compression,
@@ -257,9 +255,6 @@ class Compaction {
// Create a CompactionFilter from compaction_filter_factory
std::unique_ptr<CompactionFilter> CreateCompactionFilter() const;
// Create a SstPartitioner from sst_partitioner_factory
std::unique_ptr<SstPartitioner> CreateSstPartitioner() const;
// Is the input level corresponding to output_level_ empty?
bool IsOutputLevelEmpty() const;
+5 -24
View File
@@ -182,7 +182,7 @@ void CompactionIterator::Next() {
PrepareOutput();
}
bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
void CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
Slice* skip_until) {
if (compaction_filter_ != nullptr &&
(ikey_.type == kTypeValue || ikey_.type == kTypeBlobIndex)) {
@@ -225,31 +225,14 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
value_.clear();
iter_stats_.num_record_drop_user++;
} else if (filter == CompactionFilter::Decision::kChangeValue) {
if (ikey_.type == kTypeBlobIndex) {
// value transfer from blob file to inlined data
ikey_.type = kTypeValue;
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
}
value_ = compaction_filter_value_;
} else if (filter == CompactionFilter::Decision::kRemoveAndSkipUntil) {
*need_skip = true;
compaction_filter_skip_until_.ConvertFromUserKey(kMaxSequenceNumber,
kValueTypeForSeek);
*skip_until = compaction_filter_skip_until_.Encode();
} else if (filter == CompactionFilter::Decision::kChangeBlobIndex) {
if (ikey_.type == kTypeValue) {
// value transfer from inlined data to blob file
ikey_.type = kTypeBlobIndex;
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
}
value_ = compaction_filter_value_;
} else if (filter == CompactionFilter::Decision::kIOError) {
status_ =
Status::IOError("Failed to access blob during compaction filter");
return false;
}
}
return true;
}
void CompactionIterator::NextFromInput() {
@@ -312,9 +295,8 @@ void CompactionIterator::NextFromInput() {
// Apply the compaction filter to the first committed version of the user
// key.
if (current_key_committed_ &&
!InvokeFilterIfNeeded(&need_skip, &skip_until)) {
break;
if (current_key_committed_) {
InvokeFilterIfNeeded(&need_skip, &skip_until);
}
} else {
// Update the current key to reflect the new sequence number/type without
@@ -334,9 +316,8 @@ void CompactionIterator::NextFromInput() {
current_key_committed_ = KeyCommitted(ikey_.sequence);
// Apply the compaction filter to the first committed version of the
// user key.
if (current_key_committed_ &&
!InvokeFilterIfNeeded(&need_skip, &skip_until)) {
break;
if (current_key_committed_) {
InvokeFilterIfNeeded(&need_skip, &skip_until);
}
}
}
+1 -2
View File
@@ -121,8 +121,7 @@ class CompactionIterator {
void PrepareOutput();
// Invoke compaction filter if needed.
// Return true on success, false on failures (e.g.: kIOError).
bool InvokeFilterIfNeeded(bool* need_skip, Slice* skip_until);
void InvokeFilterIfNeeded(bool* need_skip, Slice* skip_until);
// Given a sequence number, return the sequence number of the
// earliest snapshot that this sequence number is visible in.
+37 -100
View File
@@ -7,8 +7,6 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction_job.h"
#include <algorithm>
#include <cinttypes>
#include <functional>
@@ -21,6 +19,7 @@
#include <vector>
#include "db/builder.h"
#include "db/compaction/compaction_job.h"
#include "db/db_impl/db_impl.h"
#include "db/db_iter.h"
#include "db/dbformat.h"
@@ -46,7 +45,6 @@
#include "port/port.h"
#include "rocksdb/db.h"
#include "rocksdb/env.h"
#include "rocksdb/sst_partitioner.h"
#include "rocksdb/statistics.h"
#include "rocksdb/status.h"
#include "rocksdb/table.h"
@@ -56,7 +54,6 @@
#include "table/table_builder.h"
#include "test_util/sync_point.h"
#include "util/coding.h"
#include "util/hash.h"
#include "util/mutexlock.h"
#include "util/random.h"
#include "util/stop_watch.h"
@@ -119,14 +116,10 @@ struct CompactionJob::SubcompactionState {
// The return status of this subcompaction
Status status;
// The return IO Status of this subcompaction
IOStatus io_status;
// Files produced by this subcompaction
struct Output {
FileMetaData meta;
bool finished;
uint64_t paranoid_hash;
std::shared_ptr<const TableProperties> table_properties;
};
@@ -136,7 +129,7 @@ struct CompactionJob::SubcompactionState {
std::unique_ptr<TableBuilder> builder;
Output* current_output() {
if (outputs.empty()) {
// This subcompaction's output could be empty if compaction was aborted
// This subcompaction's outptut could be empty if compaction was aborted
// before this subcompaction had a chance to generate any output files.
// When subcompactions are executed sequentially this is more likely and
// will be particulalry likely for the later subcompactions to be empty.
@@ -186,7 +179,6 @@ struct CompactionJob::SubcompactionState {
start = std::move(o.start);
end = std::move(o.end);
status = std::move(o.status);
io_status = std::move(o.io_status);
outputs = std::move(o.outputs);
outfile = std::move(o.outfile);
builder = std::move(o.builder);
@@ -206,21 +198,6 @@ struct CompactionJob::SubcompactionState {
SubcompactionState& operator=(const SubcompactionState&) = delete;
// Adds the key and value to the builder
// If paranoid is true, adds the key-value to the paranoid hash
void AddToBuilder(const Slice& key, const Slice& value, bool paranoid) {
auto curr = current_output();
assert(builder != nullptr);
assert(curr != nullptr);
if (paranoid) {
// Generate a rolling 64-bit hash of the key and values
curr->paranoid_hash = Hash64(key.data(), key.size(), curr->paranoid_hash);
curr->paranoid_hash =
Hash64(value.data(), value.size(), curr->paranoid_hash);
}
builder->Add(key, value);
}
// Returns true iff we should stop building the current output
// before processing "internal_key".
bool ShouldStopBefore(const Slice& internal_key, uint64_t curr_file_size) {
@@ -328,15 +305,12 @@ CompactionJob::CompactionJob(
const SnapshotChecker* snapshot_checker, std::shared_ptr<Cache> table_cache,
EventLogger* event_logger, bool paranoid_file_checks, bool measure_io_stats,
const std::string& dbname, CompactionJobStats* compaction_job_stats,
Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused,
const std::string& db_id, const std::string& db_session_id)
Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused)
: job_id_(job_id),
compact_(new CompactionState(compaction)),
compaction_job_stats_(compaction_job_stats),
compaction_stats_(compaction->compaction_reason(), 1),
dbname_(dbname),
db_id_(db_id),
db_session_id_(db_session_id),
db_options_(db_options),
file_options_(file_options),
env_(db_options.env),
@@ -632,42 +606,38 @@ Status CompactionJob::Run() {
// Check if any thread encountered an error during execution
Status status;
IOStatus io_s;
for (const auto& state : compact_->sub_compact_states) {
if (!state.status.ok()) {
status = state.status;
io_s = state.io_status;
break;
}
}
if (io_status_.ok()) {
io_status_ = io_s;
}
IOStatus io_s;
if (status.ok() && output_directory_) {
io_s = output_directory_->Fsync(IOOptions(), nullptr);
}
if (io_status_.ok()) {
if (!io_s.ok()) {
io_status_ = io_s;
}
if (status.ok()) {
status = io_s;
}
if (status.ok()) {
thread_pool.clear();
std::vector<const CompactionJob::SubcompactionState::Output*> files_output;
std::vector<const FileMetaData*> files_meta;
for (const auto& state : compact_->sub_compact_states) {
for (const auto& output : state.outputs) {
files_output.emplace_back(&output);
files_meta.emplace_back(&output.meta);
}
}
ColumnFamilyData* cfd = compact_->compaction->column_family_data();
auto prefix_extractor =
compact_->compaction->mutable_cf_options()->prefix_extractor.get();
std::atomic<size_t> next_file_idx(0);
std::atomic<size_t> next_file_meta_idx(0);
auto verify_table = [&](Status& output_status) {
while (true) {
size_t file_idx = next_file_idx.fetch_add(1);
if (file_idx >= files_output.size()) {
size_t file_idx = next_file_meta_idx.fetch_add(1);
if (file_idx >= files_meta.size()) {
break;
}
// Verify that the table is usable
@@ -678,8 +648,7 @@ Status CompactionJob::Run() {
// to cache it here for further user reads
InternalIterator* iter = cfd->table_cache()->NewIterator(
ReadOptions(), file_options_, cfd->internal_comparator(),
files_output[file_idx]->meta, /*range_del_agg=*/nullptr,
prefix_extractor,
*files_meta[file_idx], /*range_del_agg=*/nullptr, prefix_extractor,
/*table_reader_ptr=*/nullptr,
cfd->internal_stats()->GetFileReadHist(
compact_->compaction->output_level()),
@@ -693,16 +662,8 @@ Status CompactionJob::Run() {
auto s = iter->status();
if (s.ok() && paranoid_file_checks_) {
uint64_t hash = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
// Generate a rolling 64-bit hash of the key and values, using the
hash = Hash64(iter->key().data(), iter->key().size(), hash);
hash = Hash64(iter->value().data(), iter->value().size(), hash);
}
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {}
s = iter->status();
if (s.ok() && hash != files_output[file_idx]->paranoid_hash) {
s = Status::Corruption("Paraniod checksums do not match");
}
}
delete iter;
@@ -950,12 +911,6 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
}
const auto& c_iter_stats = c_iter->iter_stats();
std::unique_ptr<SstPartitioner> partitioner =
sub_compact->compaction->output_level() == 0
? nullptr
: sub_compact->compaction->CreateSstPartitioner();
std::string last_key_for_partitioner;
while (status.ok() && !cfd->IsDropped() && c_iter->Valid()) {
// Invariant: c_iter.status() is guaranteed to be OK if c_iter->Valid()
// returns true.
@@ -982,8 +937,9 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
break;
}
}
sub_compact->AddToBuilder(key, value, paranoid_file_checks_);
assert(sub_compact->builder != nullptr);
assert(sub_compact->current_output() != nullptr);
sub_compact->builder->Add(key, value);
sub_compact->current_output_file_size =
sub_compact->builder->EstimatedFileSize();
const ParsedInternalKey& ikey = c_iter->ikey();
@@ -1013,29 +969,20 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
"CompactionJob::Run():PausingManualCompaction:2",
reinterpret_cast<void*>(
const_cast<std::atomic<bool>*>(manual_compaction_paused_)));
if (partitioner.get()) {
last_key_for_partitioner.assign(c_iter->user_key().data_,
c_iter->user_key().size_);
}
c_iter->Next();
if (c_iter->status().IsManualCompactionPaused()) {
break;
}
if (!output_file_ended && c_iter->Valid()) {
if (((partitioner.get() &&
partitioner->ShouldPartition(PartitionerRequest(
last_key_for_partitioner, c_iter->user_key(),
sub_compact->current_output_file_size)) == kRequired) ||
(sub_compact->compaction->output_level() != 0 &&
sub_compact->ShouldStopBefore(
c_iter->key(), sub_compact->current_output_file_size))) &&
sub_compact->builder != nullptr) {
// (2) this key belongs to the next file. For historical reasons, the
// iterator status after advancing will be given to
// FinishCompactionOutputFile().
input_status = input->status();
output_file_ended = true;
}
if (!output_file_ended && c_iter->Valid() &&
sub_compact->compaction->output_level() != 0 &&
sub_compact->ShouldStopBefore(c_iter->key(),
sub_compact->current_output_file_size) &&
sub_compact->builder != nullptr) {
// (2) this key belongs to the next file. For historical reasons, the
// iterator status after advancing will be given to
// FinishCompactionOutputFile().
input_status = input->status();
output_file_ended = true;
}
if (output_file_ended) {
const Slice* next_key = nullptr;
@@ -1260,7 +1207,6 @@ Status CompactionJob::FinishCompactionOutputFile(
} else {
it->SeekToFirst();
}
TEST_SYNC_POINT("CompactionJob::FinishCompactionOutputFile1");
for (; it->Valid(); it->Next()) {
auto tombstone = it->Tombstone();
if (upper_bound != nullptr) {
@@ -1288,7 +1234,6 @@ Status CompactionJob::FinishCompactionOutputFile(
auto kv = tombstone.Serialize();
assert(lower_bound == nullptr ||
ucmp->Compare(*lower_bound, kv.second) < 0);
// Range tombstone is not supported by output validator yet.
sub_compact->builder->Add(kv.first.Encode(), kv.second);
InternalKey smallest_candidate = std::move(kv.first);
if (lower_bound != nullptr &&
@@ -1362,9 +1307,9 @@ Status CompactionJob::FinishCompactionOutputFile(
} else {
sub_compact->builder->Abandon();
}
IOStatus io_s = sub_compact->builder->io_status();
if (s.ok()) {
s = io_s;
if (!sub_compact->builder->io_status().ok()) {
io_status_ = sub_compact->builder->io_status();
s = io_status_;
}
const uint64_t current_bytes = sub_compact->builder->FileSize();
if (s.ok()) {
@@ -1374,25 +1319,24 @@ Status CompactionJob::FinishCompactionOutputFile(
sub_compact->total_bytes += current_bytes;
// Finish and check for file errors
IOStatus io_s;
if (s.ok()) {
StopWatch sw(env_, stats_, COMPACTION_OUTFILE_SYNC_MICROS);
io_s = sub_compact->outfile->Sync(db_options_.use_fsync);
}
if (s.ok() && io_s.ok()) {
if (io_s.ok()) {
io_s = sub_compact->outfile->Close();
}
if (s.ok() && io_s.ok()) {
if (io_s.ok()) {
// Add the checksum information to file metadata.
meta->file_checksum = sub_compact->outfile->GetFileChecksum();
meta->file_checksum_func_name =
sub_compact->outfile->GetFileChecksumFuncName();
}
if (s.ok()) {
if (!io_s.ok()) {
io_status_ = io_s;
s = io_s;
}
if (sub_compact->io_status.ok()) {
sub_compact->io_status = io_s;
}
sub_compact->outfile.reset();
TableProperties tp;
@@ -1541,12 +1485,7 @@ Status CompactionJob::OpenCompactionOutputFile(
TEST_SYNC_POINT_CALLBACK("CompactionJob::OpenCompactionOutputFile",
&syncpoint_arg);
#endif
Status s;
IOStatus io_s = NewWritableFile(fs_, fname, &writable_file, file_options_);
s = io_s;
if (sub_compact->io_status.ok()) {
sub_compact->io_status = io_s;
}
Status s = NewWritableFile(fs_, fname, &writable_file, file_options_);
if (!s.ok()) {
ROCKS_LOG_ERROR(
db_options_.info_log,
@@ -1586,7 +1525,6 @@ Status CompactionJob::OpenCompactionOutputFile(
out.meta.oldest_ancester_time = oldest_ancester_time;
out.meta.file_creation_time = current_time;
out.finished = false;
out.paranoid_hash = 0;
sub_compact->outputs.push_back(out);
}
@@ -1616,8 +1554,7 @@ Status CompactionJob::OpenCompactionOutputFile(
sub_compact->compaction->output_compression_opts(),
sub_compact->compaction->output_level(), skip_filters,
oldest_ancester_time, 0 /* oldest_key_time */,
sub_compact->compaction->max_output_file_size(), current_time, db_id_,
db_session_id_));
sub_compact->compaction->max_output_file_size(), current_time));
LogFlush(db_options_.info_log);
return s;
}
+1 -5
View File
@@ -78,9 +78,7 @@ class CompactionJob {
const std::string& dbname,
CompactionJobStats* compaction_job_stats,
Env::Priority thread_pri,
const std::atomic<bool>* manual_compaction_paused = nullptr,
const std::string& db_id = "",
const std::string& db_session_id = "");
const std::atomic<bool>* manual_compaction_paused = nullptr);
~CompactionJob();
@@ -154,8 +152,6 @@ class CompactionJob {
// DBImpl state
const std::string& dbname_;
const std::string db_id_;
const std::string db_session_id_;
const ImmutableDBOptions& db_options_;
const FileOptions file_options_;
+5 -4
View File
@@ -52,7 +52,6 @@
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/cast_util.h"
#include "util/compression.h"
#include "util/hash.h"
#include "util/mutexlock.h"
@@ -127,7 +126,9 @@ class CompactionJobStatsTest : public testing::Test,
static void SetUpTestCase() {}
static void TearDownTestCase() {}
DBImpl* dbfull() { return static_cast_with_check<DBImpl>(db_); }
DBImpl* dbfull() {
return reinterpret_cast<DBImpl*>(db_);
}
void CreateColumnFamilies(const std::vector<std::string>& cfs,
const Options& options) {
@@ -796,7 +797,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
}
ASSERT_OK(Flush(1));
static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact();
reinterpret_cast<DBImpl*>(db_)->TEST_WaitForCompact();
stats_checker->set_verify_next_comp_io_stats(true);
std::atomic<bool> first_prepare_write(true);
@@ -1011,7 +1012,7 @@ TEST_P(CompactionJobStatsTest, UniversalCompactionTest) {
&rnd, start_key, start_key + key_base - 1,
kKeySize, kValueSize, key_interval,
compression_ratio, 1);
static_cast_with_check<DBImpl>(db_)->TEST_WaitForCompact();
reinterpret_cast<DBImpl*>(db_)->TEST_WaitForCompact();
}
ASSERT_EQ(stats_checker->NumberOfUnverifiedStats(), 0U);
}
+3 -6
View File
@@ -76,7 +76,6 @@ class CompactionJobTest : public testing::Test {
dbname_(test::PerThreadDBPath("compaction_job_test")),
db_options_(),
mutable_cf_options_(cf_options_),
mutable_db_options_(),
table_cache_(NewLRUCache(50000, 16)),
write_buffer_manager_(db_options_.db_write_buffer_size),
versions_(new VersionSet(dbname_, &db_options_, env_options_,
@@ -317,10 +316,9 @@ class CompactionJobTest : public testing::Test {
Compaction compaction(
cfd->current()->storage_info(), *cfd->ioptions(),
*cfd->GetLatestMutableCFOptions(), mutable_db_options_,
compaction_input_files, output_level, 1024 * 1024, 10 * 1024 * 1024, 0,
kNoCompression, cfd->GetLatestMutableCFOptions()->compression_opts, 0,
{}, true);
*cfd->GetLatestMutableCFOptions(), compaction_input_files, output_level,
1024 * 1024, 10 * 1024 * 1024, 0, kNoCompression,
cfd->GetLatestMutableCFOptions()->compression_opts, 0, {}, true);
compaction.SetInputVersion(cfd->current());
LogBuffer log_buffer(InfoLogLevel::INFO_LEVEL, db_options_.info_log.get());
@@ -372,7 +370,6 @@ class CompactionJobTest : public testing::Test {
ImmutableDBOptions db_options_;
ColumnFamilyOptions cf_options_;
MutableCFOptions mutable_cf_options_;
MutableDBOptions mutable_db_options_;
std::shared_ptr<Cache> table_cache_;
WriteController write_controller_;
WriteBufferManager write_buffer_manager_;
+8 -9
View File
@@ -332,7 +332,7 @@ Compaction* CompactionPicker::CompactFiles(
const CompactionOptions& compact_options,
const std::vector<CompactionInputFiles>& input_files, int output_level,
VersionStorageInfo* vstorage, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, uint32_t output_path_id) {
uint32_t output_path_id) {
assert(input_files.size());
// This compaction output should not overlap with a running compaction as
// `SanitizeCompactionInputFiles` should've checked earlier and db mutex
@@ -356,8 +356,8 @@ Compaction* CompactionPicker::CompactFiles(
compression_type = compact_options.compression;
}
auto c = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options, input_files,
output_level, compact_options.output_file_size_limit,
vstorage, ioptions_, mutable_cf_options, input_files, output_level,
compact_options.output_file_size_limit,
mutable_cf_options.max_compaction_bytes, output_path_id, compression_type,
GetCompressionOptions(mutable_cf_options, vstorage, output_level),
compact_options.max_subcompactions,
@@ -563,8 +563,7 @@ void CompactionPicker::GetGrandparents(
Compaction* CompactionPicker::CompactRange(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
int input_level, int output_level,
VersionStorageInfo* vstorage, int input_level, int output_level,
const CompactRangeOptions& compact_range_options, const InternalKey* begin,
const InternalKey* end, InternalKey** compaction_end, bool* manual_conflict,
uint64_t max_file_num_to_ignore) {
@@ -627,8 +626,8 @@ Compaction* CompactionPicker::CompactRange(
}
Compaction* c = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options,
std::move(inputs), output_level,
vstorage, ioptions_, mutable_cf_options, std::move(inputs),
output_level,
MaxFileSizeForLevel(mutable_cf_options, output_level,
ioptions_.compaction_style),
/* max_compaction_bytes */ LLONG_MAX,
@@ -779,8 +778,8 @@ Compaction* CompactionPicker::CompactRange(
std::vector<FileMetaData*> grandparents;
GetGrandparents(vstorage, inputs, output_level_inputs, &grandparents);
Compaction* compaction = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options,
std::move(compaction_inputs), output_level,
vstorage, ioptions_, mutable_cf_options, std::move(compaction_inputs),
output_level,
MaxFileSizeForLevel(mutable_cf_options, output_level,
ioptions_.compaction_style, vstorage->base_level(),
ioptions_.level_compaction_dynamic_level_bytes),
+2 -7
View File
@@ -56,8 +56,7 @@ class CompactionPicker {
// describes the compaction. Caller should delete the result.
virtual Compaction* PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer,
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber earliest_memtable_seqno = kMaxSequenceNumber) = 0;
// Return a compaction object for compacting the range [begin,end] in
@@ -73,8 +72,7 @@ class CompactionPicker {
// *compaction_end should point to valid InternalKey!
virtual Compaction* CompactRange(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
int input_level, int output_level,
VersionStorageInfo* vstorage, int input_level, int output_level,
const CompactRangeOptions& compact_range_options,
const InternalKey* begin, const InternalKey* end,
InternalKey** compaction_end, bool* manual_conflict,
@@ -115,7 +113,6 @@ class CompactionPicker {
const std::vector<CompactionInputFiles>& input_files,
int output_level, VersionStorageInfo* vstorage,
const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options,
uint32_t output_path_id);
// Converts a set of compaction input file numbers into
@@ -253,7 +250,6 @@ class NullCompactionPicker : public CompactionPicker {
Compaction* PickCompaction(
const std::string& /*cf_name*/,
const MutableCFOptions& /*mutable_cf_options*/,
const MutableDBOptions& /*mutable_db_options*/,
VersionStorageInfo* /*vstorage*/, LogBuffer* /* log_buffer */,
SequenceNumber /* earliest_memtable_seqno */) override {
return nullptr;
@@ -262,7 +258,6 @@ class NullCompactionPicker : public CompactionPicker {
// Always return "nullptr"
Compaction* CompactRange(const std::string& /*cf_name*/,
const MutableCFOptions& /*mutable_cf_options*/,
const MutableDBOptions& /*mutable_db_options*/,
VersionStorageInfo* /*vstorage*/,
int /*input_level*/, int /*output_level*/,
const CompactRangeOptions& /*compact_range_options*/,
+15 -22
View File
@@ -36,8 +36,7 @@ bool FIFOCompactionPicker::NeedsCompaction(
Compaction* FIFOCompactionPicker::PickTTLCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer) {
VersionStorageInfo* vstorage, LogBuffer* log_buffer) {
assert(mutable_cf_options.ttl > 0);
const int kLevel0 = 0;
@@ -109,9 +108,8 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
}
Compaction* c = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options,
std::move(inputs), 0, 0, 0, 0, kNoCompression,
mutable_cf_options.compression_opts,
vstorage, ioptions_, mutable_cf_options, std::move(inputs), 0, 0, 0, 0,
kNoCompression, mutable_cf_options.compression_opts,
/* max_subcompactions */ 0, {}, /* is manual */ false,
vstorage->CompactionScore(0),
/* is deletion compaction */ true, CompactionReason::kFIFOTtl);
@@ -120,8 +118,7 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
Compaction* FIFOCompactionPicker::PickSizeCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer) {
VersionStorageInfo* vstorage, LogBuffer* log_buffer) {
const int kLevel0 = 0;
const std::vector<FileMetaData*>& level_files = vstorage->LevelFiles(kLevel0);
uint64_t total_size = GetTotalFilesSize(level_files);
@@ -150,8 +147,8 @@ Compaction* FIFOCompactionPicker::PickSizeCompaction(
max_compact_bytes_per_del_file,
mutable_cf_options.max_compaction_bytes, &comp_inputs)) {
Compaction* c = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options,
{comp_inputs}, 0, 16 * 1024 * 1024 /* output file size limit */,
vstorage, ioptions_, mutable_cf_options, {comp_inputs}, 0,
16 * 1024 * 1024 /* output file size limit */,
0 /* max compaction bytes, not applicable */,
0 /* output path ID */, mutable_cf_options.compression,
mutable_cf_options.compression_opts, 0 /* max_subcompactions */, {},
@@ -201,9 +198,8 @@ Compaction* FIFOCompactionPicker::PickSizeCompaction(
}
Compaction* c = new Compaction(
vstorage, ioptions_, mutable_cf_options, mutable_db_options,
std::move(inputs), 0, 0, 0, 0, kNoCompression,
mutable_cf_options.compression_opts,
vstorage, ioptions_, mutable_cf_options, std::move(inputs), 0, 0, 0, 0,
kNoCompression, mutable_cf_options.compression_opts,
/* max_subcompactions */ 0, {}, /* is manual */ false,
vstorage->CompactionScore(0),
/* is deletion compaction */ true, CompactionReason::kFIFOMaxSize);
@@ -212,18 +208,16 @@ Compaction* FIFOCompactionPicker::PickSizeCompaction(
Compaction* FIFOCompactionPicker::PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer, SequenceNumber /*earliest_memtable_seqno*/) {
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber /*earliest_memtable_seqno*/) {
assert(vstorage->num_levels() == 1);
Compaction* c = nullptr;
if (mutable_cf_options.ttl > 0) {
c = PickTTLCompaction(cf_name, mutable_cf_options, mutable_db_options,
vstorage, log_buffer);
c = PickTTLCompaction(cf_name, mutable_cf_options, vstorage, log_buffer);
}
if (c == nullptr) {
c = PickSizeCompaction(cf_name, mutable_cf_options, mutable_db_options,
vstorage, log_buffer);
c = PickSizeCompaction(cf_name, mutable_cf_options, vstorage, log_buffer);
}
RegisterCompaction(c);
return c;
@@ -231,8 +225,7 @@ Compaction* FIFOCompactionPicker::PickCompaction(
Compaction* FIFOCompactionPicker::CompactRange(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
int input_level, int output_level,
VersionStorageInfo* vstorage, int input_level, int output_level,
const CompactRangeOptions& /*compact_range_options*/,
const InternalKey* /*begin*/, const InternalKey* /*end*/,
InternalKey** compaction_end, bool* /*manual_conflict*/,
@@ -245,8 +238,8 @@ Compaction* FIFOCompactionPicker::CompactRange(
assert(output_level == 0);
*compaction_end = nullptr;
LogBuffer log_buffer(InfoLogLevel::INFO_LEVEL, ioptions_.info_log);
Compaction* c = PickCompaction(cf_name, mutable_cf_options,
mutable_db_options, vstorage, &log_buffer);
Compaction* c =
PickCompaction(cf_name, mutable_cf_options, vstorage, &log_buffer);
log_buffer.FlushBufferToLog();
return c;
}
+2 -6
View File
@@ -21,14 +21,12 @@ class FIFOCompactionPicker : public CompactionPicker {
virtual Compaction* PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* version,
LogBuffer* log_buffer,
VersionStorageInfo* version, LogBuffer* log_buffer,
SequenceNumber earliest_memtable_seqno = kMaxSequenceNumber) override;
virtual Compaction* CompactRange(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
int input_level, int output_level,
VersionStorageInfo* vstorage, int input_level, int output_level,
const CompactRangeOptions& compact_range_options,
const InternalKey* begin, const InternalKey* end,
InternalKey** compaction_end, bool* manual_conflict,
@@ -43,13 +41,11 @@ class FIFOCompactionPicker : public CompactionPicker {
private:
Compaction* PickTTLCompaction(const std::string& cf_name,
const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options,
VersionStorageInfo* version,
LogBuffer* log_buffer);
Compaction* PickSizeCompaction(const std::string& cf_name,
const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options,
VersionStorageInfo* version,
LogBuffer* log_buffer);
};
+7 -11
View File
@@ -49,16 +49,14 @@ class LevelCompactionBuilder {
CompactionPicker* compaction_picker,
LogBuffer* log_buffer,
const MutableCFOptions& mutable_cf_options,
const ImmutableCFOptions& ioptions,
const MutableDBOptions& mutable_db_options)
const ImmutableCFOptions& ioptions)
: cf_name_(cf_name),
vstorage_(vstorage),
earliest_mem_seqno_(earliest_mem_seqno),
compaction_picker_(compaction_picker),
log_buffer_(log_buffer),
mutable_cf_options_(mutable_cf_options),
ioptions_(ioptions),
mutable_db_options_(mutable_db_options) {}
ioptions_(ioptions) {}
// Pick and return a compaction.
Compaction* PickCompaction();
@@ -122,7 +120,6 @@ class LevelCompactionBuilder {
const MutableCFOptions& mutable_cf_options_;
const ImmutableCFOptions& ioptions_;
const MutableDBOptions& mutable_db_options_;
// Pick a path ID to place a newly generated file, with its level
static uint32_t GetPathId(const ImmutableCFOptions& ioptions,
const MutableCFOptions& mutable_cf_options,
@@ -326,8 +323,8 @@ Compaction* LevelCompactionBuilder::PickCompaction() {
Compaction* LevelCompactionBuilder::GetCompaction() {
auto c = new Compaction(
vstorage_, ioptions_, mutable_cf_options_, mutable_db_options_,
std::move(compaction_inputs_), output_level_,
vstorage_, ioptions_, mutable_cf_options_, std::move(compaction_inputs_),
output_level_,
MaxFileSizeForLevel(mutable_cf_options_, output_level_,
ioptions_.compaction_style, vstorage_->base_level(),
ioptions_.level_compaction_dynamic_level_bytes),
@@ -500,11 +497,10 @@ bool LevelCompactionBuilder::PickIntraL0Compaction() {
Compaction* LevelCompactionPicker::PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer, SequenceNumber earliest_mem_seqno) {
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber earliest_mem_seqno) {
LevelCompactionBuilder builder(cf_name, vstorage, earliest_mem_seqno, this,
log_buffer, mutable_cf_options, ioptions_,
mutable_db_options);
log_buffer, mutable_cf_options, ioptions_);
return builder.PickCompaction();
}
} // namespace ROCKSDB_NAMESPACE
+1 -2
View File
@@ -22,8 +22,7 @@ class LevelCompactionPicker : public CompactionPicker {
: CompactionPicker(ioptions, icmp) {}
virtual Compaction* PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer,
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber earliest_memtable_seqno = kMaxSequenceNumber) override;
virtual bool NeedsCompaction(
+55 -138
View File
@@ -33,7 +33,6 @@ class CompactionPickerTest : public testing::Test {
Options options_;
ImmutableCFOptions ioptions_;
MutableCFOptions mutable_cf_options_;
MutableDBOptions mutable_db_options_;
LevelCompactionPicker level_compaction_picker;
std::string cf_name_;
CountingLogger logger_;
@@ -53,7 +52,6 @@ class CompactionPickerTest : public testing::Test {
icmp_(ucmp_),
ioptions_(options_),
mutable_cf_options_(options_),
mutable_db_options_(),
level_compaction_picker(ioptions_, &icmp_),
cf_name_("dummy"),
log_buffer_(InfoLogLevel::INFO_LEVEL, &logger_),
@@ -183,8 +181,7 @@ TEST_F(CompactionPickerTest, Empty) {
NewVersionStorage(6, kCompactionStyleLevel);
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() == nullptr);
}
@@ -195,8 +192,7 @@ TEST_F(CompactionPickerTest, Single) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() == nullptr);
}
@@ -209,8 +205,7 @@ TEST_F(CompactionPickerTest, Level0Trigger) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_files(0));
ASSERT_EQ(1U, compaction->input(0, 0)->fd.GetNumber());
@@ -223,8 +218,7 @@ TEST_F(CompactionPickerTest, Level1Trigger) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
ASSERT_EQ(66U, compaction->input(0, 0)->fd.GetNumber());
@@ -242,8 +236,7 @@ TEST_F(CompactionPickerTest, Level1Trigger2) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
ASSERT_EQ(2U, compaction->num_input_files(1));
@@ -274,8 +267,7 @@ TEST_F(CompactionPickerTest, LevelMaxScore) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
ASSERT_EQ(7U, compaction->input(0, 0)->fd.GetNumber());
@@ -322,8 +314,7 @@ TEST_F(CompactionPickerTest, Level0TriggerDynamic) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_files(0));
ASSERT_EQ(1U, compaction->input(0, 0)->fd.GetNumber());
@@ -347,8 +338,7 @@ TEST_F(CompactionPickerTest, Level0TriggerDynamic2) {
ASSERT_EQ(vstorage_->base_level(), num_levels - 2);
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_files(0));
ASSERT_EQ(1U, compaction->input(0, 0)->fd.GetNumber());
@@ -373,8 +363,7 @@ TEST_F(CompactionPickerTest, Level0TriggerDynamic3) {
ASSERT_EQ(vstorage_->base_level(), num_levels - 3);
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_files(0));
ASSERT_EQ(1U, compaction->input(0, 0)->fd.GetNumber());
@@ -403,8 +392,7 @@ TEST_F(CompactionPickerTest, Level0TriggerDynamic4) {
ASSERT_EQ(vstorage_->base_level(), num_levels - 3);
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_files(0));
ASSERT_EQ(1U, compaction->input(0, 0)->fd.GetNumber());
@@ -436,8 +424,7 @@ TEST_F(CompactionPickerTest, LevelTriggerDynamic4) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
ASSERT_EQ(5U, compaction->input(0, 0)->fd.GetNumber());
@@ -494,8 +481,7 @@ TEST_F(CompactionPickerTest, CompactionUniversalIngestBehindReservedLevel) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
// output level should be the one above the bottom-most
ASSERT_EQ(1, compaction->output_level());
@@ -529,8 +515,7 @@ TEST_F(CompactionPickerTest, CannotTrivialMoveUniversal) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(!compaction->is_trivial_move());
}
@@ -556,8 +541,7 @@ TEST_F(CompactionPickerTest, AllowsTrivialMoveUniversal) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction->is_trivial_move());
}
@@ -585,8 +569,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction1) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
ASSERT_EQ(4, compaction->output_level());
@@ -616,8 +599,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction2) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_FALSE(compaction);
}
@@ -643,8 +625,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction3) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_FALSE(compaction);
}
@@ -674,8 +655,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction4) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(!compaction ||
compaction->start_level() != compaction->output_level());
}
@@ -695,8 +675,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction5) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
ASSERT_EQ(0, compaction->start_level());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -720,8 +699,7 @@ TEST_F(CompactionPickerTest, UniversalPeriodicCompaction6) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
ASSERT_EQ(4, compaction->start_level());
ASSERT_EQ(2U, compaction->num_input_files(0));
@@ -781,8 +759,7 @@ TEST_F(CompactionPickerTest, CompactionPriMinOverlapping1) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
// Pick file 8 because it overlaps with 0 files on level 3.
@@ -814,8 +791,7 @@ TEST_F(CompactionPickerTest, CompactionPriMinOverlapping2) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
// Picking file 7 because overlapping ratio is the biggest.
@@ -842,8 +818,7 @@ TEST_F(CompactionPickerTest, CompactionPriMinOverlapping3) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
// Picking file 8 because overlapping ratio is the biggest.
@@ -872,8 +847,7 @@ TEST_F(CompactionPickerTest, CompactionPriMinOverlapping4) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_files(0));
// Picking file 8 because overlapping ratio is the biggest.
@@ -900,8 +874,7 @@ TEST_F(CompactionPickerTest, ParentIndexResetBug) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
}
// This test checks ExpandWhileOverlapping() by having overlapping user keys
@@ -918,8 +891,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(2U, compaction->num_input_files(0));
@@ -938,8 +910,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys2) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(2U, compaction->num_input_files(0));
@@ -966,8 +937,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys3) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(5U, compaction->num_input_files(0));
@@ -997,8 +967,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys4) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1021,8 +990,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys5) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() == nullptr);
}
@@ -1043,8 +1011,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys6) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1064,8 +1031,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys7) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_GE(1U, compaction->num_input_files(0));
@@ -1093,8 +1059,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys8) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(3U, compaction->num_input_files(0));
@@ -1126,8 +1091,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys9) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(5U, compaction->num_input_files(0));
@@ -1167,8 +1131,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys10) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1206,8 +1169,7 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys11) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1244,8 +1206,7 @@ TEST_F(CompactionPickerTest, NotScheduleL1IfL0WithHigherPri1) {
ASSERT_EQ(0, vstorage_->CompactionScoreLevel(0));
ASSERT_EQ(1, vstorage_->CompactionScoreLevel(1));
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() == nullptr);
}
@@ -1275,8 +1236,7 @@ TEST_F(CompactionPickerTest, NotScheduleL1IfL0WithHigherPri2) {
ASSERT_EQ(0, vstorage_->CompactionScoreLevel(0));
ASSERT_EQ(1, vstorage_->CompactionScoreLevel(1));
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
}
@@ -1309,8 +1269,7 @@ TEST_F(CompactionPickerTest, NotScheduleL1IfL0WithHigherPri3) {
ASSERT_EQ(1, vstorage_->CompactionScoreLevel(0));
ASSERT_EQ(0, vstorage_->CompactionScoreLevel(1));
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
}
@@ -1604,8 +1563,7 @@ TEST_F(CompactionPickerTest, MaxCompactionBytesHit) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1629,8 +1587,7 @@ TEST_F(CompactionPickerTest, MaxCompactionBytesNotHit) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(2U, compaction->num_input_levels());
ASSERT_EQ(3U, compaction->num_input_files(0));
@@ -1659,38 +1616,11 @@ TEST_F(CompactionPickerTest, IsTrivialMoveOn) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_TRUE(compaction->IsTrivialMove());
}
TEST_F(CompactionPickerTest, IsTrivialMoveOffSstPartitioned) {
mutable_cf_options_.max_bytes_for_level_base = 10000u;
mutable_cf_options_.max_compaction_bytes = 10001u;
ioptions_.level_compaction_dynamic_level_bytes = false;
ioptions_.sst_partitioner_factory = NewSstPartitionerFixedPrefixFactory(1);
NewVersionStorage(6, kCompactionStyleLevel);
// A compaction should be triggered and pick file 2
Add(1, 1U, "100", "150", 3000U);
Add(1, 2U, "151", "200", 3001U);
Add(1, 3U, "201", "250", 3000U);
Add(1, 4U, "251", "300", 3000U);
Add(3, 5U, "120", "130", 7000U);
Add(3, 6U, "170", "180", 7000U);
Add(3, 7U, "220", "230", 7000U);
Add(3, 8U, "270", "280", 7000U);
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
// No trivial move, because partitioning is applied
ASSERT_TRUE(!compaction->IsTrivialMove());
}
TEST_F(CompactionPickerTest, IsTrivialMoveOff) {
mutable_cf_options_.max_bytes_for_level_base = 1000000u;
mutable_cf_options_.max_compaction_bytes = 10000u;
@@ -1707,8 +1637,7 @@ TEST_F(CompactionPickerTest, IsTrivialMoveOff) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_FALSE(compaction->IsTrivialMove());
}
@@ -1733,8 +1662,7 @@ TEST_F(CompactionPickerTest, CacheNextCompactionIndex) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1743,8 +1671,7 @@ TEST_F(CompactionPickerTest, CacheNextCompactionIndex) {
ASSERT_EQ(2, vstorage_->NextCompactionIndex(1 /* level */));
compaction.reset(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(1U, compaction->num_input_files(0));
@@ -1753,8 +1680,7 @@ TEST_F(CompactionPickerTest, CacheNextCompactionIndex) {
ASSERT_EQ(3, vstorage_->NextCompactionIndex(1 /* level */));
compaction.reset(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() == nullptr);
ASSERT_EQ(4, vstorage_->NextCompactionIndex(1 /* level */));
}
@@ -1779,8 +1705,7 @@ TEST_F(CompactionPickerTest, IntraL0MaxCompactionBytesNotHit) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(5U, compaction->num_input_files(0));
@@ -1810,8 +1735,7 @@ TEST_F(CompactionPickerTest, IntraL0MaxCompactionBytesHit) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(4U, compaction->num_input_files(0));
@@ -1843,8 +1767,7 @@ TEST_F(CompactionPickerTest, IntraL0ForEarliestSeqno) {
UpdateVersionStorageInfo();
std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_, 107));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_, 107));
ASSERT_TRUE(compaction.get() != nullptr);
ASSERT_EQ(1U, compaction->num_input_levels());
ASSERT_EQ(4U, compaction->num_input_files(0));
@@ -1876,8 +1799,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
// Validate that its a compaction to reduce sorted runs
@@ -1894,8 +1816,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
std::unique_ptr<Compaction> compaction2(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_FALSE(compaction2);
}
@@ -1919,8 +1840,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap2) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
// Validate that its a delete triggered compaction
@@ -1938,8 +1858,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap2) {
std::unique_ptr<Compaction> compaction2(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_FALSE(compaction2);
}
@@ -1979,8 +1898,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionStartOutputOverlap) {
std::unique_ptr<Compaction> compaction(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_TRUE(compaction);
// Validate that its a delete triggered compaction
@@ -2010,8 +1928,7 @@ TEST_F(CompactionPickerTest, UniversalMarkedCompactionStartOutputOverlap) {
random_index = 0;
std::unique_ptr<Compaction> compaction2(
universal_compaction_picker.PickCompaction(
cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
&log_buffer_));
cf_name_, mutable_cf_options_, vstorage_.get(), &log_buffer_));
ASSERT_FALSE(compaction2);
DeleteVersionStorage();
}
+20 -20
View File
@@ -31,16 +31,17 @@ namespace {
// PickCompaction().
class UniversalCompactionBuilder {
public:
UniversalCompactionBuilder(
const ImmutableCFOptions& ioptions, const InternalKeyComparator* icmp,
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
UniversalCompactionPicker* picker, LogBuffer* log_buffer)
UniversalCompactionBuilder(const ImmutableCFOptions& ioptions,
const InternalKeyComparator* icmp,
const std::string& cf_name,
const MutableCFOptions& mutable_cf_options,
VersionStorageInfo* vstorage,
UniversalCompactionPicker* picker,
LogBuffer* log_buffer)
: ioptions_(ioptions),
icmp_(icmp),
cf_name_(cf_name),
mutable_cf_options_(mutable_cf_options),
mutable_db_options_(mutable_db_options),
vstorage_(vstorage),
picker_(picker),
log_buffer_(log_buffer) {}
@@ -114,7 +115,6 @@ class UniversalCompactionBuilder {
std::vector<SortedRun> sorted_runs_;
const std::string& cf_name_;
const MutableCFOptions& mutable_cf_options_;
const MutableDBOptions& mutable_db_options_;
VersionStorageInfo* vstorage_;
UniversalCompactionPicker* picker_;
LogBuffer* log_buffer_;
@@ -277,11 +277,11 @@ bool UniversalCompactionPicker::NeedsCompaction(
Compaction* UniversalCompactionPicker::PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer, SequenceNumber /* earliest_memtable_seqno */) {
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber /* earliest_memtable_seqno */) {
UniversalCompactionBuilder builder(ioptions_, icmp_, cf_name,
mutable_cf_options, mutable_db_options,
vstorage, this, log_buffer);
mutable_cf_options, vstorage, this,
log_buffer);
return builder.PickCompaction();
}
@@ -730,8 +730,8 @@ Compaction* UniversalCompactionBuilder::PickCompactionToReduceSortedRuns(
compaction_reason = CompactionReason::kUniversalSortedRunNum;
}
return new Compaction(
vstorage_, ioptions_, mutable_cf_options_, mutable_db_options_,
std::move(inputs), output_level,
vstorage_, ioptions_, mutable_cf_options_, std::move(inputs),
output_level,
MaxFileSizeForLevel(mutable_cf_options_, output_level,
kCompactionStyleUniversal),
LLONG_MAX, path_id,
@@ -944,8 +944,8 @@ Compaction* UniversalCompactionBuilder::PickDeleteTriggeredCompaction() {
uint32_t path_id =
GetPathId(ioptions_, mutable_cf_options_, estimated_total_size);
return new Compaction(
vstorage_, ioptions_, mutable_cf_options_, mutable_db_options_,
std::move(inputs), output_level,
vstorage_, ioptions_, mutable_cf_options_, std::move(inputs),
output_level,
MaxFileSizeForLevel(mutable_cf_options_, output_level,
kCompactionStyleUniversal),
/* max_grandparent_overlap_bytes */ LLONG_MAX, path_id,
@@ -1014,14 +1014,14 @@ Compaction* UniversalCompactionBuilder::PickCompactionToOldest(
// compaction_options_universal.compression_size_percent,
// because we always compact all the files, so always compress.
return new Compaction(
vstorage_, ioptions_, mutable_cf_options_, mutable_db_options_,
std::move(inputs), output_level,
vstorage_, ioptions_, mutable_cf_options_, std::move(inputs),
output_level,
MaxFileSizeForLevel(mutable_cf_options_, output_level,
kCompactionStyleUniversal),
LLONG_MAX, path_id,
GetCompressionType(ioptions_, vstorage_, mutable_cf_options_,
output_level, 1, true /* enable_compression */),
GetCompressionOptions(mutable_cf_options_, vstorage_, output_level,
GetCompressionType(ioptions_, vstorage_, mutable_cf_options_, start_level,
1, true /* enable_compression */),
GetCompressionOptions(mutable_cf_options_, vstorage_, start_level,
true /* enable_compression */),
/* max_subcompactions */ 0, /* grandparents */ {}, /* is manual */ false,
score_, false /* deletion_compaction */, compaction_reason);
+1 -2
View File
@@ -20,8 +20,7 @@ class UniversalCompactionPicker : public CompactionPicker {
: CompactionPicker(ioptions, icmp) {}
virtual Compaction* PickCompaction(
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
const MutableDBOptions& mutable_db_options, VersionStorageInfo* vstorage,
LogBuffer* log_buffer,
VersionStorageInfo* vstorage, LogBuffer* log_buffer,
SequenceNumber earliest_memtable_seqno = kMaxSequenceNumber) override;
virtual int MaxOutputLevel() const override { return NumberLevels() - 1; }
-44
View File
@@ -1,44 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
#include "rocksdb/sst_partitioner.h"
#include <algorithm>
namespace ROCKSDB_NAMESPACE {
PartitionerResult SstPartitionerFixedPrefix::ShouldPartition(
const PartitionerRequest& request) {
Slice last_key_fixed(*request.prev_user_key);
if (last_key_fixed.size() > len_) {
last_key_fixed.size_ = len_;
}
Slice current_key_fixed(*request.current_user_key);
if (current_key_fixed.size() > len_) {
current_key_fixed.size_ = len_;
}
return last_key_fixed.compare(current_key_fixed) != 0 ? kRequired
: kNotRequired;
}
bool SstPartitionerFixedPrefix::CanDoTrivialMove(
const Slice& smallest_user_key, const Slice& largest_user_key) {
return ShouldPartition(PartitionerRequest(smallest_user_key, largest_user_key,
0)) == kNotRequired;
}
std::unique_ptr<SstPartitioner>
SstPartitionerFixedPrefixFactory::CreatePartitioner(
const SstPartitioner::Context& /* context */) const {
return std::unique_ptr<SstPartitioner>(new SstPartitionerFixedPrefix(len_));
}
std::shared_ptr<SstPartitionerFactory> NewSstPartitionerFixedPrefixFactory(
size_t prefix_len) {
return std::make_shared<SstPartitionerFixedPrefixFactory>(prefix_len);
}
} // namespace ROCKSDB_NAMESPACE
+2 -3
View File
@@ -13,7 +13,6 @@
#include "test_util/testutil.h"
#include "util/hash.h"
#include "util/kv_map.h"
#include "util/random.h"
#include "util/string_util.h"
#include "utilities/merge_operators.h"
@@ -343,12 +342,12 @@ TEST_P(ComparatorDBTest, SimpleSuffixReverseComparator) {
std::vector<std::string> source_prefixes;
// Randomly generate 5 prefixes
for (int i = 0; i < 5; i++) {
source_prefixes.push_back(rnd.HumanReadableString(8));
source_prefixes.push_back(test::RandomHumanReadableString(&rnd, 8));
}
for (int j = 0; j < 20; j++) {
int prefix_index = rnd.Uniform(static_cast<int>(source_prefixes.size()));
std::string key = source_prefixes[prefix_index] +
rnd.HumanReadableString(rnd.Uniform(8));
test::RandomHumanReadableString(&rnd, rnd.Uniform(8));
source_strings.push_back(key);
}
+17 -171
View File
@@ -9,13 +9,13 @@
#ifndef ROCKSDB_LITE
#include "rocksdb/db.h"
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <cinttypes>
#include "db/db_impl/db_impl.h"
#include "db/db_test_util.h"
#include "db/log_format.h"
@@ -24,16 +24,13 @@
#include "file/filename.h"
#include "rocksdb/cache.h"
#include "rocksdb/convenience.h"
#include "rocksdb/db.h"
#include "rocksdb/env.h"
#include "rocksdb/table.h"
#include "rocksdb/write_batch.h"
#include "table/block_based/block_based_table_builder.h"
#include "table/meta_blocks.h"
#include "table/mock_table.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/random.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -104,24 +101,22 @@ class CorruptionTest : public testing::Test {
ASSERT_OK(::ROCKSDB_NAMESPACE::RepairDB(dbname_, options_));
}
void Build(int n, int start, int flush_every) {
void Build(int n, int flush_every = 0) {
std::string key_space, value_space;
WriteBatch batch;
for (int i = 0; i < n; i++) {
if (flush_every != 0 && i != 0 && i % flush_every == 0) {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
}
//if ((i % 100) == 0) fprintf(stderr, "@ %d of %d\n", i, n);
Slice key = Key(i + start, &key_space);
Slice key = Key(i, &key_space);
batch.Clear();
batch.Put(key, Value(i, &value_space));
ASSERT_OK(db_->Write(WriteOptions(), &batch));
}
}
void Build(int n, int flush_every = 0) { Build(n, 0, flush_every); }
void Check(int min_expected, int max_expected) {
uint64_t next_expected = 0;
uint64_t missed = 0;
@@ -224,11 +219,11 @@ class CorruptionTest : public testing::Test {
// preserves the implementation that was in place when all of the
// magic values in this file were picked.
*storage = std::string(kValueSize, ' ');
return Slice(*storage);
} else {
Random r(k);
*storage = r.RandomString(kValueSize);
return test::RandomString(&r, kValueSize, storage);
}
return Slice(*storage);
}
};
@@ -286,7 +281,7 @@ TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
TEST_F(CorruptionTest, TableFile) {
Build(100);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
@@ -309,7 +304,7 @@ TEST_F(CorruptionTest, VerifyChecksumReadahead) {
Reopen(&options);
Build(10000);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
@@ -356,14 +351,14 @@ TEST_F(CorruptionTest, TableFileIndexData) {
Reopen(&options);
// build 2 tables, flush at 5000
Build(10000, 5000);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
// corrupt an index block of an entire file
Corrupt(kTableFile, -2000, 500);
options.paranoid_checks = false;
Reopen(&options);
dbi = static_cast_with_check<DBImpl>(db_);
dbi = reinterpret_cast<DBImpl*>(db_);
// one full file may be readable, since only one was corrupted
// the other file should be fully non-readable, since index was corrupted
Check(0, 5000);
@@ -403,7 +398,7 @@ TEST_F(CorruptionTest, SequenceNumberRecovery) {
TEST_F(CorruptionTest, CorruptedDescriptor) {
ASSERT_OK(db_->Put(WriteOptions(), "foo", "hello"));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
@@ -422,7 +417,7 @@ TEST_F(CorruptionTest, CompactionInputError) {
Options options;
Reopen(&options);
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
dbi->TEST_CompactRange(0, nullptr, nullptr);
dbi->TEST_CompactRange(1, nullptr, nullptr);
@@ -444,7 +439,7 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
options.write_buffer_size = 131072;
options.max_write_buffer_number = 2;
Reopen(&options);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
// Fill levels >= 1
for (int level = 1; level < dbi->NumberLevels(); level++) {
@@ -459,7 +454,7 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
Reopen(&options);
dbi = static_cast_with_check<DBImpl>(db_);
dbi = reinterpret_cast<DBImpl*>(db_);
Build(10);
dbi->TEST_FlushMemTable();
dbi->TEST_WaitForCompact();
@@ -486,7 +481,7 @@ TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
TEST_F(CorruptionTest, UnrelatedKeys) {
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_FlushMemTable();
Corrupt(kTableFile, 100, 1);
ASSERT_NOK(dbi->VerifyChecksum());
@@ -537,7 +532,7 @@ TEST_F(CorruptionTest, FileSystemStateCorrupted) {
Reopen(&options);
Build(10);
ASSERT_OK(db_->Flush(FlushOptions()));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
std::vector<LiveFileMetaData> metadata;
dbi->GetLiveFilesMetaData(&metadata);
ASSERT_GT(metadata.size(), size_t(0));
@@ -563,155 +558,6 @@ TEST_F(CorruptionTest, FileSystemStateCorrupted) {
}
}
static const auto& corruption_modes = {mock::MockTableFactory::kCorruptNone,
mock::MockTableFactory::kCorruptKey,
mock::MockTableFactory::kCorruptValue};
TEST_F(CorruptionTest, ParaniodFileChecksOnFlush) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
Status s;
for (const auto& mode : corruption_modes) {
delete db_;
s = DestroyDB(dbname_, options);
std::shared_ptr<mock::MockTableFactory> mock =
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
mock->SetCorruptionMode(mode);
ASSERT_OK(DB::Open(options, dbname_, &db_));
Build(10);
s = db_->Flush(FlushOptions());
if (mode == mock::MockTableFactory::kCorruptNone) {
ASSERT_OK(s);
} else {
ASSERT_NOK(s);
}
}
}
TEST_F(CorruptionTest, ParaniodFileChecksOnCompact) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
Status s;
for (const auto& mode : corruption_modes) {
delete db_;
s = DestroyDB(dbname_, options);
std::shared_ptr<mock::MockTableFactory> mock =
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
Build(100, 2);
// ASSERT_OK(db_->Flush(FlushOptions()));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
mock->SetCorruptionMode(mode);
s = dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true);
if (mode == mock::MockTableFactory::kCorruptNone) {
ASSERT_OK(s);
} else {
ASSERT_NOK(s);
}
}
}
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
delete db_;
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
std::string start, end;
assert(db_ != nullptr);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(3, &start), Key(7, &end)));
auto snap = db_->GetSnapshot();
ASSERT_NE(snap, nullptr);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(8, &start), Key(9, &end)));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(2, &start), Key(5, &end)));
Build(10);
if (do_flush) {
ASSERT_OK(db_->Flush(FlushOptions()));
} else {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
}
db_->ReleaseSnapshot(snap);
}
}
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
delete db_;
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
Build(10, 0, 0);
std::string start, end;
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(5, &start), Key(15, &end)));
auto snap = db_->GetSnapshot();
ASSERT_NE(snap, nullptr);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(8, &start), Key(9, &end)));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(12, &start), Key(17, &end)));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(2, &start), Key(4, &end)));
Build(10, 10, 0);
if (do_flush) {
ASSERT_OK(db_->Flush(FlushOptions()));
} else {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
}
db_->ReleaseSnapshot(snap);
}
}
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
Options options;
options.paranoid_file_checks = true;
options.create_if_missing = true;
for (bool do_flush : {true, false}) {
delete db_;
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
std::string start, end;
Build(10);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(3, &start), Key(7, &end)));
auto snap = db_->GetSnapshot();
ASSERT_NE(snap, nullptr);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(6, &start), Key(8, &end)));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(2, &start), Key(5, &end)));
if (do_flush) {
ASSERT_OK(db_->Flush(FlushOptions()));
} else {
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
ASSERT_OK(dbi->TEST_FlushMemTable());
ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
}
db_->ReleaseSnapshot(snap);
}
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+3 -2
View File
@@ -13,7 +13,6 @@
#include "table/meta_blocks.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/cast_util.h"
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -47,7 +46,9 @@ class CuckooTableDBTest : public testing::Test {
return options;
}
DBImpl* dbfull() { return static_cast_with_check<DBImpl>(db_); }
DBImpl* dbfull() {
return reinterpret_cast<DBImpl*>(db_);
}
// The following util methods are copied from plain_table_db_test.
void Reopen(Options* options = nullptr) {
+146 -352
View File
@@ -7,9 +7,6 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <cstring>
#include <regex>
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "rocksdb/merge_operator.h"
@@ -17,11 +14,10 @@
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#include "test_util/fault_injection_test_env.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "util/random.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators.h"
#include "utilities/merge_operators/string_append/stringappend.h"
@@ -45,63 +41,6 @@ TEST_F(DBBasicTest, OpenWhenOpen) {
delete db2;
}
TEST_F(DBBasicTest, UniqueSession) {
Options options = CurrentOptions();
std::string sid1, sid2, sid3, sid4;
db_->GetDbSessionId(sid1);
Reopen(options);
db_->GetDbSessionId(sid2);
ASSERT_OK(Put("foo", "v1"));
db_->GetDbSessionId(sid4);
Reopen(options);
db_->GetDbSessionId(sid3);
ASSERT_NE(sid1, sid2);
ASSERT_NE(sid1, sid3);
ASSERT_NE(sid2, sid3);
ASSERT_EQ(sid2, sid4);
// Expected compact format for session ids (see notes in implementation)
std::regex expected("[0-9A-Z]{20}");
const std::string match("match");
EXPECT_EQ(match, std::regex_replace(sid1, expected, match));
EXPECT_EQ(match, std::regex_replace(sid2, expected, match));
EXPECT_EQ(match, std::regex_replace(sid3, expected, match));
#ifndef ROCKSDB_LITE
Close();
ASSERT_OK(ReadOnlyReopen(options));
db_->GetDbSessionId(sid1);
// Test uniqueness between readonly open (sid1) and regular open (sid3)
ASSERT_NE(sid1, sid3);
Close();
ASSERT_OK(ReadOnlyReopen(options));
db_->GetDbSessionId(sid2);
ASSERT_EQ("v1", Get("foo"));
db_->GetDbSessionId(sid3);
ASSERT_NE(sid1, sid2);
ASSERT_EQ(sid2, sid3);
#endif // ROCKSDB_LITE
CreateAndReopenWithCF({"goku"}, options);
db_->GetDbSessionId(sid1);
ASSERT_OK(Put("bar", "e1"));
db_->GetDbSessionId(sid2);
ASSERT_EQ("e1", Get("bar"));
db_->GetDbSessionId(sid3);
ReopenWithColumnFamilies({"default", "goku"}, options);
db_->GetDbSessionId(sid4);
ASSERT_EQ(sid1, sid2);
ASSERT_EQ(sid2, sid3);
ASSERT_NE(sid1, sid4);
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, ReadOnlyDB) {
ASSERT_OK(Put("foo", "v1"));
@@ -1118,7 +1057,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCF) {
}
int get_sv_count = 0;
ROCKSDB_NAMESPACE::DBImpl* db = static_cast_with_check<DBImpl>(db_);
ROCKSDB_NAMESPACE::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::AfterRefSV", [&](void* /*arg*/) {
if (++get_sv_count == 2) {
@@ -1136,7 +1075,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCF) {
}
if (get_sv_count == 11) {
for (int i = 0; i < 8; ++i) {
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(
db->GetColumnFamilyHandle(i))
->cfd();
ASSERT_EQ(cfd->TEST_GetLocalSV()->Get(), SuperVersion::kSVInUse);
@@ -1187,10 +1126,9 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCF) {
ASSERT_EQ(values[2], std::get<2>(cf_kv_vec[1]) + "_2");
for (int cf = 0; cf < 8; ++cf) {
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(
static_cast_with_check<DBImpl>(db_)->GetColumnFamilyHandle(cf))
->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(
reinterpret_cast<DBImpl*>(db_)->GetColumnFamilyHandle(cf))
->cfd();
ASSERT_NE(cfd->TEST_GetLocalSV()->Get(), SuperVersion::kSVInUse);
ASSERT_NE(cfd->TEST_GetLocalSV()->Get(), SuperVersion::kSVObsolete);
}
@@ -1250,10 +1188,9 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFMutex) {
"cf" + std::to_string(j) + "_val" + std::to_string(retries));
}
for (int i = 0; i < 8; ++i) {
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(
static_cast_with_check<DBImpl>(db_)->GetColumnFamilyHandle(i))
->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(
reinterpret_cast<DBImpl*>(db_)->GetColumnFamilyHandle(i))
->cfd();
ASSERT_NE(cfd->TEST_GetLocalSV()->Get(), SuperVersion::kSVInUse);
}
}
@@ -1270,7 +1207,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFSnapshot) {
}
int get_sv_count = 0;
ROCKSDB_NAMESPACE::DBImpl* db = static_cast_with_check<DBImpl>(db_);
ROCKSDB_NAMESPACE::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::AfterRefSV", [&](void* /*arg*/) {
if (++get_sv_count == 2) {
@@ -1282,7 +1219,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFSnapshot) {
}
if (get_sv_count == 8) {
for (int i = 0; i < 8; ++i) {
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(
db->GetColumnFamilyHandle(i))
->cfd();
ASSERT_TRUE(
@@ -1310,10 +1247,9 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFSnapshot) {
ASSERT_EQ(values[j], "cf" + std::to_string(j) + "_val");
}
for (int i = 0; i < 8; ++i) {
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(
static_cast_with_check<DBImpl>(db_)->GetColumnFamilyHandle(i))
->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(
reinterpret_cast<DBImpl*>(db_)->GetColumnFamilyHandle(i))
->cfd();
ASSERT_NE(cfd->TEST_GetLocalSV()->Get(), SuperVersion::kSVInUse);
}
}
@@ -2049,7 +1985,7 @@ TEST_F(DBBasicTest, MultiGetIOBufferOverrun) {
for (int i = 0; i < 100; ++i) {
// Make the value compressible. A purely random string doesn't compress
// and the resultant data block will not be compressed
std::string value(rnd.RandomString(128) + zero_str);
std::string value(RandomString(&rnd, 128) + zero_str);
assert(Put(Key(i), value) == Status::OK());
}
Flush();
@@ -2439,7 +2375,7 @@ class DBBasicTestMultiGet : public DBTestBase {
for (int i = 0; i < 100; ++i) {
// Make the value compressible. A purely random string doesn't compress
// and the resultant data block will not be compressed
values_.emplace_back(rnd.RandomString(128) + zero_str);
values_.emplace_back(RandomString(&rnd, 128) + zero_str);
assert(((num_cfs == 1) ? Put(Key(i), values_[i])
: Put(cf, Key(i), values_[i])) == Status::OK());
}
@@ -2451,7 +2387,7 @@ class DBBasicTestMultiGet : public DBTestBase {
for (int i = 0; i < 100; ++i) {
// block cannot gain space by compression
uncompressable_values_.emplace_back(rnd.RandomString(256) + '\0');
uncompressable_values_.emplace_back(RandomString(&rnd, 256) + '\0');
std::string tmp_key = "a" + Key(i);
assert(((num_cfs == 1) ? Put(tmp_key, uncompressable_values_[i])
: Put(cf, tmp_key, uncompressable_values_[i])) ==
@@ -2815,176 +2751,132 @@ INSTANTIATE_TEST_CASE_P(ParallelIO, DBBasicTestWithParallelIO,
::testing::Bool(), ::testing::Bool(),
::testing::Values(1, 4)));
// Forward declaration
class DeadlineFS;
class DeadlineRandomAccessFile : public FSRandomAccessFileWrapper {
public:
DeadlineRandomAccessFile(DeadlineFS& fs, SpecialEnv* env,
std::unique_ptr<FSRandomAccessFile>& file)
: FSRandomAccessFileWrapper(file.get()),
fs_(fs),
file_(std::move(file)),
env_(env) {}
IOStatus Read(uint64_t offset, size_t len, const IOOptions& opts,
Slice* result, char* scratch,
IODebugContext* dbg) const override;
IOStatus MultiRead(FSReadRequest* reqs, size_t num_reqs,
const IOOptions& options, IODebugContext* dbg) override;
private:
DeadlineFS& fs_;
std::unique_ptr<FSRandomAccessFile> file_;
SpecialEnv* env_;
};
class DeadlineFS : public FileSystemWrapper {
public:
explicit DeadlineFS(SpecialEnv* env)
: FileSystemWrapper(FileSystem::Default()),
delay_idx_(0),
deadline_(std::chrono::microseconds::zero()),
env_(env),
timedout_(false),
ignore_deadline_(false) {}
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
std::unique_ptr<FSRandomAccessFile> file;
IOStatus s;
s = target()->NewRandomAccessFile(fname, opts, &file, dbg);
result->reset(new DeadlineRandomAccessFile(*this, env_, file));
int delay;
const std::chrono::microseconds deadline = GetDeadline();
if (deadline.count()) {
AssertDeadline(deadline, opts.io_options);
}
if (ShouldDelay(&delay, &s)) {
env_->SleepForMicroseconds(delay);
}
return s;
}
// Set a vector of {IO counter, delay in microseconds, return status} tuples
// that control when to inject a delay and duration of the delay
void SetDelaySequence(
const std::chrono::microseconds deadline,
const std::vector<std::tuple<int, int, IOStatus>>&& seq) {
int total_delay = 0;
for (auto& seq_iter : seq) {
// Ensure no individual delay is > 500ms
ASSERT_LT(std::get<1>(seq_iter), 500000);
total_delay += std::get<1>(seq_iter);
}
// ASSERT total delay is < 1s. This is mainly to keep the test from
// timing out in CI test frameworks
ASSERT_LT(total_delay, 1000000);
delay_seq_ = seq;
delay_idx_ = 0;
io_count_ = 0;
deadline_ = deadline;
timedout_ = false;
}
// Increment the IO counter and return a delay in microseconds
bool ShouldDelay(int* delay, IOStatus* s) {
if (!ignore_deadline_ && delay_idx_ < delay_seq_.size() &&
std::get<0>(delay_seq_[delay_idx_]) == io_count_++) {
*delay = std::get<1>(delay_seq_[delay_idx_]);
*s = std::get<2>(delay_seq_[delay_idx_]);
delay_idx_++;
timedout_ = true;
return true;
}
*s = IOStatus::OK();
return false;
}
const std::chrono::microseconds GetDeadline() {
return ignore_deadline_ ? std::chrono::microseconds::zero() : deadline_;
}
bool TimedOut() { return timedout_; }
void IgnoreDeadline(bool ignore) { ignore_deadline_ = ignore; }
void AssertDeadline(const std::chrono::microseconds deadline,
const IOOptions& opts) const {
// Give a leeway of +- 10us as it can take some time for the Get/
// MultiGet call to reach here, in order to avoid false alarms
std::chrono::microseconds now =
std::chrono::microseconds(env_->NowMicros());
if (deadline - now != opts.timeout) {
ASSERT_EQ(deadline - now, opts.timeout);
}
}
private:
std::vector<std::tuple<int, int, IOStatus>> delay_seq_;
size_t delay_idx_;
int io_count_;
std::chrono::microseconds deadline_;
SpecialEnv* env_;
bool timedout_;
bool ignore_deadline_;
};
IOStatus DeadlineRandomAccessFile::Read(uint64_t offset, size_t len,
const IOOptions& opts, Slice* result,
char* scratch,
IODebugContext* dbg) const {
int delay;
const std::chrono::microseconds deadline = fs_.GetDeadline();
IOStatus s;
if (deadline.count()) {
fs_.AssertDeadline(deadline, opts);
}
if (fs_.ShouldDelay(&delay, &s)) {
env_->SleepForMicroseconds(delay);
}
if (s.ok()) {
s = FSRandomAccessFileWrapper::Read(offset, len, opts, result, scratch,
dbg);
}
return s;
}
IOStatus DeadlineRandomAccessFile::MultiRead(FSReadRequest* reqs,
size_t num_reqs,
const IOOptions& options,
IODebugContext* dbg) {
int delay;
const std::chrono::microseconds deadline = fs_.GetDeadline();
IOStatus s;
if (deadline.count()) {
fs_.AssertDeadline(deadline, options);
}
if (fs_.ShouldDelay(&delay, &s)) {
env_->SleepForMicroseconds(delay);
}
if (s.ok()) {
s = FSRandomAccessFileWrapper::MultiRead(reqs, num_reqs, options, dbg);
}
return s;
}
// A test class for intercepting random reads and injecting artificial
// delays. Used for testing the deadline/timeout feature
class DBBasicTestMultiGetDeadline : public DBBasicTestMultiGet {
public:
DBBasicTestMultiGetDeadline()
: DBBasicTestMultiGet(
"db_basic_test_multiget_deadline" /*Test dir*/,
10 /*# of column families*/, false /*compressed cache enabled*/,
true /*uncompressed cache enabled*/, true /*compression enabled*/,
true /*ReadOptions.fill_cache*/,
1 /*# of parallel compression threads*/) {}
: DBBasicTestMultiGet("db_basic_test_multiget_deadline" /*Test dir*/,
10 /*# of column families*/,
false /*compressed cache enabled*/,
true /*uncompressed cache enabled*/,
true /*compression enabled*/,
true /*ReadOptions.fill_cache*/,
1 /*# of parallel compression threads*/) {}
// Forward declaration
class DeadlineFS;
class DeadlineRandomAccessFile : public FSRandomAccessFileWrapper {
public:
DeadlineRandomAccessFile(DeadlineFS& fs, SpecialEnv* env,
std::unique_ptr<FSRandomAccessFile>& file)
: FSRandomAccessFileWrapper(file.get()),
fs_(fs),
file_(std::move(file)),
env_(env) {}
IOStatus Read(uint64_t offset, size_t len, const IOOptions& opts,
Slice* result, char* scratch, IODebugContext* dbg) const override {
int delay;
const std::chrono::microseconds deadline = fs_.GetDeadline();
if (deadline.count()) {
AssertDeadline(deadline, opts);
}
if (fs_.ShouldDelay(&delay)) {
env_->SleepForMicroseconds(delay);
}
return FSRandomAccessFileWrapper::Read(offset, len, opts, result, scratch,
dbg);
}
IOStatus MultiRead(FSReadRequest* reqs, size_t num_reqs,
const IOOptions& options, IODebugContext* dbg) override {
int delay;
const std::chrono::microseconds deadline = fs_.GetDeadline();
if (deadline.count()) {
AssertDeadline(deadline, options);
}
if (fs_.ShouldDelay(&delay)) {
env_->SleepForMicroseconds(delay);
}
return FSRandomAccessFileWrapper::MultiRead(reqs, num_reqs, options, dbg);
}
private:
void AssertDeadline(const std::chrono::microseconds deadline,
const IOOptions& opts) const {
// Give a leeway of +- 10us as it can take some time for the Get/
// MultiGet call to reach here, in order to avoid false alarms
std::chrono::microseconds now =
std::chrono::microseconds(env_->NowMicros());
ASSERT_EQ(deadline - now, opts.timeout);
}
DeadlineFS& fs_;
std::unique_ptr<FSRandomAccessFile> file_;
SpecialEnv* env_;
};
class DeadlineFS : public FileSystemWrapper {
public:
DeadlineFS(SpecialEnv* env)
: FileSystemWrapper(FileSystem::Default()),
delay_idx_(0),
deadline_(std::chrono::microseconds::zero()),
env_(env) {}
~DeadlineFS() = default;
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
std::unique_ptr<FSRandomAccessFile> file;
IOStatus s;
s = target()->NewRandomAccessFile(fname, opts, &file, dbg);
result->reset(new DeadlineRandomAccessFile(*this, env_, file));
return s;
}
// Set a vector of {IO counter, delay in microseconds} pairs that control
// when to inject a delay and duration of the delay
void SetDelaySequence(const std::chrono::microseconds deadline,
const std::vector<std::pair<int, int>>&& seq) {
int total_delay = 0;
for (auto& seq_iter : seq) {
// Ensure no individual delay is > 500ms
ASSERT_LT(seq_iter.second, 500000);
total_delay += seq_iter.second;
}
// ASSERT total delay is < 1s. This is mainly to keep the test from
// timing out in CI test frameworks
ASSERT_LT(total_delay, 1000000);
delay_seq_ = seq;
delay_idx_ = 0;
io_count_ = 0;
deadline_ = deadline;
}
// Increment the IO counter and return a delay in microseconds
bool ShouldDelay(int* delay) {
if (delay_idx_ < delay_seq_.size() &&
delay_seq_[delay_idx_].first == io_count_++) {
*delay = delay_seq_[delay_idx_].second;
delay_idx_++;
return true;
}
return false;
}
const std::chrono::microseconds GetDeadline() { return deadline_; }
private:
std::vector<std::pair<int, int>> delay_seq_;
size_t delay_idx_;
int io_count_;
std::chrono::microseconds deadline_;
SpecialEnv* env_;
};
inline void CheckStatus(std::vector<Status>& statuses, size_t num_ok) {
for (size_t i = 0; i < statuses.size(); ++i) {
@@ -2998,7 +2890,8 @@ class DBBasicTestMultiGetDeadline : public DBBasicTestMultiGet {
};
TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
std::shared_ptr<DeadlineFS> fs = std::make_shared<DeadlineFS>(env_);
std::shared_ptr<DBBasicTestMultiGetDeadline::DeadlineFS> fs(
new DBBasicTestMultiGetDeadline::DeadlineFS(env_));
std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, fs));
Options options = CurrentOptions();
env_->SetTimeElapseOnlySleep(&options);
@@ -3029,8 +2922,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
ReadOptions ro;
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
// Delay the first IO by 200ms
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{0, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{0, 20000}});
std::vector<Status> statuses = dbfull()->MultiGet(ro, cfs, keys, &values);
// The first key is successful because we check after the lookup, but
@@ -3055,8 +2947,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
keys[i] = Slice(key_str[i].data(), key_str[i].size());
}
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{1, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{1, 20000}});
statuses = dbfull()->MultiGet(ro, cfs, keys, &values);
CheckStatus(statuses, 3);
@@ -3070,8 +2961,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
statuses.clear();
statuses.resize(keys.size());
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{0, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{0, 20000}});
dbfull()->MultiGet(ro, keys.size(), cfs.data(), keys.data(),
pin_values.data(), statuses.data());
CheckStatus(statuses, 2);
@@ -3086,8 +2976,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
statuses.clear();
statuses.resize(keys.size());
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{2, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{2, 20000}});
dbfull()->MultiGet(ro, keys.size(), cfs.data(), keys.data(),
pin_values.data(), statuses.data());
CheckStatus(statuses, 6);
@@ -3101,8 +2990,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
statuses.clear();
statuses.resize(keys.size());
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{3, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{3, 20000}});
dbfull()->MultiGet(ro, keys.size(), cfs.data(), keys.data(),
pin_values.data(), statuses.data());
CheckStatus(statuses, 8);
@@ -3128,8 +3016,7 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
statuses.clear();
statuses.resize(keys.size());
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{1, 20000, IOStatus::OK()}});
fs->SetDelaySequence(ro.deadline, {{1, 20000}});
dbfull()->MultiGet(ro, handles_[0], keys.size(), keys.data(),
pin_values.data(), statuses.data());
CheckStatus(statuses, 64);
@@ -3163,99 +3050,6 @@ TEST_F(DBBasicTest, ManifestWriteFailure) {
Reopen(options);
}
TEST_F(DBBasicTest, PointLookupDeadline) {
std::shared_ptr<DeadlineFS> fs = std::make_shared<DeadlineFS>(env_);
std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, fs));
// Since we call SetTimeElapseOnlySleep, Close() later on may not work
// properly for the DB that's opened by the DBTestBase constructor.
Close();
for (int option_config = kDefault; option_config < kEnd; ++option_config) {
if (ShouldSkipOptions(option_config, kSkipPlainTable | kSkipMmapReads)) {
continue;
}
option_config_ = option_config;
Options options = CurrentOptions();
if (options.use_direct_reads) {
continue;
}
options.env = env.get();
options.disable_auto_compactions = true;
Cache* block_cache = nullptr;
env_->SetTimeElapseOnlySleep(&options);
// Fileter block reads currently don't cause the request to get
// aborted on a read timeout, so its possible those block reads
// may get issued even if the deadline is past
SyncPoint::GetInstance()->SetCallBack(
"BlockBasedTable::Get:BeforeFilterMatch",
[&](void* /*arg*/) { fs->IgnoreDeadline(true); });
SyncPoint::GetInstance()->SetCallBack(
"BlockBasedTable::Get:AfterFilterMatch",
[&](void* /*arg*/) { fs->IgnoreDeadline(false); });
// DB open will create table readers unless we reduce the table cache
// capacity.
// SanitizeOptions will set max_open_files to minimum of 20. Table cache
// is allocated with max_open_files - 10 as capacity. So override
// max_open_files to 11 so table cache capacity will become 1. This will
// prevent file open during DB open and force the file to be opened
// during MultiGet
SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = (int*)arg;
*max_open_files = 11;
});
SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
if (options.table_factory &&
!strcmp(options.table_factory->Name(),
BlockBasedTableFactory::kName.c_str())) {
BlockBasedTableFactory* bbtf =
static_cast<BlockBasedTableFactory*>(options.table_factory.get());
block_cache = bbtf->table_options().block_cache.get();
}
Random rnd(301);
for (int i = 0; i < 400; ++i) {
std::string key = "k" + ToString(i);
Put(key, rnd.RandomString(100));
}
Flush();
bool timedout = true;
// A timeout will be forced when the IO counter reaches this value
int io_deadline_trigger = 0;
// Keep incrementing io_deadline_trigger and call Get() until there is an
// iteration that doesn't cause a timeout. This ensures that we cover
// all file reads in the point lookup path that can potentially timeout
// and cause the Get() to fail.
while (timedout) {
ReadOptions ro;
ro.deadline = std::chrono::microseconds{env->NowMicros() + 10000};
fs->SetDelaySequence(
ro.deadline, {std::tuple<int, int, IOStatus>{
io_deadline_trigger, 20000, IOStatus::TimedOut()}});
block_cache->SetCapacity(0);
block_cache->SetCapacity(1048576);
std::string value;
Status s = dbfull()->Get(ro, "k50", &value);
if (fs->TimedOut()) {
ASSERT_EQ(s, Status::TimedOut());
} else {
timedout = false;
ASSERT_OK(s);
}
io_deadline_trigger++;
}
// Reset the delay sequence in order to avoid false alarms during Reopen
fs->SetDelaySequence(std::chrono::microseconds::zero(), {});
}
Close();
}
} // namespace ROCKSDB_NAMESPACE
#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
+2 -4
View File
@@ -7,12 +7,10 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <cstdlib>
#include "cache/lru_cache.h"
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "util/compression.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -766,7 +764,7 @@ TEST_F(DBBlockCacheTest, CompressedCache) {
std::string str;
for (int i = 0; i < num_iter; i++) {
if (i % 4 == 0) { // high compression ratio
str = rnd.RandomString(1000);
str = RandomString(&rnd, 1000);
}
values.push_back(str);
ASSERT_OK(Put(1, Key(i), values[i]));
@@ -853,7 +851,7 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
for (int i = 0; i < kNumFiles; ++i) {
ASSERT_EQ(i, NumTableFilesAtLevel(0, 0));
for (int j = 0; j < kNumEntriesPerFile; ++j) {
std::string value = rnd.RandomString(kNumBytesPerEntry);
std::string value = RandomString(&rnd, kNumBytesPerEntry);
ASSERT_OK(Put(Key(j * kNumFiles + i), value.c_str()));
}
ASSERT_OK(Flush());
+79 -234
View File
@@ -14,10 +14,9 @@
#include "rocksdb/experimental.h"
#include "rocksdb/sst_file_writer.h"
#include "rocksdb/utilities/convenience.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "util/concurrent_task_limiter_impl.h"
#include "util/random.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -296,7 +295,7 @@ TEST_P(DBCompactionTestWithParam, CompactionDeletionTrigger) {
const int kTestSize = kCDTKeysPerBuffer * 1024;
std::vector<std::string> values;
for (int k = 0; k < kTestSize; ++k) {
values.push_back(rnd.RandomString(kCDTValueSize));
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
dbfull()->TEST_WaitForFlushMemTable();
@@ -344,7 +343,7 @@ TEST_P(DBCompactionTestWithParam, CompactionsPreserveDeletes) {
const int kTestSize = kCDTKeysPerBuffer;
std::vector<std::string> values;
for (int k = 0; k < kTestSize; ++k) {
values.push_back(rnd.RandomString(kCDTValueSize));
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
@@ -409,7 +408,7 @@ TEST_F(DBCompactionTest, SkipStatsUpdateTest) {
const int kTestSize = kCDTKeysPerBuffer * 512;
std::vector<std::string> values;
for (int k = 0; k < kTestSize; ++k) {
values.push_back(rnd.RandomString(kCDTValueSize));
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
@@ -556,7 +555,7 @@ TEST_P(DBCompactionTestWithParam, CompactionDeletionTriggerReopen) {
const int kTestSize = kCDTKeysPerBuffer * 512;
std::vector<std::string> values;
for (int k = 0; k < kTestSize; ++k) {
values.push_back(rnd.RandomString(kCDTValueSize));
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
dbfull()->TEST_WaitForFlushMemTable();
@@ -674,7 +673,7 @@ TEST_F(DBCompactionTest, DisableStatsUpdateReopen) {
const int kTestSize = kCDTKeysPerBuffer * 512;
std::vector<std::string> values;
for (int k = 0; k < kTestSize; ++k) {
values.push_back(rnd.RandomString(kCDTValueSize));
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
dbfull()->TEST_WaitForFlushMemTable();
@@ -737,7 +736,7 @@ TEST_P(DBCompactionTestWithParam, CompactionTrigger) {
std::vector<std::string> values;
// Write 100KB (100 values, each 1K)
for (int i = 0; i < kNumKeysPerFile; i++) {
values.push_back(rnd.RandomString(990));
values.push_back(RandomString(&rnd, 990));
ASSERT_OK(Put(1, Key(i), values[i]));
}
// put extra key to trigger flush
@@ -749,7 +748,7 @@ TEST_P(DBCompactionTestWithParam, CompactionTrigger) {
// generate one more file in level-0, and should trigger level-0 compaction
std::vector<std::string> values;
for (int i = 0; i < kNumKeysPerFile; i++) {
values.push_back(rnd.RandomString(990));
values.push_back(RandomString(&rnd, 990));
ASSERT_OK(Put(1, Key(i), values[i]));
}
// put extra key to trigger flush
@@ -868,7 +867,7 @@ TEST_P(DBCompactionTestWithParam, CompactionsGenerateMultipleFiles) {
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 0);
std::vector<std::string> values;
for (int i = 0; i < 80; i++) {
values.push_back(rnd.RandomString(100000));
values.push_back(RandomString(&rnd, 100000));
ASSERT_OK(Put(1, Key(i), values[i]));
}
@@ -977,60 +976,6 @@ TEST_F(DBCompactionTest, UserKeyCrossFile2) {
ASSERT_EQ("NOT_FOUND", Get("3"));
}
TEST_F(DBCompactionTest, CompactionSstPartitioner) {
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 3;
std::shared_ptr<SstPartitionerFactory> factory(
NewSstPartitionerFixedPrefixFactory(4));
options.sst_partitioner_factory = factory;
DestroyAndReopen(options);
// create first file and flush to l0
Put("aaaa1", "A");
Put("bbbb1", "B");
Flush();
dbfull()->TEST_WaitForFlushMemTable();
Put("aaaa1", "A2");
Flush();
dbfull()->TEST_WaitForFlushMemTable();
// move both files down to l1
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
std::vector<LiveFileMetaData> files;
dbfull()->GetLiveFilesMetaData(&files);
ASSERT_EQ(2, files.size());
ASSERT_EQ("A2", Get("aaaa1"));
ASSERT_EQ("B", Get("bbbb1"));
}
TEST_F(DBCompactionTest, CompactionSstPartitionerNonTrivial) {
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
options.level0_file_num_compaction_trigger = 1;
std::shared_ptr<SstPartitionerFactory> factory(
NewSstPartitionerFixedPrefixFactory(4));
options.sst_partitioner_factory = factory;
DestroyAndReopen(options);
// create first file and flush to l0
Put("aaaa1", "A");
Put("bbbb1", "B");
Flush();
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact(true);
std::vector<LiveFileMetaData> files;
dbfull()->GetLiveFilesMetaData(&files);
ASSERT_EQ(2, files.size());
ASSERT_EQ("A", Get("aaaa1"));
ASSERT_EQ("B", Get("bbbb1"));
}
TEST_F(DBCompactionTest, ZeroSeqIdCompaction) {
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleLevel;
@@ -1160,7 +1105,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveOneFile) {
Random rnd(301);
std::vector<std::string> values;
for (int i = 0; i < num_keys; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(Key(i), values[i]));
}
@@ -1232,7 +1177,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveNonOverlappingFiles) {
std::map<int32_t, std::string> values;
for (size_t i = 0; i < ranges.size(); i++) {
for (int32_t j = ranges[i].first; j <= ranges[i].second; j++) {
values[j] = rnd.RandomString(value_size);
values[j] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(j), values[j]));
}
ASSERT_OK(Flush());
@@ -1278,7 +1223,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveNonOverlappingFiles) {
};
for (size_t i = 0; i < ranges.size(); i++) {
for (int32_t j = ranges[i].first; j <= ranges[i].second; j++) {
values[j] = rnd.RandomString(value_size);
values[j] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(j), values[j]));
}
ASSERT_OK(Flush());
@@ -1323,14 +1268,14 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveTargetLevel) {
// file 1 [0 => 300]
for (int32_t i = 0; i <= 300; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 2 [600 => 700]
for (int32_t i = 600; i <= 700; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1404,14 +1349,14 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
// file 1 [0 => 100]
for (int32_t i = 0; i < 100; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 2 [100 => 300]
for (int32_t i = 100; i < 300; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1432,7 +1377,7 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
// file 3 [ 0 => 200]
for (int32_t i = 0; i < 200; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1464,21 +1409,21 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
TEST_SYNC_POINT("DBCompaction::ManualPartial:1");
// file 4 [300 => 400)
for (int32_t i = 300; i <= 400; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 5 [400 => 500)
for (int32_t i = 400; i <= 500; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 6 [500 => 600)
for (int32_t i = 500; i <= 600; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
// Second non-trivial compaction is triggered
@@ -1546,14 +1491,14 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
// file 1 [0 => 100]
for (int32_t i = 0; i < 100; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 2 [100 => 300]
for (int32_t i = 100; i < 300; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1572,7 +1517,7 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
// file 3 [ 0 => 200]
for (int32_t i = 0; i < 200; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1604,7 +1549,7 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
ASSERT_OK(Flush());
dbfull()->TEST_WaitForFlushMemTable();
}
values[j] = rnd.RandomString(value_size);
values[j] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(j), values[j]));
}
}
@@ -1675,14 +1620,14 @@ TEST_F(DBCompactionTest, DeleteFileRange) {
// file 1 [0 => 100]
for (int32_t i = 0; i < 100; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
// file 2 [100 => 300]
for (int32_t i = 100; i < 300; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1698,7 +1643,7 @@ TEST_F(DBCompactionTest, DeleteFileRange) {
// file 3 [ 0 => 200]
for (int32_t i = 0; i < 200; i++) {
values[i] = rnd.RandomString(value_size);
values[i] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1710,7 +1655,7 @@ TEST_F(DBCompactionTest, DeleteFileRange) {
ASSERT_OK(Flush());
dbfull()->TEST_WaitForFlushMemTable();
}
values[j] = rnd.RandomString(value_size);
values[j] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(j), values[j]));
}
}
@@ -1797,7 +1742,7 @@ TEST_F(DBCompactionTest, DeleteFilesInRanges) {
for (auto i = 0; i < 10; i++) {
for (auto j = 0; j < 100; j++) {
auto k = i * 100 + j;
values[k] = rnd.RandomString(value_size);
values[k] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(k), values[k]));
}
ASSERT_OK(Flush());
@@ -1929,7 +1874,7 @@ TEST_F(DBCompactionTest, DeleteFileRangeFileEndpointsOverlapBug) {
// would cause `1 -> vals[0]` (an older key) to reappear.
std::string vals[kNumL0Files];
for (int i = 0; i < kNumL0Files; ++i) {
vals[i] = rnd.RandomString(kValSize);
vals[i] = RandomString(&rnd, kValSize);
Put(Key(i), vals[i]);
Put(Key(i + 1), vals[i]);
Flush();
@@ -1971,7 +1916,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveToLastLevelWithFiles) {
std::vector<std::string> values;
// File with keys [ 0 => 99 ]
for (int i = 0; i < 100; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -1989,7 +1934,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveToLastLevelWithFiles) {
// File with keys [ 100 => 199 ]
for (int i = 100; i < 200; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(Key(i), values[i]));
}
ASSERT_OK(Flush());
@@ -2384,7 +2329,7 @@ TEST_P(DBCompactionTestWithParam, ConvertCompactionStyle) {
for (int i = 0; i <= max_key_level_insert; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, 10000)));
}
ASSERT_OK(Flush(1));
dbfull()->TEST_WaitForCompact();
@@ -2442,7 +2387,7 @@ TEST_P(DBCompactionTestWithParam, ConvertCompactionStyle) {
ReopenWithColumnFamilies({"default", "pikachu"}, options);
for (int i = max_key_level_insert / 2; i <= max_key_universal_insert; i++) {
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, 10000)));
}
dbfull()->Flush(FlushOptions());
ASSERT_OK(Flush(1));
@@ -2737,7 +2682,7 @@ TEST_P(DBCompactionTestWithParam, DISABLED_CompactFilesOnLevelCompaction) {
Random rnd(301);
for (int key = 64 * kEntriesPerBuffer; key >= 0; --key) {
ASSERT_OK(Put(1, ToString(key), rnd.RandomString(kTestValueSize)));
ASSERT_OK(Put(1, ToString(key), RandomString(&rnd, kTestValueSize)));
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
@@ -2813,8 +2758,8 @@ TEST_P(DBCompactionTestWithParam, PartialCompactionFailure) {
std::vector<std::string> keys;
std::vector<std::string> values;
for (int k = 0; k < kNumInsertedKeys; ++k) {
keys.emplace_back(rnd.RandomString(kKeySize));
values.emplace_back(rnd.RandomString(kKvSize - kKeySize));
keys.emplace_back(RandomString(&rnd, kKeySize));
values.emplace_back(RandomString(&rnd, kKvSize - kKeySize));
ASSERT_OK(Put(Slice(keys[k]), Slice(values[k])));
dbfull()->TEST_WaitForFlushMemTable();
}
@@ -2880,7 +2825,7 @@ TEST_P(DBCompactionTestWithParam, DeleteMovedFileAfterCompaction) {
for (int i = 0; i < 2; ++i) {
// Create 1MB sst file
for (int j = 0; j < 100; ++j) {
ASSERT_OK(Put(Key(i * 50 + j), rnd.RandomString(10 * 1024)));
ASSERT_OK(Put(Key(i * 50 + j), RandomString(&rnd, 10 * 1024)));
}
ASSERT_OK(Flush());
}
@@ -2915,7 +2860,7 @@ TEST_P(DBCompactionTestWithParam, DeleteMovedFileAfterCompaction) {
for (int i = 0; i < 2; ++i) {
// Create 1MB sst file
for (int j = 0; j < 100; ++j) {
ASSERT_OK(Put(Key(i * 50 + j + 100), rnd.RandomString(10 * 1024)));
ASSERT_OK(Put(Key(i * 50 + j + 100), RandomString(&rnd, 10 * 1024)));
}
ASSERT_OK(Flush());
}
@@ -3173,7 +3118,7 @@ TEST_P(DBCompactionTestWithParam, ForceBottommostLevelCompaction) {
std::vector<std::string> values;
// File with keys [ 0 => 99 ]
for (int i = 0; i < 100; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(ShortKey(i), values[i]));
}
ASSERT_OK(Flush());
@@ -3190,7 +3135,7 @@ TEST_P(DBCompactionTestWithParam, ForceBottommostLevelCompaction) {
// File with keys [ 100 => 199 ]
for (int i = 100; i < 200; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(ShortKey(i), values[i]));
}
ASSERT_OK(Flush());
@@ -3208,7 +3153,7 @@ TEST_P(DBCompactionTestWithParam, ForceBottommostLevelCompaction) {
// File with keys [ 200 => 299 ]
for (int i = 200; i < 300; i++) {
values.push_back(rnd.RandomString(value_size));
values.push_back(RandomString(&rnd, value_size));
ASSERT_OK(Put(ShortKey(i), values[i]));
}
ASSERT_OK(Flush());
@@ -3252,7 +3197,7 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
const size_t kValueSize = 1 << 20;
Random rnd(301);
std::string value(rnd.RandomString(kValueSize));
std::string value(RandomString(&rnd, kValueSize));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompactionBySize:0",
@@ -3278,6 +3223,7 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
ASSERT_OK(Put(Key(i + 1), value));
}
ASSERT_OK(Flush());
ASSERT_EQ(i + 1, TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS));
}
dbfull()->TEST_WaitForCompact();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
@@ -3316,7 +3262,7 @@ TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
const size_t kValueSize = 1 << 20;
Random rnd(301);
std::string value(rnd.RandomString(kValueSize));
std::string value(RandomString(&rnd, kValueSize));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompactionBySize:0",
@@ -3535,7 +3481,7 @@ TEST_F(DBCompactionTest, CompactBottomLevelFilesWithDeletions) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
if (i == kNumLevelFiles - 1) {
snapshot = db_->GetSnapshot();
@@ -3607,7 +3553,7 @@ TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -3653,7 +3599,7 @@ TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -3748,7 +3694,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
// Add two L6 files with key ranges: [1 .. 100], [101 .. 200].
Random rnd(301);
for (int i = 1; i <= 100; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
// Get the first file's creation time. This will be the oldest file in the
@@ -3761,7 +3707,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
// Add 1 hour and do another flush.
env_->addon_time_.fetch_add(1 * 60 * 60);
for (int i = 101; i <= 200; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
MoveFilesToLevel(6);
@@ -3770,12 +3716,12 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
env_->addon_time_.fetch_add(1 * 60 * 60);
// Add two L4 files with key ranges: [1 .. 50], [51 .. 150].
for (int i = 1; i <= 50; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
env_->addon_time_.fetch_add(1 * 60 * 60);
for (int i = 51; i <= 150; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
MoveFilesToLevel(4);
@@ -3784,7 +3730,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
env_->addon_time_.fetch_add(1 * 60 * 60);
// Add one L1 file with key range: [26, 75].
for (int i = 26; i <= 75; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
dbfull()->TEST_WaitForCompact();
@@ -3895,8 +3841,8 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
ASSERT_OK(Put(Key(i * kNumKeysPerFile + j),
RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -3990,7 +3936,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
for (int i = 0; i < kNumFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
// Move the first two files to L2.
@@ -4053,7 +3999,7 @@ TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -4164,7 +4110,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithCompactionFilters) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -4224,7 +4170,7 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByL0FileCount) {
Random rnd(301);
for (int j = 0; j < kNumL0FilesLimit - 1; ++j) {
for (int k = 0; k < 2; ++k) {
ASSERT_OK(Put(Key(k), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(k), RandomString(&rnd, 1024)));
}
Flush();
}
@@ -4278,7 +4224,7 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByImmMemTableCount) {
Random rnd(301);
for (int j = 0; j < kNumImmMemTableLimit - 1; ++j) {
ASSERT_OK(Put(Key(0), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(0), RandomString(&rnd, 1024)));
FlushOptions flush_opts;
flush_opts.wait = false;
flush_opts.allow_write_stall = true;
@@ -4326,7 +4272,7 @@ TEST_F(DBCompactionTest, CompactRangeShutdownWhileDelayed) {
Random rnd(301);
for (int j = 0; j < kNumL0FilesLimit - 1; ++j) {
for (int k = 0; k < 2; ++k) {
ASSERT_OK(Put(1, Key(k), rnd.RandomString(1024)));
ASSERT_OK(Put(1, Key(k), RandomString(&rnd, 1024)));
}
Flush(1);
}
@@ -4386,7 +4332,7 @@ TEST_F(DBCompactionTest, CompactRangeSkipFlushAfterDelay) {
flush_opts.allow_write_stall = true;
for (int i = 0; i < kNumL0FilesLimit - 1; ++i) {
for (int j = 0; j < 2; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 1024)));
}
dbfull()->Flush(flush_opts);
}
@@ -4397,9 +4343,9 @@ TEST_F(DBCompactionTest, CompactRangeSkipFlushAfterDelay) {
});
TEST_SYNC_POINT("DBCompactionTest::CompactRangeSkipFlushAfterDelay:PreFlush");
Put(ToString(0), rnd.RandomString(1024));
Put(ToString(0), RandomString(&rnd, 1024));
dbfull()->Flush(flush_opts);
Put(ToString(0), rnd.RandomString(1024));
Put(ToString(0), RandomString(&rnd, 1024));
TEST_SYNC_POINT("DBCompactionTest::CompactRangeSkipFlushAfterDelay:PostFlush");
manual_compaction_thread.join();
@@ -4839,7 +4785,7 @@ TEST_P(CompactionPriTest, Test) {
RandomShuffle(std::begin(keys), std::end(keys), rnd.Next());
for (int i = 0; i < kNKeys; i++) {
ASSERT_OK(Put(Key(keys[i]), rnd.RandomString(102)));
ASSERT_OK(Put(Key(keys[i]), RandomString(&rnd, 102)));
}
dbfull()->TEST_WaitForCompact();
@@ -4881,7 +4827,7 @@ TEST_F(DBCompactionTest, PartialManualCompaction) {
Random rnd(301);
for (auto i = 0; i < 8; ++i) {
for (auto j = 0; j < 10; ++j) {
Merge("foo", rnd.RandomString(1024));
Merge("foo", RandomString(&rnd, 1024));
}
Flush();
}
@@ -4913,8 +4859,8 @@ TEST_F(DBCompactionTest, ManualCompactionFailsInReadOnlyMode) {
Random rnd(301);
for (int i = 0; i < kNumL0Files; ++i) {
// Make sure files are overlapping in key-range to prevent trivial move.
Put("key1", rnd.RandomString(1024));
Put("key2", rnd.RandomString(1024));
Put("key1", RandomString(&rnd, 1024));
Put("key2", RandomString(&rnd, 1024));
Flush();
}
ASSERT_EQ(kNumL0Files, NumTableFilesAtLevel(0));
@@ -4923,7 +4869,7 @@ TEST_F(DBCompactionTest, ManualCompactionFailsInReadOnlyMode) {
mock_env->SetFilesystemActive(false);
// Make sure this is outside `CompactRange`'s range so that it doesn't fail
// early trying to flush memtable.
ASSERT_NOK(Put("key3", rnd.RandomString(1024)));
ASSERT_NOK(Put("key3", RandomString(&rnd, 1024)));
// In the bug scenario, the first manual compaction would fail and forget to
// unregister itself, causing the second one to hang forever due to conflict
@@ -4962,7 +4908,7 @@ TEST_F(DBCompactionTest, ManualCompactionBottomLevelOptimized) {
for (auto i = 0; i < 8; ++i) {
for (auto j = 0; j < 10; ++j) {
ASSERT_OK(
Put("foo" + std::to_string(i * 10 + j), rnd.RandomString(1024)));
Put("foo" + std::to_string(i * 10 + j), RandomString(&rnd, 1024)));
}
Flush();
}
@@ -4972,7 +4918,7 @@ TEST_F(DBCompactionTest, ManualCompactionBottomLevelOptimized) {
for (auto i = 0; i < 8; ++i) {
for (auto j = 0; j < 10; ++j) {
ASSERT_OK(
Put("bar" + std::to_string(i * 10 + j), rnd.RandomString(1024)));
Put("bar" + std::to_string(i * 10 + j), RandomString(&rnd, 1024)));
}
Flush();
}
@@ -5006,7 +4952,7 @@ TEST_F(DBCompactionTest, CompactionDuringShutdown) {
for (auto i = 0; i < 2; ++i) {
for (auto j = 0; j < 10; ++j) {
ASSERT_OK(
Put("foo" + std::to_string(i * 10 + j), rnd.RandomString(1024)));
Put("foo" + std::to_string(i * 10 + j), RandomString(&rnd, 1024)));
}
Flush();
}
@@ -5029,7 +4975,7 @@ TEST_P(DBCompactionTestWithParam, FixFileIngestionCompactionDeadlock) {
// Generate an external SST file containing a single key, i.e. 99
std::string sst_files_dir = dbname_ + "/sst_files/";
DestroyDir(env_, sst_files_dir);
test::DestroyDir(env_, sst_files_dir);
ASSERT_OK(env_->CreateDir(sst_files_dir));
SstFileWriter sst_writer(EnvOptions(), options);
const std::string sst_file_path = sst_files_dir + "test.sst";
@@ -5056,7 +5002,7 @@ TEST_P(DBCompactionTestWithParam, FixFileIngestionCompactionDeadlock) {
// Generate level0_stop_writes_trigger L0 files to trigger write stop
for (int i = 0; i != options.level0_file_num_compaction_trigger; ++i) {
for (int j = 0; j != kNumKeysPerFile; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(990)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 990)));
}
if (0 == i) {
// When we reach here, the memtables have kNumKeysPerFile keys. Note that
@@ -5148,7 +5094,7 @@ TEST_F(DBCompactionTest, ConsistencyFailTest2) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
std::string value = rnd.RandomString(1000);
std::string value = RandomString(&rnd, 1000);
ASSERT_OK(Put("foo1", value));
ASSERT_OK(Put("z", ""));
@@ -5195,7 +5141,7 @@ TEST_P(DBCompactionTestWithParam,
const size_t kValueSize = 1 << 20;
Random rnd(301);
std::atomic<int> pick_intra_l0_count(0);
std::string value(rnd.RandomString(kValueSize));
std::string value(RandomString(&rnd, kValueSize));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBCompactionTestWithParam::FlushAfterIntraL0:1",
@@ -5237,6 +5183,7 @@ TEST_P(DBCompactionTestWithParam,
// Put one key, to make biggest log sequence number in this memtable is bigger
// than sst which would be ingested in next step.
ASSERT_OK(Put(Key(2), "b"));
ASSERT_EQ(10, NumTableFilesAtLevel(0));
dbfull()->TEST_WaitForCompact();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
std::vector<std::vector<FileMetaData>> level_to_files;
@@ -5262,8 +5209,8 @@ TEST_P(DBCompactionTestWithParam,
const size_t kValueSize = 1 << 20;
Random rnd(301);
std::string value(rnd.RandomString(kValueSize));
std::string value2(rnd.RandomString(kValueSize));
std::string value(RandomString(&rnd, kValueSize));
std::string value2(RandomString(&rnd, kValueSize));
std::string bigvalue = value + value;
// prevents trivial move
@@ -5333,108 +5280,6 @@ TEST_P(DBCompactionTestWithParam,
}
}
TEST_F(DBCompactionTest, UpdateLevelSubCompactionTest) {
Options options = CurrentOptions();
options.max_subcompactions = 10;
options.target_file_size_base = 1 << 10; // 1KB
DestroyAndReopen(options);
bool has_compaction = false;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->max_subcompactions() == 10);
has_compaction = true;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_TRUE(dbfull()->GetDBOptions().max_subcompactions == 10);
// Trigger compaction
for (int i = 0; i < 32; i++) {
for (int j = 0; j < 5000; j++) {
Put(std::to_string(j), std::string(1, 'A'));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(has_compaction);
has_compaction = false;
ASSERT_OK(dbfull()->SetDBOptions({{"max_subcompactions", "2"}}));
ASSERT_TRUE(dbfull()->GetDBOptions().max_subcompactions == 2);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->max_subcompactions() == 2);
has_compaction = true;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Trigger compaction
for (int i = 0; i < 32; i++) {
for (int j = 0; j < 5000; j++) {
Put(std::to_string(j), std::string(1, 'A'));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(has_compaction);
}
TEST_F(DBCompactionTest, UpdateUniversalSubCompactionTest) {
Options options = CurrentOptions();
options.max_subcompactions = 10;
options.compaction_style = kCompactionStyleUniversal;
options.target_file_size_base = 1 << 10; // 1KB
DestroyAndReopen(options);
bool has_compaction = false;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"UniversalCompactionBuilder::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->max_subcompactions() == 10);
has_compaction = true;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Trigger compaction
for (int i = 0; i < 32; i++) {
for (int j = 0; j < 5000; j++) {
Put(std::to_string(j), std::string(1, 'A'));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(has_compaction);
has_compaction = false;
ASSERT_OK(dbfull()->SetDBOptions({{"max_subcompactions", "2"}}));
ASSERT_TRUE(dbfull()->GetDBOptions().max_subcompactions == 2);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"UniversalCompactionBuilder::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->max_subcompactions() == 2);
has_compaction = true;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Trigger compaction
for (int i = 0; i < 32; i++) {
for (int j = 0; j < 5000; j++) {
Put(std::to_string(j), std::string(1, 'A'));
}
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(has_compaction);
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
+17 -18
View File
@@ -15,7 +15,6 @@
#include "db/db_test_util.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
class DBTestDynamicLevel : public DBTestBase {
@@ -81,9 +80,9 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase) {
for (int i = 0; i < kNKeys; i++) {
int key = keys[i];
ASSERT_OK(Put(Key(kNKeys + key), rnd.RandomString(102)));
ASSERT_OK(Put(Key(key), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kNKeys * 2 + key), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kNKeys + key), RandomString(&rnd, 102)));
ASSERT_OK(Put(Key(key), RandomString(&rnd, 102)));
ASSERT_OK(Put(Key(kNKeys * 2 + key), RandomString(&rnd, 102)));
ASSERT_OK(Delete(Key(kNKeys + keys[i / 10])));
env_->SleepForMicroseconds(5000);
}
@@ -159,7 +158,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
// Put about 28K to L0
for (int i = 0; i < 70; i++) {
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
rnd.RandomString(380)));
RandomString(&rnd, 380)));
}
ASSERT_OK(dbfull()->SetOptions({
{"disable_auto_compactions", "false"},
@@ -176,7 +175,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
}));
for (int i = 0; i < 70; i++) {
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
rnd.RandomString(380)));
RandomString(&rnd, 380)));
}
ASSERT_OK(dbfull()->SetOptions({
@@ -198,7 +197,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
// Write about 40K more
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
rnd.RandomString(380)));
RandomString(&rnd, 380)));
}
ASSERT_OK(dbfull()->SetOptions({
{"disable_auto_compactions", "false"},
@@ -217,7 +216,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
// Each file is about 11KB, with 9KB of data.
for (int i = 0; i < 1300; i++) {
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
rnd.RandomString(380)));
RandomString(&rnd, 380)));
}
// Make sure that the compaction starts before the last bit of data is
@@ -258,7 +257,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
TEST_SYNC_POINT("DynamicLevelMaxBytesBase2:1");
for (int i = 0; i < 2; i++) {
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
rnd.RandomString(380)));
RandomString(&rnd, 380)));
}
TEST_SYNC_POINT("DynamicLevelMaxBytesBase2:2");
@@ -311,15 +310,15 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesCompactRange) {
// Put about 7K to L0
for (int i = 0; i < 140; i++) {
ASSERT_OK(
Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))), rnd.RandomString(80)));
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
RandomString(&rnd, 80)));
}
Flush();
dbfull()->TEST_WaitForCompact();
if (NumTableFilesAtLevel(0) == 0) {
// Make sure level 0 is not empty
ASSERT_OK(
Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))), rnd.RandomString(80)));
ASSERT_OK(Put(Key(static_cast<int>(rnd.Uniform(kMaxKey))),
RandomString(&rnd, 80)));
Flush();
}
@@ -383,7 +382,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBaseInc) {
const int total_keys = 3000;
const int random_part_size = 100;
for (int i = 0; i < total_keys; i++) {
std::string value = rnd.RandomString(random_part_size);
std::string value = RandomString(&rnd, random_part_size);
PutFixed32(&value, static_cast<uint32_t>(i));
ASSERT_OK(Put(Key(i), value));
}
@@ -442,8 +441,8 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
int total_keys = 1000;
for (int i = 0; i < total_keys; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kMaxKey + i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 102)));
ASSERT_OK(Put(Key(kMaxKey + i), RandomString(&rnd, 102)));
ASSERT_OK(Delete(Key(i / 10)));
}
verify_func(total_keys, false);
@@ -476,8 +475,8 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
int total_keys2 = 2000;
for (int i = total_keys; i < total_keys2; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kMaxKey + i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 102)));
ASSERT_OK(Put(Key(kMaxKey + i), RandomString(&rnd, 102)));
ASSERT_OK(Delete(Key(i / 10)));
}
+5 -5
View File
@@ -13,10 +13,10 @@
#include "db/db_test_util.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "util/cast_util.h"
#include "util/mutexlock.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -89,7 +89,7 @@ TEST_F(DBFlushTest, SyncFail) {
CreateAndReopenWithCF({"pikachu"}, options);
Put("key", "value");
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
reinterpret_cast<ColumnFamilyHandleImpl*>(db_->DefaultColumnFamily())
->cfd();
FlushOptions flush_options;
flush_options.wait = false;
@@ -379,9 +379,9 @@ TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
DBImpl* db_impl = static_cast_with_check<DBImpl>(db);
InstrumentedMutex* mutex = db_impl->mutex();
mutex->Lock();
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(
db->DefaultColumnFamily())
->cfd();
auto* cfd =
reinterpret_cast<ColumnFamilyHandleImpl*>(db->DefaultColumnFamily())
->cfd();
ASSERT_LT(seq, cfd->imm()->current()->GetEarliestSequenceNumber());
mutex->Unlock();
}
+29 -89
View File
@@ -87,10 +87,10 @@
#include "table/get_context.h"
#include "table/merging_iterator.h"
#include "table/multiget_context.h"
#include "table/sst_file_dumper.h"
#include "table/table_builder.h"
#include "table/two_level_iterator.h"
#include "test_util/sync_point.h"
#include "tools/sst_dump_tool_imp.h"
#include "util/autovector.h"
#include "util/build_version.h"
#include "util/cast_util.h"
@@ -249,8 +249,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
new ColumnFamilyMemTablesImpl(versions_->GetColumnFamilySet()));
DumpRocksDBBuildVersion(immutable_db_options_.info_log.get());
SetDbSessionId();
DumpDBFileSummary(immutable_db_options_, dbname_, db_session_id_);
DumpDBFileSummary(immutable_db_options_, dbname_);
immutable_db_options_.Dump(immutable_db_options_.info_log.get());
mutable_db_options_.Dump(immutable_db_options_.info_log.get());
DumpSupportInfo(immutable_db_options_.info_log.get());
@@ -329,8 +328,7 @@ Status DBImpl::ResumeImpl() {
// whether there are flush jobs with non-empty data to flush, triggering
// appends to MANIFEST.
VersionEdit edit;
auto cfh =
static_cast_with_check<ColumnFamilyHandleImpl>(default_cf_handle_);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(default_cf_handle_);
assert(cfh);
ColumnFamilyData* cfd = cfh->cfd();
const MutableCFOptions& cf_opts = *cfd->GetLatestMutableCFOptions();
@@ -685,17 +683,9 @@ void DBImpl::StartTimedTasks() {
stats_dump_period_sec = mutable_db_options_.stats_dump_period_sec;
if (stats_dump_period_sec > 0) {
if (!thread_dump_stats_) {
// In case of many `DB::Open()` in rapid succession we can have all
// threads dumping at once, which causes severe lock contention in
// jemalloc. Ensure successive `DB::Open()`s are staggered by at least
// one second in the common case.
static std::atomic<uint64_t> stats_dump_threads_started(0);
thread_dump_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::DumpStats(); }, "dump_st", env_,
static_cast<uint64_t>(stats_dump_period_sec) * kMicrosInSecond,
stats_dump_threads_started.fetch_add(1) %
static_cast<uint64_t>(stats_dump_period_sec) *
kMicrosInSecond));
static_cast<uint64_t>(stats_dump_period_sec) * kMicrosInSecond));
}
}
stats_persist_period_sec = mutable_db_options_.stats_persist_period_sec;
@@ -954,8 +944,7 @@ Status DBImpl::SetOptions(
(void)options_map;
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
if (options_map.empty()) {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"SetOptions() on column family [%s], empty input",
@@ -1091,18 +1080,9 @@ Status DBImpl::SetDBOptions(
mutex_.Lock();
}
if (new_options.stats_dump_period_sec > 0) {
// In case many DBs have `stats_dump_period_sec` enabled in rapid
// succession, we can have all threads dumping at once, which causes
// severe lock contention in jemalloc. Ensure successive enabling of
// `stats_dump_period_sec` are staggered by at least one second in the
// common case.
static std::atomic<uint64_t> stats_dump_threads_started(0);
thread_dump_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::DumpStats(); }, "dump_st", env_,
static_cast<uint64_t>(new_options.stats_dump_period_sec) *
kMicrosInSecond,
stats_dump_threads_started.fetch_add(1) %
static_cast<uint64_t>(new_options.stats_dump_period_sec) *
kMicrosInSecond));
} else {
thread_dump_stats_.reset();
@@ -1371,7 +1351,7 @@ InternalIterator* DBImpl::NewInternalIterator(
if (column_family == nullptr) {
cfd = default_cf_handle_->cfd();
} else {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
cfd = cfh->cfd();
}
@@ -1600,8 +1580,8 @@ Status DBImpl::GetImpl(const ReadOptions& read_options, const Slice& key,
StopWatch sw(env_, stats_, DB_GET);
PERF_TIMER_GUARD(get_snapshot_time);
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(
get_impl_options.column_family);
auto cfh =
reinterpret_cast<ColumnFamilyHandleImpl*>(get_impl_options.column_family);
auto cfd = cfh->cfd();
if (tracer_) {
@@ -1802,7 +1782,7 @@ std::vector<Status> DBImpl::MultiGet(
std::unordered_map<uint32_t, MultiGetColumnFamilyData> multiget_cf_data(
column_family.size());
for (auto cf : column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(cf);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(cf);
auto cfd = cfh->cfd();
if (multiget_cf_data.find(cfd->GetID()) == multiget_cf_data.end()) {
multiget_cf_data.emplace(cfd->GetID(),
@@ -2204,11 +2184,10 @@ void DBImpl::PrepareMultiGetKeys(
KeyContext* lhs = (*sorted_keys)[index - 1];
KeyContext* rhs = (*sorted_keys)[index];
ColumnFamilyHandleImpl* cfh =
static_cast_with_check<ColumnFamilyHandleImpl>(lhs->column_family);
reinterpret_cast<ColumnFamilyHandleImpl*>(lhs->column_family);
uint32_t cfd_id1 = cfh->cfd()->GetID();
const Comparator* comparator = cfh->cfd()->user_comparator();
cfh =
static_cast_with_check<ColumnFamilyHandleImpl>(lhs->column_family);
cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(lhs->column_family);
uint32_t cfd_id2 = cfh->cfd()->GetID();
assert(cfd_id1 <= cfd_id2);
@@ -2572,7 +2551,7 @@ Status DBImpl::CreateColumnFamilyImpl(const ColumnFamilyOptions& cf_options,
// this is outside the mutex
if (s.ok()) {
NewThreadStatusCfInfo(
static_cast_with_check<ColumnFamilyHandleImpl>(*handle)->cfd());
reinterpret_cast<ColumnFamilyHandleImpl*>(*handle)->cfd());
}
return s;
}
@@ -2609,7 +2588,7 @@ Status DBImpl::DropColumnFamilies(
}
Status DBImpl::DropColumnFamilyImpl(ColumnFamilyHandle* column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
if (cfd->GetID() == 0) {
return Status::InvalidArgument("Can't drop default column family");
@@ -2705,6 +2684,8 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
return NewErrorIterator(
Status::NotSupported("Managed iterator is not supported anymore."));
}
// We will eventually support deadline for iterators too, but safeguard
// for now
if (read_options.deadline != std::chrono::microseconds::zero()) {
return NewErrorIterator(
Status::NotSupported("ReadOptions deadline is not supported"));
@@ -2723,7 +2704,7 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
"Iterator requested internal keys which are too old and are not"
" guaranteed to be preserved, try larger iter_start_seqnum opt."));
}
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
ColumnFamilyData* cfd = cfh->cfd();
assert(cfd != nullptr);
ReadCallback* read_callback = nullptr; // No read callback provided.
@@ -2858,7 +2839,7 @@ Status DBImpl::NewIterators(
"Tailing iterator not supported in RocksDB lite");
#else
for (auto cfh : column_families) {
auto cfd = static_cast_with_check<ColumnFamilyHandleImpl>(cfh)->cfd();
auto cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(cfh)->cfd();
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
auto iter = new ForwardIterator(this, read_options, cfd, sv,
/* allow_unprepared_value */ true);
@@ -2878,8 +2859,7 @@ Status DBImpl::NewIterators(
: versions_->LastSequence();
for (size_t i = 0; i < column_families.size(); ++i) {
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_families[i])
->cfd();
reinterpret_cast<ColumnFamilyHandleImpl*>(column_families[i])->cfd();
iterators->push_back(
NewIteratorImpl(read_options, cfd, snapshot, read_callback));
}
@@ -2986,7 +2966,7 @@ void DBImpl::ReleaseSnapshot(const Snapshot* s) {
#ifndef ROCKSDB_LITE
Status DBImpl::GetPropertiesOfAllTables(ColumnFamilyHandle* column_family,
TablePropertiesCollection* props) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
// Increment the ref count
@@ -3008,7 +2988,7 @@ Status DBImpl::GetPropertiesOfAllTables(ColumnFamilyHandle* column_family,
Status DBImpl::GetPropertiesOfTablesInRange(ColumnFamilyHandle* column_family,
const Range* range, std::size_t n,
TablePropertiesCollection* props) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
// Increment the ref count
@@ -3044,7 +3024,7 @@ FileSystem* DBImpl::GetFileSystem() const {
Options DBImpl::GetOptions(ColumnFamilyHandle* column_family) const {
InstrumentedMutexLock l(&mutex_);
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
return Options(BuildDBOptions(immutable_db_options_, mutable_db_options_),
cfh->cfd()->GetLatestCFOptions());
}
@@ -3058,8 +3038,7 @@ bool DBImpl::GetProperty(ColumnFamilyHandle* column_family,
const Slice& property, std::string* value) {
const DBPropertyInfo* property_info = GetPropertyInfo(property);
value->clear();
auto cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
if (property_info == nullptr) {
return false;
} else if (property_info->handle_int) {
@@ -3093,8 +3072,7 @@ bool DBImpl::GetMapProperty(ColumnFamilyHandle* column_family,
std::map<std::string, std::string>* value) {
const DBPropertyInfo* property_info = GetPropertyInfo(property);
value->clear();
auto cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
if (property_info == nullptr) {
return false;
} else if (property_info->handle_map) {
@@ -3113,8 +3091,7 @@ bool DBImpl::GetIntProperty(ColumnFamilyHandle* column_family,
if (property_info == nullptr || property_info->handle_int == nullptr) {
return false;
}
auto cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
return GetIntPropertyInternal(cfd, *property_info, false, value);
}
@@ -3287,7 +3264,7 @@ void DBImpl::GetApproximateMemTableStats(ColumnFamilyHandle* column_family,
uint64_t* const count,
uint64_t* const size) {
ColumnFamilyHandleImpl* cfh =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
ColumnFamilyData* cfd = cfh->cfd();
SuperVersion* sv = GetAndRefSuperVersion(cfd);
@@ -3312,7 +3289,7 @@ Status DBImpl::GetApproximateSizes(const SizeApproximationOptions& options,
}
Version* v;
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
SuperVersion* sv = GetAndRefSuperVersion(cfd);
v = sv->current;
@@ -3470,7 +3447,7 @@ Status DBImpl::DeleteFilesInRanges(ColumnFamilyHandle* column_family,
const RangePtr* ranges, size_t n,
bool include_end) {
Status status;
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
ColumnFamilyData* cfd = cfh->cfd();
VersionEdit edit;
std::set<FileMetaData*> deleted_files;
@@ -3560,16 +3537,10 @@ void DBImpl::GetLiveFilesMetaData(std::vector<LiveFileMetaData>* metadata) {
versions_->GetLiveFilesMetaData(metadata);
}
Status DBImpl::GetLiveFilesChecksumInfo(FileChecksumList* checksum_list) {
InstrumentedMutexLock l(&mutex_);
return versions_->GetLiveFilesChecksumInfo(checksum_list);
}
void DBImpl::GetColumnFamilyMetaData(ColumnFamilyHandle* column_family,
ColumnFamilyMetaData* cf_meta) {
assert(column_family);
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
auto* sv = GetAndRefSuperVersion(cfd);
{
// Without mutex, Version::GetColumnFamilyMetaData will have data race with
@@ -3685,37 +3656,6 @@ Status DBImpl::GetDbIdentityFromIdentityFile(std::string* identity) const {
return s;
}
Status DBImpl::GetDbSessionId(std::string& session_id) const {
session_id.assign(db_session_id_);
return Status::OK();
}
void DBImpl::SetDbSessionId() {
// GenerateUniqueId() generates an identifier that has a negligible
// probability of being duplicated, ~128 bits of entropy
std::string uuid = env_->GenerateUniqueId();
// Hash and reformat that down to a more compact format, 20 characters
// in base-36 ([0-9A-Z]), which is ~103 bits of entropy, which is enough
// to expect no collisions across a billion servers each opening DBs
// a million times (~2^50). Benefits vs. raw unique id:
// * Save ~ dozen bytes per SST file
// * Shorter shared backup file names (some platforms have low limits)
// * Visually distinct from DB id format
uint64_t a = NPHash64(uuid.data(), uuid.size(), 1234U);
uint64_t b = NPHash64(uuid.data(), uuid.size(), 5678U);
db_session_id_.resize(20);
static const char* const base36 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ";
size_t i = 0;
for (; i < 10U; ++i, a /= 36U) {
db_session_id_[i] = base36[a % 36];
}
for (; i < 20U; ++i, b /= 36U) {
db_session_id_[i] = base36[b % 36];
}
TEST_SYNC_POINT_CALLBACK("DBImpl::SetDbSessionId", &db_session_id_);
}
// Default implementation -- returns not supported status
Status DB::CreateColumnFamily(const ColumnFamilyOptions& /*cf_options*/,
const std::string& /*column_family_name*/,
@@ -4551,7 +4491,7 @@ Status DBImpl::CreateColumnFamilyWithImport(
}
// Import sst files from metadata.
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(*handle);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(*handle);
auto cfd = cfh->cfd();
ImportColumnFamilyJob import_job(env_, versions_.get(), cfd,
immutable_db_options_, file_options_,
+5 -17
View File
@@ -350,8 +350,6 @@ class DBImpl : public DB {
virtual Status GetDbIdentityFromIdentityFile(std::string* identity) const;
virtual Status GetDbSessionId(std::string& session_id) const override;
ColumnFamilyHandle* DefaultColumnFamily() const override;
ColumnFamilyHandle* PersistentStatsColumnFamily() const;
@@ -393,9 +391,6 @@ class DBImpl : public DB {
virtual void GetLiveFilesMetaData(
std::vector<LiveFileMetaData>* metadata) override;
virtual Status GetLiveFilesChecksumInfo(
FileChecksumList* checksum_list) override;
// Obtains the meta data of the specified column family of the DB.
// Status::NotFound() will be returned if the current DB does not have
// any column family match the specified name.
@@ -852,8 +847,8 @@ class DBImpl : public DB {
InstrumentedMutex* mutex() const { return &mutex_; }
// Initialize a brand new DB. The DB directory is expected to be empty before
// calling it. Push new manifest file name into `new_filenames`.
Status NewDB(std::vector<std::string>* new_filenames);
// calling it.
Status NewDB();
// This is to be used only by internal rocksdb classes.
static Status Open(const DBOptions& db_options, const std::string& name,
@@ -989,9 +984,6 @@ class DBImpl : public DB {
protected:
const std::string dbname_;
std::string db_id_;
// db_session_id_ is an identifier that gets reset
// every time the DB is opened
std::string db_session_id_;
std::unique_ptr<VersionSet> versions_;
// Flag to check whether we allocated and own the info log file
bool own_info_log_;
@@ -1173,10 +1165,6 @@ class DBImpl : public DB {
// bump up the version set's next_file_number_ to be 1 + largest_file_number.
Status FinishBestEffortsRecovery();
// SetDbSessionId() should be called in the constuctor DBImpl()
// to ensure that db_session_id_ gets updated every time the DB is opened
void SetDbSessionId();
private:
friend class DB;
friend class ErrorHandler;
@@ -1202,7 +1190,7 @@ class DBImpl : public DB {
friend class StatsHistoryTest_PersistentStatsCreateColumnFamilies_Test;
#ifndef NDEBUG
friend class DBTest2_ReadCallbackTest_Test;
friend class WriteCallbackPTest_WriteWithCallbackTest_Test;
friend class WriteCallbackTest_WriteWithCallbackTest_Test;
friend class XFTransactionWriteHandler;
friend class DBBlobIndexTest;
friend class WriteUnpreparedTransactionTest_RecoveryTest_Test;
@@ -1710,8 +1698,8 @@ class DBImpl : public DB {
size_t GetWalPreallocateBlockSize(uint64_t write_buffer_size) const;
Env::WriteLifeTimeHint CalculateWALWriteHint() { return Env::WLTH_SHORT; }
IOStatus CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
size_t preallocate_block_size, log::Writer** new_log);
Status CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
size_t preallocate_block_size, log::Writer** new_log);
// Validate self-consistency of DB options
static Status ValidateOptions(const DBOptions& db_options);
+16 -28
View File
@@ -154,8 +154,7 @@ Status DBImpl::FlushMemTableToOutputFile(
GetDataDir(cfd, 0U),
GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_,
&event_logger_, mutable_cf_options.report_bg_io_stats,
true /* sync_output_directory */, true /* write_manifest */, thread_pri,
db_id_, db_session_id_);
true /* sync_output_directory */, true /* write_manifest */, thread_pri);
FileMetaData file_meta;
TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:BeforePickMemtables");
@@ -194,9 +193,7 @@ Status DBImpl::FlushMemTableToOutputFile(
} else {
flush_job.Cancel();
}
if (io_s.ok()) {
io_s = flush_job.io_status();
}
io_s = flush_job.io_status();
if (s.ok()) {
InstallSuperVersionAndScheduleWork(cfd, superversion_context,
@@ -356,7 +353,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
data_dir, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
stats_, &event_logger_, mutable_cf_options.report_bg_io_stats,
false /* sync_output_directory */, false /* write_manifest */,
thread_pri, db_id_, db_session_id_));
thread_pri));
jobs.back()->PickMemTable();
}
@@ -697,7 +694,7 @@ void DBImpl::NotifyOnFlushCompleted(
Status DBImpl::CompactRange(const CompactRangeOptions& options,
ColumnFamilyHandle* column_family,
const Slice* begin, const Slice* end) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
if (options.target_path_id >= cfd->ioptions()->cf_paths.size()) {
@@ -887,8 +884,7 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
return Status::InvalidArgument("ColumnFamilyHandle must be non-null.");
}
auto cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
assert(cfd);
Status s;
@@ -1021,7 +1017,7 @@ Status DBImpl::CompactFilesImpl(
assert(cfd->compaction_picker());
c.reset(cfd->compaction_picker()->CompactFiles(
compact_options, input_files, output_level, version->storage_info(),
*cfd->GetLatestMutableCFOptions(), mutable_db_options_, output_path_id));
*cfd->GetLatestMutableCFOptions(), output_path_id));
// we already sanitized the set of input files and checked for conflicts
// without releasing the lock, so we're guaranteed a compaction can be formed.
assert(c != nullptr);
@@ -1051,8 +1047,7 @@ Status DBImpl::CompactFilesImpl(
snapshot_checker, table_cache_, &event_logger_,
c->mutable_cf_options()->paranoid_file_checks,
c->mutable_cf_options()->report_bg_io_stats, dbname_,
&compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_,
db_id_, db_session_id_);
&compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_);
// Creating a compaction influences the compaction score because the score
// takes running compactions into account (by skipping files that are already
@@ -1109,12 +1104,7 @@ Status DBImpl::CompactFilesImpl(
"[%s] [JOB %d] Compaction error: %s",
c->column_family_data()->GetName().c_str(),
job_context->job_id, status.ToString().c_str());
IOStatus io_s = compaction_job.io_status();
if (!io_s.ok()) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kCompaction);
} else {
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction);
}
error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction);
}
if (output_file_names != nullptr) {
@@ -1367,7 +1357,7 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
}
int DBImpl::NumberLevels(ColumnFamilyHandle* column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
return cfh->cfd()->NumberLevels();
}
@@ -1376,7 +1366,7 @@ int DBImpl::MaxMemCompactionLevel(ColumnFamilyHandle* /*column_family*/) {
}
int DBImpl::Level0StopWriteTrigger(ColumnFamilyHandle* column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
InstrumentedMutexLock l(&mutex_);
return cfh->cfd()
->GetSuperVersion()
@@ -1385,7 +1375,7 @@ int DBImpl::Level0StopWriteTrigger(ColumnFamilyHandle* column_family) {
Status DBImpl::Flush(const FlushOptions& flush_options,
ColumnFamilyHandle* column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
ROCKS_LOG_INFO(immutable_db_options_.info_log, "[%s] Manual flush start.",
cfh->GetName().c_str());
Status s;
@@ -1536,9 +1526,9 @@ Status DBImpl::RunManualCompaction(
scheduled ||
(((manual.manual_end = &manual.tmp_storage1) != nullptr) &&
((compaction = manual.cfd->CompactRange(
*manual.cfd->GetLatestMutableCFOptions(), mutable_db_options_,
manual.input_level, manual.output_level, compact_range_options,
manual.begin, manual.end, &manual.manual_end, &manual_conflict,
*manual.cfd->GetLatestMutableCFOptions(), manual.input_level,
manual.output_level, compact_range_options, manual.begin,
manual.end, &manual.manual_end, &manual_conflict,
max_file_num_to_ignore)) == nullptr &&
manual_conflict))) {
// exclusive manual compactions should not see a conflict during
@@ -2642,8 +2632,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
// compaction is not necessary. Need to make sure mutex is held
// until we make a copy in the following code
TEST_SYNC_POINT("DBImpl::BackgroundCompaction():BeforePickCompaction");
c.reset(cfd->PickCompaction(*mutable_cf_options, mutable_db_options_,
log_buffer));
c.reset(cfd->PickCompaction(*mutable_cf_options, log_buffer));
TEST_SYNC_POINT("DBImpl::BackgroundCompaction():AfterPickCompaction");
if (c != nullptr) {
@@ -2847,8 +2836,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
&event_logger_, c->mutable_cf_options()->paranoid_file_checks,
c->mutable_cf_options()->report_bg_io_stats, dbname_,
&compaction_job_stats, thread_pri,
is_manual ? &manual_compaction_paused_ : nullptr, db_id_,
db_session_id_);
is_manual ? &manual_compaction_paused_ : nullptr);
compaction_job.Prepare();
NotifyOnCompactionBegin(c->column_family_data(), c.get(), status,
+6 -6
View File
@@ -47,7 +47,7 @@ int64_t DBImpl::TEST_MaxNextLevelOverlappingBytes(
if (column_family == nullptr) {
cfd = default_cf_handle_->cfd();
} else {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
cfd = cfh->cfd();
}
InstrumentedMutexLock l(&mutex_);
@@ -57,7 +57,7 @@ int64_t DBImpl::TEST_MaxNextLevelOverlappingBytes(
void DBImpl::TEST_GetFilesMetaData(
ColumnFamilyHandle* column_family,
std::vector<std::vector<FileMetaData>>* metadata) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
InstrumentedMutexLock l(&mutex_);
metadata->resize(NumberLevels());
@@ -88,7 +88,7 @@ Status DBImpl::TEST_CompactRange(int level, const Slice* begin,
if (column_family == nullptr) {
cfd = default_cf_handle_->cfd();
} else {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
cfd = cfh->cfd();
}
int output_level =
@@ -131,7 +131,7 @@ Status DBImpl::TEST_FlushMemTable(bool wait, bool allow_write_stall,
if (cfh == nullptr) {
cfd = default_cf_handle_->cfd();
} else {
auto cfhi = static_cast_with_check<ColumnFamilyHandleImpl>(cfh);
auto cfhi = reinterpret_cast<ColumnFamilyHandleImpl*>(cfh);
cfd = cfhi->cfd();
}
return FlushMemTable(cfd, fo, FlushReason::kTest);
@@ -152,7 +152,7 @@ Status DBImpl::TEST_WaitForFlushMemTable(ColumnFamilyHandle* column_family) {
if (column_family == nullptr) {
cfd = default_cf_handle_->cfd();
} else {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
cfd = cfh->cfd();
}
return WaitForFlushMemTable(cfd, nullptr, false);
@@ -242,7 +242,7 @@ Status DBImpl::TEST_GetLatestMutableCFOptions(
ColumnFamilyHandle* column_family, MutableCFOptions* mutable_cf_options) {
InstrumentedMutexLock l(&mutex_);
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
*mutable_cf_options = *cfh->cfd()->GetLatestMutableCFOptions();
return Status::OK();
}
+3 -3
View File
@@ -7,22 +7,22 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/db_impl/db_impl.h"
#include <cinttypes>
#include <vector>
#include "db/column_family.h"
#include "db/db_impl/db_impl.h"
#include "db/job_context.h"
#include "db/version_set.h"
#include "rocksdb/status.h"
#include "util/cast_util.h"
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
Status DBImpl::SuggestCompactRange(ColumnFamilyHandle* column_family,
const Slice* begin, const Slice* end) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
InternalKey start_key, end_key;
if (begin != nullptr) {
+1 -1
View File
@@ -190,7 +190,7 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
// set of all files in the directory. We'll exclude files that are still
// alive in the subsequent processings.
std::vector<std::string> files;
env_->GetChildren(path, &files).PermitUncheckedError(); // Ignore errors
env_->GetChildren(path, &files); // Ignore errors
for (const std::string& file : files) {
uint64_t number;
FileType type;
+26 -54
View File
@@ -253,7 +253,7 @@ Status DBImpl::ValidateOptions(const DBOptions& db_options) {
return Status::OK();
}
Status DBImpl::NewDB(std::vector<std::string>* new_filenames) {
Status DBImpl::NewDB() {
VersionEdit new_db;
Status s = SetIdentityFile(env_, dbname_);
if (!s.ok()) {
@@ -293,10 +293,6 @@ Status DBImpl::NewDB(std::vector<std::string>* new_filenames) {
if (s.ok()) {
// Make "CURRENT" file that points to the new manifest file.
s = SetCurrentFile(fs_.get(), dbname_, 1, directories_.GetDbDir());
if (new_filenames) {
new_filenames->emplace_back(
manifest.substr(manifest.find_last_of("/\\") + 1));
}
} else {
fs_->DeleteFile(manifest, IOOptions(), nullptr);
}
@@ -360,7 +356,6 @@ Status DBImpl::Recover(
bool is_new_db = false;
assert(db_lock_ == nullptr);
std::vector<std::string> files_in_dbname;
if (!read_only) {
Status s = directories_.SetDirectories(fs_.get(), dbname_,
immutable_db_options_.wal_dir,
@@ -384,12 +379,10 @@ Status DBImpl::Recover(
s = env_->FileExists(current_fname);
} else {
s = Status::NotFound();
Status io_s = env_->GetChildren(dbname_, &files_in_dbname);
if (!io_s.ok()) {
s = io_s;
files_in_dbname.clear();
}
for (const std::string& file : files_in_dbname) {
std::vector<std::string> files;
// No need to check return value
env_->GetChildren(dbname_, &files);
for (const std::string& file : files) {
uint64_t number = 0;
FileType type = kLogFile; // initialize
if (ParseFileName(file, &number, &type) && type == kDescriptorFile) {
@@ -403,7 +396,7 @@ Status DBImpl::Recover(
}
if (s.IsNotFound()) {
if (immutable_db_options_.create_if_missing) {
s = NewDB(&files_in_dbname);
s = NewDB();
is_new_db = true;
if (!s.ok()) {
return s;
@@ -445,16 +438,6 @@ Status DBImpl::Recover(
}
}
}
} else if (immutable_db_options_.best_efforts_recovery) {
assert(files_in_dbname.empty());
Status s = env_->GetChildren(dbname_, &files_in_dbname);
if (s.IsNotFound()) {
return Status::InvalidArgument(dbname_,
"does not exist (open for read only)");
} else if (s.IsIOError()) {
return s;
}
assert(s.ok());
}
assert(db_id_.empty());
Status s;
@@ -462,9 +445,8 @@ Status DBImpl::Recover(
if (!immutable_db_options_.best_efforts_recovery) {
s = versions_->Recover(column_families, read_only, &db_id_);
} else {
assert(!files_in_dbname.empty());
s = versions_->TryRecover(column_families, read_only, files_in_dbname,
&db_id_, &missing_table_file);
s = versions_->TryRecover(column_families, read_only, &db_id_,
&missing_table_file);
if (s.ok()) {
// TryRecover may delete previous column_family_set_.
column_family_memtables_.reset(
@@ -524,7 +506,6 @@ Status DBImpl::Recover(
s = InitPersistStatsColumnFamily();
}
std::vector<std::string> files_in_wal_dir;
if (s.ok()) {
// Initial max_total_in_memory_state_ before recovery logs. Log recovery
// may check this value to decide whether to flush.
@@ -551,8 +532,9 @@ Status DBImpl::Recover(
// Note that prev_log_number() is no longer used, but we pay
// attention to it in case we are recovering a database
// produced by an older version of rocksdb.
std::vector<std::string> filenames;
if (!immutable_db_options_.best_efforts_recovery) {
s = env_->GetChildren(immutable_db_options_.wal_dir, &files_in_wal_dir);
s = env_->GetChildren(immutable_db_options_.wal_dir, &filenames);
}
if (s.IsNotFound()) {
return Status::InvalidArgument("wal_dir not found",
@@ -562,15 +544,15 @@ Status DBImpl::Recover(
}
std::vector<uint64_t> logs;
for (const auto& file : files_in_wal_dir) {
for (size_t i = 0; i < filenames.size(); i++) {
uint64_t number;
FileType type;
if (ParseFileName(file, &number, &type) && type == kLogFile) {
if (ParseFileName(filenames[i], &number, &type) && type == kLogFile) {
if (is_new_db) {
return Status::Corruption(
"While creating a new Db, wal_dir contains "
"existing log file: ",
file);
filenames[i]);
} else {
logs.push_back(number);
}
@@ -622,24 +604,15 @@ Status DBImpl::Recover(
// to reflect the most recent OPTIONS file. It does not matter for regular
// read-write db instance because options_file_number_ will later be
// updated to versions_->NewFileNumber() in RenameTempFileToOptionsFile.
std::vector<std::string> filenames;
std::vector<std::string> file_names;
if (s.ok()) {
const std::string normalized_dbname = NormalizePath(dbname_);
const std::string normalized_wal_dir =
NormalizePath(immutable_db_options_.wal_dir);
if (immutable_db_options_.best_efforts_recovery) {
filenames = std::move(files_in_dbname);
} else if (normalized_dbname == normalized_wal_dir) {
filenames = std::move(files_in_wal_dir);
} else {
s = env_->GetChildren(GetName(), &filenames);
}
s = env_->GetChildren(GetName(), &file_names);
}
if (s.ok()) {
uint64_t number = 0;
uint64_t options_file_number = 0;
FileType type;
for (const auto& fname : filenames) {
for (const auto& fname : file_names) {
if (ParseFileName(fname, &number, &type) && type == kOptionsFile) {
options_file_number = std::max(number, options_file_number);
}
@@ -647,6 +620,7 @@ Status DBImpl::Recover(
versions_->options_file_number_ = options_file_number;
}
}
return s;
}
@@ -1315,8 +1289,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
mutable_cf_options.compression_opts, paranoid_file_checks,
cfd->internal_stats(), TableFileCreationReason::kRecovery, &io_s,
&event_logger_, job_id, Env::IO_HIGH, nullptr /* table_properties */,
-1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
0 /* file_creation_time */, db_id_, db_session_id_);
-1 /* level */, current_time, write_hint);
LogFlush(immutable_db_options_.info_log);
ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[%s] [WriteLevel0TableForRecovery]"
@@ -1388,10 +1361,9 @@ Status DB::Open(const DBOptions& db_options, const std::string& dbname,
!kSeqPerBatch, kBatchPerTxn);
}
IOStatus DBImpl::CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
size_t preallocate_block_size,
log::Writer** new_log) {
IOStatus io_s;
Status DBImpl::CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
size_t preallocate_block_size, log::Writer** new_log) {
Status s;
std::unique_ptr<FSWritableFile> lfile;
DBOptions db_options =
@@ -1409,13 +1381,13 @@ IOStatus DBImpl::CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
LogFileName(immutable_db_options_.wal_dir, recycle_log_number);
TEST_SYNC_POINT("DBImpl::CreateWAL:BeforeReuseWritableFile1");
TEST_SYNC_POINT("DBImpl::CreateWAL:BeforeReuseWritableFile2");
io_s = fs_->ReuseWritableFile(log_fname, old_log_fname, opt_file_options,
&lfile, /*dbg=*/nullptr);
s = fs_->ReuseWritableFile(log_fname, old_log_fname, opt_file_options,
&lfile, /*dbg=*/nullptr);
} else {
io_s = NewWritableFile(fs_.get(), log_fname, &lfile, opt_file_options);
s = NewWritableFile(fs_.get(), log_fname, &lfile, opt_file_options);
}
if (io_s.ok()) {
if (s.ok()) {
lfile->SetWriteLifeTimeHint(CalculateWALWriteHint());
lfile->SetPreallocationBlockSize(preallocate_block_size);
@@ -1427,7 +1399,7 @@ IOStatus DBImpl::CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
immutable_db_options_.recycle_log_file_num > 0,
immutable_db_options_.manual_wal_flush);
}
return io_s;
return s;
}
Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
+5 -6
View File
@@ -4,14 +4,13 @@
// (found in the LICENSE.Apache file in the root directory).
#include "db/db_impl/db_impl_readonly.h"
#include "db/arena_wrapped_db_iter.h"
#include "db/compacted_db_impl.h"
#include "db/db_impl/db_impl.h"
#include "db/db_iter.h"
#include "db/merge_context.h"
#include "monitoring/perf_context_imp.h"
#include "util/cast_util.h"
namespace ROCKSDB_NAMESPACE {
@@ -36,7 +35,7 @@ Status DBImplReadOnly::Get(const ReadOptions& read_options,
PERF_TIMER_GUARD(get_snapshot_time);
Status s;
SequenceNumber snapshot = versions_->LastSequence();
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
if (tracer_) {
InstrumentedMutexLock lock(&trace_mutex_);
@@ -71,7 +70,7 @@ Status DBImplReadOnly::Get(const ReadOptions& read_options,
Iterator* DBImplReadOnly::NewIterator(const ReadOptions& read_options,
ColumnFamilyHandle* column_family) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
SuperVersion* super_version = cfd->GetSuperVersion()->Ref();
SequenceNumber latest_snapshot = versions_->LastSequence();
@@ -112,7 +111,7 @@ Status DBImplReadOnly::NewIterators(
: latest_snapshot;
for (auto cfh : column_families) {
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(cfh)->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(cfh)->cfd();
auto* sv = cfd->GetSuperVersion()->Ref();
auto* db_iter = NewArenaWrappedDbIterator(
env_, read_options, *cfd->ioptions(), sv->mutable_cf_options, read_seq,
@@ -237,7 +236,7 @@ Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
*dbptr = impl;
for (auto* h : *handles) {
impl->NewThreadStatusCfInfo(
static_cast_with_check<ColumnFamilyHandleImpl>(h)->cfd());
reinterpret_cast<ColumnFamilyHandleImpl*>(h)->cfd());
}
} else {
for (auto h : *handles) {
+2 -2
View File
@@ -388,7 +388,7 @@ Iterator* DBImplSecondary::NewIterator(const ReadOptions& read_options,
"ReadTier::kPersistedData is not yet supported in iterators."));
}
Iterator* result = nullptr;
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
auto cfd = cfh->cfd();
ReadCallback* read_callback = nullptr; // No read callback provided.
if (read_options.tailing) {
@@ -642,7 +642,7 @@ Status DB::OpenAsSecondary(
*dbptr = impl;
for (auto h : *handles) {
impl->NewThreadStatusCfInfo(
static_cast_with_check<ColumnFamilyHandleImpl>(h)->cfd());
reinterpret_cast<ColumnFamilyHandleImpl*>(h)->cfd());
}
} else {
for (auto h : *handles) {
+7 -19
View File
@@ -6,15 +6,14 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <cinttypes>
#include "db/db_impl/db_impl.h"
#include <cinttypes>
#include "db/error_handler.h"
#include "db/event_helpers.h"
#include "monitoring/perf_context_imp.h"
#include "options/options_helper.h"
#include "test_util/sync_point.h"
#include "util/cast_util.h"
namespace ROCKSDB_NAMESPACE {
// Convenience methods
@@ -25,7 +24,7 @@ Status DBImpl::Put(const WriteOptions& o, ColumnFamilyHandle* column_family,
Status DBImpl::Merge(const WriteOptions& o, ColumnFamilyHandle* column_family,
const Slice& key, const Slice& val) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
if (!cfh->cfd()->ioptions()->merge_operator) {
return Status::NotSupported("Provide a merge_operator when opening DB");
} else {
@@ -1638,7 +1637,6 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
std::unique_ptr<WritableFile> lfile;
log::Writer* new_log = nullptr;
MemTable* new_mem = nullptr;
IOStatus io_s;
// Recoverable state is persisted in WAL. After memtable switch, WAL might
// be deleted, so we write the state to memtable to be persisted as well.
@@ -1684,11 +1682,8 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
if (creating_new_log) {
// TODO: Write buffer size passed in should be max of all CF's instead
// of mutable_cf_options.write_buffer_size.
io_s = CreateWAL(new_log_number, recycle_log_number, preallocate_block_size,
&new_log);
if (s.ok()) {
s = io_s;
}
s = CreateWAL(new_log_number, recycle_log_number, preallocate_block_size,
&new_log);
}
if (s.ok()) {
SequenceNumber seq = versions_->LastSequence();
@@ -1714,10 +1709,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
if (!logs_.empty()) {
// Alway flush the buffer of the last log before switching to a new one
log::Writer* cur_log_writer = logs_.back().writer;
io_s = cur_log_writer->WriteBuffer();
if (s.ok()) {
s = io_s;
}
s = cur_log_writer->WriteBuffer();
if (!s.ok()) {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"[%s] Failed to switch from #%" PRIu64 " to #%" PRIu64
@@ -1752,11 +1744,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
}
// We may have lost data from the WritableFileBuffer in-memory buffer for
// the current log, so treat it as a fatal error and set bg_error
if (!io_s.ok()) {
error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable);
} else {
error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
}
error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
// Read back bg_error in order to get the right severity
s = error_handler_.GetBGError();
return s;
+1 -1
View File
@@ -10,8 +10,8 @@
#include "db/db_impl/db_impl_secondary.h"
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
+1 -4
View File
@@ -17,8 +17,7 @@
namespace ROCKSDB_NAMESPACE {
void DumpDBFileSummary(const ImmutableDBOptions& options,
const std::string& dbname,
const std::string& session_id) {
const std::string& dbname) {
if (options.info_log == nullptr) {
return;
}
@@ -33,8 +32,6 @@ void DumpDBFileSummary(const ImmutableDBOptions& options,
std::string file_info, wal_info;
Header(options.info_log, "DB SUMMARY\n");
Header(options.info_log, "DB Session ID: %s\n", session_id.c_str());
// Get files in dbname dir
if (!env->GetChildren(dbname, &files).ok()) {
Error(options.info_log,
+1 -2
View File
@@ -10,6 +10,5 @@
namespace ROCKSDB_NAMESPACE {
void DumpDBFileSummary(const ImmutableDBOptions& options,
const std::string& dbname,
const std::string& session_id = "");
const std::string& dbname);
} // namespace ROCKSDB_NAMESPACE
+4 -5
View File
@@ -9,7 +9,6 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -282,8 +281,8 @@ TEST_F(DBIOFailureTest, FlushSstRangeSyncError) {
Random rnd(301);
std::string rnd_str =
rnd.RandomString(static_cast<int>(options.bytes_per_sync / 2));
std::string rnd_str_512kb = rnd.RandomString(512 * 1024);
RandomString(&rnd, static_cast<int>(options.bytes_per_sync / 2));
std::string rnd_str_512kb = RandomString(&rnd, 512 * 1024);
ASSERT_OK(Put(1, "foo", "bar"));
// First 1MB doesn't get range synced
@@ -331,8 +330,8 @@ TEST_F(DBIOFailureTest, CompactSstRangeSyncError) {
Random rnd(301);
std::string rnd_str =
rnd.RandomString(static_cast<int>(options.bytes_per_sync / 2));
std::string rnd_str_512kb = rnd.RandomString(512 * 1024);
RandomString(&rnd, static_cast<int>(options.bytes_per_sync / 2));
std::string rnd_str_512kb = RandomString(&rnd, 512 * 1024);
ASSERT_OK(Put(1, "foo", "bar"));
// First 1MB doesn't get range synced
+4 -7
View File
@@ -97,8 +97,7 @@ struct StressTestIterator : public InternalIterator {
bool MaybeFail() {
if (rnd->Next() >=
static_cast<double>(std::numeric_limits<uint64_t>::max()) *
error_probability) {
std::numeric_limits<uint64_t>::max() * error_probability) {
return false;
}
if (rnd->Next() % 2) {
@@ -115,8 +114,7 @@ struct StressTestIterator : public InternalIterator {
void MaybeMutate() {
if (rnd->Next() >=
static_cast<double>(std::numeric_limits<uint64_t>::max()) *
mutation_probability) {
std::numeric_limits<uint64_t>::max() * mutation_probability) {
return;
}
do {
@@ -128,9 +126,8 @@ struct StressTestIterator : public InternalIterator {
if (data->hidden.empty()) {
hide_probability = 1;
}
bool do_hide = rnd->Next() <
static_cast<double>(std::numeric_limits<uint64_t>::max()) *
hide_probability;
bool do_hide =
rnd->Next() < std::numeric_limits<uint64_t>::max() * hide_probability;
if (do_hide) {
// Hide a random entry.
size_t idx = rnd->Next() % data->entries.size();
+18 -20
View File
@@ -17,7 +17,6 @@
#include "rocksdb/iostats_context.h"
#include "rocksdb/perf_context.h"
#include "table/block_based/flush_block_policy.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -41,8 +40,7 @@ class DBIteratorTest : public DBTestBase,
if (column_family == nullptr) {
column_family = db_->DefaultColumnFamily();
}
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
auto* cfd = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family)->cfd();
SequenceNumber seq = read_options.snapshot != nullptr
? read_options.snapshot->GetSequenceNumber()
: db_->GetLatestSequenceNumber();
@@ -195,10 +193,10 @@ TEST_P(DBIteratorTest, IterReseekNewUpperBound) {
options.compression = kNoCompression;
Reopen(options);
ASSERT_OK(Put("a", rnd.RandomString(400)));
ASSERT_OK(Put("aabb", rnd.RandomString(400)));
ASSERT_OK(Put("aaef", rnd.RandomString(400)));
ASSERT_OK(Put("b", rnd.RandomString(400)));
ASSERT_OK(Put("a", RandomString(&rnd, 400)));
ASSERT_OK(Put("aabb", RandomString(&rnd, 400)));
ASSERT_OK(Put("aaef", RandomString(&rnd, 400)));
ASSERT_OK(Put("b", RandomString(&rnd, 400)));
dbfull()->Flush(FlushOptions());
ReadOptions opts;
Slice ub = Slice("aa");
@@ -1361,7 +1359,7 @@ class DBIteratorTestForPinnedData : public DBIteratorTest {
std::vector<std::string> generated_keys(key_pool);
for (int i = 0; i < key_pool; i++) {
generated_keys[i] = rnd.RandomString(key_size);
generated_keys[i] = RandomString(&rnd, key_size);
}
std::map<std::string, std::string> true_data;
@@ -1369,7 +1367,7 @@ class DBIteratorTestForPinnedData : public DBIteratorTest {
std::vector<std::string> deleted_keys;
for (int i = 0; i < puts; i++) {
auto& k = generated_keys[rnd.Next() % key_pool];
auto v = rnd.RandomString(val_size);
auto v = RandomString(&rnd, val_size);
// Insert data to true_data map and to DB
true_data[k] = v;
@@ -1532,7 +1530,7 @@ TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
Random rnd(301);
for (int i = 1; i <= 1000; i++) {
std::string k = Key(i * 3);
std::string v = rnd.RandomString(100);
std::string v = RandomString(&rnd, 100);
ASSERT_OK(Put(k, v));
true_data[k] = v;
if (i % 250 == 0) {
@@ -1546,7 +1544,7 @@ TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
// Generate 4 sst files in L0
for (int i = 1; i <= 1000; i++) {
std::string k = Key(i * 2);
std::string v = rnd.RandomString(100);
std::string v = RandomString(&rnd, 100);
ASSERT_OK(Put(k, v));
true_data[k] = v;
if (i % 250 == 0) {
@@ -1558,7 +1556,7 @@ TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
// Add some keys/values in memtables
for (int i = 1; i <= 1000; i++) {
std::string k = Key(i);
std::string v = rnd.RandomString(100);
std::string v = RandomString(&rnd, 100);
ASSERT_OK(Put(k, v));
true_data[k] = v;
}
@@ -1660,8 +1658,8 @@ TEST_P(DBIteratorTest, PinnedDataIteratorReadAfterUpdate) {
std::map<std::string, std::string> true_data;
for (int i = 0; i < 1000; i++) {
std::string k = rnd.RandomString(10);
std::string v = rnd.RandomString(1000);
std::string k = RandomString(&rnd, 10);
std::string v = RandomString(&rnd, 1000);
ASSERT_OK(Put(k, v));
true_data[k] = v;
}
@@ -1675,7 +1673,7 @@ TEST_P(DBIteratorTest, PinnedDataIteratorReadAfterUpdate) {
if (rnd.OneIn(2)) {
ASSERT_OK(Delete(kv.first));
} else {
std::string new_val = rnd.RandomString(1000);
std::string new_val = RandomString(&rnd, 1000);
ASSERT_OK(Put(kv.first, new_val));
}
}
@@ -1932,7 +1930,7 @@ TEST_P(DBIteratorTest, IterPrevKeyCrossingBlocksRandomized) {
for (int i = 0; i < kNumKeys; i++) {
gen_key = Key(i);
gen_val = rnd.RandomString(kValSize);
gen_val = RandomString(&rnd, kValSize);
ASSERT_OK(Put(gen_key, gen_val));
true_data[gen_key] = gen_val;
@@ -1950,7 +1948,7 @@ TEST_P(DBIteratorTest, IterPrevKeyCrossingBlocksRandomized) {
for (int j = 0; j < kNumMergeOperands; j++) {
gen_key = Key(i);
gen_val = rnd.RandomString(kValSize);
gen_val = RandomString(&rnd, kValSize);
ASSERT_OK(db_->Merge(WriteOptions(), gen_key, gen_val));
true_data[gen_key] += "," + gen_val;
@@ -2050,7 +2048,7 @@ TEST_P(DBIteratorTest, IteratorWithLocalStatistics) {
Random rnd(301);
for (int i = 0; i < 1000; i++) {
// Key 10 bytes / Value 10 bytes
ASSERT_OK(Put(rnd.RandomString(10), rnd.RandomString(10)));
ASSERT_OK(Put(RandomString(&rnd, 10), RandomString(&rnd, 10)));
}
std::atomic<uint64_t> total_next(0);
@@ -2706,7 +2704,7 @@ TEST_P(DBIteratorTest, AvoidReseekLevelIterator) {
Reopen(options);
Random rnd(301);
std::string random_str = rnd.RandomString(180);
std::string random_str = RandomString(&rnd, 180);
ASSERT_OK(Put("1", random_str));
ASSERT_OK(Put("2", random_str));
@@ -2913,7 +2911,7 @@ TEST_F(DBIteratorWithReadCallbackTest, ReadCallback) {
SequenceNumber seq2 = db_->GetLatestSequenceNumber();
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
reinterpret_cast<ColumnFamilyHandleImpl*>(db_->DefaultColumnFamily())
->cfd();
// The iterator are suppose to see data before seq1.
Iterator* iter =
+2 -2
View File
@@ -303,14 +303,14 @@ TEST_F(DBMemTableTest, InsertWithHint) {
ASSERT_EQ(hint_bar, rep->last_hint_in());
ASSERT_EQ(hint_bar, rep->last_hint_out());
ASSERT_EQ(5, rep->num_insert_with_hint());
ASSERT_OK(Put("NotInPrefixDomain", "vvv"));
ASSERT_OK(Put("whitelisted", "vvv"));
ASSERT_EQ(5, rep->num_insert_with_hint());
ASSERT_EQ("foo_v1", Get("foo_k1"));
ASSERT_EQ("foo_v2", Get("foo_k2"));
ASSERT_EQ("foo_v3", Get("foo_k3"));
ASSERT_EQ("bar_v1", Get("bar_k1"));
ASSERT_EQ("bar_v2", Get("bar_k2"));
ASSERT_EQ("vvv", Get("NotInPrefixDomain"));
ASSERT_EQ("vvv", Get("whitelisted"));
}
TEST_F(DBMemTableTest, ColumnFamilyId) {
+1 -1
View File
@@ -8,11 +8,11 @@
#include "rocksdb/perf_context.h"
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_builder.h"
#include "test_util/fault_injection_test_env.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "rocksdb/merge_operator.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators.h"
#include "utilities/merge_operators/sortlist.h"
#include "utilities/merge_operators/string_append/stringappend2.h"
+5 -6
View File
@@ -9,7 +9,6 @@
#include "db/forward_iterator.h"
#include "port/stack_trace.h"
#include "rocksdb/merge_operator.h"
#include "util/random.h"
#include "utilities/merge_operators.h"
#include "utilities/merge_operators/string_append/stringappend2.h"
@@ -243,7 +242,7 @@ TEST_P(MergeOperatorPinningTest, OperandsMultiBlocks) {
std::string key = Key(key_id % 35);
key_id++;
for (int k = 0; k < kOperandsPerKeyPerFile; k++) {
std::string val = rnd.RandomString(kOperandSize);
std::string val = RandomString(&rnd, kOperandSize);
ASSERT_OK(db_->Merge(WriteOptions(), key, val));
if (true_data[key].size() == 0) {
true_data[key] = val;
@@ -328,7 +327,7 @@ TEST_P(MergeOperatorPinningTest, EvictCacheBeforeMerge) {
for (int i = 0; i < kNumOperands; i++) {
for (int j = 0; j < kNumKeys; j++) {
std::string k = Key(j);
std::string v = rnd.RandomString(kOperandSize);
std::string v = RandomString(&rnd, kOperandSize);
ASSERT_OK(db_->Merge(WriteOptions(), k, v));
true_data[k] = std::max(true_data[k], v);
@@ -621,7 +620,7 @@ TEST_P(PerConfigMergeOperatorPinningTest, Randomized) {
// kNumPutBefore keys will have base values
for (int i = 0; i < kNumPutBefore; i++) {
std::string key = Key(rnd.Next() % kKeyRange);
std::string value = rnd.RandomString(kOperandSize);
std::string value = RandomString(&rnd, kOperandSize);
ASSERT_OK(db_->Put(WriteOptions(), key, value));
true_data[key] = value;
@@ -630,7 +629,7 @@ TEST_P(PerConfigMergeOperatorPinningTest, Randomized) {
// Do kTotalMerges merges
for (int i = 0; i < kTotalMerges; i++) {
std::string key = Key(rnd.Next() % kKeyRange);
std::string value = rnd.RandomString(kOperandSize);
std::string value = RandomString(&rnd, kOperandSize);
ASSERT_OK(db_->Merge(WriteOptions(), key, value));
if (true_data[key] < value) {
@@ -641,7 +640,7 @@ TEST_P(PerConfigMergeOperatorPinningTest, Randomized) {
// Overwrite random kNumPutAfter keys
for (int i = 0; i < kNumPutAfter; i++) {
std::string key = Key(rnd.Next() % kKeyRange);
std::string value = rnd.RandomString(kOperandSize);
std::string value = RandomString(&rnd, kOperandSize);
ASSERT_OK(db_->Put(WriteOptions(), key, value));
true_data[key] = value;
+8 -7
View File
@@ -39,7 +39,7 @@ class DBOptionsTest : public DBTestBase {
std::unordered_map<std::string, std::string> options_map;
StringToMap(options_str, &options_map);
std::unordered_map<std::string, std::string> mutable_map;
for (const auto& opt : db_options_type_info) {
for (const auto opt : db_options_type_info) {
if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
mutable_map[opt.first] = options_map[opt.first];
}
@@ -57,7 +57,7 @@ class DBOptionsTest : public DBTestBase {
std::unordered_map<std::string, std::string> options_map;
StringToMap(options_str, &options_map);
std::unordered_map<std::string, std::string> mutable_map;
for (const auto& opt : cf_options_type_info) {
for (const auto opt : cf_options_type_info) {
if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
mutable_map[opt.first] = options_map[opt.first];
}
@@ -493,7 +493,8 @@ TEST_F(DBOptionsTest, SetDelayedWriteRateOption) {
TEST_F(DBOptionsTest, MaxTotalWalSizeChange) {
Random rnd(1044);
const auto value_size = size_t(1024);
std::string value = rnd.RandomString(value_size);
std::string value;
test::RandomString(&rnd, value_size, &value);
Options options;
options.create_if_missing = true;
@@ -714,7 +715,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
}
@@ -745,7 +746,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
}
@@ -777,7 +778,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
}
@@ -841,7 +842,7 @@ TEST_F(DBOptionsTest, FIFOTtlBackwardCompatible) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
}
+18 -18
View File
@@ -126,8 +126,8 @@ TEST_F(DBPropertiesTest, GetAggregatedIntPropertyTest) {
Random rnd(301);
for (auto* handle : handles_) {
for (int i = 0; i < kKeyNum; ++i) {
db_->Put(WriteOptions(), handle, rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
db_->Put(WriteOptions(), handle, RandomString(&rnd, kKeySize),
RandomString(&rnd, kValueSize));
}
}
@@ -346,18 +346,18 @@ TEST_F(DBPropertiesTest, AggregatedTableProperties) {
Random rnd(5632);
for (int table = 1; table <= kTableCount; ++table) {
for (int i = 0; i < kPutsPerTable; ++i) {
db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
db_->Put(WriteOptions(), RandomString(&rnd, kKeySize),
RandomString(&rnd, kValueSize));
}
for (int i = 0; i < kDeletionsPerTable; i++) {
db_->Delete(WriteOptions(), rnd.RandomString(kKeySize));
db_->Delete(WriteOptions(), RandomString(&rnd, kKeySize));
}
for (int i = 0; i < kMergeOperandsPerTable; i++) {
db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
db_->Merge(WriteOptions(), RandomString(&rnd, kKeySize),
RandomString(&rnd, kValueSize));
}
for (int i = 0; i < kRangeDeletionsPerTable; i++) {
std::string start = rnd.RandomString(kKeySize);
std::string start = RandomString(&rnd, kKeySize);
std::string end = start;
end.resize(kValueSize);
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), start, end);
@@ -546,18 +546,18 @@ TEST_F(DBPropertiesTest, AggregatedTablePropertiesAtLevel) {
TableProperties tp, sum_tp, expected_tp;
for (int table = 1; table <= kTableCount; ++table) {
for (int i = 0; i < kPutsPerTable; ++i) {
db_->Put(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
db_->Put(WriteOptions(), RandomString(&rnd, kKeySize),
RandomString(&rnd, kValueSize));
}
for (int i = 0; i < kDeletionsPerTable; i++) {
db_->Delete(WriteOptions(), rnd.RandomString(kKeySize));
db_->Delete(WriteOptions(), RandomString(&rnd, kKeySize));
}
for (int i = 0; i < kMergeOperandsPerTable; i++) {
db_->Merge(WriteOptions(), rnd.RandomString(kKeySize),
rnd.RandomString(kValueSize));
db_->Merge(WriteOptions(), RandomString(&rnd, kKeySize),
RandomString(&rnd, kValueSize));
}
for (int i = 0; i < kRangeDeletionsPerTable; i++) {
std::string start = rnd.RandomString(kKeySize);
std::string start = RandomString(&rnd, kKeySize);
std::string end = start;
end.resize(kValueSize);
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), start, end);
@@ -920,7 +920,7 @@ TEST_F(DBPropertiesTest, ApproximateMemoryUsage) {
for (int r = 0; r < kNumRounds; ++r) {
for (int f = 0; f < kFlushesPerRound; ++f) {
for (int w = 0; w < kWritesPerFlush; ++w) {
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize));
Put(RandomString(&rnd, kKeySize), RandomString(&rnd, kValueSize));
}
}
// Make sure that there is no flush between getting the two properties.
@@ -938,7 +938,7 @@ TEST_F(DBPropertiesTest, ApproximateMemoryUsage) {
iters.push_back(db_->NewIterator(ReadOptions()));
for (int f = 0; f < kFlushesPerRound; ++f) {
for (int w = 0; w < kWritesPerFlush; ++w) {
Put(rnd.RandomString(kKeySize), rnd.RandomString(kValueSize));
Put(RandomString(&rnd, kKeySize), RandomString(&rnd, kValueSize));
}
}
// Force flush to prevent flush from happening between getting the
@@ -1296,8 +1296,8 @@ TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
const int kMaxKey = 1000;
for (int i = 0; i < kMaxKey; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(kMaxKey + i), rnd.RandomString(102)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 102)));
ASSERT_OK(Put(Key(kMaxKey + i), RandomString(&rnd, 102)));
}
Flush();
dbfull()->TEST_WaitForCompact();
+14 -15
View File
@@ -7,7 +7,6 @@
#include "port/stack_trace.h"
#include "rocksdb/utilities/write_batch_with_index.h"
#include "test_util/testutil.h"
#include "util/random.h"
#include "utilities/merge_operators.h"
namespace ROCKSDB_NAMESPACE {
@@ -125,7 +124,7 @@ TEST_F(DBRangeDelTest, CompactionOutputFilesExactlyFilled) {
std::vector<std::string> values;
// Write 12K (4 values, each 3K)
for (int j = 0; j < kNumPerFile; j++) {
values.push_back(rnd.RandomString(3 << 10));
values.push_back(RandomString(&rnd, 3 << 10));
ASSERT_OK(Put(Key(i * kNumPerFile + j), values[j]));
if (j == 0 && i > 0) {
dbfull()->TEST_WaitForFlushMemTable();
@@ -173,7 +172,7 @@ TEST_F(DBRangeDelTest, MaxCompactionBytesCutsOutputFiles) {
std::vector<std::string> values;
// Write 1MB (256 values, each 4K)
for (int j = 0; j < kNumPerFile; j++) {
values.push_back(rnd.RandomString(kBytesPerVal));
values.push_back(RandomString(&rnd, kBytesPerVal));
ASSERT_OK(Put(GetNumericStr(kNumPerFile * i + j), values[j]));
}
// extra entry to trigger SpecialSkipListFactory's flush
@@ -379,7 +378,7 @@ TEST_F(DBRangeDelTest, ValidLevelSubcompactionBoundaries) {
std::vector<std::string> values;
// Write 100KB (100 values, each 1K)
for (int k = 0; k < kNumPerFile; k++) {
values.push_back(rnd.RandomString(990));
values.push_back(RandomString(&rnd, 990));
ASSERT_OK(Put(Key(j * kNumPerFile + k), values[k]));
}
// put extra key to trigger flush
@@ -439,7 +438,7 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
std::vector<std::string> values;
// Write 100KB (100 values, each 1K)
for (int k = 0; k < kNumPerFile; k++) {
values.push_back(rnd.RandomString(990));
values.push_back(RandomString(&rnd, 990));
ASSERT_OK(Put(Key(j * kNumPerFile + k), values[k]));
}
// put extra key to trigger flush
@@ -460,7 +459,7 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
// probably means universal compaction + subcompaction + range deletion are
// compatible.
ASSERT_OK(dbfull()->RunManualCompaction(
static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
reinterpret_cast<ColumnFamilyHandleImpl*>(db_->DefaultColumnFamily())
->cfd(),
1 /* input_level */, 2 /* output_level */, CompactRangeOptions(),
nullptr /* begin */, nullptr /* end */, true /* exclusive */,
@@ -991,7 +990,7 @@ TEST_F(DBRangeDelTest, CompactionTreatsSplitInputLevelDeletionAtomically) {
Key(2 * kNumFilesPerLevel));
Random rnd(301);
std::string value = rnd.RandomString(kValueBytes);
std::string value = RandomString(&rnd, kValueBytes);
for (int j = 0; j < kNumFilesPerLevel; ++j) {
// give files overlapping key-ranges to prevent trivial move
ASSERT_OK(Put(Key(j), value));
@@ -1064,7 +1063,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
// [key000000#3,1, key000004#72057594037927935,15]
// [key000001#5,1, key000002#6,1]
Random rnd(301);
std::string value = rnd.RandomString(kValueBytes);
std::string value = RandomString(&rnd, kValueBytes);
for (int j = 0; j < kNumFilesPerLevel; ++j) {
// Give files overlapping key-ranges to prevent a trivial move when we
// compact from L0 to L1.
@@ -1199,7 +1198,7 @@ TEST_F(DBRangeDelTest, KeyAtOverlappingEndpointReappears) {
const Snapshot* snapshot = nullptr;
for (int i = 0; i < kNumFiles; ++i) {
for (int j = 0; j < kFileBytes / kValueBytes; ++j) {
auto value = rnd.RandomString(kValueBytes);
auto value = RandomString(&rnd, kValueBytes);
ASSERT_OK(db_->Merge(WriteOptions(), "key", value));
}
if (i == kNumFiles - 1) {
@@ -1283,7 +1282,7 @@ TEST_F(DBRangeDelTest, UntruncatedTombstoneDoesNotDeleteNewerKey) {
const Snapshot* snapshots[] = {nullptr, nullptr};
for (int i = 0; i < kNumFiles; ++i) {
for (int j = 0; j < kFileBytes / kValueBytes; ++j) {
auto value = rnd.RandomString(kValueBytes);
auto value = RandomString(&rnd, kValueBytes);
std::string key;
if (i < kNumFiles / 2) {
key = Key(0);
@@ -1329,7 +1328,7 @@ TEST_F(DBRangeDelTest, UntruncatedTombstoneDoesNotDeleteNewerKey) {
// Now overwrite a few keys that are in L1 files that definitely don't have
// overlapping boundary keys.
for (int i = kMaxKey; i > kMaxKey - kKeysOverwritten; --i) {
auto value = rnd.RandomString(kValueBytes);
auto value = RandomString(&rnd, kValueBytes);
ASSERT_OK(db_->Merge(WriteOptions(), Key(i), value));
}
ASSERT_OK(db_->Flush(FlushOptions()));
@@ -1376,7 +1375,7 @@ TEST_F(DBRangeDelTest, DeletedMergeOperandReappearsIterPrev) {
const Snapshot* snapshot = nullptr;
for (int i = 0; i < kNumFiles; ++i) {
for (int j = 0; j < kFileBytes / kValueBytes; ++j) {
auto value = rnd.RandomString(kValueBytes);
auto value = RandomString(&rnd, kValueBytes);
ASSERT_OK(db_->Merge(WriteOptions(), Key(j % kNumKeys), value));
if (i == 0 && j == kNumKeys) {
// Take snapshot to prevent covered merge operands from being dropped or
@@ -1516,7 +1515,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneWrittenToMinimalSsts) {
for (int i = 0; i < kFileBytes / kValueBytes; ++i) {
std::string key(1, first_char);
key.append(Key(i));
std::string value = rnd.RandomString(kValueBytes);
std::string value = RandomString(&rnd, kValueBytes);
ASSERT_OK(Put(key, value));
}
db_->Flush(FlushOptions());
@@ -1598,7 +1597,7 @@ TEST_F(DBRangeDelTest, OverlappedTombstones) {
std::vector<std::string> values;
// Write 12K (4 values, each 3K)
for (int j = 0; j < kNumPerFile; j++) {
values.push_back(rnd.RandomString(3 << 10));
values.push_back(RandomString(&rnd, 3 << 10));
ASSERT_OK(Put(Key(i * kNumPerFile + j), values[j]));
}
}
@@ -1637,7 +1636,7 @@ TEST_F(DBRangeDelTest, OverlappedKeys) {
std::vector<std::string> values;
// Write 12K (4 values, each 3K)
for (int j = 0; j < kNumPerFile; j++) {
values.push_back(rnd.RandomString(3 << 10));
values.push_back(RandomString(&rnd, 3 << 10));
ASSERT_OK(Put(Key(i * kNumPerFile + j), values[j]));
}
}
+9 -10
View File
@@ -12,7 +12,6 @@
#include "port/port.h"
#include "port/stack_trace.h"
#include "rocksdb/sst_file_manager.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -164,7 +163,7 @@ TEST_F(DBSSTTest, DontDeleteMovedFile) {
for (int i = 0; i < 2; ++i) {
// Create 1MB sst file
for (int j = 0; j < 100; ++j) {
ASSERT_OK(Put(Key(i * 50 + j), rnd.RandomString(10 * 1024)));
ASSERT_OK(Put(Key(i * 50 + j), RandomString(&rnd, 10 * 1024)));
}
ASSERT_OK(Flush());
}
@@ -212,7 +211,7 @@ TEST_F(DBSSTTest, DeleteObsoleteFilesPendingOutputs) {
for (int i = 0; i < 2; ++i) {
// Create 1MB sst file
for (int j = 0; j < 100; ++j) {
ASSERT_OK(Put(Key(i * 50 + j), rnd.RandomString(10 * 1024)));
ASSERT_OK(Put(Key(i * 50 + j), RandomString(&rnd, 10 * 1024)));
}
ASSERT_OK(Flush());
}
@@ -243,7 +242,7 @@ TEST_F(DBSSTTest, DeleteObsoleteFilesPendingOutputs) {
// write_buffer_size. The flush will be blocked with block_first_time
// pending_file is protecting all the files created after
for (int j = 0; j < 256; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(10 * 1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 10 * 1024)));
}
blocking_thread.WaitUntilSleeping();
@@ -759,7 +758,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowed) {
// Generate a file containing 100 keys.
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
@@ -800,7 +799,7 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
// Generate a file containing 10 keys.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
uint64_t total_file_size = 0;
@@ -810,7 +809,7 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
// Generate another file to trigger compaction.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
dbfull()->TEST_WaitForCompact(true);
@@ -847,7 +846,7 @@ TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
// Generate a file containing 10 keys.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
uint64_t total_file_size = 0;
@@ -857,7 +856,7 @@ TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
// Generate another file to trigger compaction.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
@@ -954,7 +953,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
// It is easy to detect if the test is stuck in a loop. No need for
// complex termination logic.
while (true) {
auto s = Put(rnd.RandomString(10), rnd.RandomString(50));
auto s = Put(RandomString(&rnd, 10), RandomString(&rnd, 50));
if (!s.ok()) {
break;
}
+2 -3
View File
@@ -9,7 +9,6 @@
#include "monitoring/thread_status_util.h"
#include "port/stack_trace.h"
#include "rocksdb/statistics.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -56,7 +55,7 @@ TEST_F(DBStatisticsTest, CompressionStatsTest) {
Random rnd(301);
for (int i = 0; i < kNumKeysWritten; ++i) {
// compressible string
ASSERT_OK(Put(Key(i), rnd.RandomString(128) + std::string(128, 'a')));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 128) + std::string(128, 'a')));
}
ASSERT_OK(Flush());
ASSERT_GT(options.statistics->getTickerCount(NUMBER_BLOCK_COMPRESSED), 0);
@@ -76,7 +75,7 @@ TEST_F(DBStatisticsTest, CompressionStatsTest) {
// Check that compressions do not occur when turned off
for (int i = 0; i < kNumKeysWritten; ++i) {
// compressible string
ASSERT_OK(Put(Key(i), rnd.RandomString(128) + std::string(128, 'a')));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 128) + std::string(128, 'a')));
}
ASSERT_OK(Flush());
ASSERT_EQ(options.statistics->getTickerCount(NUMBER_BLOCK_COMPRESSED)
+2 -23
View File
@@ -16,7 +16,6 @@
#include "rocksdb/utilities/table_properties_collectors.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/random.h"
#ifndef ROCKSDB_LITE
@@ -156,12 +155,12 @@ TEST_F(DBTablePropertiesTest, GetPropertiesOfTablesInRange) {
// build a decent LSM
for (int i = 0; i < 10000; i++) {
ASSERT_OK(Put(test::RandomKey(&rnd, 5), rnd.RandomString(102)));
ASSERT_OK(Put(test::RandomKey(&rnd, 5), RandomString(&rnd, 102)));
}
Flush();
dbfull()->TEST_WaitForCompact();
if (NumTableFilesAtLevel(0) == 0) {
ASSERT_OK(Put(test::RandomKey(&rnd, 5), rnd.RandomString(102)));
ASSERT_OK(Put(test::RandomKey(&rnd, 5), RandomString(&rnd, 102)));
Flush();
}
@@ -253,26 +252,6 @@ TEST_F(DBTablePropertiesTest, GetColumnFamilyNameProperty) {
}
}
TEST_F(DBTablePropertiesTest, GetDbIdentifiersProperty) {
CreateAndReopenWithCF({"goku"}, CurrentOptions());
for (uint32_t cf = 0; cf < 2; ++cf) {
Put(cf, "key", "val");
Put(cf, "foo", "bar");
Flush(cf);
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
ASSERT_EQ(1U, fname_to_props.size());
std::string id, sid;
db_->GetDbIdentity(id);
db_->GetDbSessionId(sid);
ASSERT_EQ(id, fname_to_props.begin()->second->db_id);
ASSERT_EQ(sid, fname_to_props.begin()->second->db_session_id);
}
}
class DeletionTriggeredCompactionTestListener : public EventListener {
public:
void OnCompactionBegin(DB* , const CompactionJobInfo& ci) override {
+89 -112
View File
@@ -64,7 +64,6 @@
#include "test_util/testutil.h"
#include "util/compression.h"
#include "util/mutexlock.h"
#include "util/random.h"
#include "util/rate_limiter.h"
#include "util/string_util.h"
#include "utilities/merge_operators.h"
@@ -127,7 +126,7 @@ TEST_F(DBTest, MockEnvTest) {
// TEST_FlushMemTable() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
DBImpl* dbi = static_cast_with_check<DBImpl>(db);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db);
ASSERT_OK(dbi->TEST_FlushMemTable());
for (size_t i = 0; i < 3; ++i) {
@@ -175,7 +174,7 @@ TEST_F(DBTest, MemEnvTest) {
ASSERT_TRUE(!iterator->Valid());
delete iterator;
DBImpl* dbi = static_cast_with_check<DBImpl>(db);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db);
ASSERT_OK(dbi->TEST_FlushMemTable());
for (size_t i = 0; i < 3; ++i) {
@@ -915,7 +914,7 @@ TEST_F(DBTest, FlushSchedule) {
WriteOptions wo;
// this should fill up 2 memtables
for (int k = 0; k < 5000; ++k) {
ASSERT_OK(db_->Put(wo, handles_[a & 1], rnd.RandomString(13), ""));
ASSERT_OK(db_->Put(wo, handles_[a & 1], RandomString(&rnd, 13), ""));
}
};
@@ -1172,7 +1171,7 @@ void MinLevelHelper(DBTest* self, Options& options) {
std::vector<std::string> values;
// Write 120KB (12 values, each 10K)
for (int i = 0; i < 12; i++) {
values.push_back(rnd.RandomString(10000));
values.push_back(DBTestBase::RandomString(&rnd, 10000));
ASSERT_OK(self->Put(DBTestBase::Key(i), values[i]));
}
self->dbfull()->TEST_WaitForFlushMemTable();
@@ -1182,7 +1181,7 @@ void MinLevelHelper(DBTest* self, Options& options) {
// generate one more file in level-0, and should trigger level-0 compaction
std::vector<std::string> values;
for (int i = 0; i < 12; i++) {
values.push_back(rnd.RandomString(10000));
values.push_back(DBTestBase::RandomString(&rnd, 10000));
ASSERT_OK(self->Put(DBTestBase::Key(i), values[i]));
}
self->dbfull()->TEST_WaitForCompact();
@@ -1295,7 +1294,7 @@ TEST_F(DBTest, DISABLED_RepeatedWritesToSameKey) {
Random rnd(301);
std::string value =
rnd.RandomString(static_cast<int>(2 * options.write_buffer_size));
RandomString(&rnd, static_cast<int>(2 * options.write_buffer_size));
for (int i = 0; i < 5 * kMaxFiles; i++) {
ASSERT_OK(Put(1, "key", value));
ASSERT_LE(TotalTableFiles(1), kMaxFiles);
@@ -1371,7 +1370,7 @@ TEST_F(DBTest, ApproximateSizesMemTable) {
const int N = 128;
Random rnd(301);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 1024)));
}
uint64_t size;
@@ -1395,7 +1394,7 @@ TEST_F(DBTest, ApproximateSizesMemTable) {
ASSERT_EQ(size, 0);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(1000 + i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(1000 + i), RandomString(&rnd, 1024)));
}
start = Key(500);
@@ -1422,12 +1421,10 @@ TEST_F(DBTest, ApproximateSizesMemTable) {
keys[i * 3 + 1] = i * 5 + 1;
keys[i * 3 + 2] = i * 5 + 2;
}
// MemTable entry counting is estimated and can vary greatly depending on
// layout. Thus, using deterministic seed for test stability.
RandomShuffle(std::begin(keys), std::end(keys), rnd.Next());
RandomShuffle(std::begin(keys), std::end(keys));
for (int i = 0; i < N * 3; i++) {
ASSERT_OK(Put(Key(keys[i] + 1000), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(keys[i] + 1000), RandomString(&rnd, 1024)));
}
start = Key(100);
@@ -1461,7 +1458,7 @@ TEST_F(DBTest, ApproximateSizesMemTable) {
Flush();
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(i + 1000), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i + 1000), RandomString(&rnd, 1024)));
}
start = Key(1050);
@@ -1509,7 +1506,7 @@ TEST_F(DBTest, ApproximateSizesFilesWithErrorMargin) {
const int N = 64000;
Random rnd(301);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(24)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 24)));
}
// Flush everything to files
Flush();
@@ -1518,7 +1515,7 @@ TEST_F(DBTest, ApproximateSizesFilesWithErrorMargin) {
// Write more keys
for (int i = N; i < (N + N / 4); i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(24)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 24)));
}
// Flush everything to files again
Flush();
@@ -1577,7 +1574,7 @@ TEST_F(DBTest, GetApproximateMemTableStats) {
const int N = 128;
Random rnd(301);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 1024)));
}
uint64_t count;
@@ -1609,7 +1606,7 @@ TEST_F(DBTest, GetApproximateMemTableStats) {
ASSERT_EQ(size, 0);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(Key(1000 + i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(1000 + i), RandomString(&rnd, 1024)));
}
start = Key(100);
@@ -1640,7 +1637,7 @@ TEST_F(DBTest, ApproximateSizes) {
static const int S2 = 105000; // Allow some expansion from metadata
Random rnd(301);
for (int i = 0; i < N; i++) {
ASSERT_OK(Put(1, Key(i), rnd.RandomString(S1)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, S1)));
}
// 0 because GetApproximateSizes() does not account for memtable space
@@ -1683,15 +1680,15 @@ TEST_F(DBTest, ApproximateSizes_MixOfSmallAndLarge) {
CreateAndReopenWithCF({"pikachu"}, options);
Random rnd(301);
std::string big1 = rnd.RandomString(100000);
ASSERT_OK(Put(1, Key(0), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(1), rnd.RandomString(10000)));
std::string big1 = RandomString(&rnd, 100000);
ASSERT_OK(Put(1, Key(0), RandomString(&rnd, 10000)));
ASSERT_OK(Put(1, Key(1), RandomString(&rnd, 10000)));
ASSERT_OK(Put(1, Key(2), big1));
ASSERT_OK(Put(1, Key(3), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(3), RandomString(&rnd, 10000)));
ASSERT_OK(Put(1, Key(4), big1));
ASSERT_OK(Put(1, Key(5), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(6), rnd.RandomString(300000)));
ASSERT_OK(Put(1, Key(7), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(5), RandomString(&rnd, 10000)));
ASSERT_OK(Put(1, Key(6), RandomString(&rnd, 300000)));
ASSERT_OK(Put(1, Key(7), RandomString(&rnd, 10000)));
// Check sizes across recovery by reopening a few times
for (int run = 0; run < 3; run++) {
@@ -1791,14 +1788,13 @@ TEST_F(DBTest, Snapshot) {
TEST_F(DBTest, HiddenValuesAreRemoved) {
anon::OptionsOverride options_override;
options_override.skip_policy = kSkipNoSnapshot;
env_->skip_fsync_ = true;
do {
Options options = CurrentOptions(options_override);
CreateAndReopenWithCF({"pikachu"}, options);
Random rnd(301);
FillLevels("a", "z", 1);
std::string big = rnd.RandomString(50000);
std::string big = RandomString(&rnd, 50000);
Put(1, "foo", big);
Put(1, "pastfoo", "v");
const Snapshot* snapshot = db_->GetSnapshot();
@@ -2188,7 +2184,7 @@ TEST_F(DBTest, SnapshotFiles) {
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 0);
std::vector<std::string> values;
for (int i = 0; i < 80; i++) {
values.push_back(rnd.RandomString(100000));
values.push_back(RandomString(&rnd, 100000));
ASSERT_OK(Put((i < 40), Key(i), values[i]));
}
@@ -2210,10 +2206,7 @@ TEST_F(DBTest, SnapshotFiles) {
// copy these files to a new snapshot directory
std::string snapdir = dbname_ + ".snapdir/";
if (env_->FileExists(snapdir).ok()) {
ASSERT_OK(DestroyDir(env_, snapdir));
}
ASSERT_OK(env_->CreateDir(snapdir));
ASSERT_OK(env_->CreateDirIfMissing(snapdir));
for (size_t i = 0; i < files.size(); i++) {
// our clients require that GetLiveFiles returns
@@ -2244,7 +2237,7 @@ TEST_F(DBTest, SnapshotFiles) {
// overwrite one key, this key should not appear in the snapshot
std::vector<std::string> extras;
for (unsigned int i = 0; i < 1; i++) {
extras.push_back(rnd.RandomString(100000));
extras.push_back(RandomString(&rnd, 100000));
ASSERT_OK(Put(0, Key(i), extras[i]));
}
@@ -2350,31 +2343,24 @@ TEST_F(DBTest, GetLiveBlobFiles) {
ColumnFamilyData* const cfd = versions->GetColumnFamilySet()->GetDefault();
assert(cfd);
Version* const version = cfd->current();
assert(version);
VersionStorageInfo* const storage_info = version->storage_info();
assert(storage_info);
// Add a live blob file.
VersionEdit edit;
constexpr uint64_t blob_file_number = 234;
constexpr uint64_t total_blob_count = 555;
constexpr uint64_t total_blob_bytes = 66666;
constexpr char checksum_method[] = "CRC32";
constexpr char checksum_value[] = "3d87ff57";
auto shared_meta = SharedBlobFileMetaData::Create(
blob_file_number, total_blob_count, total_blob_bytes, checksum_method,
checksum_value);
edit.AddBlobFile(blob_file_number, total_blob_count, total_blob_bytes,
checksum_method, checksum_value);
constexpr uint64_t garbage_blob_count = 0;
constexpr uint64_t garbage_blob_bytes = 0;
dbfull()->TEST_LockMutex();
Status s = versions->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
&edit, dbfull()->mutex());
dbfull()->TEST_UnlockMutex();
auto meta = BlobFileMetaData::Create(std::move(shared_meta),
BlobFileMetaData::LinkedSsts(),
garbage_blob_count, garbage_blob_bytes);
storage_info->AddBlobFile(std::move(meta));
ASSERT_OK(s);
// Make sure it appears in the results returned by GetLiveFiles.
uint64_t manifest_size = 0;
@@ -3014,11 +3000,6 @@ class ModelDB : public DB {
return Status::OK();
}
Status GetLiveFilesChecksumInfo(
FileChecksumList* /*checksum_list*/) override {
return Status::OK();
}
Status GetSortedWalFiles(VectorLogPtr& /*files*/) override {
return Status::OK();
}
@@ -3051,10 +3032,6 @@ class ModelDB : public DB {
return Status::OK();
}
Status GetDbSessionId(std::string& /*session_id*/) const override {
return Status::OK();
}
SequenceNumber GetLatestSequenceNumber() const override { return 0; }
bool SetPreserveDeletesSequenceNumber(SequenceNumber /*seqnum*/) override {
@@ -3210,8 +3187,8 @@ TEST_P(DBTestRandomized, Randomized) {
}
if (p < 45) { // Put
k = RandomKey(&rnd, minimum);
v = rnd.RandomString(rnd.OneIn(20) ? 100 + rnd.Uniform(100)
: rnd.Uniform(8));
v = RandomString(&rnd,
rnd.OneIn(20) ? 100 + rnd.Uniform(100) : rnd.Uniform(8));
ASSERT_OK(model.Put(WriteOptions(), k, v));
ASSERT_OK(db_->Put(WriteOptions(), k, v));
} else if (p < 90) { // Delete
@@ -3229,7 +3206,7 @@ TEST_P(DBTestRandomized, Randomized) {
// we have multiple entries in the write batch for the same key
}
if (rnd.OneIn(2)) {
v = rnd.RandomString(rnd.Uniform(10));
v = RandomString(&rnd, rnd.Uniform(10));
b.Put(k, v);
} else {
b.Delete(k);
@@ -3399,7 +3376,7 @@ TEST_P(DBTestWithParam, FIFOCompactionTest) {
Random rnd(301);
for (int i = 0; i < 6; ++i) {
for (int j = 0; j < 110; ++j) {
ASSERT_OK(Put(ToString(i * 100 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 100 + j), RandomString(&rnd, 980)));
}
// flush should happen here
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
@@ -3437,7 +3414,7 @@ TEST_F(DBTest, FIFOCompactionTestWithCompaction) {
for (int i = 0; i < 60; i++) {
// Generate and flush a file about 20KB.
for (int j = 0; j < 20; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3448,7 +3425,7 @@ TEST_F(DBTest, FIFOCompactionTestWithCompaction) {
for (int i = 0; i < 60; i++) {
// Generate and flush a file about 20KB.
for (int j = 0; j < 20; j++) {
ASSERT_OK(Put(ToString(i * 20 + j + 2000), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j + 2000), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3478,9 +3455,9 @@ TEST_F(DBTest, FIFOCompactionStyleWithCompactionAndDelete) {
Random rnd(301);
for (int i = 0; i < 3; i++) {
// Each file contains a different key which will be dropped later.
ASSERT_OK(Put("a" + ToString(i), rnd.RandomString(500)));
ASSERT_OK(Put("a" + ToString(i), RandomString(&rnd, 500)));
ASSERT_OK(Put("key" + ToString(i), ""));
ASSERT_OK(Put("z" + ToString(i), rnd.RandomString(500)));
ASSERT_OK(Put("z" + ToString(i), RandomString(&rnd, 500)));
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
@@ -3490,9 +3467,9 @@ TEST_F(DBTest, FIFOCompactionStyleWithCompactionAndDelete) {
}
for (int i = 0; i < 3; i++) {
// Each file contains a different key which will be dropped later.
ASSERT_OK(Put("a" + ToString(i), rnd.RandomString(500)));
ASSERT_OK(Put("a" + ToString(i), RandomString(&rnd, 500)));
ASSERT_OK(Delete("key" + ToString(i)));
ASSERT_OK(Put("z" + ToString(i), rnd.RandomString(500)));
ASSERT_OK(Put("z" + ToString(i), RandomString(&rnd, 500)));
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
@@ -3562,7 +3539,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3597,7 +3574,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3613,7 +3590,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
// Create 1 more file to trigger TTL compaction. The old files are dropped.
for (int i = 0; i < 1; i++) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
}
@@ -3639,7 +3616,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 3; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3654,7 +3631,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 5; i++) {
for (int j = 0; j < 140; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3677,7 +3654,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 10; i++) {
// Generate and flush a file about 10KB.
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3696,7 +3673,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
// Create 10 more files. The old 5 files are dropped as their ttl expired.
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3721,7 +3698,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 60; i++) {
// Generate and flush a file about 20KB.
for (int j = 0; j < 20; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3732,7 +3709,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
for (int i = 0; i < 60; i++) {
// Generate and flush a file about 20KB.
for (int j = 0; j < 20; j++) {
ASSERT_OK(Put(ToString(i * 20 + j + 2000), rnd.RandomString(980)));
ASSERT_OK(Put(ToString(i * 20 + j + 2000), RandomString(&rnd, 980)));
}
Flush();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -3775,7 +3752,8 @@ TEST_F(DBTest, DISABLED_RateLimitingTest) {
uint64_t start = env_->NowMicros();
// Write ~96M data
for (int64_t i = 0; i < (96 << 10); ++i) {
ASSERT_OK(Put(rnd.RandomString(32), rnd.RandomString((1 << 10) + 1), wo));
ASSERT_OK(
Put(RandomString(&rnd, 32), RandomString(&rnd, (1 << 10) + 1), wo));
}
uint64_t elapsed = env_->NowMicros() - start;
double raw_rate = env_->bytes_written_ * 1000000.0 / elapsed;
@@ -3793,7 +3771,8 @@ TEST_F(DBTest, DISABLED_RateLimitingTest) {
start = env_->NowMicros();
// Write ~96M data
for (int64_t i = 0; i < (96 << 10); ++i) {
ASSERT_OK(Put(rnd.RandomString(32), rnd.RandomString((1 << 10) + 1), wo));
ASSERT_OK(
Put(RandomString(&rnd, 32), RandomString(&rnd, (1 << 10) + 1), wo));
}
rate_limiter_drains =
TestGetTickerCount(options, NUMBER_RATE_LIMITER_DRAINS) -
@@ -3818,7 +3797,8 @@ TEST_F(DBTest, DISABLED_RateLimitingTest) {
start = env_->NowMicros();
// Write ~96M data
for (int64_t i = 0; i < (96 << 10); ++i) {
ASSERT_OK(Put(rnd.RandomString(32), rnd.RandomString((1 << 10) + 1), wo));
ASSERT_OK(
Put(RandomString(&rnd, 32), RandomString(&rnd, (1 << 10) + 1), wo));
}
elapsed = env_->NowMicros() - start;
rate_limiter_drains =
@@ -4001,7 +3981,6 @@ TEST_F(DBTest, DynamicMemtableOptions) {
const uint64_t k128KB = 1 << 17;
const uint64_t k5KB = 5 * 1024;
Options options;
env_->skip_fsync_ = true;
options.env = env_;
options.create_if_missing = true;
options.compression = kNoCompression;
@@ -4019,7 +3998,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
const int kNumPutsBeforeWaitForFlush = 64;
Random rnd(301);
for (int i = 0; i < size; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 1024)));
// The following condition prevents a race condition between flush jobs
// acquiring work and this thread filling up multiple memtables. Without
@@ -4093,7 +4072,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
while (!sleeping_task_low.WokenUp() && count < 256) {
ASSERT_OK(Put(Key(count), rnd.RandomString(1024), WriteOptions()));
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), WriteOptions()));
count++;
}
ASSERT_GT(static_cast<double>(count), 128 * 0.8);
@@ -4113,7 +4092,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
Env::Priority::LOW);
count = 0;
while (!sleeping_task_low.WokenUp() && count < 1024) {
ASSERT_OK(Put(Key(count), rnd.RandomString(1024), WriteOptions()));
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), WriteOptions()));
count++;
}
// Windows fails this test. Will tune in the future and figure out
@@ -4137,7 +4116,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
count = 0;
while (!sleeping_task_low.WokenUp() && count < 1024) {
ASSERT_OK(Put(Key(count), rnd.RandomString(1024), WriteOptions()));
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), WriteOptions()));
count++;
}
// Windows fails this test. Will tune in the future and figure out
@@ -4324,7 +4303,7 @@ TEST_P(DBTestWithParam, ThreadStatusSingleCompaction) {
for (int file = 0; file < kNumL0Files; ++file) {
for (int key = 0; key < kEntriesPerBuffer; ++key) {
ASSERT_OK(Put(ToString(key + file * kEntriesPerBuffer),
rnd.RandomString(kTestValueSize)));
RandomString(&rnd, kTestValueSize)));
}
Flush();
}
@@ -4472,7 +4451,7 @@ TEST_P(DBTestWithParam, PreShutdownMultipleCompaction) {
int operation_count[ThreadStatus::NUM_OP_TYPES] = {0};
for (int file = 0; file < 16 * kNumL0Files; ++file) {
for (int k = 0; k < kEntriesPerBuffer; ++k) {
ASSERT_OK(Put(ToString(key++), rnd.RandomString(kTestValueSize)));
ASSERT_OK(Put(ToString(key++), RandomString(&rnd, kTestValueSize)));
}
Status s = env_->GetThreadList(&thread_list);
@@ -4559,7 +4538,7 @@ TEST_P(DBTestWithParam, PreShutdownCompactionMiddle) {
int operation_count[ThreadStatus::NUM_OP_TYPES] = {0};
for (int file = 0; file < 16 * kNumL0Files; ++file) {
for (int k = 0; k < kEntriesPerBuffer; ++k) {
ASSERT_OK(Put(ToString(key++), rnd.RandomString(kTestValueSize)));
ASSERT_OK(Put(ToString(key++), RandomString(&rnd, kTestValueSize)));
}
Status s = env_->GetThreadList(&thread_list);
@@ -4745,7 +4724,7 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int i = 0; i < 100; i++) {
std::string value = rnd.RandomString(200);
std::string value = RandomString(&rnd, 200);
ASSERT_OK(Put(Key(keys[i]), value));
if (i % 25 == 24) {
Flush();
@@ -4790,7 +4769,7 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int i = 101; i < 500; i++) {
std::string value = rnd.RandomString(200);
std::string value = RandomString(&rnd, 200);
ASSERT_OK(Put(Key(keys[i]), value));
if (i % 100 == 99) {
Flush();
@@ -4842,7 +4821,7 @@ TEST_F(DBTest, DynamicCompactionOptions) {
auto gen_l0_kb = [this](int start, int size, int stride) {
Random rnd(301);
for (int i = 0; i < size; i++) {
ASSERT_OK(Put(Key(start + stride * i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(start + stride * i), RandomString(&rnd, 1024)));
}
dbfull()->TEST_WaitForFlushMemTable();
};
@@ -4937,7 +4916,7 @@ TEST_F(DBTest, DynamicCompactionOptions) {
Random rnd(301);
WriteOptions wo;
while (count < 64) {
ASSERT_OK(Put(Key(count), rnd.RandomString(1024), wo));
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), wo));
dbfull()->TEST_FlushMemTable(true, true);
count++;
if (dbfull()->TEST_write_controler().IsStopped()) {
@@ -4965,7 +4944,7 @@ TEST_F(DBTest, DynamicCompactionOptions) {
sleeping_task_low.WaitUntilSleeping();
count = 0;
while (count < 64) {
ASSERT_OK(Put(Key(count), rnd.RandomString(1024), wo));
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), wo));
dbfull()->TEST_FlushMemTable(true, true);
count++;
if (dbfull()->TEST_write_controler().IsStopped()) {
@@ -4987,7 +4966,7 @@ TEST_F(DBTest, DynamicCompactionOptions) {
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
for (int i = 0; i < 4; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 1024)));
// Wait for compaction so that put won't stop
dbfull()->TEST_FlushMemTable(true);
}
@@ -5001,7 +4980,7 @@ TEST_F(DBTest, DynamicCompactionOptions) {
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
for (int i = 0; i < 4; ++i) {
ASSERT_OK(Put(Key(i), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 1024)));
// Wait for compaction so that put won't stop
dbfull()->TEST_FlushMemTable(true);
}
@@ -5148,7 +5127,6 @@ TEST_F(DBTest, DynamicUniversalCompactionOptions) {
TEST_F(DBTest, FileCreationRandomFailure) {
Options options;
env_->skip_fsync_ = true;
options.env = env_;
options.create_if_missing = true;
options.write_buffer_size = 100000; // Small write buffer
@@ -5175,7 +5153,7 @@ TEST_F(DBTest, FileCreationRandomFailure) {
}
for (int k = 0; k < kTestSize; ++k) {
// here we expect some of the Put fails.
std::string value = rnd.RandomString(100);
std::string value = RandomString(&rnd, 100);
Status s = Put(Key(k), Slice(value));
if (s.ok()) {
// update the latest successful put
@@ -5224,11 +5202,11 @@ TEST_F(DBTest, DynamicMiscOptions) {
int key1 = key_start + 1;
int key2 = key_start + 2;
Random rnd(301);
ASSERT_OK(Put(Key(key0), rnd.RandomString(8)));
ASSERT_OK(Put(Key(key0), RandomString(&rnd, 8)));
for (int i = 0; i < 10; ++i) {
ASSERT_OK(Put(Key(key1), rnd.RandomString(8)));
ASSERT_OK(Put(Key(key1), RandomString(&rnd, 8)));
}
ASSERT_OK(Put(Key(key2), rnd.RandomString(8)));
ASSERT_OK(Put(Key(key2), RandomString(&rnd, 8)));
std::unique_ptr<Iterator> iter(db_->NewIterator(ReadOptions()));
iter->Seek(Key(key1));
ASSERT_TRUE(iter->Valid());
@@ -5368,7 +5346,7 @@ TEST_F(DBTest, EncodeDecompressedBlockSizeTest) {
Random rnd(301);
for (int i = 0; i < kNumKeysWritten; ++i) {
// compressible string
ASSERT_OK(Put(Key(i), rnd.RandomString(128) + std::string(128, 'a')));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 128) + std::string(128, 'a')));
}
table_options.format_version = first_table_version == 1 ? 2 : 1;
@@ -5508,7 +5486,6 @@ TEST_F(DBTest, MergeTestTime) {
#ifndef ROCKSDB_LITE
TEST_P(DBTestWithParam, MergeCompactionTimeTest) {
SetPerfLevel(kEnableTime);
env_->skip_fsync_ = true;
Options options = CurrentOptions();
options.compaction_filter_factory = std::make_shared<KeepFilterFactory>();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -5713,7 +5690,7 @@ TEST_F(DBTest, PromoteL0) {
std::map<int32_t, std::string> values;
for (const auto& range : ranges) {
for (int32_t j = range.first; j < range.second; j++) {
values[j] = rnd.RandomString(value_size);
values[j] = RandomString(&rnd, value_size);
ASSERT_OK(Put(Key(j), values[j]));
}
ASSERT_OK(Flush());
@@ -5774,7 +5751,7 @@ TEST_F(DBTest, CompactRangeWithEmptyBottomLevel) {
Random rnd(301);
for (int i = 0; i < kNumL0Files; ++i) {
ASSERT_OK(Put(Key(0), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(0), RandomString(&rnd, 1024)));
Flush();
}
ASSERT_EQ(NumTableFilesAtLevel(0), kNumL0Files);
@@ -5813,7 +5790,7 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
for (int i = 0; i < 2; ++i) {
// put two keys to ensure no trivial move
for (int j = 0; j < 2; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 1024)));
}
ASSERT_OK(Flush());
}
@@ -5827,7 +5804,7 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
for (int i = 0; i < kNumL0Files; ++i) {
// put two keys to ensure no trivial move
for (int j = 0; j < 2; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 1024)));
}
ASSERT_OK(Flush());
}
@@ -5856,7 +5833,7 @@ TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
for (int i = 0; i < 2; ++i) {
// put two keys to ensure no trivial move
for (int j = 0; j < 2; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 1024)));
}
ASSERT_OK(Flush());
}
@@ -5886,7 +5863,7 @@ TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
// generate enough files to trigger compaction
for (int i = 0; i < 20; ++i) {
for (int j = 0; j < 2; ++j) {
ASSERT_OK(Put(Key(j), rnd.RandomString(1024)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 1024)));
}
ASSERT_OK(Flush());
}
@@ -6497,7 +6474,7 @@ TEST_F(DBTest, PauseBackgroundWorkTest) {
threads.emplace_back([&]() {
Random rnd(301);
for (int i = 0; i < 10000; ++i) {
Put(rnd.RandomString(10), rnd.RandomString(10));
Put(RandomString(&rnd, 10), RandomString(&rnd, 10));
}
done.store(true);
});
@@ -6627,7 +6604,7 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
@@ -6652,7 +6629,7 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), rnd.RandomString(kValueSize)));
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
}
Flush();
}
+31 -34
View File
@@ -16,8 +16,7 @@
#include "port/stack_trace.h"
#include "rocksdb/persistent_cache.h"
#include "rocksdb/wal_filter.h"
#include "util/random.h"
#include "utilities/fault_injection_env.h"
#include "test_util/fault_injection_test_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -167,7 +166,7 @@ TEST_F(DBTest2, PartitionedIndexUserToInternalKey) {
for (int i = 0; i < 3000; i++) {
int j = i % 30;
std::string value = rnd.RandomString(10500);
std::string value = RandomString(&rnd, 10500);
ASSERT_OK(Put("keykey_" + std::to_string(j), value));
snapshots.push_back(db_->GetSnapshot());
}
@@ -1275,7 +1274,7 @@ TEST_F(DBTest2, PresetCompressionDict) {
std::string seq_datas[10];
for (int j = 0; j < 10; ++j) {
seq_datas[j] =
rnd.RandomString(kBlockSizeBytes - kApproxPerBlockOverheadBytes);
RandomString(&rnd, kBlockSizeBytes - kApproxPerBlockOverheadBytes);
}
ASSERT_EQ(0, NumTableFilesAtLevel(0, 1));
@@ -1350,7 +1349,7 @@ TEST_F(DBTest2, PresetCompressionDictLocality) {
for (int i = 0; i < kNumFiles; ++i) {
for (int j = 0; j < kNumEntriesPerFile; ++j) {
ASSERT_OK(Put(Key(i * kNumEntriesPerFile + j),
rnd.RandomString(kNumBytesPerEntry)));
RandomString(&rnd, kNumBytesPerEntry)));
}
ASSERT_OK(Flush());
MoveFilesToLevel(1);
@@ -1520,9 +1519,9 @@ TEST_P(CompressionFailuresTest, CompressionFailures) {
// Write 10 random files
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 5; j++) {
std::string key = rnd.RandomString(kKeySize);
std::string key = RandomString(&rnd, kKeySize);
// Ensure good compression ratio
std::string valueUnit = rnd.RandomString(kValUnitSize);
std::string valueUnit = RandomString(&rnd, kValUnitSize);
std::string value;
for (int k = 0; k < kValSize; k += kValUnitSize) {
value += valueUnit;
@@ -1624,8 +1623,8 @@ TEST_F(DBTest2, CompressionOptions) {
// Write 10 random files
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 5; j++) {
std::string key = rnd.RandomString(kKeySize);
std::string value = rnd.RandomString(kValSize);
std::string key = RandomString(&rnd, kKeySize);
std::string value = RandomString(&rnd, kValSize);
key_value_written[key] = value;
ASSERT_OK(Put(key, value));
}
@@ -1697,7 +1696,7 @@ TEST_F(DBTest2, CompactionStall) {
// 4 Files in L0
for (int i = 0; i < 4; i++) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(rnd.RandomString(10), rnd.RandomString(10)));
ASSERT_OK(Put(RandomString(&rnd, 10), RandomString(&rnd, 10)));
}
ASSERT_OK(Flush());
}
@@ -1712,7 +1711,7 @@ TEST_F(DBTest2, CompactionStall) {
// Another 6 L0 files to trigger compaction again
for (int i = 0; i < 6; i++) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(rnd.RandomString(10), rnd.RandomString(10)));
ASSERT_OK(Put(RandomString(&rnd, 10), RandomString(&rnd, 10)));
}
ASSERT_OK(Flush());
}
@@ -1758,7 +1757,7 @@ TEST_F(DBTest2, DuplicateSnapshot) {
Options options;
options = CurrentOptions(options);
std::vector<const Snapshot*> snapshots;
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
SequenceNumber oldest_ww_snap, first_ww_snap;
Put("k", "v"); // inc seq
@@ -2101,10 +2100,6 @@ class MockPersistentCache : public PersistentCache {
return PersistentCache::StatsType();
}
uint64_t NewId() override {
return last_id_.fetch_add(1, std::memory_order_relaxed);
}
Status Insert(const Slice& page_key, const char* data,
const size_t size) override {
MutexLock _(&lock_);
@@ -2145,7 +2140,6 @@ class MockPersistentCache : public PersistentCache {
const bool is_compressed_ = true;
size_t size_ = 0;
const size_t max_size_ = 10 * 1024; // 10KiB
std::atomic<uint64_t> last_id_{1};
};
#ifdef OS_LINUX
@@ -2269,7 +2263,10 @@ TEST_F(DBTest2, TestPerfContextIterCpuTime) {
}
#endif // OS_LINUX
#if !defined OS_SOLARIS
// GetUniqueIdFromFile is not implemented on these platforms. Persistent cache
// breaks when that function is not implemented and no regular block cache is
// provided.
#if !defined(OS_SOLARIS) && !defined(OS_WIN)
TEST_F(DBTest2, PersistentCache) {
int num_iter = 80;
@@ -2312,7 +2309,7 @@ TEST_F(DBTest2, PersistentCache) {
std::string str;
for (int i = 0; i < num_iter; i++) {
if (i % 4 == 0) { // high compression ratio
str = rnd.RandomString(1000);
str = RandomString(&rnd, 1000);
}
values.push_back(str);
ASSERT_OK(Put(1, Key(i), values[i]));
@@ -2333,7 +2330,7 @@ TEST_F(DBTest2, PersistentCache) {
}
}
}
#endif // !defined OS_SOLARIS
#endif // !defined(OS_SOLARIS) && !defined(OS_WIN)
namespace {
void CountSyncPoint() {
@@ -2410,7 +2407,7 @@ TEST_F(DBTest2, ReadAmpBitmap) {
Random rnd(301);
for (size_t i = 0; i < kNumEntries; i++) {
ASSERT_OK(Put(Key(static_cast<int>(i)), rnd.RandomString(100)));
ASSERT_OK(Put(Key(static_cast<int>(i)), RandomString(&rnd, 100)));
}
ASSERT_OK(Flush());
@@ -2517,7 +2514,7 @@ TEST_F(DBTest2, ReadAmpBitmapLiveInCacheAfterDBClose) {
Random rnd(301);
for (int i = 0; i < kNumEntries; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(100)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 100)));
}
ASSERT_OK(Flush());
@@ -2740,13 +2737,13 @@ TEST_F(DBTest2, PausingManualCompaction1) {
Random rnd(301);
// Generate a file containing 10 keys.
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
// Generate another file containing same keys
for (int i = 0; i < 10; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
@@ -2819,7 +2816,7 @@ TEST_F(DBTest2, PausingManualCompaction2) {
for (int i = 0; i < 2; i++) {
// Generate a file containing 10 keys.
for (int j = 0; j < 100; j++) {
ASSERT_OK(Put(Key(j), rnd.RandomString(50)));
ASSERT_OK(Put(Key(j), RandomString(&rnd, 50)));
}
ASSERT_OK(Flush());
}
@@ -2841,7 +2838,7 @@ TEST_F(DBTest2, PausingManualCompaction3) {
for (int i = 0; i < options.num_levels; i++) {
for (int j = 0; j < options.num_levels - i + 1; j++) {
for (int k = 0; k < 1000; k++) {
ASSERT_OK(Put(Key(k + j * 1000), rnd.RandomString(50)));
ASSERT_OK(Put(Key(k + j * 1000), RandomString(&rnd, 50)));
}
Flush();
}
@@ -2895,7 +2892,7 @@ TEST_F(DBTest2, PausingManualCompaction4) {
for (int i = 0; i < options.num_levels; i++) {
for (int j = 0; j < options.num_levels - i + 1; j++) {
for (int k = 0; k < 1000; k++) {
ASSERT_OK(Put(Key(k + j * 1000), rnd.RandomString(50)));
ASSERT_OK(Put(Key(k + j * 1000), RandomString(&rnd, 50)));
}
Flush();
}
@@ -3599,7 +3596,7 @@ TEST_F(DBTest2, TraceAndReplay) {
// Open another db, replay, and verify the data
std::string value;
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay");
std::string dbname2 = test::TmpDir(env_) + "/db_replay";
ASSERT_OK(DestroyDB(dbname2, options));
// Using a different name than db2, to pacify infer's use-after-lifetime
@@ -3673,7 +3670,7 @@ TEST_F(DBTest2, TraceWithLimit) {
ASSERT_OK(Put(0, "c", "1"));
ASSERT_OK(db_->EndTrace());
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay2");
std::string dbname2 = test::TmpDir(env_) + "/db_replay2";
std::string value;
ASSERT_OK(DestroyDB(dbname2, options));
@@ -3742,7 +3739,7 @@ TEST_F(DBTest2, TraceWithSampling) {
ASSERT_OK(Put(0, "e", "5"));
ASSERT_OK(db_->EndTrace());
std::string dbname2 = test::PerThreadDBPath(env_, "/db_replay_sampling");
std::string dbname2 = test::TmpDir(env_) + "/db_replay_sampling";
std::string value;
ASSERT_OK(DestroyDB(dbname2, options));
@@ -4022,7 +4019,7 @@ TEST_F(DBTest2, DISABLED_IteratorPinnedMemory) {
Reopen(options);
Random rnd(301);
std::string v = rnd.RandomString(400);
std::string v = RandomString(&rnd, 400);
// Since v is the size of a block, each key should take a block
// of 400+ bytes.
@@ -4209,7 +4206,7 @@ TEST_F(DBTest2, TestGetColumnFamilyHandleUnlocked) {
CreateColumnFamilies({"test1", "test2"}, Options());
ASSERT_EQ(handles_.size(), 2);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
port::Thread user_thread1([&]() {
auto cfh = dbi->GetColumnFamilyHandleUnlocked(handles_[0]->GetID());
ASSERT_EQ(cfh->GetID(), handles_[0]->GetID());
@@ -4750,7 +4747,7 @@ TEST_F(DBTest2, BlockBasedTablePrefixIndexSeekForPrev) {
Reopen(options);
Random rnd(301);
std::string large_value = rnd.RandomString(500);
std::string large_value = RandomString(&rnd, 500);
ASSERT_OK(Put("a1", large_value));
ASSERT_OK(Put("x1", large_value));
@@ -5012,7 +5009,7 @@ TEST_F(DBTest2, AutoPrefixMode1) {
Reopen(options);
Random rnd(301);
std::string large_value = rnd.RandomString(500);
std::string large_value = RandomString(&rnd, 500);
ASSERT_OK(Put("a1", large_value));
ASSERT_OK(Put("x1", large_value));
+5 -7
View File
@@ -8,11 +8,9 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/db_test_util.h"
#include "db/forward_iterator.h"
#include "rocksdb/env_encryption.h"
#include "rocksdb/utilities/object_registry.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -410,7 +408,7 @@ Options DBTestBase::GetOptions(
options.use_direct_reads = true;
options.use_direct_io_for_flush_and_compaction = true;
options.compaction_readahead_size = 2 * 1024 * 1024;
SetupSyncPointsToMockDirectIO();
test::SetupSyncPointsToMockDirectIO();
break;
}
#endif // ROCKSDB_LITE
@@ -1194,7 +1192,7 @@ int DBTestBase::GetSstFileCount(std::string path) {
void DBTestBase::GenerateNewFile(int cf, Random* rnd, int* key_idx,
bool nowait) {
for (int i = 0; i < KNumKeysByGenerateNewFile; i++) {
ASSERT_OK(Put(cf, Key(*key_idx), rnd->RandomString((i == 99) ? 1 : 990)));
ASSERT_OK(Put(cf, Key(*key_idx), RandomString(rnd, (i == 99) ? 1 : 990)));
(*key_idx)++;
}
if (!nowait) {
@@ -1206,7 +1204,7 @@ void DBTestBase::GenerateNewFile(int cf, Random* rnd, int* key_idx,
// this will generate non-overlapping files since it keeps increasing key_idx
void DBTestBase::GenerateNewFile(Random* rnd, int* key_idx, bool nowait) {
for (int i = 0; i < KNumKeysByGenerateNewFile; i++) {
ASSERT_OK(Put(Key(*key_idx), rnd->RandomString((i == 99) ? 1 : 990)));
ASSERT_OK(Put(Key(*key_idx), RandomString(rnd, (i == 99) ? 1 : 990)));
(*key_idx)++;
}
if (!nowait) {
@@ -1219,9 +1217,9 @@ const int DBTestBase::kNumKeysByGenerateNewRandomFile = 51;
void DBTestBase::GenerateNewRandomFile(Random* rnd, bool nowait) {
for (int i = 0; i < kNumKeysByGenerateNewRandomFile; i++) {
ASSERT_OK(Put("key" + rnd->RandomString(7), rnd->RandomString(2000)));
ASSERT_OK(Put("key" + RandomString(rnd, 7), RandomString(rnd, 2000)));
}
ASSERT_OK(Put("key" + rnd->RandomString(7), rnd->RandomString(200)));
ASSERT_OK(Put("key" + RandomString(rnd, 7), RandomString(rnd, 200)));
if (!nowait) {
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
+15 -66
View File
@@ -10,9 +10,9 @@
#pragma once
#include <fcntl.h>
#include <cinttypes>
#include <algorithm>
#include <cinttypes>
#include <map>
#include <set>
#include <string>
@@ -43,11 +43,12 @@
#include "table/plain/plain_table_factory.h"
#include "table/scoped_arena_iterator.h"
#include "test_util/mock_time_env.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "util/cast_util.h"
#include "util/compression.h"
#include "util/mutexlock.h"
#include "test_util/sync_point.h"
#include "test_util/testharness.h"
#include "test_util/testutil.h"
#include "util/string_util.h"
#include "utilities/merge_operators.h"
@@ -275,10 +276,7 @@ class SpecialEnv : public EnvWrapper {
while (env_->delay_sstable_sync_.load(std::memory_order_acquire)) {
env_->SleepForMicroseconds(100000);
}
Status s;
if (!env_->skip_fsync_) {
s = base_->Sync();
}
Status s = base_->Sync();
#if !(defined NDEBUG) || !defined(OS_WIN)
TEST_SYNC_POINT_CALLBACK("SpecialEnv::SStableFile::Sync", &s);
#endif // !(defined NDEBUG) || !defined(OS_WIN)
@@ -316,11 +314,7 @@ class SpecialEnv : public EnvWrapper {
if (env_->manifest_sync_error_.load(std::memory_order_acquire)) {
return Status::IOError("simulated sync error");
} else {
if (env_->skip_fsync_) {
return Status::OK();
} else {
return base_->Sync();
}
return base_->Sync();
}
}
uint64_t GetFileSize() override { return base_->GetFileSize(); }
@@ -375,11 +369,7 @@ class SpecialEnv : public EnvWrapper {
Status Flush() override { return base_->Flush(); }
Status Sync() override {
++env_->sync_counter_;
if (env_->skip_fsync_) {
return Status::OK();
} else {
return base_->Sync();
}
return base_->Sync();
}
bool IsSyncThreadSafe() const override {
return env_->is_wal_sync_thread_safe_.load();
@@ -392,30 +382,6 @@ class SpecialEnv : public EnvWrapper {
SpecialEnv* env_;
std::unique_ptr<WritableFile> base_;
};
class OtherFile : public WritableFile {
public:
OtherFile(SpecialEnv* env, std::unique_ptr<WritableFile>&& b)
: env_(env), base_(std::move(b)) {}
Status Append(const Slice& data) override { return base_->Append(data); }
Status Truncate(uint64_t size) override { return base_->Truncate(size); }
Status Close() override { return base_->Close(); }
Status Flush() override { return base_->Flush(); }
Status Sync() override {
if (env_->skip_fsync_) {
return Status::OK();
} else {
return base_->Sync();
}
}
uint64_t GetFileSize() override { return base_->GetFileSize(); }
Status Allocate(uint64_t offset, uint64_t len) override {
return base_->Allocate(offset, len);
}
private:
SpecialEnv* env_;
std::unique_ptr<WritableFile> base_;
};
if (non_writeable_rate_.load(std::memory_order_acquire) > 0) {
uint32_t random_number;
@@ -450,8 +416,6 @@ class SpecialEnv : public EnvWrapper {
r->reset(new ManifestFile(this, std::move(*r)));
} else if (strstr(f.c_str(), "log") != nullptr) {
r->reset(new WalFile(this, std::move(*r)));
} else {
r->reset(new OtherFile(this, std::move(*r)));
}
}
return s;
@@ -582,24 +546,6 @@ class SpecialEnv : public EnvWrapper {
options->stats_persist_period_sec = 0;
}
Status NewDirectory(const std::string& name,
std::unique_ptr<Directory>* result) override {
if (!skip_fsync_) {
return target()->NewDirectory(name, result);
} else {
class NoopDirectory : public Directory {
public:
NoopDirectory() {}
~NoopDirectory() {}
Status Fsync() override { return Status::OK(); }
};
result->reset(new NoopDirectory());
return Status::OK();
}
}
// Something to return when mocking current time
const int64_t maybe_starting_time_;
@@ -647,9 +593,6 @@ class SpecialEnv : public EnvWrapper {
std::atomic<int> sync_counter_;
// If true, all fsync to files and directories are skipped.
bool skip_fsync_ = false;
std::atomic<uint32_t> non_writeable_rate_;
std::atomic<uint32_t> new_writable_count_;
@@ -875,6 +818,12 @@ class DBTestBase : public testing::Test {
~DBTestBase();
static std::string RandomString(Random* rnd, int len) {
std::string r;
test::RandomString(rnd, len, &r);
return r;
}
static std::string Key(int i) {
char buf[100];
snprintf(buf, sizeof(buf), "key%06d", i);
@@ -915,7 +864,7 @@ class DBTestBase : public testing::Test {
const anon::OptionsOverride& options_override =
anon::OptionsOverride()) const;
DBImpl* dbfull() { return static_cast_with_check<DBImpl>(db_); }
DBImpl* dbfull() { return reinterpret_cast<DBImpl*>(db_); }
void CreateColumnFamilies(const std::vector<std::string>& cfs,
const Options& options);
+16 -17
View File
@@ -12,7 +12,6 @@
#if !defined(ROCKSDB_LITE)
#include "rocksdb/utilities/table_properties_collectors.h"
#include "test_util/sync_point.h"
#include "util/random.h"
namespace ROCKSDB_NAMESPACE {
@@ -362,7 +361,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionSizeAmplification) {
num++) {
// Write 110KB (11 values, each 10K)
for (int i = 0; i < 11; i++) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(key_idx), RandomString(&rnd, 10000)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
@@ -420,7 +419,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionSizeAmplification) {
num++) {
// Write 110KB (11 values, each 10K)
for (int i = 0; i < 11; i++) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(key_idx), RandomString(&rnd, 10000)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
@@ -499,7 +498,7 @@ TEST_P(DBTestUniversalCompaction, DynamicUniversalCompactionReadAmplification) {
for (int num = 0; num < options.level0_file_num_compaction_trigger; num++) {
// Write 110KB (11 values, each 10K)
for (int i = 0; i < 11; i++) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(key_idx), RandomString(&rnd, 10000)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
@@ -577,7 +576,7 @@ TEST_P(DBTestUniversalCompaction, CompactFilesOnUniversalCompaction) {
ASSERT_EQ(options.compaction_style, kCompactionStyleUniversal);
Random rnd(301);
for (int key = 1024 * kEntriesPerBuffer; key >= 0; --key) {
ASSERT_OK(Put(1, ToString(key), rnd.RandomString(kTestValueSize)));
ASSERT_OK(Put(1, ToString(key), RandomString(&rnd, kTestValueSize)));
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
@@ -640,17 +639,17 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionTargetLevel) {
// Generate 3 overlapping files
Random rnd(301);
for (int i = 0; i < 210; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(100)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 100)));
}
ASSERT_OK(Flush());
for (int i = 200; i < 300; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(100)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 100)));
}
ASSERT_OK(Flush());
for (int i = 250; i < 260; i++) {
ASSERT_OK(Put(Key(i), rnd.RandomString(100)));
ASSERT_OK(Put(Key(i), RandomString(&rnd, 100)));
}
ASSERT_OK(Flush());
@@ -961,7 +960,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionOptions) {
for (int num = 0; num < options.level0_file_num_compaction_trigger; num++) {
// Write 100KB (100 values, each 1K)
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(1, Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(1, Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
@@ -999,7 +998,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
num++) {
// Write 100KB (100 values, each 1K)
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable();
@@ -1009,7 +1008,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
// Generate one more file at level-0, which should trigger level-0
// compaction.
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForCompact();
@@ -1030,7 +1029,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
num++) {
// Write 110KB (11 values, each 10K)
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForFlushMemTable();
@@ -1040,7 +1039,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
// Generate one more file at level-0, which should trigger level-0
// compaction.
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForCompact();
@@ -1051,7 +1050,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
// Now we have 3 files at level 0, with size 4, 0.4, 2. Generate one
// more file at level-0, which should trigger level-0 compaction.
for (int i = 0; i < 100; i++) {
ASSERT_OK(Put(Key(key_idx), rnd.RandomString(990)));
ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 990)));
key_idx++;
}
dbfull()->TEST_WaitForCompact();
@@ -1531,7 +1530,7 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
for (int i = 0; i <= max_key1; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, 10000)));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
@@ -1549,7 +1548,7 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
// Insert more keys
for (int i = max_key1 + 1; i <= max_key2; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, 10000)));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
@@ -1581,7 +1580,7 @@ TEST_P(DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels) {
// Insert more keys
for (int i = max_key2 + 1; i <= max_key3; i++) {
// each value is 10K
ASSERT_OK(Put(1, Key(i), rnd.RandomString(10000)));
ASSERT_OK(Put(1, Key(i), RandomString(&rnd, 10000)));
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
dbfull()->TEST_WaitForCompact();
}
+161 -179
View File
@@ -12,16 +12,15 @@
#include "options/options_helper.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
class DBWALTestBase : public DBTestBase {
protected:
explicit DBWALTestBase(const std::string& dir_name) : DBTestBase(dir_name) {}
class DBWALTest : public DBTestBase {
public:
DBWALTest() : DBTestBase("/db_wal_test") {}
#if defined(ROCKSDB_PLATFORM_POSIX)
public:
uint64_t GetAllocatedFileSize(std::string file_name) {
struct stat sbuf;
int err = stat(file_name.c_str(), &sbuf);
@@ -31,11 +30,6 @@ class DBWALTestBase : public DBTestBase {
#endif
};
class DBWALTest : public DBWALTestBase {
public:
DBWALTest() : DBWALTestBase("/db_wal_test") {}
};
// A SpecialEnv enriched to give more insight about deleted files
class EnrichedSpecialEnv : public SpecialEnv {
public:
@@ -911,16 +905,16 @@ TEST_F(DBWALTest, PartOfWritesWithWALDisabled) {
class RecoveryTestHelper {
public:
// Number of WAL files to generate
static constexpr int kWALFilesCount = 10;
static const int kWALFilesCount = 10;
// Starting number for the WAL file name like 00010.log
static constexpr int kWALFileOffset = 10;
static const int kWALFileOffset = 10;
// Keys to be written per WAL file
static constexpr int kKeysPerWALFile = 133;
static const int kKeysPerWALFile = 133;
// Size of the value
static constexpr int kValueSize = 96;
static const int kValueSize = 96;
// Create WAL files with values filled in
static void FillData(DBWALTestBase* test, const Options& options,
static void FillData(DBWALTest* test, const Options& options,
const size_t wal_count, size_t* count) {
// Calling internal functions requires sanitized options.
Options sanitized_options = SanitizeOptions(test->dbname_, options);
@@ -974,7 +968,7 @@ class RecoveryTestHelper {
}
// Recreate and fill the store with some data
static size_t FillData(DBWALTestBase* test, Options* options) {
static size_t FillData(DBWALTest* test, Options* options) {
options->create_if_missing = true;
test->DestroyAndReopen(*options);
test->Close();
@@ -985,7 +979,7 @@ class RecoveryTestHelper {
}
// Read back all the keys we wrote and return the number of keys found
static size_t GetData(DBWALTestBase* test) {
static size_t GetData(DBWALTest* test) {
size_t count = 0;
for (size_t i = 0; i < kWALFilesCount * kKeysPerWALFile; i++) {
if (test->Get("key" + ToString(i)) != "NOT_FOUND") {
@@ -996,7 +990,7 @@ class RecoveryTestHelper {
}
// Manuall corrupt the specified WAL
static void CorruptWAL(DBWALTestBase* test, const Options& options,
static void CorruptWAL(DBWALTest* test, const Options& options,
const double off, const double len,
const int wal_file_id, const bool trunc = false) {
Env* env = options.env;
@@ -1041,101 +1035,76 @@ class RecoveryTestHelper {
}
};
class DBWALTestWithParams
: public DBWALTestBase,
public ::testing::WithParamInterface<std::tuple<bool, int, int>> {
public:
DBWALTestWithParams() : DBWALTestBase("/db_wal_test_with_params") {}
};
INSTANTIATE_TEST_CASE_P(
Wal, DBWALTestWithParams,
::testing::Combine(::testing::Bool(), ::testing::Range(0, 4, 1),
::testing::Range(RecoveryTestHelper::kWALFileOffset,
RecoveryTestHelper::kWALFileOffset +
RecoveryTestHelper::kWALFilesCount,
1)));
class DBWALTestWithParamsVaryingRecoveryMode
: public DBWALTestBase,
public ::testing::WithParamInterface<
std::tuple<bool, int, int, WALRecoveryMode>> {
public:
DBWALTestWithParamsVaryingRecoveryMode()
: DBWALTestBase("/db_wal_test_with_params_mode") {}
};
INSTANTIATE_TEST_CASE_P(
Wal, DBWALTestWithParamsVaryingRecoveryMode,
::testing::Combine(
::testing::Bool(), ::testing::Range(0, 4, 1),
::testing::Range(RecoveryTestHelper::kWALFileOffset,
RecoveryTestHelper::kWALFileOffset +
RecoveryTestHelper::kWALFilesCount,
1),
::testing::Values(WALRecoveryMode::kTolerateCorruptedTailRecords,
WALRecoveryMode::kAbsoluteConsistency,
WALRecoveryMode::kPointInTimeRecovery,
WALRecoveryMode::kSkipAnyCorruptedRecords)));
// Test scope:
// - We expect to open the data store when there is incomplete trailing writes
// at the end of any of the logs
// - We do not expect to open the data store for corruption
TEST_P(DBWALTestWithParams, kTolerateCorruptedTailRecords) {
bool trunc = std::get<0>(GetParam()); // Corruption style
// Corruption offset position
int corrupt_offset = std::get<1>(GetParam());
int wal_file_id = std::get<2>(GetParam()); // WAL file
TEST_F(DBWALTest, kTolerateCorruptedTailRecords) {
const int jstart = RecoveryTestHelper::kWALFileOffset;
const int jend = jstart + RecoveryTestHelper::kWALFilesCount;
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
// test checksum failure or parsing
RecoveryTestHelper::CorruptWAL(this, options, corrupt_offset * .3,
/*len%=*/.1, wal_file_id, trunc);
for (auto trunc : {true, false}) { /* Corruption style */
for (int i = 0; i < 3; i++) { /* Corruption offset position */
for (int j = jstart; j < jend; j++) { /* WAL file */
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
// test checksum failure or parsing
RecoveryTestHelper::CorruptWAL(this, options, /*off=*/i * .3,
/*len%=*/.1, /*wal=*/j, trunc);
options.wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
if (trunc) {
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
const size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_TRUE(corrupt_offset == 0 || recovered_row_count > 0);
ASSERT_LT(recovered_row_count, row_count);
} else {
ASSERT_NOK(TryReopen(options));
if (trunc) {
options.wal_recovery_mode =
WALRecoveryMode::kTolerateCorruptedTailRecords;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
const size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_TRUE(i == 0 || recovered_row_count > 0);
ASSERT_LT(recovered_row_count, row_count);
} else {
options.wal_recovery_mode =
WALRecoveryMode::kTolerateCorruptedTailRecords;
ASSERT_NOK(TryReopen(options));
}
}
}
}
}
// Test scope:
// We don't expect the data store to be opened if there is any corruption
// (leading, middle or trailing -- incomplete writes or corruption)
TEST_P(DBWALTestWithParams, kAbsoluteConsistency) {
TEST_F(DBWALTest, kAbsoluteConsistency) {
const int jstart = RecoveryTestHelper::kWALFileOffset;
const int jend = jstart + RecoveryTestHelper::kWALFilesCount;
// Verify clean slate behavior
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
options.wal_recovery_mode = WALRecoveryMode::kAbsoluteConsistency;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
ASSERT_EQ(RecoveryTestHelper::GetData(this), row_count);
bool trunc = std::get<0>(GetParam()); // Corruption style
// Corruption offset position
int corrupt_offset = std::get<1>(GetParam());
int wal_file_id = std::get<2>(GetParam()); // WAL file
for (auto trunc : {true, false}) { /* Corruption style */
for (int i = 0; i < 4; i++) { /* Corruption offset position */
if (trunc && i == 0) {
continue;
}
if (trunc && corrupt_offset == 0) {
return;
for (int j = jstart; j < jend; j++) { /* wal files */
// fill with new date
RecoveryTestHelper::FillData(this, &options);
// corrupt the wal
RecoveryTestHelper::CorruptWAL(this, options, /*off=*/i * .3,
/*len%=*/.1, j, trunc);
// verify
options.wal_recovery_mode = WALRecoveryMode::kAbsoluteConsistency;
options.create_if_missing = false;
ASSERT_NOK(TryReopen(options));
}
}
}
// fill with new date
RecoveryTestHelper::FillData(this, &options);
// corrupt the wal
RecoveryTestHelper::CorruptWAL(this, options, corrupt_offset * .3,
/*len%=*/.1, wal_file_id, trunc);
// verify
options.wal_recovery_mode = WALRecoveryMode::kAbsoluteConsistency;
options.create_if_missing = false;
ASSERT_NOK(TryReopen(options));
}
// Test scope:
@@ -1174,79 +1143,86 @@ TEST_F(DBWALTest, kPointInTimeRecoveryCFConsistency) {
// Test scope:
// - We expect to open data store under all circumstances
// - We expect only data upto the point where the first error was encountered
TEST_P(DBWALTestWithParams, kPointInTimeRecovery) {
TEST_F(DBWALTest, kPointInTimeRecovery) {
const int jstart = RecoveryTestHelper::kWALFileOffset;
const int jend = jstart + RecoveryTestHelper::kWALFilesCount;
const int maxkeys =
RecoveryTestHelper::kWALFilesCount * RecoveryTestHelper::kKeysPerWALFile;
bool trunc = std::get<0>(GetParam()); // Corruption style
// Corruption offset position
int corrupt_offset = std::get<1>(GetParam());
int wal_file_id = std::get<2>(GetParam()); // WAL file
for (auto trunc : {true, false}) { /* Corruption style */
for (int i = 0; i < 4; i++) { /* Offset of corruption */
for (int j = jstart; j < jend; j++) { /* WAL file */
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
// Corrupt the wal
RecoveryTestHelper::CorruptWAL(this, options, /*off=*/i * .3,
/*len%=*/.1, j, trunc);
// Corrupt the wal
RecoveryTestHelper::CorruptWAL(this, options, corrupt_offset * .3,
/*len%=*/.1, wal_file_id, trunc);
// Verify
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
// Verify
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
// Probe data for invariants
size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_LT(recovered_row_count, row_count);
// Probe data for invariants
size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_LT(recovered_row_count, row_count);
bool expect_data = true;
for (size_t k = 0; k < maxkeys; ++k) {
bool found = Get("key" + ToString(i)) != "NOT_FOUND";
if (expect_data && !found) {
expect_data = false;
}
ASSERT_EQ(found, expect_data);
}
bool expect_data = true;
for (size_t k = 0; k < maxkeys; ++k) {
bool found = Get("key" + ToString(corrupt_offset)) != "NOT_FOUND";
if (expect_data && !found) {
expect_data = false;
const size_t min = RecoveryTestHelper::kKeysPerWALFile *
(j - RecoveryTestHelper::kWALFileOffset);
ASSERT_GE(recovered_row_count, min);
if (!trunc && i != 0) {
const size_t max = RecoveryTestHelper::kKeysPerWALFile *
(j - RecoveryTestHelper::kWALFileOffset + 1);
ASSERT_LE(recovered_row_count, max);
}
}
}
ASSERT_EQ(found, expect_data);
}
const size_t min = RecoveryTestHelper::kKeysPerWALFile *
(wal_file_id - RecoveryTestHelper::kWALFileOffset);
ASSERT_GE(recovered_row_count, min);
if (!trunc && corrupt_offset != 0) {
const size_t max = RecoveryTestHelper::kKeysPerWALFile *
(wal_file_id - RecoveryTestHelper::kWALFileOffset + 1);
ASSERT_LE(recovered_row_count, max);
}
}
// Test scope:
// - We expect to open the data store under all scenarios
// - We expect to have recovered records past the corruption zone
TEST_P(DBWALTestWithParams, kSkipAnyCorruptedRecords) {
bool trunc = std::get<0>(GetParam()); // Corruption style
// Corruption offset position
int corrupt_offset = std::get<1>(GetParam());
int wal_file_id = std::get<2>(GetParam()); // WAL file
TEST_F(DBWALTest, kSkipAnyCorruptedRecords) {
const int jstart = RecoveryTestHelper::kWALFileOffset;
const int jend = jstart + RecoveryTestHelper::kWALFilesCount;
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
for (auto trunc : {true, false}) { /* Corruption style */
for (int i = 0; i < 4; i++) { /* Corruption offset */
for (int j = jstart; j < jend; j++) { /* wal files */
// Fill data for testing
Options options = CurrentOptions();
const size_t row_count = RecoveryTestHelper::FillData(this, &options);
// Corrupt the WAL
RecoveryTestHelper::CorruptWAL(this, options, corrupt_offset * .3,
/*len%=*/.1, wal_file_id, trunc);
// Corrupt the WAL
RecoveryTestHelper::CorruptWAL(this, options, /*off=*/i * .3,
/*len%=*/.1, j, trunc);
// Verify behavior
options.wal_recovery_mode = WALRecoveryMode::kSkipAnyCorruptedRecords;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
// Verify behavior
options.wal_recovery_mode = WALRecoveryMode::kSkipAnyCorruptedRecords;
options.create_if_missing = false;
ASSERT_OK(TryReopen(options));
// Probe data for invariants
size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_LT(recovered_row_count, row_count);
// Probe data for invariants
size_t recovered_row_count = RecoveryTestHelper::GetData(this);
ASSERT_LT(recovered_row_count, row_count);
if (!trunc) {
ASSERT_TRUE(corrupt_offset != 0 || recovered_row_count > 0);
if (!trunc) {
ASSERT_TRUE(i != 0 || recovered_row_count > 0);
}
}
}
}
}
@@ -1425,8 +1401,9 @@ TEST_F(DBWALTest, RecoverWithoutFlushMultipleCF) {
// 2. Open with avoid_flush_during_recovery = true;
// 3. Append more data without flushing, which creates new WAL log.
// 4. Open again. See if it can correctly handle previous corruption.
TEST_P(DBWALTestWithParamsVaryingRecoveryMode,
RecoverFromCorruptedWALWithoutFlush) {
TEST_F(DBWALTest, RecoverFromCorruptedWALWithoutFlush) {
const int jstart = RecoveryTestHelper::kWALFileOffset;
const int jend = jstart + RecoveryTestHelper::kWALFilesCount;
const int kAppendKeys = 100;
Options options = CurrentOptions();
options.avoid_flush_during_recovery = true;
@@ -1445,39 +1422,44 @@ TEST_P(DBWALTestWithParamsVaryingRecoveryMode,
delete iter;
return data;
};
bool trunc = std::get<0>(GetParam()); // Corruption style
// Corruption offset position
int corrupt_offset = std::get<1>(GetParam());
int wal_file_id = std::get<2>(GetParam()); // WAL file
WALRecoveryMode recovery_mode = std::get<3>(GetParam());
options.wal_recovery_mode = recovery_mode;
// Create corrupted WAL
RecoveryTestHelper::FillData(this, &options);
RecoveryTestHelper::CorruptWAL(this, options, corrupt_offset * .3,
/*len%=*/.1, wal_file_id, trunc);
// Skip the test if DB won't open.
if (!TryReopen(options).ok()) {
ASSERT_TRUE(options.wal_recovery_mode ==
WALRecoveryMode::kAbsoluteConsistency ||
(!trunc && options.wal_recovery_mode ==
WALRecoveryMode::kTolerateCorruptedTailRecords));
return;
for (auto& mode : {WALRecoveryMode::kTolerateCorruptedTailRecords,
WALRecoveryMode::kAbsoluteConsistency,
WALRecoveryMode::kPointInTimeRecovery,
WALRecoveryMode::kSkipAnyCorruptedRecords}) {
options.wal_recovery_mode = mode;
for (auto trunc : {true, false}) {
for (int i = 0; i < 4; i++) {
for (int j = jstart; j < jend; j++) {
// Create corrupted WAL
RecoveryTestHelper::FillData(this, &options);
RecoveryTestHelper::CorruptWAL(this, options, /*off=*/i * .3,
/*len%=*/.1, /*wal=*/j, trunc);
// Skip the test if DB won't open.
if (!TryReopen(options).ok()) {
ASSERT_TRUE(options.wal_recovery_mode ==
WALRecoveryMode::kAbsoluteConsistency ||
(!trunc &&
options.wal_recovery_mode ==
WALRecoveryMode::kTolerateCorruptedTailRecords));
continue;
}
ASSERT_OK(TryReopen(options));
// Append some more data.
for (int k = 0; k < kAppendKeys; k++) {
std::string key = "extra_key" + ToString(k);
std::string value = DummyString(RecoveryTestHelper::kValueSize);
ASSERT_OK(Put(key, value));
}
// Save data for comparison.
auto data = getAll();
// Reopen. Verify data.
ASSERT_OK(TryReopen(options));
auto actual_data = getAll();
ASSERT_EQ(data, actual_data);
}
}
}
}
ASSERT_OK(TryReopen(options));
// Append some more data.
for (int k = 0; k < kAppendKeys; k++) {
std::string key = "extra_key" + ToString(k);
std::string value = DummyString(RecoveryTestHelper::kValueSize);
ASSERT_OK(Put(key, value));
}
// Save data for comparison.
auto data = getAll();
// Reopen. Verify data.
ASSERT_OK(TryReopen(options));
auto actual_data = getAll();
ASSERT_EQ(data, actual_data);
}
// Tests that total log size is recovered if we set
+1 -1
View File
@@ -13,10 +13,10 @@
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#include "test_util/fault_injection_test_env.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
class DBBasicTestWithTimestampBase : public DBTestBase {
+3 -5
View File
@@ -4,20 +4,18 @@
// (found in the LICENSE.Apache file in the root directory).
#include <atomic>
#include <fstream>
#include <memory>
#include <thread>
#include <vector>
#include <fstream>
#include "db/db_test_util.h"
#include "db/write_batch_internal.h"
#include "db/write_thread.h"
#include "port/port.h"
#include "port/stack_trace.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
#include "util/random.h"
#include "util/string_util.h"
#include "utilities/fault_injection_env.h"
namespace ROCKSDB_NAMESPACE {
@@ -248,7 +246,7 @@ TEST_P(DBWriteTest, IOErrorOnSwitchMemtable) {
mock_env->SetFilesystemActive(false, Status::IOError("Not active"));
Status s;
for (int i = 0; i < 4 * 512; ++i) {
s = Put(Key(i), rnd.RandomString(1024));
s = Put(Key(i), RandomString(&rnd, 1024));
if (!s.ok()) {
break;
}
+15 -6
View File
@@ -26,6 +26,12 @@ namespace ROCKSDB_NAMESPACE {
const ValueType kValueTypeForSeek = kTypeDeletionWithTimestamp;
const ValueType kValueTypeForSeekForPrev = kTypeDeletion;
uint64_t PackSequenceAndType(uint64_t seq, ValueType t) {
assert(seq <= kMaxSequenceNumber);
assert(IsExtendedValueType(t));
return (seq << 8) | t;
}
EntryType GetEntryType(ValueType value_type) {
switch (value_type) {
case kTypeValue:
@@ -56,6 +62,14 @@ bool ParseFullKey(const Slice& internal_key, FullKey* fkey) {
return true;
}
void UnPackSequenceAndType(uint64_t packed, uint64_t* seq, ValueType* t) {
*seq = packed >> 8;
*t = static_cast<ValueType>(packed & 0xff);
assert(*seq <= kMaxSequenceNumber);
assert(IsExtendedValueType(*t));
}
void AppendInternalKey(std::string* result, const ParsedInternalKey& key) {
result->append(key.user_key.data(), key.user_key.size());
PutFixed64(result, PackSequenceAndType(key.sequence, key.type));
@@ -97,12 +111,7 @@ std::string InternalKey::DebugString(bool hex) const {
return result;
}
const char* InternalKeyComparator::Name() const {
if (name_.empty()) {
return "rocksdb.anonymous.InternalKeyComparator";
}
return name_.c_str();
}
const char* InternalKeyComparator::Name() const { return name_.c_str(); }
int InternalKeyComparator::Compare(const ParsedInternalKey& a,
const ParsedInternalKey& b) const {
+7 -61
View File
@@ -124,22 +124,11 @@ inline size_t InternalKeyEncodingLength(const ParsedInternalKey& key) {
}
// Pack a sequence number and a ValueType into a uint64_t
inline uint64_t PackSequenceAndType(uint64_t seq, ValueType t) {
assert(seq <= kMaxSequenceNumber);
assert(IsExtendedValueType(t));
return (seq << 8) | t;
}
extern uint64_t PackSequenceAndType(uint64_t seq, ValueType t);
// Given the result of PackSequenceAndType, store the sequence number in *seq
// and the ValueType in *t.
inline void UnPackSequenceAndType(uint64_t packed, uint64_t* seq,
ValueType* t) {
*seq = packed >> 8;
*t = static_cast<ValueType>(packed & 0xff);
assert(*seq <= kMaxSequenceNumber);
assert(IsExtendedValueType(*t));
}
extern void UnPackSequenceAndType(uint64_t packed, uint64_t* seq, ValueType* t);
EntryType GetEntryType(ValueType value_type);
@@ -211,22 +200,11 @@ class InternalKeyComparator
std::string name_;
public:
// `InternalKeyComparator`s constructed with the default constructor are not
// usable and will segfault on any attempt to use them for comparisons.
InternalKeyComparator() = default;
// @param named If true, assign a name to this comparator based on the
// underlying comparator's name. This involves an allocation and copy in
// this constructor to precompute the result of `Name()`. To avoid this
// overhead, set `named` to false. In that case, `Name()` will return a
// generic name that is non-specific to the underlying comparator.
explicit InternalKeyComparator(const Comparator* c, bool named = true)
: Comparator(c->timestamp_size()), user_comparator_(c) {
if (named) {
name_ = "rocksdb.InternalKeyComparator:" +
std::string(user_comparator_.Name());
}
}
explicit InternalKeyComparator(const Comparator* c)
: Comparator(c->timestamp_size()),
user_comparator_(c),
name_("rocksdb.InternalKeyComparator:" +
std::string(user_comparator_.Name())) {}
virtual ~InternalKeyComparator() {}
virtual const char* Name() const override;
@@ -243,12 +221,6 @@ class InternalKeyComparator
int Compare(const InternalKey& a, const InternalKey& b) const;
int Compare(const ParsedInternalKey& a, const ParsedInternalKey& b) const;
// In this `Compare()` overload, the sequence numbers provided in
// `a_global_seqno` and `b_global_seqno` override the sequence numbers in `a`
// and `b`, respectively. To disable sequence number override(s), provide the
// value `kDisableGlobalSequenceNumber`.
int Compare(const Slice& a, SequenceNumber a_global_seqno, const Slice& b,
SequenceNumber b_global_seqno) const;
virtual const Comparator* GetRootComparator() const override {
return user_comparator_.GetRootComparator();
}
@@ -705,32 +677,6 @@ inline int InternalKeyComparator::CompareKeySeq(const Slice& akey,
return r;
}
inline int InternalKeyComparator::Compare(const Slice& a,
SequenceNumber a_global_seqno,
const Slice& b,
SequenceNumber b_global_seqno) const {
int r = user_comparator_.Compare(ExtractUserKey(a), ExtractUserKey(b));
if (r == 0) {
uint64_t a_footer, b_footer;
if (a_global_seqno == kDisableGlobalSequenceNumber) {
a_footer = ExtractInternalKeyFooter(a);
} else {
a_footer = PackSequenceAndType(a_global_seqno, ExtractValueType(a));
}
if (b_global_seqno == kDisableGlobalSequenceNumber) {
b_footer = ExtractInternalKeyFooter(b);
} else {
b_footer = PackSequenceAndType(b_global_seqno, ExtractValueType(b));
}
if (a_footer > b_footer) {
r = -1;
} else if (a_footer < b_footer) {
r = +1;
}
}
return r;
}
// Wrap InternalKeyComparator as a comparator class for ParsedInternalKey.
struct ParsedInternalKeyComparator {
explicit ParsedInternalKeyComparator(const InternalKeyComparator* c)
+9 -148
View File
@@ -154,10 +154,6 @@ void ErrorHandler::CancelErrorRecovery() {
recovery_in_prog_ = false;
}
}
// If auto recovery is also runing to resume from the retryable error,
// we should wait and end the auto recovery.
EndAutoRecovery();
#endif
}
@@ -181,6 +177,7 @@ void ErrorHandler::CancelErrorRecovery() {
// end whether recovery succeeded or not
Status ErrorHandler::SetBGError(const Status& bg_err, BackgroundErrorReason reason) {
db_mutex_->AssertHeld();
if (bg_err.ok()) {
return Status::OK();
}
@@ -263,11 +260,8 @@ Status ErrorHandler::SetBGError(const IOStatus& bg_io_err,
if (bg_io_err.ok()) {
return Status::OK();
}
ROCKS_LOG_WARN(db_options_.info_log, "Background IO error %s",
bg_io_err.ToString().c_str());
if (recovery_in_prog_ && recovery_io_error_.ok()) {
recovery_io_error_ = bg_io_err;
if (recovery_in_prog_ && recovery_error_.ok()) {
recovery_error_ = bg_io_err;
}
if (BackgroundErrorReason::kManifestWrite == reason) {
// Always returns ok
@@ -281,9 +275,6 @@ Status ErrorHandler::SetBGError(const IOStatus& bg_io_err,
bool auto_recovery = false;
Status bg_err(new_bg_io_err, Status::Severity::kUnrecoverableError);
bg_error_ = bg_err;
if (recovery_in_prog_ && recovery_error_.ok()) {
recovery_error_ = bg_err;
}
EventHelpers::NotifyOnBackgroundError(db_options_.listeners, reason, &s,
db_mutex_, &auto_recovery);
return bg_error_;
@@ -291,28 +282,16 @@ Status ErrorHandler::SetBGError(const IOStatus& bg_io_err,
// Second, check if the error is a retryable IO error or not. if it is
// retryable error and its severity is higher than bg_error_, overwrite
// the bg_error_ with new error.
// In current stage, for retryable IO error of compaction, treat it as
// soft error. In other cases, treat the retryable IO error as hard
// error.
// In current stage, treat retryable error as HardError. No automatic
// recovery.
bool auto_recovery = false;
Status bg_err(new_bg_io_err, Status::Severity::kHardError);
EventHelpers::NotifyOnBackgroundError(db_options_.listeners, reason, &s,
db_mutex_, &auto_recovery);
if (BackgroundErrorReason::kCompaction == reason) {
Status bg_err(new_bg_io_err, Status::Severity::kSoftError);
if (bg_err.severity() > bg_error_.severity()) {
bg_error_ = bg_err;
}
return bg_error_;
} else {
Status bg_err(new_bg_io_err, Status::Severity::kHardError);
if (recovery_in_prog_ && recovery_error_.ok()) {
recovery_error_ = bg_err;
}
if (bg_err.severity() > bg_error_.severity()) {
bg_error_ = bg_err;
}
return StartRecoverFromRetryableBGIOError(bg_io_err);
if (bg_err.severity() > bg_error_.severity()) {
bg_error_ = bg_err;
}
return bg_error_;
} else {
s = SetBGError(new_bg_io_err, reason);
}
@@ -422,122 +401,4 @@ Status ErrorHandler::RecoverFromBGError(bool is_manual) {
return bg_error_;
#endif
}
Status ErrorHandler::StartRecoverFromRetryableBGIOError(IOStatus io_error) {
#ifndef ROCKSDB_LITE
db_mutex_->AssertHeld();
if (bg_error_.ok() || io_error.ok()) {
return Status::OK();
}
if (db_options_.max_bgerror_resume_count <= 0 || recovery_in_prog_ ||
recovery_thread_) {
// Auto resume BG error is not enabled, directly return bg_error_.
return bg_error_;
}
recovery_in_prog_ = true;
recovery_thread_.reset(
new port::Thread(&ErrorHandler::RecoverFromRetryableBGIOError, this));
if (recovery_io_error_.ok() && recovery_error_.ok()) {
return Status::OK();
} else {
TEST_SYNC_POINT("StartRecoverRetryableBGIOError:RecoverFail");
return bg_error_;
}
#else
(void)io_error;
return bg_error_;
#endif
}
// Automatic recover from Retryable BG IO error. Must be called after db
// mutex is released.
void ErrorHandler::RecoverFromRetryableBGIOError() {
#ifndef ROCKSDB_LITE
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeStart");
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeStart1");
InstrumentedMutexLock l(db_mutex_);
if (end_recovery_) {
return;
}
int resume_count = db_options_.max_bgerror_resume_count;
uint64_t wait_interval = db_options_.bgerror_resume_retry_interval;
// Recover from the retryable error. Create a separate thread to do it.
while (resume_count > 0) {
if (end_recovery_) {
return;
}
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeResume0");
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeResume1");
recovery_io_error_ = IOStatus::OK();
recovery_error_ = Status::OK();
Status s = db_->ResumeImpl();
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:AfterResume0");
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:AfterResume1");
if (s.IsShutdownInProgress() ||
bg_error_.severity() >= Status::Severity::kFatalError) {
// If DB shutdown in progress or the error severity is higher than
// Hard Error, stop auto resume and returns.
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:RecoverFail0");
recovery_in_prog_ = false;
return;
}
if (!recovery_io_error_.ok() &&
recovery_error_.severity() <= Status::Severity::kHardError &&
recovery_io_error_.GetRetryable()) {
// If new BG IO error happens during auto recovery and it is retryable
// and its severity is Hard Error or lower, the auto resmue sleep for
// a period of time and redo auto resume if it is allowed.
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeWait0");
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:BeforeWait1");
int64_t wait_until = db_->env_->NowMicros() + wait_interval;
cv_.TimedWait(wait_until);
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:AfterWait0");
} else {
// There are three possibility: 1) recover_io_error is set during resume
// and the error is not retryable, 2) recover is successful, 3) other
// error happens during resume and cannot be resumed here.
if (recovery_io_error_.ok() && recovery_error_.ok() && s.ok()) {
// recover from the retryable IO error and no other BG errors. Clean
// the bg_error and notify user.
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:RecoverSuccess");
Status old_bg_error = bg_error_;
bg_error_ = Status::OK();
EventHelpers::NotifyOnErrorRecoveryCompleted(db_options_.listeners,
old_bg_error, db_mutex_);
recovery_in_prog_ = false;
return;
} else {
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:RecoverFail1");
// In this case: 1) recovery_io_error is more serious or not retryable
// 2) other Non IO recovery_error happens. The auto recovery stops.
recovery_in_prog_ = false;
return;
}
}
resume_count--;
}
recovery_in_prog_ = false;
TEST_SYNC_POINT("RecoverFromRetryableBGIOError:LoopOut");
return;
#else
return;
#endif
}
void ErrorHandler::EndAutoRecovery() {
db_mutex_->AssertHeld();
if (!end_recovery_) {
end_recovery_ = true;
}
cv_.SignalAll();
db_mutex_->Unlock();
if (recovery_thread_) {
recovery_thread_->join();
}
db_mutex_->Lock();
return;
}
} // namespace ROCKSDB_NAMESPACE
-17
View File
@@ -22,10 +22,6 @@ class ErrorHandler {
db_options_(db_options),
bg_error_(Status::OK()),
recovery_error_(Status::OK()),
recovery_io_error_(IOStatus::OK()),
cv_(db_mutex),
end_recovery_(false),
recovery_thread_(nullptr),
db_mutex_(db_mutex),
auto_recovery_(false),
recovery_in_prog_(false) {}
@@ -63,8 +59,6 @@ class ErrorHandler {
Status RecoverFromBGError(bool is_manual = false);
void CancelErrorRecovery();
void EndAutoRecovery();
private:
DBImpl* db_;
const ImmutableDBOptions& db_options_;
@@ -72,15 +66,6 @@ class ErrorHandler {
// A separate Status variable used to record any errors during the
// recovery process from hard errors
Status recovery_error_;
// A separate IO Status variable used to record any IO errors during
// the recovery process. At the same time, recovery_error_ is also set.
IOStatus recovery_io_error_;
// The condition variable used with db_mutex during auto resume for time
// wait.
InstrumentedCondVar cv_;
bool end_recovery_;
std::unique_ptr<port::Thread> recovery_thread_;
InstrumentedMutex* db_mutex_;
// A flag indicating whether automatic recovery from errors is enabled
bool auto_recovery_;
@@ -88,8 +73,6 @@ class ErrorHandler {
Status OverrideNoSpaceError(Status bg_error, bool* auto_recovery);
void RecoverFromNoSpace();
Status StartRecoverFromRetryableBGIOError(IOStatus io_error);
void RecoverFromRetryableBGIOError();
};
} // namespace ROCKSDB_NAMESPACE
+15 -771
View File
@@ -13,12 +13,11 @@
#include "rocksdb/io_status.h"
#include "rocksdb/perf_context.h"
#include "rocksdb/sst_file_manager.h"
#include "test_util/fault_injection_test_env.h"
#include "test_util/fault_injection_test_fs.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "util/random.h"
#include "utilities/fault_injection_env.h"
#include "utilities/fault_injection_fs.h"
namespace ROCKSDB_NAMESPACE {
@@ -192,7 +191,6 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeError) {
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 0;
Status s;
listener->EnableAutoRecovery(false);
@@ -299,7 +297,6 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableError) {
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 0;
Status s;
std::string old_manifest;
std::string new_manifest;
@@ -469,7 +466,6 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
options.create_if_missing = true;
options.level0_file_num_compaction_trigger = 2;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 0;
Status s;
std::string old_manifest;
std::string new_manifest;
@@ -588,7 +584,6 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
options.create_if_missing = true;
options.level0_file_num_compaction_trigger = 2;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 0;
Status s;
DestroyAndReopen(options);
@@ -606,7 +601,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
{{"DBImpl::FlushMemTable:FlushMemTableFinished",
"BackgroundCallCompaction:0"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::OpenCompactionOutputFile",
"BackgroundCallCompaction:0",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -615,7 +610,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
ASSERT_EQ(s, Status::OK());
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
fault_fs->SetFilesystemActive(true);
SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -749,7 +744,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteError) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
batch.Put(Key(i), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -762,7 +757,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteError) {
int write_error = 0;
for (auto i = 100; i < 199; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
batch.Put(Key(i), RandomString(&rnd, 1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -812,7 +807,6 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
options.writable_file_max_buffer_size = 32768;
options.listeners.emplace_back(listener);
options.paranoid_checks = true;
options.max_bgerror_resume_count = 0;
Status s;
Random rnd(301);
@@ -826,7 +820,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
batch.Put(Key(i), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -841,7 +835,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
batch.Put(Key(i), RandomString(&rnd, 1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -877,7 +871,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
batch.Put(Key(i), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -918,7 +912,7 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
for (auto i = 1; i < 4; ++i) {
for (auto j = 0; j < 100; ++j) {
batch.Put(handles_[i], Key(j), rnd.RandomString(1024));
batch.Put(handles_[i], Key(j), RandomString(&rnd, 1024));
}
}
@@ -933,7 +927,7 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
// Write to one CF
for (auto i = 100; i < 199; ++i) {
batch.Put(handles_[2], Key(i), rnd.RandomString(1024));
batch.Put(handles_[2], Key(i), RandomString(&rnd, 1024));
}
SyncPoint::GetInstance()->SetCallBack(
@@ -1022,7 +1016,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
WriteBatch batch;
for (auto j = 0; j <= 100; ++j) {
batch.Put(Key(j), rnd.RandomString(1024));
batch.Put(Key(j), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -1037,7 +1031,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
// Write to one CF
for (auto j = 100; j < 199; ++j) {
batch.Put(Key(j), rnd.RandomString(1024));
batch.Put(Key(j), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -1135,7 +1129,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
WriteBatch batch;
for (auto j = 0; j <= 100; ++j) {
batch.Put(Key(j), rnd.RandomString(1024));
batch.Put(Key(j), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -1150,7 +1144,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
// Write to one CF
for (auto j = 100; j < 199; ++j) {
batch.Put(Key(j), rnd.RandomString(1024));
batch.Put(Key(j), RandomString(&rnd, 1024));
}
WriteOptions wopts;
@@ -1211,756 +1205,6 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
options.clear();
delete def_env;
}
TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover1) {
// Fail the first resume and make the second resume successful
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeWait0",
"FLushWritRetryableeErrorAutoRecover1:0"},
{"FLushWritRetryableeErrorAutoRecover1:1",
"RecoverFromRetryableBGIOError:BeforeWait1"},
{"RecoverFromRetryableBGIOError:RecoverSuccess",
"FLushWritRetryableeErrorAutoRecover1:2"}});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:0");
fault_fs->SetFilesystemActive(true);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:1");
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover1:2");
SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ("val1", Get(Key(1)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
// Activate the FS before the first resume
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(1), "val1");
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
SyncPoint::GetInstance()->DisableProcessing();
fault_fs->SetFilesystemActive(true);
ASSERT_EQ(listener->WaitForRecovery(5000000), true);
ASSERT_EQ("val1", Get(Key(1)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
// Fail all the resume and let user to resume
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"FLushWritRetryableeErrorAutoRecover3:0",
"RecoverFromRetryableBGIOError:BeforeStart"},
{"RecoverFromRetryableBGIOError:LoopOut",
"FLushWritRetryableeErrorAutoRecover3:1"}});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover3:0");
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover3:1");
fault_fs->SetFilesystemActive(true);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ("val1", Get(Key(1)));
// Auto resume fails due to FS does not recover during resume. User call
// resume manually here.
s = dbfull()->Resume();
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ(s, Status::OK());
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover4) {
// Fail the first resume and does not do resume second time because
// the IO error severity is Fatal Error and not Retryable.
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 10; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"FLushWritRetryableeErrorAutoRecover4:0"},
{"FLushWritRetryableeErrorAutoRecover4:2",
"RecoverFromRetryableBGIOError:RecoverFail0"}});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->SetCallBack(
"RecoverFromRetryableBGIOError:BeforeResume1",
[&](void*) { fault_fs->SetFilesystemActive(false, nr_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover4:0");
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover4:2");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
fault_fs->SetFilesystemActive(true);
// Even the FS is recoverd, due to the Fatal Error in bg_error_ the resume
// and flush will all fail.
ASSERT_EQ("val1", Get(Key(1)));
s = dbfull()->Resume();
ASSERT_NE(s, Status::OK());
ASSERT_EQ("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_NE(s, Status::OK());
ASSERT_EQ("NOT_FOUND", Get(Key(2)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
// During the resume, call DB->CLose, make sure the resume thread exist
// before close continues. Due to the shutdown, the resume is not successful
// and the FS does not become active, so close status is still IO error
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 10; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"FLushWritRetryableeErrorAutoRecover5:0"}});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover5:0");
// The first resume will cause recovery_error and its severity is the
// Fatal error
s = dbfull()->Close();
ASSERT_NE(s, Status::OK());
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
fault_fs->SetFilesystemActive(true);
Reopen(options);
ASSERT_NE("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover6) {
// During the resume, call DB->CLose, make sure the resume thread exist
// before close continues. Due to the shutdown, the resume is not successful
// and the FS does not become active, so close status is still IO error
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 10; // 0.1 second
Status s;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
IOStatus nr_msg = IOStatus::IOError("No Retryable Fatal IO Error");
nr_msg.SetRetryable(false);
Put(Key(1), "val1");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"FLushWritRetryableeErrorAutoRecover6:0",
"RecoverFromRetryableBGIOError:BeforeStart"},
{"RecoverFromRetryableBGIOError:BeforeWait0",
"FLushWritRetryableeErrorAutoRecover6:1"},
{"FLushWritRetryableeErrorAutoRecover6:2",
"RecoverFromRetryableBGIOError:BeforeWait1"},
{"RecoverFromRetryableBGIOError:AfterWait0",
"FLushWritRetryableeErrorAutoRecover6:3"}});
SyncPoint::GetInstance()->SetCallBack(
"BuildTable:BeforeFinishBuildTable",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:0");
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:1");
fault_fs->SetFilesystemActive(true);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:2");
TEST_SYNC_POINT("FLushWritRetryableeErrorAutoRecover6:3");
// The first resume will cause recovery_error and its severity is the
// Fatal error
s = dbfull()->Close();
ASSERT_EQ(s, Status::OK());
SyncPoint::GetInstance()->DisableProcessing();
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
Put(Key(2), "val2");
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableErrorAutoRecover) {
// Fail the first resume and let the second resume be successful
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
std::string old_manifest;
std::string new_manifest;
listener->EnableAutoRecovery(false);
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(0), "val");
Flush();
Put(Key(1), "val");
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeStart",
"ManifestWriteRetryableErrorAutoRecover:0"},
{"ManifestWriteRetryableErrorAutoRecover:1",
"RecoverFromRetryableBGIOError:BeforeWait1"},
{"RecoverFromRetryableBGIOError:RecoverSuccess",
"ManifestWriteRetryableErrorAutoRecover:2"}});
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest",
[&](void*) { fault_fs->SetFilesystemActive(false, error_msg); });
SyncPoint::GetInstance()->EnableProcessing();
s = Flush();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("ManifestWriteRetryableErrorAutoRecover:0");
fault_fs->SetFilesystemActive(true);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("ManifestWriteRetryableErrorAutoRecover:1");
TEST_SYNC_POINT("ManifestWriteRetryableErrorAutoRecover:2");
SyncPoint::GetInstance()->DisableProcessing();
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
Reopen(options);
ASSERT_EQ("val", Get(Key(0)));
ASSERT_EQ("val", Get(Key(1)));
Close();
}
TEST_F(DBErrorHandlingFSTest,
CompactionManifestWriteRetryableErrorAutoRecover) {
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.level0_file_num_compaction_trigger = 2;
options.listeners.emplace_back(listener);
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
std::string old_manifest;
std::string new_manifest;
std::atomic<bool> fail_manifest(false);
DestroyAndReopen(options);
old_manifest = GetManifestNameFromLiveFiles();
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(0), "val");
Put(Key(2), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
// Wait for flush of 2nd L0 file before starting compaction
{{"DBImpl::FlushMemTable:FlushMemTableFinished",
"BackgroundCallCompaction:0"},
// Wait for compaction to detect manifest write error
{"BackgroundCallCompaction:1", "CompactionManifestWriteErrorAR:0"},
// Make compaction thread wait for error to be cleared
{"CompactionManifestWriteErrorAR:1",
"DBImpl::BackgroundCallCompaction:FoundObsoleteFiles"},
{"CompactionManifestWriteErrorAR:2",
"RecoverFromRetryableBGIOError:BeforeStart"},
// Fail the first resume, before the wait in resume
{"RecoverFromRetryableBGIOError:BeforeResume0",
"CompactionManifestWriteErrorAR:3"},
// Activate the FS before the second resume
{"CompactionManifestWriteErrorAR:4",
"RecoverFromRetryableBGIOError:BeforeResume1"},
// Wait the auto resume be sucessful
{"RecoverFromRetryableBGIOError:RecoverSuccess",
"CompactionManifestWriteErrorAR:5"}});
// trigger manifest write failure in compaction thread
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"BackgroundCallCompaction:0", [&](void*) { fail_manifest.store(true); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest", [&](void*) {
if (fail_manifest.load()) {
fault_fs->SetFilesystemActive(false, error_msg);
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:0");
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:1");
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:2");
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:3");
fault_fs->SetFilesystemActive(true);
SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:4");
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:5");
SyncPoint::GetInstance()->DisableProcessing();
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
Reopen(options);
ASSERT_EQ("val", Get(Key(0)));
ASSERT_EQ("val", Get(Key(1)));
ASSERT_EQ("val", Get(Key(2)));
Close();
}
TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableErrorAutoRecover) {
// In this test, in the first round of compaction, the FS is set to error.
// So the first compaction fails due to retryable IO error and it is mapped
// to soft error. Then, compaction is rescheduled, in the second round of
// compaction, the FS is set to active and compaction is successful, so
// the test will hit the CompactionJob::FinishCompactionOutputFile1 sync
// point.
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.level0_file_num_compaction_trigger = 2;
options.listeners.emplace_back(listener);
Status s;
std::atomic<bool> fail_first(false);
std::atomic<bool> fail_second(true);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
Put(Key(0), "va;");
Put(Key(2), "va;");
s = Flush();
ASSERT_EQ(s, Status::OK());
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::FlushMemTable:FlushMemTableFinished",
"BackgroundCallCompaction:0"},
{"CompactionJob::FinishCompactionOutputFile1",
"CompactionWriteRetryableErrorAutoRecover0"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:Start",
[&](void*) { fault_fs->SetFilesystemActive(true); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"BackgroundCallCompaction:0", [&](void*) { fail_first.store(true); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::OpenCompactionOutputFile", [&](void*) {
if (fail_first.load() && fail_second.load()) {
fault_fs->SetFilesystemActive(false, error_msg);
fail_second.store(false);
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Put(Key(1), "val");
s = Flush();
ASSERT_EQ(s, Status::OK());
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
TEST_SYNC_POINT("CompactionWriteRetryableErrorAutoRecover0");
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
Destroy(options);
}
TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.writable_file_max_buffer_size = 32768;
options.listeners.emplace_back(listener);
options.paranoid_checks = true;
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
Random rnd(301);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
// For the first batch, write is successful, require sync
{
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
};
// For the second batch, the first 2 file Append are successful, then the
// following Append fails due to file system retryable IOError.
{
WriteBatch batch;
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeResume0", "WALWriteError1:0"},
{"WALWriteError1:1", "RecoverFromRetryableBGIOError:BeforeResume1"},
{"RecoverFromRetryableBGIOError:RecoverSuccess", "WALWriteError1:2"}});
SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::Append:BeforePrepareWrite", [&](void*) {
write_error++;
if (write_error > 2) {
fault_fs->SetFilesystemActive(false, error_msg);
}
});
SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wopts;
wopts.sync = true;
s = dbfull()->Write(wopts, &batch);
ASSERT_EQ(true, s.IsIOError());
TEST_SYNC_POINT("WALWriteError1:0");
fault_fs->SetFilesystemActive(true);
SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("WALWriteError1:1");
TEST_SYNC_POINT("WALWriteError1:2");
}
SyncPoint::GetInstance()->DisableProcessing();
// Data in corrupted WAL are not stored
for (auto i = 0; i < 199; ++i) {
if (i < 100) {
ASSERT_NE(Get(Key(i)), "NOT_FOUND");
} else {
ASSERT_EQ(Get(Key(i)), "NOT_FOUND");
}
}
// Resume and write a new batch, should be in the WAL
{
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
};
Reopen(options);
for (auto i = 0; i < 300; ++i) {
if (i < 100 || i >= 200) {
ASSERT_NE(Get(Key(i)), "NOT_FOUND");
} else {
ASSERT_EQ(Get(Key(i)), "NOT_FOUND");
}
}
Close();
}
TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
// Fail the first recover and try second time.
std::shared_ptr<FaultInjectionTestFS> fault_fs(
new FaultInjectionTestFS(FileSystem::Default()));
std::unique_ptr<Env> fault_fs_env(NewCompositeEnv(fault_fs));
std::shared_ptr<ErrorHandlerFSListener> listener(
new ErrorHandlerFSListener());
Options options = GetDefaultOptions();
options.env = fault_fs_env.get();
options.create_if_missing = true;
options.writable_file_max_buffer_size = 32768;
options.listeners.emplace_back(listener);
options.paranoid_checks = true;
options.max_bgerror_resume_count = 2;
options.bgerror_resume_retry_interval = 100000; // 0.1 second
Status s;
Random rnd(301);
DestroyAndReopen(options);
IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
error_msg.SetRetryable(true);
// For the first batch, write is successful, require sync
{
WriteBatch batch;
for (auto i = 0; i < 100; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
};
// For the second batch, the first 2 file Append are successful, then the
// following Append fails due to file system retryable IOError.
{
WriteBatch batch;
int write_error = 0;
for (auto i = 100; i < 200; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"RecoverFromRetryableBGIOError:BeforeWait0", "WALWriteError2:0"},
{"WALWriteError2:1", "RecoverFromRetryableBGIOError:BeforeWait1"},
{"RecoverFromRetryableBGIOError:RecoverSuccess", "WALWriteError2:2"}});
SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::Append:BeforePrepareWrite", [&](void*) {
write_error++;
if (write_error > 2) {
fault_fs->SetFilesystemActive(false, error_msg);
}
});
SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wopts;
wopts.sync = true;
s = dbfull()->Write(wopts, &batch);
ASSERT_EQ(true, s.IsIOError());
TEST_SYNC_POINT("WALWriteError2:0");
fault_fs->SetFilesystemActive(true);
SyncPoint::GetInstance()->ClearAllCallBacks();
TEST_SYNC_POINT("WALWriteError2:1");
TEST_SYNC_POINT("WALWriteError2:2");
}
SyncPoint::GetInstance()->DisableProcessing();
// Data in corrupted WAL are not stored
for (auto i = 0; i < 199; ++i) {
if (i < 100) {
ASSERT_NE(Get(Key(i)), "NOT_FOUND");
} else {
ASSERT_EQ(Get(Key(i)), "NOT_FOUND");
}
}
// Resume and write a new batch, should be in the WAL
{
WriteBatch batch;
for (auto i = 200; i < 300; ++i) {
batch.Put(Key(i), rnd.RandomString(1024));
}
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
};
Reopen(options);
for (auto i = 0; i < 300; ++i) {
if (i < 100 || i >= 200) {
ASSERT_NE(Get(Key(i)), "NOT_FOUND");
} else {
ASSERT_EQ(Get(Key(i)), "NOT_FOUND");
}
}
Close();
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+1 -3
View File
@@ -121,9 +121,7 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
<< table_properties.compression_options << "creation_time"
<< table_properties.creation_time << "oldest_key_time"
<< table_properties.oldest_key_time << "file_creation_time"
<< table_properties.file_creation_time << "db_id"
<< table_properties.db_id << "db_session_id"
<< table_properties.db_session_id;
<< table_properties.file_creation_time;
// user collected properties
for (const auto& prop : table_properties.readable_properties) {
Loaded 100 of 339 files, more files were not shown because too many files have changed in this diff. Show more