Compare commits

...
Author SHA1 Message Date
Ramkumar Vadivelu 9da32babe1 Update "Further Documentation"
Remove stale pointers in the "Further Documentation"  and just point to RocksDB Wiki
2020-12-30 17:14:07 -08:00
jbosh edb0b1fb7f rocksdb_transaction_get_for_update now exports (#6293)
Summary:
Added missing ROCKSDB_LIBRARY_API decorator to rocksdb_transaction_get_for_update.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6293

Reviewed By: jay-zhuang

Differential Revision: D25234298

Pulled By: ajkr

fbshipit-source-id: 8a4817adaec1f445f338c8d8c59d3392925b5721
2020-12-30 15:42:59 -08:00
Adam Retter fd2db79fa4 Attempt to fix build errors around missing compression library includes (#7803)
Summary:
This fixes an issue introduced in https://github.com/facebook/rocksdb/pull/7769 that caused many errors about missing compression libraries to be displayed during compilation, although compilation actually succeeded. This PR fixes the compilation so the compression libraries are only introduced where strictly needed.

It likely needs to be merged into the same branches as https://github.com/facebook/rocksdb/pull/7769 which I think are:
1. master
2. 6.15.fb
3. 6.16.fb

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7803

Reviewed By: ramvadiv

Differential Revision: D25733743

Pulled By: pdillinger

fbshipit-source-id: 6c04f6864b2ff4a345841d791a89b19e0e3f5bf7
2020-12-30 13:40:10 -08:00
anand76 01298c8ff7 Return Status from FilePrefetchBuffer::TryReadFromCache() (#7816)
Summary:
Return the Status from TryReadFromCache() in an argument to make it easier to report prefetch errors to the user.

Tests:
make crash_test
make check

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7816

Reviewed By: akankshamahajan15

Differential Revision: D25717222

Pulled By: anand1976

fbshipit-source-id: c320d3c12d4146bda16df78ff6927eee584c1810
2020-12-30 09:25:09 -08:00
anand76 d7738666b0 Fix db_bench duration for multireadrandom benchmark (#7817)
Summary:
The multireadrandom benchmark, when run for a specific number of reads (--reads argument), should base the duration on the actual number of keys read rather than number of batches.

Tests:
Run db_bench multireadrandom benchmark

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7817

Reviewed By: zhichao-cao

Differential Revision: D25717230

Pulled By: anand1976

fbshipit-source-id: 13f4d8162268cf9a34918655e60302d0aba3864b
2020-12-28 13:38:10 -08:00
cheng-chang 736c6dc59f Disable BasicLockEscalation if cannot determine whether TSAN is enabled (#7814)
Summary:
BasicLockEscalation will cause false-positive warnings under TSAN (this is a known issue in TSAN, see details in https://gist.github.com/spetrunia/77274cf2d5848e0a7e090d622695ed4e), skip this test if TSAN is enabled, or if we are not sure whether TSAN is enabled.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7814

Test Plan: watch the tsan contrun test to pass.

Reviewed By: zhichao-cao

Differential Revision: D25708094

Pulled By: cheng-chang

fbshipit-source-id: 4fc813ff373301d033d086154cc7bb60a5e95889
2020-12-27 16:18:00 -08:00
Zhichao Cao 44ebc24dca Add rate_limiter to GenerateOneFileChecksum (#7811)
Summary:
In GenerateOneFileChecksum(), RocksDB reads the file and computes its checksum. A rate limiter can be passed to the constructor of RandomAccessFileReader so that read I/O can be rate limited.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7811

Test Plan: make check

Reviewed By: cheng-chang

Differential Revision: D25699896

Pulled By: zhichao-cao

fbshipit-source-id: e2688bc1126c543979a3bcf91dda784bd7b74164
2020-12-26 22:07:24 -08:00
Zhichao Cao 601585bca4 fix memory leak in db_stress checkpoint test (#7813)
Summary:
fix memory leak in db_stress checkpoint test. If s is not ok, checkpoint is not deleted, may cause memory leak.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7813

Test Plan: make asan_check

Reviewed By: cheng-chang

Differential Revision: D25702999

Pulled By: zhichao-cao

fbshipit-source-id: 08253b0852835acb8cfd412503cdabf720afb678
2020-12-25 13:15:48 -08:00
mrambacher 55e99688cc No elide constructors (#7798)
Summary:
Added "no-elide-constructors to the ASSERT_STATUS_CHECK builds.  This flag gives more errors/warnings for some of the Status checks where an inner class checks a Status and later returns it.  In this case,  without the elide check on, the returned status may not have been checked in the caller, thereby bypassing the checked code.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7798

Reviewed By: jay-zhuang

Differential Revision: D25680451

Pulled By: pdillinger

fbshipit-source-id: c3f14ed9e2a13f0a8c54d839d5fb4d1fc1e93917
2020-12-23 16:55:53 -08:00
Akanksha Mahajan 30a5ed9c53 Update "num_data_read" stat in RetrieveMultipleBlocks (#7770)
Summary:
RetrieveMultipleBlocks which is used by MultiGet to read data blocks is not updating num_data_read stat in
GetContextStats.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7770

Test Plan: make check -j64

Reviewed By: anand1976

Differential Revision: D25538982

Pulled By: akankshamahajan15

fbshipit-source-id: e3daedb035b1be8ab6af6f115cb3793ccc7b1ec6
2020-12-23 15:16:46 -08:00
cheng-chang bdb7e544bd Skip WALs according to MinLogNumberToKeep when creating checkpoint (#7789)
Summary:
In a stress test failure, we observe that a WAL is skipped when creating checkpoint, although its log number >= MinLogNumberToKeep(). This might happen in the following case:

1. when creating the checkpoint, there are 2 column families: CF0 and CF1, and there are 2 WALs: 1, 2;
2. CF0's log number is 1, CF0's active memtable is empty, CF1's log number is 2, CF1's active memtable is not empty, WAL 2 is not empty, the sequence number points to WAL 2;
2. the checkpoint process flushes CF0, since CF0' active memtable is empty, there is no need to SwitchMemtable, thus no new WAL will be created, so CF0's log number is now 2, concurrently, some data is written to CF0 and WAL 2;
3. the checkpoint process flushes CF1, WAL 3 is created and CF1's log number is now 3, CF0's log number is still 2 because CF0 is not empty and WAL 2 contains its unflushed data concurrently written in step 2;
4.  the checkpoint process determines that WAL 1 and 2 are no longer needed according to [live_wal_files[i]->StartSequence() >= *sequence_number](https://github.com/facebook/rocksdb/blob/master/utilities/checkpoint/checkpoint_impl.cc#L388), so it skips linking them to the checkpoint directory;
5. but according to `MinLogNumberToKeep()`, WAL 2 still needs to be kept because CF0's log number is 2.

If the checkpoint is reopened in read-only mode, and only read from the snapshot with the initial sequence number, then there will be no data loss or data inconsistency.

But if the checkpoint is reopened and read from the most recent sequence number, suppose in step 3, there are also data concurrently written to CF1 and WAL 3, then the most recent sequence number refers to the latest entry in WAL 3, so the data written in step 2 should also be visible, but since WAL 2 is discarded, those data are lost.

When tracking WAL in MANIFEST is enabled, when reopening the checkpoint, since WAL 2 is still tracked in MANIFEST as alive, but it's missing from the checkpoint directory, a corruption will be reported.

This PR makes the checkpoint process to only skip a WAL if its log number < `MinLogNumberToKeep`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7789

Test Plan: watch existing tests to pass.

Reviewed By: ajkr

Differential Revision: D25662346

Pulled By: cheng-chang

fbshipit-source-id: 136471095baa01886cf44809455cf855f24857a0
2020-12-23 11:33:26 -08:00
anand76 bd2645bc34 Update regression_test.sh to run multireadrandom benchmark (#7802)
Summary:
Update the regression_test.sh script to run the multireadrandom benchmark.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7802

Reviewed By: zhichao-cao

Differential Revision: D25685482

Pulled By: anand1976

fbshipit-source-id: ef2973b551a1bbdbce198a0adf29fc277f3e65e2
2020-12-23 11:26:12 -08:00
Peter Dillinger a727efca99 Remove flaky, redundant, and dubious DBTest.SparseMerge (#7800)
Summary:
This test would occasionally fail like this:

    WARNING: c:\users\circleci\project\db\db_test.cc(1343): error: Expected:
    (dbfull()->TEST_MaxNextLevelOverlappingBytes(handles_[1])) <= (20 * 1048576), actual: 33501540 vs 20971520

And being a super old test, it's not structured in a sound way. And it appears that DBTest2.MaxCompactionBytesTest is a better test of what SparseMerge was intended to test. In fact, SparseMerge fails if I set

    options.max_compaction_bytes = options.target_file_size_base * 1000;

Thus, we are removing this negative-value test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7800

Test Plan: Q.E.D.

Reviewed By: ajkr

Differential Revision: D25693366

Pulled By: pdillinger

fbshipit-source-id: 9da07d4dce0559547fc938b2163a2015e956c548
2020-12-23 11:08:12 -08:00
mrambacher 02418194d7 Add more tests for assert status checked (#7524)
Summary:
Added 10 more tests that pass the ASSERT_STATUS_CHECKED test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7524

Reviewed By: akankshamahajan15

Differential Revision: D24323093

Pulled By: ajkr

fbshipit-source-id: 28d4106d0ca1740c3b896c755edf82d504b74801
2020-12-22 23:45:58 -08:00
Sergei Petrunia daab7603f6 Range Locking: Implementation of range locking (#7506)
Summary:
Range Locking - an implementation based on the locktree library

- Add a RangeTreeLockManager and RangeTreeLockTracker which implement
  range locking using the locktree library.
- Point locks are handled as locks on single-point ranges.
- Add a unit test: range_locking_test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7506

Reviewed By: akankshamahajan15

Differential Revision: D25320703

Pulled By: cheng-chang

fbshipit-source-id: f86347384b42ba2b0257d67eca0f45f806b69da7
2020-12-22 19:12:36 -08:00
sdong f4db3e4119 Avoid to force PORTABLE mode in tools/regression_test.sh (#7806)
Summary:
Right now tools/regression_test.sh always builds RocksDB with PORTABLE=1. There isn't a reason for that. Remove it. Users can always specify PORTABLE through  envirionement variable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7806

Test Plan: Run tools/regression_test.sh and see it still builds.

Reviewed By: ajkr

Differential Revision: D25687911

fbshipit-source-id: 1c0b03e5df890babc8b7d8af48b48774d9a4600c
2020-12-22 16:54:07 -08:00
Adam Retter 81592d9ffa Add more tests to ASSERT_STATUS_CHECKED (4) (#7718)
Summary:
Fourth batch of adding more tests to ASSERT_STATUS_CHECKED.

* db_range_del_test
* db_write_test
* random_access_file_reader_test
* merge_test
* external_sst_file_test
* write_buffer_manager_test
* stringappend_test
* deletefile_test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7718

Reviewed By: pdillinger

Differential Revision: D25671608

fbshipit-source-id: 687a794e98a9e0cd5428ead9898ef05ced987c31
2020-12-22 15:09:39 -08:00
cheng-chang 41ff125a8a SyncWAL shouldn't be supported in compacted db (#7788)
Summary:
`CompactedDB` is a kind of read-only DB, so it shouldn't support `SyncWAL`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7788

Test Plan: watch existing tests to pass.

Reviewed By: akankshamahajan15

Differential Revision: D25661209

Pulled By: cheng-chang

fbshipit-source-id: 9eb2cc3f73736dcc205c8410e5944aa203f002d3
2020-12-22 14:53:43 -08:00
Sergei Petrunia 1022090981 Apply the changes from: PS-5501 : Re-license PerconaFT 'locktree' to Apache V2 (#7801)
Summary:
commit d5178f513c0b4144a5ac9358ec0f6a3b54a28e76
Author: George O. Lorch III <george.lorch@percona.com>
Date:   Tue Mar 19 12:18:40 2019 -0700

    PS-5501 : Re-license PerconaFT 'locktree' to Apache V2

    - Fixed some incomplete relicensed files from previous round.

    - Added missing license text to some.

    - Relicensed more files to Apache V2 that locktree depends on.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7801

Reviewed By: jay-zhuang

Differential Revision: D25682430

Pulled By: cheng-chang

fbshipit-source-id: deb8a0de3e76f3638672997bfbd300e2fffbe5f5
2020-12-22 14:47:41 -08:00
sdong 9057d0a079 Minimize Timing Issue in test WALTrashCleanupOnOpen (#7796)
Summary:
We saw DBWALTestWithParam/DBWALTestWithParam.WALTrashCleanupOnOpen sometimes fail with:

db/db_sst_test.cc:575: Failure
Expected: (trash_log_count) >= (1), actual: 0 vs 1

The suspicious is that delete scheduling actually deleted all trash files based on rate, but it is not expected. This can be reproduced if we manually add sleep after DB is closed for serveral seconds. Minimize its chance by setting the delete rate to be lowest possible.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7796

Test Plan: The test doesn't fail with the manual sleeping anymore

Reviewed By: anand1976

Differential Revision: D25675000

fbshipit-source-id: a39fd05e1a83719c41014e48843792e752368e22
2020-12-22 14:44:08 -08:00
Akanksha Mahajan fbac1b3f80 Add tests in ASSERT_STATUS_CHECKED (#7793)
Summary:
add io_tracer_parser_test and prefetch_test under
ASSERT_STATUS_CHECKED

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7793

Test Plan: ASSERT_STATUS_CHECKED=1 make check -j64

Reviewed By: jay-zhuang

Differential Revision: D25673464

Pulled By: akankshamahajan15

fbshipit-source-id: 50e0b6f17160ddda206a521a7b47ee33e699a2d4
2020-12-22 10:31:13 -08:00
Peter Dillinger 4d897e51df Migrate away from Travis+Linux+amd64 (#7791)
Summary:
This disables Linux/amd64 builds in Travis for PRs, and adds a
gcc-10+c++20 build in CircleCI, which should fill out sufficient coverage
vs. what we had in Travis

Fixed a use of std::is_pod, which is deprecated in c++20

Fixed ++ on a volatile in db_repl_stress.cc, with bigger refactoring.
Although ++ on this volatile was probably ok with one thread writer and
one thread reader, the code was still overly complex. There was a
deadcode check for error
`if (replThread.no_read < dataPump.no_records)` which can be proven
never to happen based on the structure of the code. It infinite loops
instead for the case intended to be checked. I just simplified the code
for what should be the same checking power.

Also most configurations seem to be using make parallelism = 2 * vcores,
so fixing / using that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7791

Test Plan:
CI
and `while ./db_repl_stress; do echo again; done` for a while

Reviewed By: siying

Differential Revision: D25669834

Pulled By: pdillinger

fbshipit-source-id: b2c688053d0b1d52c989903449d3cd27a04130d6
2020-12-22 00:20:57 -08:00
Jay Zhuang 861b0d1a49 Fix Windows build in block_cache_tracer_test (#7795)
Summary:
The test was added to cmake in https://github.com/facebook/rocksdb/issues/7783

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7795

Reviewed By: akankshamahajan15

Differential Revision: D25671010

Pulled By: jay-zhuang

fbshipit-source-id: 2146ff9559cdd7266c4d78476672488c62654a6d
2020-12-21 15:22:16 -08:00
Jay Zhuang fd0d35d390 Fix block_cache_test failure (#7783)
Summary:
`block_cache_tracer_test` and `block_cache_trace_analyzer_test` are using the same test directory. Which causes build failure if these 2 tests are running in parallel, for example: https://app.circleci.com/pipelines/github/facebook/rocksdb/5211/workflows/8639afbe-9fec-43e2-a6a4-6d47ea9cbcbe/jobs/74598/tests

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7783

Reviewed By: akankshamahajan15

Differential Revision: D25656762

Pulled By: jay-zhuang

fbshipit-source-id: 68aa020aa5b4b3bce324315edecb4e1a60cc18e6
2020-12-21 08:47:08 -08:00
153 changed files with 3079 additions and 1360 deletions

No files matched your search

+54 -25
View File
@@ -169,13 +169,13 @@ jobs:
build-linux-release:
machine:
image: ubuntu-1604:202007-01
resource_class: 2xlarge
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: make V=1 -j8 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- install-gflags
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: make V=1 -j8 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- post-steps
@@ -186,11 +186,11 @@ jobs:
steps:
- checkout # check out the code in the project directory
- run: make clean
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j8 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
- run: make clean
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j8 static_lib tools db_bench | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
build-linux-lite:
@@ -209,21 +209,21 @@ jobs:
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: LITE=1 make V=1 -j8 release | .circleci/cat_ignore_eagain
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- install-gflags
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
- run: LITE=1 make V=1 -j8 release | .circleci/cat_ignore_eagain
- run: ./db_stress --version # ensure with gflags
- post-steps
build-linux-clang-no-test:
build-linux-clang-no_test_run:
machine:
image: ubuntu-1604:202007-01
resource_class: 2xlarge
resource_class: xlarge
steps:
- checkout # check out the code in the project directory
- run: sudo apt-get update -y && sudo apt-get install -y clang libgflags-dev
- run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make V=1 -j32 all | .circleci/cat_ignore_eagain
- run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make V=1 -j16 all | .circleci/cat_ignore_eagain
- post-steps
build-linux-clang10-asan:
@@ -271,33 +271,63 @@ jobs:
- run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 CLANG_ANALYZER="/usr/bin/clang++-10" CLANG_SCAN_BUILD=scan-build-10 USE_CLANG=1 make V=1 -j32 analyze | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out. For unknown, reason passing "clang++-10" as CLANG_ANALYZER doesn't work, and we need a full path.
- post-steps
build-linux-cmake:
build-linux-cmake-no_test_run:
machine:
image: ubuntu-1604:202007-01
resource_class: 2xlarge
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: (mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make V=1 -j32) | .circleci/cat_ignore_eagain
- run: (mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make V=1 -j8) | .circleci/cat_ignore_eagain
- post-steps
build-linux-unity:
docker: # executor type
- image: gcc:latest
resource_class: xlarge
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: apt-get update -y && apt-get install -y libgflags-dev
- run: TEST_TMPDIR=/dev/shm && make V=1 -j16 unity_test | .circleci/cat_ignore_eagain
- run: TEST_TMPDIR=/dev/shm && make V=1 -j8 unity_test | .circleci/cat_ignore_eagain
- post-steps
build-linux-gcc-4-8:
build-linux-gcc-4_8-no_test_run:
machine:
image: ubuntu-1604:202007-01
resource_class: large
steps:
- pre-steps
- run: sudo apt-get update -y && sudo apt-get install gcc-4.8 g++-4.8 libgflags-dev
- run: CC=gcc-4.8 CXX=g++-4.8 V=1 SKIP_LINK=1 make -j4 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- run: CC=gcc-4.8 CXX=g++-4.8 V=1 SKIP_LINK=1 make -j8 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- post-steps
build-linux-gcc-8-no_test_run:
machine:
image: ubuntu-2004:202010-01
resource_class: large
steps:
- pre-steps
- run: sudo apt-get update -y && sudo apt-get install gcc-8 g++-8 libgflags-dev
- run: CC=gcc-8 CXX=g++-8 V=1 SKIP_LINK=1 make -j8 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- post-steps
build-linux-gcc-9-no_test_run:
machine:
image: ubuntu-2004:202010-01
resource_class: large
steps:
- pre-steps
- run: sudo apt-get update -y && sudo apt-get install gcc-9 g++-9 libgflags-dev
- run: CC=gcc-9 CXX=g++-9 V=1 SKIP_LINK=1 make -j8 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- post-steps
build-linux-gcc-10-cxx20-no_test_run:
machine:
image: ubuntu-2004:202010-01
resource_class: xlarge
steps:
- pre-steps
- run: sudo apt-get update -y && sudo apt-get install gcc-10 g++-10 libgflags-dev
- run: CC=gcc-10 CXX=g++-10 V=1 SKIP_LINK=1 ROCKSDB_CXX_STANDARD=c++20 make -j16 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI
- post-steps
build-windows:
@@ -547,9 +577,6 @@ workflows:
build-linux-lite-release:
jobs:
- build-linux-lite-release
build-linux-clang-no-test:
jobs:
- build-linux-clang-no-test
build-linux-clang10-asan:
jobs:
- build-linux-clang10-asan
@@ -562,9 +589,6 @@ workflows:
build-linux-clang10-clang-analyze:
jobs:
- build-linux-clang10-clang-analyze
build-linux-cmake:
jobs:
- build-linux-cmake
build-linux-unity:
jobs:
- build-linux-unity
@@ -608,9 +632,14 @@ workflows:
- build-linux-non-shm:
start_test: "statistics_test" # make sure unique in src.mk
end_test: ""
build-linux-gcc-4-8:
build-linux-compilers-no_test_run:
jobs:
- build-linux-gcc-4-8
- build-linux-clang-no_test_run
- build-linux-cmake-no_test_run
- build-linux-gcc-4_8-no_test_run
- build-linux-gcc-8-no_test_run
- build-linux-gcc-9-no_test_run
- build-linux-gcc-10-cxx20-no_test_run
build-macos:
jobs:
- build-macos
+2 -10
View File
@@ -86,19 +86,11 @@ matrix:
env: JOB_NAME=make-gcc4.8
- os: linux
compiler: clang
# Exclude all but most unique cmake variants for pull requests, but build all in branches
# With migration to CircleCI, exclude Linux/amd64 for pull requests
# (but build in branches for now)
- if: type = pull_request AND commit_message !~ /FULL_CI/
os : linux
arch: amd64
env: JOB_NAME=cmake
- if: type = pull_request AND commit_message !~ /FULL_CI/
os : linux
arch: amd64
env: JOB_NAME=cmake-gcc8
- if: type = pull_request AND commit_message !~ /FULL_CI/
os : linux
arch: amd64
env: JOB_NAME=cmake-gcc9
# Exclude most osx, arm64 and ppc64le tests for pull requests, but build in branches
# Temporarily disable ppc64le cmake test while snapd is broken
- if: type = pull_request AND commit_message !~ /FULL_CI/
+5 -1
View File
@@ -814,6 +814,8 @@ set(SOURCES
utilities/transactions/lock/lock_manager.cc
utilities/transactions/lock/point/point_lock_tracker.cc
utilities/transactions/lock/point/point_lock_manager.cc
utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc
utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc
utilities/transactions/optimistic_transaction_db_impl.cc
utilities/transactions/optimistic_transaction.cc
utilities/transactions/pessimistic_transaction.cc
@@ -831,7 +833,7 @@ set(SOURCES
utilities/write_batch_with_index/write_batch_with_index_internal.cc
$<TARGET_OBJECTS:build_version>)
list(APPEND SOURCE
list(APPEND SOURCES
utilities/transactions/lock/range/range_tree/lib/locktree/concurrent_tree.cc
utilities/transactions/lock/range/range_tree/lib/locktree/keyrange.cc
utilities/transactions/lock/range/range_tree/lib/locktree/lock_request.cc
@@ -1174,6 +1176,7 @@ if(WITH_TESTS)
table/table_test.cc
table/block_fetcher_test.cc
test_util/testutil_test.cc
trace_replay/block_cache_tracer_test.cc
trace_replay/io_tracer_test.cc
tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc
tools/io_tracer_parser_test.cc
@@ -1224,6 +1227,7 @@ if(WITH_TESTS)
utilities/transactions/lock/point/point_lock_manager_test.cc
utilities/transactions/write_prepared_transaction_test.cc
utilities/transactions/write_unprepared_transaction_test.cc
utilities/transactions/lock/range/range_locking_test.cc
utilities/ttl/ttl_test.cc
utilities/write_batch_with_index/write_batch_with_index_test.cc
)
+65 -5
View File
@@ -190,6 +190,11 @@ else
endif
ifdef ASSERT_STATUS_CHECKED
# For ASC, turn off constructor elision, preventing the case where a constructor returned
# by a method may pass the ASC check if the status is checked in the inner method. Forcing
# the copy constructor to be invoked disables the optimization and will cause the calling method
# to check the status in order to prevent an error from being raised.
PLATFORM_CXXFLAGS += -fno-elide-constructors
ifeq ($(filter -DROCKSDB_ASSERT_STATUS_CHECKED,$(OPT)),)
OPT += -DROCKSDB_ASSERT_STATUS_CHECKED
endif
@@ -558,6 +563,7 @@ PARALLEL_TEST = \
table_test \
transaction_test \
point_lock_manager_test \
range_locking_test \
write_prepared_transaction_test \
write_unprepared_transaction_test \
@@ -586,6 +592,7 @@ ifdef ASSERT_STATUS_CHECKED
blob_file_reader_test \
bloom_test \
cassandra_format_test \
cassandra_functional_test \
cassandra_row_merge_test \
cassandra_serialize_test \
cleanable_test \
@@ -594,6 +601,14 @@ ifdef ASSERT_STATUS_CHECKED
crc32c_test \
dbformat_test \
db_basic_test \
compact_files_test \
compaction_picker_test \
comparator_db_test \
db_encryption_test \
db_iter_test \
db_iter_stress_test \
db_log_iter_test \
db_bloom_filter_test \
db_blob_basic_test \
db_blob_index_test \
db_block_cache_test \
@@ -606,10 +621,27 @@ ifdef ASSERT_STATUS_CHECKED
db_wal_test \
db_with_timestamp_basic_test \
db_with_timestamp_compaction_test \
db_write_test \
db_options_test \
db_properties_test \
db_range_del_test \
db_secondary_test \
deletefile_test \
external_sst_file_test \
options_file_test \
db_statistics_test \
db_table_properties_test \
db_tailing_iter_test \
fault_injection_test \
listener_test \
log_test \
manual_compaction_test \
obsolete_files_test \
perf_context_test \
periodic_work_scheduler_test \
perf_context_test \
version_set_test \
wal_manager_test \
defer_test \
filename_test \
dynamic_bloom_test \
@@ -631,6 +663,7 @@ ifdef ASSERT_STATUS_CHECKED
iostats_context_test \
ldb_cmd_test \
memkind_kmem_allocator_test \
merge_test \
merger_test \
mock_env_test \
object_registry_test \
@@ -643,6 +676,7 @@ ifdef ASSERT_STATUS_CHECKED
options_settable_test \
options_test \
point_lock_manager_test \
random_access_file_reader_test \
random_test \
range_del_aggregator_test \
sst_file_reader_test \
@@ -651,9 +685,11 @@ ifdef ASSERT_STATUS_CHECKED
ribbon_test \
skiplist_test \
slice_test \
slice_transform_test \
sst_dump_test \
statistics_test \
stats_history_test \
stringappend_test \
thread_local_test \
trace_analyzer_test \
transaction_test \
@@ -671,25 +707,38 @@ ifdef ASSERT_STATUS_CHECKED
version_builder_test \
version_edit_test \
work_queue_test \
write_buffer_manager_test \
write_controller_test \
write_prepared_transaction_test \
write_unprepared_transaction_test \
compaction_iterator_test \
compaction_job_test \
compaction_job_stats_test \
io_tracer_test \
io_tracer_test \
io_tracer_parser_test \
prefetch_test \
merge_helper_test \
memtable_list_test \
flush_job_test \
block_based_filter_block_test \
block_fetcher_test \
block_test \
data_block_hash_index_test \
full_filter_block_test \
partitioned_filter_block_test \
column_family_test \
file_reader_writer_test \
rate_limiter_test \
corruption_test \
reduce_levels_test \
thread_list_test \
compact_on_deletion_collector_test \
db_universal_compaction_test \
import_column_family_test \
option_change_migration_test \
cuckoo_table_builder_test \
cuckoo_table_db_test \
cuckoo_table_reader_test \
memory_test \
table_test \
write_batch_test \
@@ -1926,6 +1975,9 @@ blob_db_test: $(OBJ_DIR)/utilities/blob_db/blob_db_test.o $(TEST_LIBRARY) $(LIBR
repeatable_thread_test: $(OBJ_DIR)/util/repeatable_thread_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
range_locking_test: utilities/transactions/lock/range/range_locking_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
range_tombstone_fragmenter_test: $(OBJ_DIR)/db/range_tombstone_fragmenter_test.o $(TEST_LIBRARY) $(LIBRARY)
$(AM_LINK)
@@ -2206,9 +2258,10 @@ endif
JAVA_STATIC_FLAGS = -DZLIB -DBZIP2 -DSNAPPY -DLZ4 -DZSTD
JAVA_STATIC_INCLUDES = -I./zlib-$(ZLIB_VER) -I./bzip2-$(BZIP2_VER) -I./snappy-$(SNAPPY_VER) -I./snappy-$(SNAPPY_VER)/build -I./lz4-$(LZ4_VER)/lib -I./zstd-$(ZSTD_VER)/lib -I./zstd-$(ZSTD_VER)/lib/dictBuilder
ifneq ($(findstring rocksdbjavastatic, $(MAKECMDGOALS)),)
ifneq ($(findstring rocksdbjavastatic, $(filter-out rocksdbjavastatic_deps, $(MAKECMDGOALS))),)
CXXFLAGS += $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES)
CFLAGS += $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES)
CFLAGS += $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES)
endif
rocksdbjavastatic:
ifeq ($(JAVA_HOME),)
@@ -2216,8 +2269,11 @@ ifeq ($(JAVA_HOME),)
endif
$(MAKE) rocksdbjavastatic_deps
$(MAKE) rocksdbjavastatic_libobjects
cd java;$(MAKE) javalib;
rm -f ./java/target/$(ROCKSDBJNILIB)
$(MAKE) rocksdbjavastatic_javalib
rocksdbjavastatic_javalib:
cd java;$(MAKE) javalib
rm -f java/target/$(ROCKSDBJNILIB)
$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC \
-o ./java/target/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) \
$(LIB_OBJECTS) $(COVERAGEFLAGS) \
@@ -2440,6 +2496,8 @@ ifneq ($(MAKECMDGOALS),clean)
ifneq ($(MAKECMDGOALS),format)
ifneq ($(MAKECMDGOALS),jclean)
ifneq ($(MAKECMDGOALS),jtest)
ifneq ($(MAKECMDGOALS),rocksdbjavastatic)
ifneq ($(MAKECMDGOALS),rocksdbjavastatic_deps)
ifneq ($(MAKECMDGOALS),package)
ifneq ($(MAKECMDGOALS),analyze)
-include $(DEPFILES)
@@ -2449,3 +2507,5 @@ endif
endif
endif
endif
endif
endif
+11
View File
@@ -400,6 +400,8 @@ cpp_library(
"utilities/transactions/lock/range/range_tree/lib/standalone_port.cc",
"utilities/transactions/lock/range/range_tree/lib/util/dbt.cc",
"utilities/transactions/lock/range/range_tree/lib/util/memarena.cc",
"utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc",
"utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc",
"utilities/transactions/optimistic_transaction.cc",
"utilities/transactions/optimistic_transaction_db_impl.cc",
"utilities/transactions/pessimistic_transaction.cc",
@@ -702,6 +704,8 @@ cpp_library(
"utilities/transactions/lock/range/range_tree/lib/standalone_port.cc",
"utilities/transactions/lock/range/range_tree/lib/util/dbt.cc",
"utilities/transactions/lock/range/range_tree/lib/util/memarena.cc",
"utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc",
"utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc",
"utilities/transactions/optimistic_transaction.cc",
"utilities/transactions/optimistic_transaction_db_impl.cc",
"utilities/transactions/pessimistic_transaction.cc",
@@ -1800,6 +1804,13 @@ ROCKS_TESTS = [
[],
[],
],
[
"range_locking_test",
"utilities/transactions/lock/range/range_locking_test.cc",
"parallel",
[],
[],
],
[
"range_tombstone_fragmenter_test",
"db/range_tombstone_fragmenter_test.cc",
+7 -2
View File
@@ -45,8 +45,13 @@ if test -z "$OUTPUT"; then
exit 1
fi
# we depend on C++11
PLATFORM_CXXFLAGS="-std=c++11"
# we depend on C++11, but should be compatible with newer standards
if [ "$ROCKSDB_CXX_STANDARD" ]; then
PLATFORM_CXXFLAGS="-std=$ROCKSDB_CXX_STANDARD"
else
PLATFORM_CXXFLAGS="-std=c++11"
fi
# we currently depend on POSIX platform
COMMON_FLAGS="-DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX"
+3 -1
View File
@@ -335,7 +335,9 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
// was not used)
auto sfm = static_cast<SstFileManagerImpl*>(db_options.sst_file_manager.get());
for (size_t i = 0; i < result.cf_paths.size(); i++) {
DeleteScheduler::CleanupDirectory(db_options.env, sfm, result.cf_paths[i].path);
DeleteScheduler::CleanupDirectory(db_options.env, sfm,
result.cf_paths[i].path)
.PermitUncheckedError();
}
#endif
+22 -22
View File
@@ -73,7 +73,7 @@ class ColumnFamilyTestBase : public testing::Test {
db_options_.create_if_missing = true;
db_options_.fail_if_options_file_error = true;
db_options_.env = env_;
DestroyDB(dbname_, Options(db_options_, column_family_options_));
EXPECT_OK(DestroyDB(dbname_, Options(db_options_, column_family_options_)));
}
~ColumnFamilyTestBase() override {
@@ -653,8 +653,8 @@ TEST_P(FlushEmptyCFTestWithParam, FlushEmptyCFTest) {
// after flushing file B is deleted. At the same time, the min log number of
// default CF is not written to manifest. Log file A still remains.
// Flushed to SST file Y.
Flush(1);
Flush(0);
ASSERT_OK(Flush(1));
ASSERT_OK(Flush(0));
ASSERT_OK(Put(1, "bar", "v3")); // seqID 4
ASSERT_OK(Put(1, "foo", "v4")); // seqID 5
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
@@ -708,15 +708,15 @@ TEST_P(FlushEmptyCFTestWithParam, FlushEmptyCFTest2) {
// and is set to current. Both CFs' min log number is set to file C so after
// flushing file B is deleted. Log file A still remains.
// Flushed to SST file Y.
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(0, "bar", "v2")); // seqID 4
ASSERT_OK(Put(2, "bar", "v2")); // seqID 5
ASSERT_OK(Put(1, "bar", "v3")); // seqID 6
// Flushing all column families. This forces all CFs' min log to current. This
// is written to the manifest file. Log file C is cleared.
Flush(0);
Flush(1);
Flush(2);
ASSERT_OK(Flush(0));
ASSERT_OK(Flush(1));
ASSERT_OK(Flush(2));
// Write to log file D
ASSERT_OK(Put(1, "bar", "v4")); // seqID 7
ASSERT_OK(Put(1, "bar", "v5")); // seqID 8
@@ -985,7 +985,7 @@ TEST_P(ColumnFamilyTest, FlushTest) {
for (int i = 0; i < 3; ++i) {
uint64_t max_total_in_memory_state =
MaxTotalInMemoryState();
Flush(i);
ASSERT_OK(Flush(i));
AssertMaxTotalInMemoryState(max_total_in_memory_state);
}
ASSERT_OK(Put(1, "foofoo", "bar"));
@@ -1093,7 +1093,7 @@ TEST_P(ColumnFamilyTest, CrashAfterFlush) {
ASSERT_OK(batch.Put(handles_[0], Slice("foo"), Slice("bar")));
ASSERT_OK(batch.Put(handles_[1], Slice("foo"), Slice("bar")));
ASSERT_OK(db_->Write(WriteOptions(), &batch));
Flush(0);
ASSERT_OK(Flush(0));
fault_env->SetFilesystemActive(false);
std::vector<std::string> names;
@@ -1103,7 +1103,7 @@ TEST_P(ColumnFamilyTest, CrashAfterFlush) {
}
}
Close();
fault_env->DropUnsyncedFileData();
ASSERT_OK(fault_env->DropUnsyncedFileData());
fault_env->ResetState();
Open(names, {});
@@ -2236,7 +2236,7 @@ TEST_P(ColumnFamilyTest, FlushStaleColumnFamilies) {
// files for column family [one], because it's empty
AssertCountLiveFiles(4);
Flush(0);
ASSERT_OK(Flush(0));
ASSERT_EQ(0, dbfull()->TEST_total_log_size());
Close();
}
@@ -3040,7 +3040,7 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile1) {
Iterator* it = db_->NewIterator(ReadOptions(), handles_[1]);
ASSERT_OK(it->status());
// A flush will make `it` hold the last reference of its super version.
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "fodor", "mirko"));
ASSERT_OK(Put(0, "fodor", "mirko"));
@@ -3093,7 +3093,7 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
Iterator* it = db_->NewIterator(ro, handles_[1]);
ASSERT_OK(it->status());
// A flush will make `it` hold the last reference of its super version.
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "fodor", "mirko"));
ASSERT_OK(Put(0, "fodor", "mirko"));
@@ -3147,7 +3147,7 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
CreateColumnFamilies({"one"});
ASSERT_OK(Put(1, "fodor", "mirko"));
ASSERT_OK(Put(1, "fodar2", "mirko"));
Flush(1);
ASSERT_OK(Flush(1));
// Create an iterator holding the current super version, as well as
// the SST file just flushed.
@@ -3159,7 +3159,7 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
ASSERT_OK(Put(1, "fodor", "mirko"));
ASSERT_OK(Put(1, "fodar2", "mirko"));
Flush(1);
ASSERT_OK(Flush(1));
WaitForCompaction();
@@ -3232,9 +3232,9 @@ TEST_P(ColumnFamilyTest, LogSyncConflictFlush) {
ROCKSDB_NAMESPACE::port::Thread thread([&] { ASSERT_OK(db_->SyncWAL()); });
TEST_SYNC_POINT("ColumnFamilyTest::LogSyncConflictFlush:1");
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "foo", "bar"));
Flush(1);
ASSERT_OK(Flush(1));
TEST_SYNC_POINT("ColumnFamilyTest::LogSyncConflictFlush:2");
@@ -3256,7 +3256,7 @@ TEST_P(ColumnFamilyTest, DISABLED_LogTruncationTest) {
Build(0, 100);
// Flush the 0th column family to force a roll of the wal log
Flush(0);
ASSERT_OK(Flush(0));
// Add some more entries
Build(100, 100);
@@ -3332,14 +3332,14 @@ TEST_P(ColumnFamilyTest, DefaultCfPathsTest) {
// Fill Column family 1.
PutRandomData(1, 100, 100);
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_EQ(1, GetSstFileCount(cf_opt1.cf_paths[0].path));
ASSERT_EQ(0, GetSstFileCount(dbname_));
// Fill column family 2
PutRandomData(2, 100, 100);
Flush(2);
ASSERT_OK(Flush(2));
// SST from Column family 2 should be generated in
// db_paths which is dbname_ in this case.
@@ -3358,14 +3358,14 @@ TEST_P(ColumnFamilyTest, MultipleCFPathsTest) {
Reopen({ColumnFamilyOptions(), cf_opt1, cf_opt2});
PutRandomData(1, 100, 100, true /* save */);
Flush(1);
ASSERT_OK(Flush(1));
// Check that files are generated in appropriate paths.
ASSERT_EQ(1, GetSstFileCount(cf_opt1.cf_paths[0].path));
ASSERT_EQ(0, GetSstFileCount(dbname_));
PutRandomData(2, 100, 100, true /* save */);
Flush(2);
ASSERT_OK(Flush(2));
ASSERT_EQ(1, GetSstFileCount(cf_opt2.cf_paths[0].path));
ASSERT_EQ(0, GetSstFileCount(dbname_));
+28 -25
View File
@@ -91,9 +91,9 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
// create couple files
// Background compaction starts and waits in BackgroundCallCompaction:0
for (int i = 0; i < kLevel0Trigger * 4; ++i) {
db->Put(WriteOptions(), ToString(i), "");
db->Put(WriteOptions(), ToString(100 - i), "");
db->Flush(FlushOptions());
ASSERT_OK(db->Put(WriteOptions(), ToString(i), ""));
ASSERT_OK(db->Put(WriteOptions(), ToString(100 - i), ""));
ASSERT_OK(db->Flush(FlushOptions()));
}
ROCKSDB_NAMESPACE::ColumnFamilyMetaData meta;
@@ -138,18 +138,18 @@ TEST_F(CompactFilesTest, ObsoleteFiles) {
DB* db = nullptr;
DestroyDB(db_name_, options);
Status s = DB::Open(options, db_name_, &db);
assert(s.ok());
assert(db);
ASSERT_OK(s);
ASSERT_NE(db, nullptr);
// create couple files
for (int i = 1000; i < 2000; ++i) {
db->Put(WriteOptions(), ToString(i),
std::string(kWriteBufferSize / 10, 'a' + (i % 26)));
ASSERT_OK(db->Put(WriteOptions(), ToString(i),
std::string(kWriteBufferSize / 10, 'a' + (i % 26))));
}
auto l0_files = collector->GetFlushedFiles();
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files, 1));
static_cast_with_check<DBImpl>(db)->TEST_WaitForCompact();
ASSERT_OK(static_cast_with_check<DBImpl>(db)->TEST_WaitForCompact());
// verify all compaction input files are deleted
for (auto fname : l0_files) {
@@ -182,15 +182,17 @@ TEST_F(CompactFilesTest, NotCutOutputOnLevel0) {
// create couple files
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
ASSERT_OK(db->Put(WriteOptions(), ToString(i),
std::string(1000, 'a' + (i % 26))));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
ASSERT_OK(static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable());
auto l0_files_1 = collector->GetFlushedFiles();
collector->ClearFlushedFiles();
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
ASSERT_OK(db->Put(WriteOptions(), ToString(i),
std::string(1000, 'a' + (i % 26))));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
ASSERT_OK(static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable());
auto l0_files_2 = collector->GetFlushedFiles();
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files_1, 0));
ASSERT_OK(db->CompactFiles(CompactionOptions(), l0_files_2, 0));
@@ -213,13 +215,13 @@ TEST_F(CompactFilesTest, CapturingPendingFiles) {
DB* db = nullptr;
DestroyDB(db_name_, options);
Status s = DB::Open(options, db_name_, &db);
assert(s.ok());
ASSERT_OK(s);
assert(db);
// Create 5 files.
for (int i = 0; i < 5; ++i) {
db->Put(WriteOptions(), "key" + ToString(i), "value");
db->Flush(FlushOptions());
ASSERT_OK(db->Put(WriteOptions(), "key" + ToString(i), "value"));
ASSERT_OK(db->Flush(FlushOptions()));
}
auto l0_files = collector->GetFlushedFiles();
@@ -237,8 +239,8 @@ TEST_F(CompactFilesTest, CapturingPendingFiles) {
// In the meantime flush another file.
TEST_SYNC_POINT("CompactFilesTest.CapturingPendingFiles:0");
db->Put(WriteOptions(), "key5", "value");
db->Flush(FlushOptions());
ASSERT_OK(db->Put(WriteOptions(), "key5", "value"));
ASSERT_OK(db->Flush(FlushOptions()));
TEST_SYNC_POINT("CompactFilesTest.CapturingPendingFiles:1");
compaction_thread.join();
@@ -249,7 +251,7 @@ TEST_F(CompactFilesTest, CapturingPendingFiles) {
// Make sure we can reopen the DB.
s = DB::Open(options, db_name_, &db);
ASSERT_TRUE(s.ok());
ASSERT_OK(s);
assert(db);
delete db;
}
@@ -293,8 +295,8 @@ TEST_F(CompactFilesTest, CompactionFilterWithGetSv) {
cf->SetDB(db);
// Write one L0 file
db->Put(WriteOptions(), "K1", "V1");
db->Flush(FlushOptions());
ASSERT_OK(db->Put(WriteOptions(), "K1", "V1"));
ASSERT_OK(db->Flush(FlushOptions()));
// Compact all L0 files using CompactFiles
ROCKSDB_NAMESPACE::ColumnFamilyMetaData meta;
@@ -337,8 +339,8 @@ TEST_F(CompactFilesTest, SentinelCompressionType) {
DB* db = nullptr;
ASSERT_OK(DB::Open(options, db_name_, &db));
db->Put(WriteOptions(), "key", "val");
db->Flush(FlushOptions());
ASSERT_OK(db->Put(WriteOptions(), "key", "val"));
ASSERT_OK(db->Flush(FlushOptions()));
auto l0_files = collector->GetFlushedFiles();
ASSERT_EQ(1, l0_files.size());
@@ -377,14 +379,15 @@ TEST_F(CompactFilesTest, GetCompactionJobInfo) {
DB* db = nullptr;
DestroyDB(db_name_, options);
Status s = DB::Open(options, db_name_, &db);
assert(s.ok());
ASSERT_OK(s);
assert(db);
// create couple files
for (int i = 0; i < 500; ++i) {
db->Put(WriteOptions(), ToString(i), std::string(1000, 'a' + (i % 26)));
ASSERT_OK(db->Put(WriteOptions(), ToString(i),
std::string(1000, 'a' + (i % 26))));
}
static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable();
ASSERT_OK(static_cast_with_check<DBImpl>(db)->TEST_WaitForFlushMemTable());
auto l0_files_1 = collector->GetFlushedFiles();
CompactionOptions co;
co.compression = CompressionType::kLZ4Compression;
+5
View File
@@ -82,6 +82,11 @@ class CompactedDBImpl : public DBImpl {
ColumnFamilyHandle* /*column_family*/) override {
return Status::NotSupported("Not supported in compacted db mode.");
}
virtual Status SyncWAL() override {
return Status::NotSupported("Not supported in compacted db mode.");
}
using DB::IngestExternalFile;
virtual Status IngestExternalFile(
ColumnFamilyHandle* /*column_family*/,
+14 -3
View File
@@ -1449,9 +1449,8 @@ Status CompactionJob::FinishCompactionOutputFile(
} else {
sub_compact->builder->Abandon();
}
IOStatus io_s = sub_compact->builder->io_status();
if (s.ok()) {
s = io_s;
s = sub_compact->builder->io_status();
}
const uint64_t current_bytes = sub_compact->builder->FileSize();
if (s.ok()) {
@@ -1462,6 +1461,7 @@ Status CompactionJob::FinishCompactionOutputFile(
sub_compact->total_bytes += current_bytes;
// Finish and check for file errors
IOStatus io_s;
if (s.ok()) {
StopWatch sw(env_, stats_, COMPACTION_OUTFILE_SYNC_MICROS);
io_s = sub_compact->outfile->Sync(db_options_.use_fsync);
@@ -1500,7 +1500,18 @@ Status CompactionJob::FinishCompactionOutputFile(
std::string fname =
TableFileName(sub_compact->compaction->immutable_cf_options()->cf_paths,
meta->fd.GetNumber(), meta->fd.GetPathId());
env_->DeleteFile(fname);
// TODO(AR) it is not clear if there are any larger implications if
// DeleteFile fails here
Status ds = env_->DeleteFile(fname);
if (!ds.ok()) {
ROCKS_LOG_WARN(
db_options_.info_log,
"[%s] [JOB %d] Unable to remove SST file for table #%" PRIu64
" at bottom level%s",
cfd->GetName().c_str(), job_id_, output_number,
meta->marked_for_compaction ? " (need compaction)" : "");
}
// Also need to remove the file from outputs, or it will be added to the
// VersionEdit.
+1 -1
View File
@@ -269,7 +269,7 @@ class CompactionJobTestBase : public testing::Test {
&write_buffer_manager_, &write_controller_,
/*block_cache_tracer=*/nullptr, /*io_tracer=*/nullptr));
compaction_job_stats_.Reset();
SetIdentityFile(env_, dbname_);
ASSERT_OK(SetIdentityFile(env_, dbname_));
VersionEdit new_db;
new_db.SetLogNumber(0);
+1 -1
View File
@@ -141,7 +141,7 @@ class CompactionPickerTest : public testing::Test {
if (temp_vstorage_) {
VersionBuilder builder(FileOptions(), &ioptions_, nullptr,
vstorage_.get(), nullptr);
builder.SaveTo(temp_vstorage_.get());
ASSERT_OK(builder.SaveTo(temp_vstorage_.get()));
vstorage_ = std::move(temp_vstorage_);
}
vstorage_->CalculateBaseBytes(ioptions_, mutable_cf_options_);
+9 -9
View File
@@ -579,7 +579,7 @@ TEST_F(CorruptionTest, FileSystemStateCorrupted) {
if (iter == 0) { // corrupt file size
std::unique_ptr<WritableFile> file;
env_->NewWritableFile(filename, &file, EnvOptions());
ASSERT_OK(env_->NewWritableFile(filename, &file, EnvOptions()));
ASSERT_OK(file->Append(Slice("corrupted sst")));
file.reset();
Status x = TryReopen(&options);
@@ -616,7 +616,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
options.table_factory = mock;
mock->SetCorruptionMode(mode);
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
Build(10);
s = db_->Flush(FlushOptions());
if (mode == mock::MockTableFactory::kCorruptNone) {
@@ -642,7 +642,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
Build(100, 2);
// ASSERT_OK(db_->Flush(FlushOptions()));
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
@@ -669,7 +669,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
std::string start, end;
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(3, &start), Key(7, &end)));
auto snap = db_->GetSnapshot();
@@ -701,7 +701,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
Build(10, 0, 0);
std::string start, end;
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
@@ -737,7 +737,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
db_ = nullptr;
ASSERT_OK(DestroyDB(dbname_, options));
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
std::string start, end;
Build(10);
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
@@ -775,7 +775,7 @@ TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
Build(100, 2);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
@@ -798,7 +798,7 @@ TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
std::make_shared<mock::MockTableFactory>();
options.table_factory = mock;
ASSERT_OK(DB::Open(options, dbname_, &db_));
assert(db_ != nullptr);
assert(db_ != nullptr); // suppress false clang-analyze report
mock->SetCorruptionMode(mock::MockTableFactory::kCorruptReorderKey);
Build(100, 2);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
@@ -884,7 +884,7 @@ TEST_F(CorruptionTest, VerifyWholeTableChecksum) {
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::VerifySstFileChecksum:mismatch", [&](void* arg) {
auto* s = reinterpret_cast<Status*>(arg);
assert(s);
ASSERT_NE(s, nullptr);
++count;
ASSERT_NOK(*s);
});
+20 -20
View File
@@ -129,10 +129,10 @@ TEST_F(CuckooTableDBTest, Flush) {
ASSERT_OK(Put("key1", "v1"));
ASSERT_OK(Put("key2", "v2"));
ASSERT_OK(Put("key3", "v3"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
TablePropertiesCollection ptc;
reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc);
ASSERT_OK(reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc));
ASSERT_EQ(1U, ptc.size());
ASSERT_EQ(3U, ptc.begin()->second->num_entries);
ASSERT_EQ("1", FilesPerLevel());
@@ -146,9 +146,9 @@ TEST_F(CuckooTableDBTest, Flush) {
ASSERT_OK(Put("key4", "v4"));
ASSERT_OK(Put("key5", "v5"));
ASSERT_OK(Put("key6", "v6"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc);
ASSERT_OK(reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc));
ASSERT_EQ(2U, ptc.size());
auto row = ptc.begin();
ASSERT_EQ(3U, row->second->num_entries);
@@ -164,8 +164,8 @@ TEST_F(CuckooTableDBTest, Flush) {
ASSERT_OK(Delete("key6"));
ASSERT_OK(Delete("key5"));
ASSERT_OK(Delete("key4"));
dbfull()->TEST_FlushMemTable();
reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc);
ASSERT_OK(dbfull()->TEST_FlushMemTable());
ASSERT_OK(reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc));
ASSERT_EQ(3U, ptc.size());
row = ptc.begin();
ASSERT_EQ(3U, row->second->num_entries);
@@ -186,10 +186,10 @@ TEST_F(CuckooTableDBTest, FlushWithDuplicateKeys) {
ASSERT_OK(Put("key1", "v1"));
ASSERT_OK(Put("key2", "v2"));
ASSERT_OK(Put("key1", "v3")); // Duplicate
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
TablePropertiesCollection ptc;
reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc);
ASSERT_OK(reinterpret_cast<DB*>(dbfull())->GetPropertiesOfAllTables(&ptc));
ASSERT_EQ(1U, ptc.size());
ASSERT_EQ(2U, ptc.begin()->second->num_entries);
ASSERT_EQ("1", FilesPerLevel());
@@ -219,7 +219,7 @@ TEST_F(CuckooTableDBTest, Uint64Comparator) {
ASSERT_OK(Put(Uint64Key(1), "v1"));
ASSERT_OK(Put(Uint64Key(2), "v2"));
ASSERT_OK(Put(Uint64Key(3), "v3"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
ASSERT_EQ("v1", Get(Uint64Key(1)));
ASSERT_EQ("v2", Get(Uint64Key(2)));
@@ -228,10 +228,10 @@ TEST_F(CuckooTableDBTest, Uint64Comparator) {
// Add more keys.
ASSERT_OK(Delete(Uint64Key(2))); // Delete.
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
ASSERT_OK(Put(Uint64Key(3), "v0")); // Update.
ASSERT_OK(Put(Uint64Key(4), "v4"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
ASSERT_EQ("v1", Get(Uint64Key(1)));
ASSERT_EQ("NOT_FOUND", Get(Uint64Key(2)));
ASSERT_EQ("v0", Get(Uint64Key(3)));
@@ -251,11 +251,11 @@ TEST_F(CuckooTableDBTest, CompactionIntoMultipleFiles) {
for (int idx = 0; idx < 28; ++idx) {
ASSERT_OK(Put(Key(idx), std::string(10000, 'a' + char(idx))));
}
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ("1", FilesPerLevel());
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow trivial move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow trivial move */));
ASSERT_EQ("0,2", FilesPerLevel());
for (int idx = 0; idx < 28; ++idx) {
ASSERT_EQ(std::string(10000, 'a' + char(idx)), Get(Key(idx)));
@@ -274,15 +274,15 @@ TEST_F(CuckooTableDBTest, SameKeyInsertedInTwoDifferentFilesAndCompacted) {
for (int idx = 0; idx < 11; ++idx) {
ASSERT_OK(Put(Key(idx), std::string(10000, 'a')));
}
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ("1", FilesPerLevel());
// Generate one more file in level-0, and should trigger level-0 compaction
for (int idx = 0; idx < 11; ++idx) {
ASSERT_OK(Put(Key(idx), std::string(10000, 'a' + char(idx))));
}
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_CompactRange(0, nullptr, nullptr);
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_EQ("0,1", FilesPerLevel());
for (int idx = 0; idx < 11; ++idx) {
@@ -303,7 +303,7 @@ TEST_F(CuckooTableDBTest, AdaptiveTable) {
ASSERT_OK(Put("key1", "v1"));
ASSERT_OK(Put("key2", "v2"));
ASSERT_OK(Put("key3", "v3"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
// Write some keys using plain table.
std::shared_ptr<TableFactory> block_based_factory(
@@ -319,7 +319,7 @@ TEST_F(CuckooTableDBTest, AdaptiveTable) {
Reopen(&options);
ASSERT_OK(Put("key4", "v4"));
ASSERT_OK(Put("key1", "v5"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
// Write some keys using block based table.
options.table_factory.reset(NewAdaptiveTableFactory(
@@ -328,7 +328,7 @@ TEST_F(CuckooTableDBTest, AdaptiveTable) {
Reopen(&options);
ASSERT_OK(Put("key5", "v6"));
ASSERT_OK(Put("key2", "v7"));
dbfull()->TEST_FlushMemTable();
ASSERT_OK(dbfull()->TEST_FlushMemTable());
ASSERT_EQ("v5", Get("key1"));
ASSERT_EQ("v7", Get("key2"));
+48 -48
View File
@@ -146,7 +146,7 @@ TEST_F(DBBasicTest, ReadOnlyDB) {
// Reopen and flush memtable.
Reopen(options);
Flush();
ASSERT_OK(Flush());
Close();
// Now check keys in read only mode.
ASSERT_OK(ReadOnlyReopen(options));
@@ -182,7 +182,7 @@ TEST_F(DBBasicTest, ReadOnlyDBWithWriteDBIdToManifestSet) {
// Reopen and flush memtable.
Reopen(options);
Flush();
ASSERT_OK(Flush());
Close();
// Now check keys in read only mode.
ASSERT_OK(ReadOnlyReopen(options));
@@ -205,7 +205,7 @@ TEST_F(DBBasicTest, CompactedDB) {
Reopen(options);
// 1 L0 file, use CompactedDB if max_open_files = -1
ASSERT_OK(Put("aaa", DummyString(kFileSize / 2, '1')));
Flush();
ASSERT_OK(Flush());
Close();
ASSERT_OK(ReadOnlyReopen(options));
Status s = Put("new", "value");
@@ -223,12 +223,12 @@ TEST_F(DBBasicTest, CompactedDB) {
Reopen(options);
// Add more L0 files
ASSERT_OK(Put("bbb", DummyString(kFileSize / 2, '2')));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(Put("aaa", DummyString(kFileSize / 2, 'a')));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(Put("bbb", DummyString(kFileSize / 2, 'b')));
ASSERT_OK(Put("eee", DummyString(kFileSize / 2, 'e')));
Flush();
ASSERT_OK(Flush());
Close();
ASSERT_OK(ReadOnlyReopen(options));
@@ -1413,10 +1413,10 @@ TEST_F(DBBasicTest, MultiGetBatchedSortedMultiFile) {
// mix with memtable
ASSERT_OK(Put(1, "k1", "v1"));
ASSERT_OK(Put(1, "k2", "v2"));
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "k3", "v3"));
ASSERT_OK(Put(1, "k4", "v4"));
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Delete(1, "k4"));
ASSERT_OK(Put(1, "k5", "v5"));
ASSERT_OK(Delete(1, "no_key"));
@@ -1459,19 +1459,19 @@ TEST_F(DBBasicTest, MultiGetBatchedDuplicateKeys) {
// mix with memtable
ASSERT_OK(Merge(1, "k1", "v1"));
ASSERT_OK(Merge(1, "k2", "v2"));
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
ASSERT_OK(Merge(1, "k3", "v3"));
ASSERT_OK(Merge(1, "k4", "v4"));
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
ASSERT_OK(Merge(1, "k4", "v4_2"));
ASSERT_OK(Merge(1, "k6", "v6"));
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
ASSERT_OK(Merge(1, "k7", "v7"));
ASSERT_OK(Merge(1, "k8", "v8"));
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
get_perf_context()->Reset();
@@ -1511,12 +1511,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevel) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l2_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(2);
@@ -1525,12 +1525,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevel) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l1_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(1);
@@ -1539,12 +1539,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevel) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l0_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
ASSERT_EQ(0, num_keys);
@@ -1590,12 +1590,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevelMerge) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l2_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(2);
@@ -1604,12 +1604,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevelMerge) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l1_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(1);
@@ -1618,12 +1618,12 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevelMerge) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l0_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
ASSERT_EQ(0, num_keys);
@@ -1705,7 +1705,7 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSize) {
ASSERT_OK(Put(1, "k7", "v7_"));
ASSERT_OK(Put(1, "k3", "v3_"));
ASSERT_OK(Put(1, "k4", "v4"));
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Delete(1, "k4"));
ASSERT_OK(Put(1, "k11", "v11"));
ASSERT_OK(Delete(1, "no_key"));
@@ -1715,7 +1715,7 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSize) {
ASSERT_OK(Put(1, "k15", "v15"));
ASSERT_OK(Put(1, "k16", "v16"));
ASSERT_OK(Put(1, "k17", "v17"));
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "k1", "v1_"));
ASSERT_OK(Put(1, "k2", "v2_"));
@@ -1785,12 +1785,12 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSizeMultiLevelMerge) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l2_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(2);
@@ -1799,12 +1799,12 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSizeMultiLevelMerge) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l1_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
MoveFilesToLevel(1);
@@ -1813,12 +1813,12 @@ TEST_F(DBBasicTest, MultiGetBatchedValueSizeMultiLevelMerge) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l0_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
ASSERT_OK(Flush());
num_keys = 0;
}
ASSERT_EQ(0, num_keys);
@@ -1908,10 +1908,10 @@ TEST_F(DBBasicTest, MultiGetStats) {
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
ASSERT_OK(Flush(1));
}
}
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
for (int i = 501; i < 1000; ++i) {
@@ -1919,11 +1919,11 @@ TEST_F(DBBasicTest, MultiGetStats) {
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
ASSERT_OK(Flush(1));
}
}
Flush(1);
ASSERT_OK(Flush(1));
MoveFilesToLevel(2, 1);
for (int i = 1001; i < total_keys; ++i) {
@@ -1931,10 +1931,10 @@ TEST_F(DBBasicTest, MultiGetStats) {
keys[i] = Slice(keys_str[i]);
ASSERT_OK(Put(1, "k" + std::to_string(i), rnd.RandomString(1000)));
if (i % 100 == 0) {
Flush(1);
ASSERT_OK(Flush(1));
}
}
Flush(1);
ASSERT_OK(Flush(1));
Close();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
@@ -1961,7 +1961,7 @@ TEST_F(DBBasicTest, MultiGetStats) {
ASSERT_GT(hist_sst.max, 0);
// Minimun number of blocks read in a level.
ASSERT_EQ(hist_data_blocks.min, 0);
ASSERT_EQ(hist_data_blocks.min, 3);
ASSERT_GT(hist_index_and_filter_blocks.min, 0);
// Minimun number of sst files read in a level.
ASSERT_GT(hist_sst.max, 0);
@@ -2042,11 +2042,11 @@ TEST_P(DBMultiGetRowCacheTest, MultiGetBatched) {
ASSERT_OK(Put(1, "k2", "v2"));
ASSERT_OK(Put(1, "k3", "v3"));
ASSERT_OK(Put(1, "k4", "v4"));
Flush(1);
ASSERT_OK(Flush(1));
ASSERT_OK(Put(1, "k5", "v5"));
const Snapshot* snap1 = dbfull()->GetSnapshot();
ASSERT_OK(Delete(1, "k4"));
Flush(1);
ASSERT_OK(Flush(1));
const Snapshot* snap2 = dbfull()->GetSnapshot();
get_perf_context()->Reset();
@@ -2185,7 +2185,7 @@ TEST_F(DBBasicTest, MultiGetIOBufferOverrun) {
std::string value(rnd.RandomString(128) + zero_str);
assert(Put(Key(i), value) == Status::OK());
}
Flush();
ASSERT_OK(Flush());
std::vector<std::string> key_data(10);
std::vector<Slice> keys;
@@ -2579,7 +2579,7 @@ class DBBasicTestTrackWal : public DBTestBase,
int CountWalFiles() {
VectorLogPtr log_files;
dbfull()->GetSortedWalFiles(log_files);
EXPECT_OK(dbfull()->GetSortedWalFiles(log_files));
return static_cast<int>(log_files.size());
};
};
@@ -2714,9 +2714,9 @@ class DBBasicTestMultiGet : public DBTestBase {
: Put(cf, Key(i), values_[i])) == Status::OK());
}
if (num_cfs == 1) {
Flush();
EXPECT_OK(Flush());
} else {
dbfull()->Flush(FlushOptions(), handles_[cf]);
EXPECT_OK(dbfull()->Flush(FlushOptions(), handles_[cf]));
}
for (int i = 0; i < 100; ++i) {
@@ -2728,9 +2728,9 @@ class DBBasicTestMultiGet : public DBTestBase {
Status::OK());
}
if (num_cfs == 1) {
Flush();
EXPECT_OK(Flush());
} else {
dbfull()->Flush(FlushOptions(), handles_[cf]);
EXPECT_OK(dbfull()->Flush(FlushOptions(), handles_[cf]));
}
}
}
@@ -3643,7 +3643,7 @@ TEST_P(DBBasicTestDeadline, PointLookupDeadline) {
std::string key = "k" + ToString(i);
ASSERT_OK(Put(key, rnd.RandomString(100)));
}
Flush();
ASSERT_OK(Flush());
bool timedout = true;
// A timeout will be forced when the IO counter reaches this value
+26 -25
View File
@@ -128,8 +128,8 @@ TEST_P(DBBloomFilterTestDefFormatVersion, KeyMayExist) {
ASSERT_EQ(cache_added, TestGetTickerCount(options, BLOCK_CACHE_ADD));
ASSERT_OK(Flush(1));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, handles_[1],
true /* disallow trivial move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, handles_[1],
true /* disallow trivial move */));
numopen = TestGetTickerCount(options, NO_FILE_OPENS);
cache_added = TestGetTickerCount(options, BLOCK_CACHE_ADD);
@@ -178,7 +178,7 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloomCustomPrefixExtractor) {
ASSERT_OK(dbfull()->Put(wo, "barbarbar2", "foo2"));
ASSERT_OK(dbfull()->Put(wo, "foofoofoo", "bar"));
dbfull()->Flush(fo);
ASSERT_OK(dbfull()->Flush(fo));
ASSERT_EQ("foo", Get("barbarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 0);
@@ -244,7 +244,7 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloom) {
ASSERT_OK(dbfull()->Put(wo, "barbarbar2", "foo2"));
ASSERT_OK(dbfull()->Put(wo, "foofoofoo", "bar"));
dbfull()->Flush(fo);
ASSERT_OK(dbfull()->Flush(fo));
ASSERT_EQ("foo", Get("barbarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 0);
@@ -297,7 +297,7 @@ TEST_F(DBBloomFilterTest, WholeKeyFilterProp) {
// ranges.
ASSERT_OK(dbfull()->Put(wo, "aaa", ""));
ASSERT_OK(dbfull()->Put(wo, "zzz", ""));
dbfull()->Flush(fo);
ASSERT_OK(dbfull()->Flush(fo));
Reopen(options);
ASSERT_EQ("NOT_FOUND", Get("foo"));
@@ -328,7 +328,7 @@ TEST_F(DBBloomFilterTest, WholeKeyFilterProp) {
// ranges.
ASSERT_OK(dbfull()->Put(wo, "aaa", ""));
ASSERT_OK(dbfull()->Put(wo, "zzz", ""));
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
// Reopen with both of whole key off and prefix extractor enabled.
// Still no bloom filter should be used.
@@ -351,7 +351,7 @@ TEST_F(DBBloomFilterTest, WholeKeyFilterProp) {
// ranges.
ASSERT_OK(dbfull()->Put(wo, "aaa", ""));
ASSERT_OK(dbfull()->Put(wo, "zzz", ""));
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
options.prefix_extractor.reset();
bbto.whole_key_filtering = true;
@@ -364,7 +364,7 @@ TEST_F(DBBloomFilterTest, WholeKeyFilterProp) {
// not filtered out by key ranges.
ASSERT_OK(dbfull()->Put(wo, "aaa", ""));
ASSERT_OK(dbfull()->Put(wo, "zzz", ""));
Flush();
ASSERT_OK(Flush());
// Now we have two files:
// File 1: An older file with prefix bloom.
@@ -467,7 +467,7 @@ TEST_P(DBBloomFilterTestWithParam, BloomFilter) {
for (int i = 0; i < N; i += 100) {
ASSERT_OK(Put(1, Key(i), Key(i)));
}
Flush(1);
ASSERT_OK(Flush(1));
// Prevent auto compactions triggered by seeks
env_->delay_sstable_sync_.store(true, std::memory_order_release);
@@ -880,7 +880,7 @@ TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
// Destroy
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
dbfull()->DestroyColumnFamilyHandle(handles_[1]);
ASSERT_OK(dbfull()->DestroyColumnFamilyHandle(handles_[1]));
handles_[1] = nullptr;
}
}
@@ -1444,9 +1444,9 @@ void PrefixScanInit(DBBloomFilterTest* dbtest) {
snprintf(buf, sizeof(buf), "%02d______:end", 10);
keystr = std::string(buf);
ASSERT_OK(dbtest->Put(keystr, keystr));
dbtest->Flush();
dbtest->dbfull()->CompactRange(CompactRangeOptions(), nullptr,
nullptr); // move to level 1
ASSERT_OK(dbtest->Flush());
ASSERT_OK(dbtest->dbfull()->CompactRange(CompactRangeOptions(), nullptr,
nullptr)); // move to level 1
// GROUP 1
for (int i = 1; i <= small_range_sstfiles; i++) {
@@ -1563,21 +1563,21 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
for (int key : keys) {
ASSERT_OK(Put(1, Key(key), "val"));
if (++num_inserted % 1000 == 0) {
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
}
ASSERT_OK(Put(1, Key(0), "val"));
ASSERT_OK(Put(1, Key(numkeys), "val"));
ASSERT_OK(Flush(1));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
if (NumTableFilesAtLevel(0, 1) == 0) {
// No Level 0 file. Create one.
ASSERT_OK(Put(1, Key(0), "val"));
ASSERT_OK(Put(1, Key(numkeys), "val"));
ASSERT_OK(Flush(1));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
for (int i = 1; i < numkeys; i += 2) {
@@ -1682,7 +1682,8 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
BottommostLevelCompaction::kSkip;
compact_options.change_level = true;
compact_options.target_level = 7;
db_->CompactRange(compact_options, handles_[1], nullptr, nullptr);
ASSERT_TRUE(db_->CompactRange(compact_options, handles_[1], nullptr, nullptr)
.IsNotSupported());
ASSERT_EQ(trivial_move, 1);
ASSERT_EQ(non_trivial_move, 0);
@@ -1714,10 +1715,10 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
int CountIter(std::unique_ptr<Iterator>& iter, const Slice& key) {
int count = 0;
for (iter->Seek(key); iter->Valid() && iter->status() == Status::OK();
iter->Next()) {
for (iter->Seek(key); iter->Valid(); iter->Next()) {
count++;
}
EXPECT_OK(iter->status());
return count;
}
@@ -1747,7 +1748,7 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterUpperBound) {
ASSERT_OK(Put("abcdxxx1", "val2"));
ASSERT_OK(Put("abcdxxx2", "val3"));
ASSERT_OK(Put("abcdxxx3", "val4"));
dbfull()->Flush(FlushOptions());
ASSERT_OK(dbfull()->Flush(FlushOptions()));
{
// prefix_extractor has not changed, BF will always be read
Slice upper_bound("abce");
@@ -1905,7 +1906,7 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterMultipleSST) {
ASSERT_OK(Put("foo4", "bar4"));
ASSERT_OK(Put("foq5", "bar5"));
ASSERT_OK(Put("fpb", "1"));
dbfull()->Flush(FlushOptions());
ASSERT_OK(dbfull()->Flush(FlushOptions()));
{
// BF is cappped:3 now
std::unique_ptr<Iterator> iter_tmp(db_->NewIterator(read_options));
@@ -1929,7 +1930,7 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterMultipleSST) {
ASSERT_OK(Put("foo7", "bar7"));
ASSERT_OK(Put("foq8", "bar8"));
ASSERT_OK(Put("fpc", "2"));
dbfull()->Flush(FlushOptions());
ASSERT_OK(dbfull()->Flush(FlushOptions()));
{
// BF is fixed:2 now
std::unique_ptr<Iterator> iter_tmp(db_->NewIterator(read_options));
@@ -2040,10 +2041,10 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterNewColumnFamily) {
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_PREFIX_USEFUL), 0);
}
ASSERT_OK(dbfull()->DropColumnFamily(handles_[2]));
dbfull()->DestroyColumnFamilyHandle(handles_[2]);
ASSERT_OK(dbfull()->DestroyColumnFamilyHandle(handles_[2]));
handles_[2] = nullptr;
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
dbfull()->DestroyColumnFamilyHandle(handles_[1]);
ASSERT_OK(dbfull()->DestroyColumnFamilyHandle(handles_[1]));
handles_[1] = nullptr;
iteration++;
}
+15 -21
View File
@@ -3483,7 +3483,6 @@ Status DBImpl::DeleteFile(std::string name) {
return Status::InvalidArgument("Invalid file name");
}
Status status;
if (type == kWalFile) {
// Only allow deleting archived log files
if (log_type != kArchivedLogFile) {
@@ -3492,7 +3491,7 @@ Status DBImpl::DeleteFile(std::string name) {
name.c_str());
return Status::NotSupported("Delete only supported for archived logs");
}
status = wal_manager_.DeleteFile(name, number);
Status status = wal_manager_.DeleteFile(name, number);
if (!status.ok()) {
ROCKS_LOG_ERROR(immutable_db_options_.info_log,
"DeleteFile %s failed -- %s.\n", name.c_str(),
@@ -3501,6 +3500,7 @@ Status DBImpl::DeleteFile(std::string name) {
return status;
}
Status status;
int level;
FileMetaData* metadata;
ColumnFamilyData* cfd;
@@ -4354,7 +4354,7 @@ Status DBImpl::IngestExternalFiles(
}
}
// Ingest multiple external SST files atomically.
size_t num_cfs = args.size();
const size_t num_cfs = args.size();
for (size_t i = 0; i != num_cfs; ++i) {
if (args[i].external_files.empty()) {
char err_msg[128] = {0};
@@ -4395,10 +4395,7 @@ Status DBImpl::IngestExternalFiles(
env_, versions_.get(), cfd, immutable_db_options_, file_options_,
&snapshots_, arg.options, &directories_, &event_logger_, io_tracer_);
}
std::vector<std::pair<bool, Status>> exec_results;
for (size_t i = 0; i != num_cfs; ++i) {
exec_results.emplace_back(false, Status::OK());
}
// TODO(yanqin) maybe make jobs run in parallel
uint64_t start_file_number = next_file_number;
for (size_t i = 1; i != num_cfs; ++i) {
@@ -4406,10 +4403,13 @@ Status DBImpl::IngestExternalFiles(
auto* cfd =
static_cast<ColumnFamilyHandleImpl*>(args[i].column_family)->cfd();
SuperVersion* super_version = cfd->GetReferencedSuperVersion(this);
exec_results[i].second = ingestion_jobs[i].Prepare(
Status es = ingestion_jobs[i].Prepare(
args[i].external_files, args[i].files_checksums,
args[i].files_checksum_func_names, start_file_number, super_version);
exec_results[i].first = true;
// capture first error only
if (!es.ok() && status.ok()) {
status = es;
}
CleanupSuperVersion(super_version);
}
TEST_SYNC_POINT("DBImpl::IngestExternalFiles:BeforeLastJobPrepare:0");
@@ -4418,23 +4418,17 @@ Status DBImpl::IngestExternalFiles(
auto* cfd =
static_cast<ColumnFamilyHandleImpl*>(args[0].column_family)->cfd();
SuperVersion* super_version = cfd->GetReferencedSuperVersion(this);
exec_results[0].second = ingestion_jobs[0].Prepare(
Status es = ingestion_jobs[0].Prepare(
args[0].external_files, args[0].files_checksums,
args[0].files_checksum_func_names, next_file_number, super_version);
exec_results[0].first = true;
CleanupSuperVersion(super_version);
}
for (const auto& exec_result : exec_results) {
if (!exec_result.second.ok()) {
status = exec_result.second;
break;
if (!es.ok()) {
status = es;
}
CleanupSuperVersion(super_version);
}
if (!status.ok()) {
for (size_t i = 0; i != num_cfs; ++i) {
if (exec_results[i].first) {
ingestion_jobs[i].Cleanup(status);
}
ingestion_jobs[i].Cleanup(status);
}
InstrumentedMutexLock l(&mutex_);
ReleaseFileNumberFromPendingOutputs(pending_output_elem);
@@ -4878,7 +4872,7 @@ Status DBImpl::VerifySstFileChecksum(const FileMetaData& fmeta,
fs_.get(), fname, immutable_db_options_.file_checksum_gen_factory.get(),
fmeta.file_checksum_func_name, &file_checksum, &func_name,
read_options.readahead_size, immutable_db_options_.allow_mmap_reads,
io_tracer_);
io_tracer_, immutable_db_options_.rate_limiter.get());
if (s.ok()) {
assert(fmeta.file_checksum_func_name == func_name);
if (file_checksum != fmeta.file_checksum) {
+18 -16
View File
@@ -284,7 +284,7 @@ Status DBImpl::FlushMemTableToOutputFile(
// Notify sst_file_manager that a new file was added
std::string file_path = MakeTableFileName(
cfd->ioptions()->cf_paths[0].path, file_meta.fd.GetNumber());
sfm->OnAddFile(file_path);
s = sfm->OnAddFile(file_path);
if (sfm->IsMaxAllowedSpaceReached()) {
Status new_bg_error =
Status::SpaceLimit("Max allowed space was reached");
@@ -618,7 +618,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
assert(all_mutable_cf_options.size() == static_cast<size_t>(num_cfs));
for (int i = 0; i != num_cfs; ++i) {
for (int i = 0; s.ok() && i != num_cfs; ++i) {
if (cfds[i]->IsDropped()) {
continue;
}
@@ -627,7 +627,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
if (sfm) {
std::string file_path = MakeTableFileName(
cfds[i]->ioptions()->cf_paths[0].path, file_meta[i].fd.GetNumber());
sfm->OnAddFile(file_path);
s = sfm->OnAddFile(file_path);
if (sfm->IsMaxAllowedSpaceReached() &&
error_handler_.GetBGError().ok()) {
Status new_bg_error =
@@ -809,20 +809,20 @@ Status DBImpl::CompactRangeInternal(const CompactRangeOptions& options,
}
bool flush_needed = true;
Status s;
if (begin != nullptr && end != nullptr) {
// TODO(ajkr): We could also optimize away the flush in certain cases where
// one/both sides of the interval are unbounded. But it requires more
// changes to RangesOverlapWithMemtables.
Range range(*begin, *end);
SuperVersion* super_version = cfd->GetReferencedSuperVersion(this);
cfd->RangesOverlapWithMemtables({range}, super_version,
immutable_db_options_.allow_data_in_errors,
&flush_needed);
s = cfd->RangesOverlapWithMemtables(
{range}, super_version, immutable_db_options_.allow_data_in_errors,
&flush_needed);
CleanupSuperVersion(super_version);
}
Status s;
if (flush_needed) {
if (s.ok() && flush_needed) {
FlushOptions fo;
fo.allow_write_stall = options.allow_write_stall;
if (immutable_db_options_.atomic_flush) {
@@ -1194,7 +1194,8 @@ Status DBImpl::CompactFilesImpl(
mutex_.Unlock();
TEST_SYNC_POINT("CompactFilesImpl:0");
TEST_SYNC_POINT("CompactFilesImpl:1");
compaction_job.Run();
// Ignore the status here, as it will be checked in the Install down below...
compaction_job.Run().PermitUncheckedError();
TEST_SYNC_POINT("CompactFilesImpl:2");
TEST_SYNC_POINT("CompactFilesImpl:3");
mutex_.Lock();
@@ -1391,8 +1392,6 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
SuperVersionContext sv_context(/* create_superversion */ true);
Status status;
InstrumentedMutexLock guard_lock(&mutex_);
// only allow one thread refitting
@@ -1456,8 +1455,9 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
"[%s] Apply version edit:\n%s", cfd->GetName().c_str(),
edit.DebugString().data());
status = versions_->LogAndApply(cfd, mutable_cf_options, &edit, &mutex_,
directories_.GetDbDir());
Status status = versions_->LogAndApply(cfd, mutable_cf_options, &edit,
&mutex_, directories_.GetDbDir());
InstallSuperVersionAndScheduleWork(cfd, &sv_context, mutable_cf_options);
ROCKS_LOG_DEBUG(immutable_db_options_.info_log, "[%s] LogAndApply: %s\n",
@@ -1468,12 +1468,14 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
"[%s] After refitting:\n%s", cfd->GetName().c_str(),
cfd->current()->DebugString().data());
}
sv_context.Clean();
refitting_level_ = false;
return status;
}
sv_context.Clean();
refitting_level_ = false;
return status;
return Status::OK();
}
int DBImpl::NumberLevels(ColumnFamilyHandle* column_family) {
+9 -8
View File
@@ -191,7 +191,8 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
// set of all files in the directory. We'll exclude files that are still
// alive in the subsequent processings.
std::vector<std::string> files;
env_->GetChildren(path, &files).PermitUncheckedError(); // Ignore errors
Status s = env_->GetChildren(path, &files);
s.PermitUncheckedError(); // TODO: What should we do on error?
for (const std::string& file : files) {
uint64_t number;
FileType type;
@@ -207,7 +208,8 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
continue;
}
// TODO(icanadi) clean up this mess to avoid having one-off "/" prefixes
// TODO(icanadi) clean up this mess to avoid having one-off "/"
// prefixes
job_context->full_scan_candidate_files.emplace_back("/" + file, path);
}
}
@@ -215,9 +217,8 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
// Add log files in wal_dir
if (immutable_db_options_.wal_dir != dbname_) {
std::vector<std::string> log_files;
env_->GetChildren(immutable_db_options_.wal_dir,
&log_files)
.PermitUncheckedError(); // Ignore errors
Status s = env_->GetChildren(immutable_db_options_.wal_dir, &log_files);
s.PermitUncheckedError(); // TODO: What should we do on error?
for (const std::string& log_file : log_files) {
job_context->full_scan_candidate_files.emplace_back(
log_file, immutable_db_options_.wal_dir);
@@ -227,9 +228,9 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
if (!immutable_db_options_.db_log_dir.empty() &&
immutable_db_options_.db_log_dir != dbname_) {
std::vector<std::string> info_log_files;
// Ignore errors
env_->GetChildren(immutable_db_options_.db_log_dir, &info_log_files)
.PermitUncheckedError();
Status s =
env_->GetChildren(immutable_db_options_.db_log_dir, &info_log_files);
s.PermitUncheckedError(); // TODO: What should we do on error?
for (std::string& log_file : info_log_files) {
job_context->full_scan_candidate_files.emplace_back(
log_file, immutable_db_options_.db_log_dir);
+5 -4
View File
@@ -147,7 +147,8 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
// DeleteScheduler::CleanupDirectory on the same dir later, it will be
// safe
std::vector<std::string> filenames;
result.env->GetChildren(result.wal_dir, &filenames).PermitUncheckedError();
Status s = result.env->GetChildren(result.wal_dir, &filenames);
s.PermitUncheckedError(); //**TODO: What to do on error?
for (std::string& filename : filenames) {
if (filename.find(".log.trash", filename.length() -
std::string(".log.trash").length()) !=
@@ -163,7 +164,8 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
// was not used)
auto sfm = static_cast<SstFileManagerImpl*>(result.sst_file_manager.get());
for (size_t i = 0; i < result.db_paths.size(); i++) {
DeleteScheduler::CleanupDirectory(result.env, sfm, result.db_paths[i].path);
DeleteScheduler::CleanupDirectory(result.env, sfm, result.db_paths[i].path)
.PermitUncheckedError();
}
// Create a default SstFileManager for purposes of tracking compaction size
@@ -1739,9 +1741,8 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
paths.erase(std::unique(paths.begin(), paths.end()), paths.end());
for (auto& path : paths) {
std::vector<std::string> existing_files;
// TODO: Check for errors here?
impl->immutable_db_options_.env->GetChildren(path, &existing_files)
.PermitUncheckedError();
.PermitUncheckedError(); //**TODO: What do to on error?
for (auto& file_name : existing_files) {
uint64_t file_number;
FileType file_type;
+13 -6
View File
@@ -163,7 +163,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
StopWatch write_sw(env_, immutable_db_options_.statistics.get(), DB_WRITE);
write_thread_.JoinBatchGroup(&w);
Status status;
if (w.state == WriteThread::STATE_PARALLEL_MEMTABLE_WRITER) {
// we are a non-leader in a parallel group
@@ -193,8 +192,6 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
}
assert(w.state == WriteThread::STATE_COMPLETED);
// STATE_COMPLETED conditional below handles exit
status = w.FinalStatus();
}
if (w.state == WriteThread::STATE_COMPLETED) {
if (log_used != nullptr) {
@@ -204,13 +201,11 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
*seq_used = w.sequence;
}
// write is complete and leader has updated sequence
// Should we handle it?
status.PermitUncheckedError();
return w.FinalStatus();
}
// else we are the leader of the write batch group
assert(w.state == WriteThread::STATE_GROUP_LEADER);
Status status;
// Once reaches this point, the current writer "w" will try to do its write
// job. It may also pick up some of the remaining writers in the "writers_"
// when it finds suitable, and finish them in the same write batch.
@@ -531,6 +526,8 @@ Status DBImpl::PipelinedWriteImpl(const WriteOptions& write_options,
PERF_TIMER_STOP(write_pre_and_post_process_time);
IOStatus io_s;
io_s.PermitUncheckedError(); // Allow io_s to be uninitialized
if (w.status.ok() && !write_options.disableWAL) {
PERF_TIMER_GUARD(write_wal_time);
stats->AddDBStats(InternalStats::kIntStatsWriteDoneBySelf, 1);
@@ -566,7 +563,12 @@ Status DBImpl::PipelinedWriteImpl(const WriteOptions& write_options,
write_thread_.ExitAsBatchGroupLeader(wal_write_group, w.status);
}
// NOTE: the memtable_write_group is declared before the following
// `if` statement because its lifetime needs to be longer
// that the inner context of the `if` as a reference to it
// may be used further below within the outer _write_thread
WriteThread::WriteGroup memtable_write_group;
if (w.state == WriteThread::STATE_MEMTABLE_WRITER_LEADER) {
PERF_TIMER_GUARD(write_memtable_time);
assert(w.ShouldWriteToMemtable());
@@ -583,6 +585,10 @@ Status DBImpl::PipelinedWriteImpl(const WriteOptions& write_options,
versions_->SetLastSequence(memtable_write_group.last_sequence);
write_thread_.ExitAsMemTableWriter(&w, memtable_write_group);
}
} else {
// NOTE: the memtable_write_group is never really used,
// so we need to set its status to pass ASSERT_STATUS_CHECKED
memtable_write_group.status.PermitUncheckedError();
}
if (w.state == WriteThread::STATE_PARALLEL_MEMTABLE_WRITER) {
@@ -767,6 +773,7 @@ Status DBImpl::WriteImplWALOnly(
}
Status status;
IOStatus io_s;
io_s.PermitUncheckedError(); // Allow io_s to be uninitialized
if (!write_options.disableWAL) {
io_s = ConcurrentWriteToWAL(write_group, log_used, &last_sequence, seq_inc);
status = io_s;
+17 -1
View File
@@ -392,6 +392,7 @@ TEST_F(DBIteratorTest, DBIteratorPrevNext) {
db_iter->SeekToLast();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
// Test case to check SeekToLast with iterate_upper_bound set
// (same key put may times - SeekToLast should start with the
@@ -489,6 +490,7 @@ TEST_F(DBIteratorTest, DBIteratorPrevNext) {
db_iter->SeekToLast();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
// Test to check the SeekToLast() with the iterate_upper_bound set
// (Deletion cases)
@@ -596,6 +598,7 @@ TEST_F(DBIteratorTest, DBIteratorPrevNext) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
{
@@ -689,6 +692,7 @@ TEST_F(DBIteratorTest, DBIteratorEmpty) {
nullptr /* read_callback */));
db_iter->SeekToLast();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
{
@@ -702,6 +706,7 @@ TEST_F(DBIteratorTest, DBIteratorEmpty) {
nullptr /* read_callback */));
db_iter->SeekToFirst();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
}
@@ -744,6 +749,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkipCountSkips) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
ASSERT_EQ(TestGetTickerCount(options, NUMBER_OF_RESEEKS_IN_ITERATION), 3u);
}
@@ -788,6 +794,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
}
@@ -820,6 +827,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
{
@@ -855,6 +863,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
}
@@ -873,9 +882,11 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
nullptr /* read_callback */));
db_iter->SeekToLast();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
db_iter->SeekToFirst();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
TestIterator* internal_iter = new TestIterator(BytewiseComparator());
@@ -896,6 +907,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
db_iter->SeekToFirst();
ASSERT_TRUE(db_iter->Valid());
@@ -904,6 +916,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Next();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
{
@@ -943,6 +956,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
}
@@ -983,6 +997,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_OK(db_iter->status());
}
}
}
@@ -1039,7 +1054,7 @@ TEST_F(DBIteratorTest, DBIteratorSkipInternalKeys) {
db_iter->Prev();
ASSERT_TRUE(!db_iter->Valid());
ASSERT_TRUE(db_iter->status().ok());
ASSERT_OK(db_iter->status());
}
// Test to make sure that the request will *not* fail as incomplete if
@@ -3136,6 +3151,7 @@ TEST_F(DBIteratorTest, SeekToFirstLowerBound) {
if (i == kNumKeys + 1) {
// lower bound was beyond the last key
ASSERT_FALSE(db_iter->Valid());
ASSERT_OK(db_iter->status());
} else {
ASSERT_TRUE(db_iter->Valid());
int expected;
+51 -47
View File
@@ -33,9 +33,8 @@ class DBTestXactLogIterator : public DBTestBase {
};
namespace {
SequenceNumber ReadRecords(
std::unique_ptr<TransactionLogIterator>& iter,
int& count) {
SequenceNumber ReadRecords(std::unique_ptr<TransactionLogIterator>& iter,
int& count, bool expect_ok = true) {
count = 0;
SequenceNumber lastSequence = 0;
BatchResult res;
@@ -47,6 +46,11 @@ SequenceNumber ReadRecords(
EXPECT_OK(iter->status());
iter->Next();
}
if (expect_ok) {
EXPECT_OK(iter->status());
} else {
EXPECT_NOK(iter->status());
}
return res.sequence;
}
@@ -64,9 +68,9 @@ TEST_F(DBTestXactLogIterator, TransactionLogIterator) {
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
Put(0, "key1", DummyString(1024));
Put(1, "key2", DummyString(1024));
Put(1, "key2", DummyString(1024));
ASSERT_OK(Put(0, "key1", DummyString(1024)));
ASSERT_OK(Put(1, "key2", DummyString(1024)));
ASSERT_OK(Put(1, "key2", DummyString(1024)));
ASSERT_EQ(dbfull()->GetLatestSequenceNumber(), 3U);
{
auto iter = OpenTransactionLogIter(0);
@@ -75,9 +79,9 @@ TEST_F(DBTestXactLogIterator, TransactionLogIterator) {
ReopenWithColumnFamilies({"default", "pikachu"}, options);
env_->SleepForMicroseconds(2 * 1000 * 1000);
{
Put(0, "key4", DummyString(1024));
Put(1, "key5", DummyString(1024));
Put(0, "key6", DummyString(1024));
ASSERT_OK(Put(0, "key4", DummyString(1024)));
ASSERT_OK(Put(1, "key5", DummyString(1024)));
ASSERT_OK(Put(0, "key6", DummyString(1024)));
}
{
auto iter = OpenTransactionLogIter(0);
@@ -109,15 +113,15 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorRace) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
Put("key1", DummyString(1024));
dbfull()->Flush(FlushOptions());
Put("key2", DummyString(1024));
dbfull()->Flush(FlushOptions());
Put("key3", DummyString(1024));
dbfull()->Flush(FlushOptions());
Put("key4", DummyString(1024));
ASSERT_OK(Put("key1", DummyString(1024)));
ASSERT_OK(dbfull()->Flush(FlushOptions()));
ASSERT_OK(Put("key2", DummyString(1024)));
ASSERT_OK(dbfull()->Flush(FlushOptions()));
ASSERT_OK(Put("key3", DummyString(1024)));
ASSERT_OK(dbfull()->Flush(FlushOptions()));
ASSERT_OK(Put("key4", DummyString(1024)));
ASSERT_EQ(dbfull()->GetLatestSequenceNumber(), 4U);
dbfull()->FlushWAL(false);
ASSERT_OK(dbfull()->FlushWAL(false));
{
auto iter = OpenTransactionLogIter(0);
@@ -130,11 +134,11 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorRace) {
// condition
FlushOptions flush_options;
flush_options.wait = false;
dbfull()->Flush(flush_options);
ASSERT_OK(dbfull()->Flush(flush_options));
// "key5" would be written in a new memtable and log
Put("key5", DummyString(1024));
dbfull()->FlushWAL(false);
ASSERT_OK(Put("key5", DummyString(1024)));
ASSERT_OK(dbfull()->FlushWAL(false));
{
// this iter would miss "key4" if not fixed
auto iter = OpenTransactionLogIter(0);
@@ -149,14 +153,14 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorStallAtLastRecord) {
do {
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
Put("key1", DummyString(1024));
ASSERT_OK(Put("key1", DummyString(1024)));
auto iter = OpenTransactionLogIter(0);
ASSERT_OK(iter->status());
ASSERT_TRUE(iter->Valid());
iter->Next();
ASSERT_TRUE(!iter->Valid());
ASSERT_OK(iter->status());
Put("key2", DummyString(1024));
ASSERT_OK(Put("key2", DummyString(1024)));
iter->Next();
ASSERT_OK(iter->status());
ASSERT_TRUE(iter->Valid());
@@ -167,9 +171,9 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorCheckAfterRestart) {
do {
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
Put("key1", DummyString(1024));
Put("key2", DummyString(1023));
dbfull()->Flush(FlushOptions());
ASSERT_OK(Put("key1", DummyString(1024)));
ASSERT_OK(Put("key2", DummyString(1023)));
ASSERT_OK(dbfull()->Flush(FlushOptions()));
Reopen(options);
auto iter = OpenTransactionLogIter(0);
ExpectRecords(2, iter);
@@ -181,10 +185,10 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorCorruptedLog) {
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
for (int i = 0; i < 1024; i++) {
Put("key"+ToString(i), DummyString(10));
ASSERT_OK(Put("key" + ToString(i), DummyString(10)));
}
dbfull()->Flush(FlushOptions());
dbfull()->FlushWAL(false);
ASSERT_OK(dbfull()->Flush(FlushOptions()));
ASSERT_OK(dbfull()->FlushWAL(false));
// Corrupt this log to create a gap
ROCKSDB_NAMESPACE::VectorLogPtr wal_files;
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
@@ -197,13 +201,13 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorCorruptedLog) {
}
// Insert a new entry to a new log file
Put("key1025", DummyString(10));
dbfull()->FlushWAL(false);
ASSERT_OK(Put("key1025", DummyString(10)));
ASSERT_OK(dbfull()->FlushWAL(false));
// Try to read from the beginning. Should stop before the gap and read less
// than 1025 entries
auto iter = OpenTransactionLogIter(0);
int count;
SequenceNumber last_sequence_read = ReadRecords(iter, count);
SequenceNumber last_sequence_read = ReadRecords(iter, count, false);
ASSERT_LT(last_sequence_read, 1025U);
// Try to read past the gap, should be able to seek to key1025
auto iter2 = OpenTransactionLogIter(last_sequence_read + 1);
@@ -217,15 +221,15 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBatchOperations) {
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
WriteBatch batch;
batch.Put(handles_[1], "key1", DummyString(1024));
batch.Put(handles_[0], "key2", DummyString(1024));
batch.Put(handles_[1], "key3", DummyString(1024));
batch.Delete(handles_[0], "key2");
dbfull()->Write(WriteOptions(), &batch);
Flush(1);
Flush(0);
ASSERT_OK(batch.Put(handles_[1], "key1", DummyString(1024)));
ASSERT_OK(batch.Put(handles_[0], "key2", DummyString(1024)));
ASSERT_OK(batch.Put(handles_[1], "key3", DummyString(1024)));
ASSERT_OK(batch.Delete(handles_[0], "key2"));
ASSERT_OK(dbfull()->Write(WriteOptions(), &batch));
ASSERT_OK(Flush(1));
ASSERT_OK(Flush(0));
ReopenWithColumnFamilies({"default", "pikachu"}, options);
Put(1, "key4", DummyString(1024));
ASSERT_OK(Put(1, "key4", DummyString(1024)));
auto iter = OpenTransactionLogIter(3);
ExpectRecords(2, iter);
} while (ChangeCompactOptions());
@@ -237,13 +241,13 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBlobs) {
CreateAndReopenWithCF({"pikachu"}, options);
{
WriteBatch batch;
batch.Put(handles_[1], "key1", DummyString(1024));
batch.Put(handles_[0], "key2", DummyString(1024));
batch.PutLogData(Slice("blob1"));
batch.Put(handles_[1], "key3", DummyString(1024));
batch.PutLogData(Slice("blob2"));
batch.Delete(handles_[0], "key2");
dbfull()->Write(WriteOptions(), &batch);
ASSERT_OK(batch.Put(handles_[1], "key1", DummyString(1024)));
ASSERT_OK(batch.Put(handles_[0], "key2", DummyString(1024)));
ASSERT_OK(batch.PutLogData(Slice("blob1")));
ASSERT_OK(batch.Put(handles_[1], "key3", DummyString(1024)));
ASSERT_OK(batch.PutLogData(Slice("blob2")));
ASSERT_OK(batch.Delete(handles_[0], "key2"));
ASSERT_OK(dbfull()->Write(WriteOptions(), &batch));
ReopenWithColumnFamilies({"default", "pikachu"}, options);
}
@@ -268,7 +272,7 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBlobs) {
return Status::OK();
}
} handler;
res.writeBatchPtr->Iterate(&handler);
ASSERT_OK(res.writeBatchPtr->Iterate(&handler));
ASSERT_EQ(
"Put(1, key1, 1024)"
"Put(0, key2, 1024)"
+1 -1
View File
@@ -729,7 +729,7 @@ TEST_F(DBOptionsTest, SetFIFOCompactionOptions) {
for (int j = 0; j < 10; j++) {
ASSERT_OK(Put(ToString(i * 20 + j), rnd.RandomString(980)));
}
Flush();
ASSERT_OK(Flush());
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 10);
+130 -112
View File
@@ -56,7 +56,7 @@ TEST_F(DBRangeDelTest, EndSameAsStartCoversNothing) {
}
TEST_F(DBRangeDelTest, EndComesBeforeStartInvalidArgument) {
db_->Put(WriteOptions(), "b", "val");
ASSERT_OK(db_->Put(WriteOptions(), "b", "val"));
ASSERT_TRUE(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "b", "a")
.IsInvalidArgument());
@@ -82,13 +82,14 @@ TEST_F(DBRangeDelTest, CompactionOutputHasOnlyRangeTombstone) {
// snapshot protects range tombstone from dropping due to becoming obsolete.
const Snapshot* snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z");
db_->Flush(FlushOptions());
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
ASSERT_OK(db_->Flush(FlushOptions()));
ASSERT_EQ(1, NumTableFilesAtLevel(0));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(1, NumTableFilesAtLevel(1));
ASSERT_EQ(0, TestGetTickerCount(opts, COMPACTION_RANGE_DEL_DROP_OBSOLETE));
@@ -118,7 +119,8 @@ TEST_F(DBRangeDelTest, CompactionOutputFilesExactlyFilled) {
// snapshot protects range tombstone from dropping due to becoming obsolete.
const Snapshot* snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0), Key(1));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
Key(1)));
Random rnd(301);
for (int i = 0; i < kNumFiles; ++i) {
@@ -128,18 +130,18 @@ TEST_F(DBRangeDelTest, CompactionOutputFilesExactlyFilled) {
values.push_back(rnd.RandomString(3 << 10));
ASSERT_OK(Put(Key(i * kNumPerFile + j), values[j]));
if (j == 0 && i > 0) {
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
}
}
// put extra key to trigger final flush
ASSERT_OK(Put("", ""));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(kNumFiles, NumTableFilesAtLevel(0));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(2, NumTableFilesAtLevel(1));
db_->ReleaseSnapshot(snapshot);
@@ -178,12 +180,12 @@ TEST_F(DBRangeDelTest, MaxCompactionBytesCutsOutputFiles) {
}
// extra entry to trigger SpecialSkipListFactory's flush
ASSERT_OK(Put(GetNumericStr(kNumPerFile), ""));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(i + 1, NumTableFilesAtLevel(0));
}
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GE(NumTableFilesAtLevel(1), 2);
@@ -221,10 +223,10 @@ TEST_F(DBRangeDelTest, SentinelsOmittedFromOutputFile) {
}
TEST_F(DBRangeDelTest, FlushRangeDelsSameStartKey) {
db_->Put(WriteOptions(), "b1", "val");
ASSERT_OK(db_->Put(WriteOptions(), "b1", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "c"));
db_->Put(WriteOptions(), "b2", "val");
ASSERT_OK(db_->Put(WriteOptions(), "b2", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "b"));
// first iteration verifies query correctness in memtable, second verifies
@@ -241,8 +243,9 @@ TEST_F(DBRangeDelTest, FlushRangeDelsSameStartKey) {
}
TEST_F(DBRangeDelTest, CompactRangeDelsSameStartKey) {
db_->Put(WriteOptions(), "unused", "val"); // prevents empty after compaction
db_->Put(WriteOptions(), "b1", "val");
ASSERT_OK(db_->Put(WriteOptions(), "unused",
"val")); // prevents empty after compaction
ASSERT_OK(db_->Put(WriteOptions(), "b1", "val"));
ASSERT_OK(db_->Flush(FlushOptions()));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "c"));
@@ -254,8 +257,8 @@ TEST_F(DBRangeDelTest, CompactRangeDelsSameStartKey) {
for (int i = 0; i < 2; ++i) {
if (i > 0) {
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(1, NumTableFilesAtLevel(1));
}
@@ -279,12 +282,13 @@ TEST_F(DBRangeDelTest, FlushRemovesCoveredKeys) {
if (i == kNum / 3) {
snapshot = db_->GetSnapshot();
} else if (i == 2 * kNum / 3) {
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin), GetNumericStr(kRangeEnd));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin),
GetNumericStr(kRangeEnd)));
}
db_->Put(WriteOptions(), GetNumericStr(i), "val");
ASSERT_OK(db_->Put(WriteOptions(), GetNumericStr(i), "val"));
}
db_->Flush(FlushOptions());
ASSERT_OK(db_->Flush(FlushOptions()));
for (int i = 0; i < kNum; ++i) {
ReadOptions read_opts;
@@ -314,24 +318,27 @@ TEST_F(DBRangeDelTest, CompactionRemovesCoveredKeys) {
for (int i = 0; i < kNumFiles; ++i) {
if (i > 0) {
// range tombstone covers first half of the previous file
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr((i - 1) * kNumPerFile),
GetNumericStr((i - 1) * kNumPerFile + kNumPerFile / 2));
ASSERT_OK(db_->DeleteRange(
WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr((i - 1) * kNumPerFile),
GetNumericStr((i - 1) * kNumPerFile + kNumPerFile / 2)));
}
// Make sure a given key appears in each file so compaction won't be able to
// use trivial move, which would happen if the ranges were non-overlapping.
// Also, we need an extra element since flush is only triggered when the
// number of keys is one greater than SpecialSkipListFactory's limit.
// We choose a key outside the key-range used by the test to avoid conflict.
db_->Put(WriteOptions(), GetNumericStr(kNumPerFile * kNumFiles), "val");
ASSERT_OK(db_->Put(WriteOptions(), GetNumericStr(kNumPerFile * kNumFiles),
"val"));
for (int j = 0; j < kNumPerFile; ++j) {
db_->Put(WriteOptions(), GetNumericStr(i * kNumPerFile + j), "val");
ASSERT_OK(
db_->Put(WriteOptions(), GetNumericStr(i * kNumPerFile + j), "val"));
}
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(i + 1, NumTableFilesAtLevel(0));
}
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(1), 0);
ASSERT_EQ((kNumFiles - 1) * kNumPerFile / 2,
@@ -373,8 +380,8 @@ TEST_F(DBRangeDelTest, ValidLevelSubcompactionBoundaries) {
if (i > 0) {
// delete [95,105) in two files, [295,305) in next two
int mid = (j + (1 - j % 2)) * kNumPerFile;
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(mid - 5), Key(mid + 5));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(mid - 5), Key(mid + 5)));
}
std::vector<std::string> values;
// Write 100KB (100 values, each 1K)
@@ -384,7 +391,7 @@ TEST_F(DBRangeDelTest, ValidLevelSubcompactionBoundaries) {
}
// put extra key to trigger flush
ASSERT_OK(Put("", ""));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
if (j < kNumFiles - 1) {
// background compaction may happen early for kNumFiles'th file
ASSERT_EQ(NumTableFilesAtLevel(0), j + 1);
@@ -400,7 +407,7 @@ TEST_F(DBRangeDelTest, ValidLevelSubcompactionBoundaries) {
// oversized L0 (relative to base_level) causes the compaction to run
// earlier.
ASSERT_OK(db_->EnableAutoCompaction({db_->DefaultColumnFamily()}));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_OK(db_->SetOptions(db_->DefaultColumnFamily(),
{{"disable_auto_compactions", "true"}}));
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
@@ -433,8 +440,8 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
// insert range deletions [95,105) in two files, [295,305) in next two
// to prepare L1 for later manual compaction.
int mid = (j + (1 - j % 2)) * kNumPerFile;
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(mid - 5), Key(mid + 5));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(mid - 5), Key(mid + 5)));
}
std::vector<std::string> values;
// Write 100KB (100 values, each 1K)
@@ -444,13 +451,13 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
}
// put extra key to trigger flush
ASSERT_OK(Put("", ""));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
if (j < kFilesPerLevel - 1) {
// background compaction may happen early for kFilesPerLevel'th file
ASSERT_EQ(NumTableFilesAtLevel(0), j + 1);
}
}
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
ASSERT_GT(NumTableFilesAtLevel(kNumLevels - 1 - i), kFilesPerLevel - 1);
}
@@ -483,17 +490,17 @@ TEST_F(DBRangeDelTest, CompactionRemovesCoveredMergeOperands) {
for (int i = 0; i <= kNumFiles * kNumPerFile; ++i) {
if (i % kNumPerFile == 0 && i / kNumPerFile == kNumFiles - 1) {
// Delete merge operands from all but the last file
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "key",
"key_");
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
"key", "key_"));
}
std::string val;
PutFixed64(&val, i);
db_->Merge(WriteOptions(), "key", val);
ASSERT_OK(db_->Merge(WriteOptions(), "key", val));
// we need to prevent trivial move using Puts so compaction will actually
// process the merge operands.
db_->Put(WriteOptions(), "prevent_trivial_move", "");
ASSERT_OK(db_->Put(WriteOptions(), "prevent_trivial_move", ""));
if (i > 0 && i % kNumPerFile == 0) {
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
}
@@ -504,7 +511,7 @@ TEST_F(DBRangeDelTest, CompactionRemovesCoveredMergeOperands) {
PutFixed64(&expected, 45); // 1+2+...+9
ASSERT_EQ(expected, actual);
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
expected.clear();
ASSERT_OK(db_->Get(read_opts, "key", &actual));
@@ -550,19 +557,19 @@ TEST_F(DBRangeDelTest, ObsoleteTombstoneCleanup) {
opts.statistics = CreateDBStatistics();
Reopen(opts);
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "dr1",
"dr10"); // obsolete after compaction
db_->Put(WriteOptions(), "key", "val");
db_->Flush(FlushOptions());
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "dr1",
"dr10")); // obsolete after compaction
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
ASSERT_OK(db_->Flush(FlushOptions()));
const Snapshot* snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "dr2",
"dr20"); // protected by snapshot
db_->Put(WriteOptions(), "key", "val");
db_->Flush(FlushOptions());
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "dr2",
"dr20")); // protected by snapshot
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
ASSERT_OK(db_->Flush(FlushOptions()));
ASSERT_EQ(2, NumTableFilesAtLevel(0));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(1, NumTableFilesAtLevel(1));
ASSERT_EQ(1, TestGetTickerCount(opts, COMPACTION_RANGE_DEL_DROP_OBSOLETE));
@@ -609,22 +616,24 @@ TEST_F(DBRangeDelTest, TableEvictedDuringScan) {
// to bottommost level (i.e., L1).
const Snapshot* snapshot = db_->GetSnapshot();
for (int i = 0; i < kNum; ++i) {
db_->Put(WriteOptions(), GetNumericStr(i), "val");
ASSERT_OK(db_->Put(WriteOptions(), GetNumericStr(i), "val"));
if (i > 0) {
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
}
if (i >= kNum / 2 && i < kNum / 2 + kNumRanges) {
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin), GetNumericStr(kRangeEnd));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin),
GetNumericStr(kRangeEnd)));
}
}
// Must be > 1 so the first L1 file can be closed before scan finishes
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GT(NumTableFilesAtLevel(1), 1);
std::vector<uint64_t> file_numbers = ListTableFiles(env_, dbname_);
ReadOptions read_opts;
auto* iter = db_->NewIterator(read_opts);
ASSERT_OK(iter->status());
int expected = kRangeEnd;
iter->SeekToFirst();
for (auto file_number : file_numbers) {
@@ -647,7 +656,7 @@ TEST_F(DBRangeDelTest, TableEvictedDuringScan) {
TEST_F(DBRangeDelTest, GetCoveredKeyFromMutableMemtable) {
do {
DestroyAndReopen(CurrentOptions());
db_->Put(WriteOptions(), "key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
@@ -669,10 +678,10 @@ TEST_F(DBRangeDelTest, GetCoveredKeyFromImmutableMemtable) {
opts.memtable_factory.reset(new SpecialSkipListFactory(1));
DestroyAndReopen(opts);
db_->Put(WriteOptions(), "key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
db_->Put(WriteOptions(), "blah", "val");
ASSERT_OK(db_->Put(WriteOptions(), "blah", "val"));
ReadOptions read_opts;
std::string value;
@@ -683,7 +692,7 @@ TEST_F(DBRangeDelTest, GetCoveredKeyFromImmutableMemtable) {
TEST_F(DBRangeDelTest, GetCoveredKeyFromSst) {
do {
DestroyAndReopen(CurrentOptions());
db_->Put(WriteOptions(), "key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
// snapshot prevents key from being deleted during flush
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(
@@ -706,11 +715,11 @@ TEST_F(DBRangeDelTest, GetCoveredMergeOperandFromMemtable) {
for (int i = 0; i < kNumMergeOps; ++i) {
std::string val;
PutFixed64(&val, i);
db_->Merge(WriteOptions(), "key", val);
ASSERT_OK(db_->Merge(WriteOptions(), "key", val));
if (i == kNumMergeOps / 2) {
// deletes [0, 5]
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "key",
"key_");
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
"key", "key_"));
}
}
@@ -734,16 +743,16 @@ TEST_F(DBRangeDelTest, GetIgnoresRangeDeletions) {
opts.memtable_factory.reset(new SpecialSkipListFactory(1));
Reopen(opts);
db_->Put(WriteOptions(), "sst_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "sst_key", "val"));
// snapshot prevents key from being deleted during flush
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
ASSERT_OK(db_->Flush(FlushOptions()));
db_->Put(WriteOptions(), "imm_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "imm_key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
db_->Put(WriteOptions(), "mem_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "mem_key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
@@ -768,13 +777,15 @@ TEST_F(DBRangeDelTest, IteratorRemovesCoveredKeys) {
// should be deleted.
for (int i = 0; i < kNum; ++i) {
if (i == kNum / 2) {
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin), GetNumericStr(kRangeEnd));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin),
GetNumericStr(kRangeEnd)));
}
db_->Put(WriteOptions(), GetNumericStr(i), "val");
ASSERT_OK(db_->Put(WriteOptions(), GetNumericStr(i), "val"));
}
ReadOptions read_opts;
auto* iter = db_->NewIterator(read_opts);
ASSERT_OK(iter->status());
int expected = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
@@ -802,14 +813,16 @@ TEST_F(DBRangeDelTest, IteratorOverUserSnapshot) {
for (int i = 0; i < kNum; ++i) {
if (i == kNum / 2) {
snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin), GetNumericStr(kRangeEnd));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
GetNumericStr(kRangeBegin),
GetNumericStr(kRangeEnd)));
}
db_->Put(WriteOptions(), GetNumericStr(i), "val");
ASSERT_OK(db_->Put(WriteOptions(), GetNumericStr(i), "val"));
}
ReadOptions read_opts;
read_opts.snapshot = snapshot;
auto* iter = db_->NewIterator(read_opts);
ASSERT_OK(iter->status());
int expected = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
@@ -828,22 +841,23 @@ TEST_F(DBRangeDelTest, IteratorIgnoresRangeDeletions) {
opts.memtable_factory.reset(new SpecialSkipListFactory(1));
Reopen(opts);
db_->Put(WriteOptions(), "sst_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "sst_key", "val"));
// snapshot prevents key from being deleted during flush
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
ASSERT_OK(db_->Flush(FlushOptions()));
db_->Put(WriteOptions(), "imm_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "imm_key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
db_->Put(WriteOptions(), "mem_key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "mem_key", "val"));
ASSERT_OK(
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), "a", "z"));
ReadOptions read_opts;
read_opts.ignore_range_deletions = true;
auto* iter = db_->NewIterator(read_opts);
ASSERT_OK(iter->status());
int i = 0;
std::string expected[] = {"imm_key", "mem_key", "sst_key"};
for (iter->SeekToFirst(); iter->Valid(); iter->Next(), ++i) {
@@ -857,7 +871,7 @@ TEST_F(DBRangeDelTest, IteratorIgnoresRangeDeletions) {
#ifndef ROCKSDB_UBSAN_RUN
TEST_F(DBRangeDelTest, TailingIteratorRangeTombstoneUnsupported) {
db_->Put(WriteOptions(), "key", "val");
ASSERT_OK(db_->Put(WriteOptions(), "key", "val"));
// snapshot prevents key from being deleted during flush
const Snapshot* snapshot = db_->GetSnapshot();
ASSERT_OK(
@@ -873,6 +887,7 @@ TEST_F(DBRangeDelTest, TailingIteratorRangeTombstoneUnsupported) {
iter->SeekToFirst();
}
ASSERT_TRUE(iter->status().IsNotSupported());
delete iter;
if (i == 0) {
ASSERT_OK(db_->Flush(FlushOptions()));
@@ -882,7 +897,6 @@ TEST_F(DBRangeDelTest, TailingIteratorRangeTombstoneUnsupported) {
}
db_->ReleaseSnapshot(snapshot);
}
#endif // !ROCKSDB_UBSAN_RUN
TEST_F(DBRangeDelTest, SubcompactionHasEmptyDedicatedRangeDelFile) {
@@ -926,8 +940,8 @@ TEST_F(DBRangeDelTest, SubcompactionHasEmptyDedicatedRangeDelFile) {
ASSERT_EQ(kNumFiles, NumTableFilesAtLevel(0));
ASSERT_EQ(1, NumTableFilesAtLevel(1));
db_->EnableAutoCompaction({db_->DefaultColumnFamily()});
dbfull()->TEST_WaitForCompact();
ASSERT_OK(db_->EnableAutoCompaction({db_->DefaultColumnFamily()}));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
db_->ReleaseSnapshot(snapshot);
}
@@ -949,7 +963,7 @@ TEST_F(DBRangeDelTest, MemtableBloomFilter) {
for (int i = 0; i < kNumKeys; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
Key(kNumKeys)));
for (int i = 0; i < kNumKeys; ++i) {
@@ -987,8 +1001,8 @@ TEST_F(DBRangeDelTest, CompactionTreatsSplitInputLevelDeletionAtomically) {
// snapshot protects range tombstone from dropping due to becoming obsolete.
const Snapshot* snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
Key(2 * kNumFilesPerLevel));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(0), Key(2 * kNumFilesPerLevel)));
Random rnd(301);
std::string value = rnd.RandomString(kValueBytes);
@@ -997,14 +1011,14 @@ TEST_F(DBRangeDelTest, CompactionTreatsSplitInputLevelDeletionAtomically) {
ASSERT_OK(Put(Key(j), value));
ASSERT_OK(Put(Key(2 * kNumFilesPerLevel - 1 - j), value));
if (j > 0) {
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(j, NumTableFilesAtLevel(0));
}
}
// put extra key to trigger final flush
ASSERT_OK(Put("", ""));
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_EQ(kNumFilesPerLevel, NumTableFilesAtLevel(1));
@@ -1022,7 +1036,7 @@ TEST_F(DBRangeDelTest, CompactionTreatsSplitInputLevelDeletionAtomically) {
} else if (i == 2) {
ASSERT_OK(db_->SetOptions(db_->DefaultColumnFamily(),
{{"max_bytes_for_level_base", "10000"}}));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(1, NumTableFilesAtLevel(1));
}
ASSERT_GT(NumTableFilesAtLevel(2), 0);
@@ -1056,8 +1070,8 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
// A snapshot protects the range tombstone from dropping due to
// becoming obsolete.
const Snapshot* snapshot = db_->GetSnapshot();
db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
Key(0), Key(2 * kNumFilesPerLevel));
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
Key(2 * kNumFilesPerLevel)));
// Create 2 additional sstables in L0. Note that the first sstable
// contains the range tombstone.
@@ -1096,7 +1110,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
ASSERT_EQ(value, Get(Key(2)));
auto begin_str = Key(3);
const ROCKSDB_NAMESPACE::Slice begin = begin_str;
dbfull()->TEST_CompactRange(1, &begin, nullptr);
ASSERT_OK(dbfull()->TEST_CompactRange(1, &begin, nullptr));
ASSERT_EQ(1, NumTableFilesAtLevel(1));
ASSERT_EQ(2, NumTableFilesAtLevel(2));
ASSERT_EQ(value, Get(Key(2)));
@@ -1115,7 +1129,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
// [key000002#6,1, key000004#72057594037927935,15]
auto begin_str = Key(0);
const ROCKSDB_NAMESPACE::Slice begin = begin_str;
dbfull()->TEST_CompactRange(1, &begin, &begin);
ASSERT_OK(dbfull()->TEST_CompactRange(1, &begin, &begin));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
ASSERT_EQ(3, NumTableFilesAtLevel(2));
}
@@ -1216,9 +1230,9 @@ TEST_F(DBRangeDelTest, KeyAtOverlappingEndpointReappears) {
std::string value;
ASSERT_TRUE(db_->Get(ReadOptions(), "key", &value).IsNotFound());
dbfull()->TEST_CompactRange(0 /* level */, nullptr /* begin */,
nullptr /* end */, nullptr /* column_family */,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(
0 /* level */, nullptr /* begin */, nullptr /* end */,
nullptr /* column_family */, true /* disallow_trivial_move */));
ASSERT_EQ(0, NumTableFilesAtLevel(0));
// Now we have multiple files at L1 all containing a single user key, thus
// guaranteeing overlap in the file endpoints.
@@ -1229,9 +1243,9 @@ TEST_F(DBRangeDelTest, KeyAtOverlappingEndpointReappears) {
// Compact and verify again. It's worthwhile because now the files have
// tighter endpoints, so we can verify that doesn't mess anything up.
dbfull()->TEST_CompactRange(1 /* level */, nullptr /* begin */,
nullptr /* end */, nullptr /* column_family */,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(
1 /* level */, nullptr /* begin */, nullptr /* end */,
nullptr /* column_family */, true /* disallow_trivial_move */));
ASSERT_GT(NumTableFilesAtLevel(2), 1);
ASSERT_TRUE(db_->Get(ReadOptions(), "key", &value).IsNotFound());
@@ -1307,6 +1321,7 @@ TEST_F(DBRangeDelTest, UntruncatedTombstoneDoesNotDeleteNewerKey) {
auto get_key_count = [this]() -> int {
auto* iter = db_->NewIterator(ReadOptions());
assert(iter->status().ok());
iter->SeekToFirst();
int keys_found = 0;
for (; iter->Valid(); iter->Next()) {
@@ -1409,6 +1424,7 @@ TEST_F(DBRangeDelTest, DeletedMergeOperandReappearsIterPrev) {
ASSERT_GT(NumTableFilesAtLevel(1), 1);
auto* iter = db_->NewIterator(ReadOptions());
ASSERT_OK(iter->status());
iter->SeekToLast();
int keys_found = 0;
for (; iter->Valid(); iter->Prev()) {
@@ -1435,11 +1451,12 @@ TEST_F(DBRangeDelTest, SnapshotPreventsDroppedKeys) {
ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
Key(10)));
db_->Flush(FlushOptions());
ASSERT_OK(db_->Flush(FlushOptions()));
ReadOptions read_opts;
read_opts.snapshot = snapshot;
auto* iter = db_->NewIterator(read_opts);
ASSERT_OK(iter->status());
iter->SeekToFirst();
ASSERT_TRUE(iter->Valid());
@@ -1482,6 +1499,7 @@ TEST_F(DBRangeDelTest, SnapshotPreventsDroppedKeysInImmMemTables) {
ReadOptions read_opts;
read_opts.snapshot = snapshot.get();
std::unique_ptr<Iterator> iter(db_->NewIterator(read_opts));
ASSERT_OK(iter->status());
TEST_SYNC_POINT("SnapshotPreventsDroppedKeysInImmMemTables:AfterNewIterator");
@@ -1519,7 +1537,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneWrittenToMinimalSsts) {
std::string value = rnd.RandomString(kValueBytes);
ASSERT_OK(Put(key, value));
}
db_->Flush(FlushOptions());
ASSERT_OK(db_->Flush(FlushOptions()));
MoveFilesToLevel(2);
}
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -1538,7 +1556,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneWrittenToMinimalSsts) {
// TODO(ajkr): remove this `Put` after file cutting accounts for range
// tombstones (#3977).
ASSERT_OK(Put("c" + Key(1), "value"));
db_->Flush(FlushOptions());
ASSERT_OK(db_->Flush(FlushOptions()));
// Ensure manual L0->L1 compaction cuts the outputs before the range tombstone
// and the range tombstone is only placed in the second SST.
@@ -1546,9 +1564,9 @@ TEST_F(DBRangeDelTest, RangeTombstoneWrittenToMinimalSsts) {
Slice begin_key(begin_key_storage);
std::string end_key_storage("d");
Slice end_key(end_key_storage);
dbfull()->TEST_CompactRange(0 /* level */, &begin_key /* begin */,
&end_key /* end */, nullptr /* column_family */,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(
0 /* level */, &begin_key /* begin */, &end_key /* end */,
nullptr /* column_family */, true /* disallow_trivial_move */));
ASSERT_EQ(2, NumTableFilesAtLevel(1));
std::vector<LiveFileMetaData> all_metadata;
@@ -1613,15 +1631,15 @@ TEST_F(DBRangeDelTest, OverlappedTombstones) {
ASSERT_EQ(1, NumTableFilesAtLevel(0));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
// The tombstone range is not broken up into multiple SSTs which may incur a
// large compaction with L2.
ASSERT_EQ(1, NumTableFilesAtLevel(1));
std::vector<std::vector<FileMetaData>> files;
dbfull()->TEST_CompactRange(1, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(1, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(1, NumTableFilesAtLevel(2));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
}
@@ -1654,13 +1672,13 @@ TEST_F(DBRangeDelTest, OverlappedKeys) {
// The key range is broken up into three SSTs to avoid a future big compaction
// with the grandparent
dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(3, NumTableFilesAtLevel(1));
std::vector<std::vector<FileMetaData>> files;
dbfull()->TEST_CompactRange(1, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */);
ASSERT_OK(dbfull()->TEST_CompactRange(1, nullptr, nullptr, nullptr,
true /* disallow_trivial_move */));
ASSERT_EQ(1, NumTableFilesAtLevel(2));
ASSERT_EQ(0, NumTableFilesAtLevel(1));
}
+14 -9
View File
@@ -170,7 +170,7 @@ TEST_F(DBSSTTest, DontDeleteMovedFile) {
ASSERT_OK(Flush());
}
// this should execute both L0->L1 and L1->(move)->L2 compactions
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ("0,0,1", FilesPerLevel(0));
// If the moved file is actually deleted (the move-safeguard in
@@ -218,7 +218,7 @@ TEST_F(DBSSTTest, DeleteObsoleteFilesPendingOutputs) {
ASSERT_OK(Flush());
}
// this should execute both L0->L1 and L1->(move)->L2 compactions
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ("0,0,1", FilesPerLevel(0));
test::SleepingBackgroundTask blocking_thread;
@@ -264,9 +264,9 @@ TEST_F(DBSSTTest, DeleteObsoleteFilesPendingOutputs) {
// finish the flush!
blocking_thread.WakeUp();
blocking_thread.WaitUntilDone();
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
// File just flushed is too big for L0 and L1 so gets moved to L2.
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ("0,0,1,0,1", FilesPerLevel(0));
metadata.clear();
@@ -302,8 +302,8 @@ TEST_F(DBSSTTest, DBWithSstFileManager) {
for (int i = 0; i < 25; i++) {
GenerateNewRandomFile(&rnd);
ASSERT_OK(Flush());
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
// Verify that we are tracking all sst files in dbname_
std::unordered_map<std::string, uint64_t> files_in_db;
ASSERT_OK(GetAllSSTFiles(&files_in_db));
@@ -544,6 +544,11 @@ TEST_P(DBWALTestWithParam, WALTrashCleanupOnOpen) {
// Create 4 files in L0
for (char v = 'a'; v <= 'd'; v++) {
if (v == 'd') {
// Maximize the change that the last log file will be preserved in trash
// before restarting the DB.
options.sst_file_manager->SetDeleteRateBytesPerSecond(1);
}
ASSERT_OK(Put("Key2", DummyString(1024, v)));
ASSERT_OK(Put("Key3", DummyString(1024, v)));
ASSERT_OK(Put("Key4", DummyString(1024, v)));
@@ -603,7 +608,7 @@ TEST_F(DBSSTTest, OpenDBWithExistingTrash) {
Destroy(last_options_);
// Add some trash files to the db directory so the DB can clean them up
env_->CreateDirIfMissing(dbname_);
ASSERT_OK(env_->CreateDirIfMissing(dbname_));
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "001.sst.trash"));
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "002.sst.trash"));
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "003.sst.trash"));
@@ -728,7 +733,7 @@ TEST_F(DBSSTTest, DestroyDBWithRateLimitedDelete) {
int num_sst_files = 0;
int num_wal_files = 0;
std::vector<std::string> db_files;
env_->GetChildren(dbname_, &db_files);
ASSERT_OK(env_->GetChildren(dbname_, &db_files));
for (std::string f : db_files) {
if (f.substr(f.find_last_of(".") + 1) == "sst") {
num_sst_files++;
@@ -817,7 +822,7 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
ASSERT_OK(Put(Key(i), rnd.RandomString(50)));
}
ASSERT_OK(Flush());
dbfull()->TEST_WaitForCompact(true);
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
// Because we set a callback in CancelledCompaction, we actually
// let the compaction run
+1 -1
View File
@@ -137,7 +137,7 @@ TEST_F(DBStatisticsTest, ResetStats) {
ASSERT_EQ(1, TestGetTickerCount(options, NUMBER_KEYS_WRITTEN));
options.statistics->histogramData(DB_WRITE, &histogram_data);
ASSERT_GT(histogram_data.max, 0.0);
options.statistics->Reset();
ASSERT_OK(options.statistics->Reset());
}
}
}
+32 -32
View File
@@ -65,9 +65,9 @@ TEST_F(DBTablePropertiesTest, GetPropertiesOfAllTablesTest) {
// Create 4 tables
for (int table = 0; table < 4; ++table) {
for (int i = 0; i < 10 + table; ++i) {
db_->Put(WriteOptions(), ToString(table * 100 + i), "val");
ASSERT_OK(db_->Put(WriteOptions(), ToString(table * 100 + i), "val"));
}
db_->Flush(FlushOptions());
ASSERT_OK(db_->Flush(FlushOptions()));
}
// 1. Read table properties directly from file
@@ -161,14 +161,14 @@ TEST_F(DBTablePropertiesTest, GetPropertiesOfTablesInRange) {
for (int i = 0; i < 10000; i++) {
ASSERT_OK(Put(test::RandomKey(&rnd, 5), rnd.RandomString(102)));
}
Flush();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
if (NumTableFilesAtLevel(0) == 0) {
ASSERT_OK(Put(test::RandomKey(&rnd, 5), rnd.RandomString(102)));
Flush();
ASSERT_OK(Flush());
}
db_->PauseBackgroundWork();
ASSERT_OK(db_->PauseBackgroundWork());
// Ensure that we have at least L0, L1 and L2
ASSERT_GT(NumTableFilesAtLevel(0), 0);
@@ -236,8 +236,8 @@ TEST_F(DBTablePropertiesTest, GetColumnFamilyNameProperty) {
// Create one table per CF, then verify it was created with the column family
// name property.
for (uint32_t cf = 0; cf < 2; ++cf) {
Put(cf, "key", "val");
Flush(cf);
ASSERT_OK(Put(cf, "key", "val"));
ASSERT_OK(Flush(cf));
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
@@ -260,17 +260,17 @@ TEST_F(DBTablePropertiesTest, GetDbIdentifiersProperty) {
CreateAndReopenWithCF({"goku"}, CurrentOptions());
for (uint32_t cf = 0; cf < 2; ++cf) {
Put(cf, "key", "val");
Put(cf, "foo", "bar");
Flush(cf);
ASSERT_OK(Put(cf, "key", "val"));
ASSERT_OK(Put(cf, "foo", "bar"));
ASSERT_OK(Flush(cf));
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
ASSERT_EQ(1U, fname_to_props.size());
std::string id, sid;
db_->GetDbIdentity(id);
db_->GetDbSessionId(sid);
ASSERT_OK(db_->GetDbIdentity(id));
ASSERT_OK(db_->GetDbSessionId(sid));
ASSERT_EQ(id, fname_to_props.begin()->second->db_id);
ASSERT_EQ(sid, fname_to_props.begin()->second->db_session_id);
}
@@ -298,9 +298,9 @@ TEST_P(DBTableHostnamePropertyTest, DbHostLocationProperty) {
CreateAndReopenWithCF({"goku"}, opts);
for (uint32_t cf = 0; cf < 2; ++cf) {
Put(cf, "key", "val");
Put(cf, "foo", "bar");
Flush(cf);
ASSERT_OK(Put(cf, "key", "val"));
ASSERT_OK(Put(cf, "foo", "bar"));
ASSERT_OK(Flush(cf));
TablePropertiesCollection fname_to_props;
ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
@@ -356,8 +356,8 @@ TEST_P(DBTablePropertiesTest, DeletionTriggeredCompactionMarking) {
// add an L1 file to prevent tombstones from dropping due to obsolescence
// during flush
Put(Key(0), "val");
Flush();
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
MoveFilesToLevel(1);
DeletionTriggeredCompactionTestListener *listener =
@@ -368,14 +368,14 @@ TEST_P(DBTablePropertiesTest, DeletionTriggeredCompactionMarking) {
for (int i = 0; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
Delete(Key(i));
ASSERT_OK(Delete(Key(i)));
} else {
Put(Key(i), "val");
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush());
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
// Change the window size and deletion trigger and ensure new values take
@@ -389,14 +389,14 @@ TEST_P(DBTablePropertiesTest, DeletionTriggeredCompactionMarking) {
for (int i = 0; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
Delete(Key(i));
ASSERT_OK(Delete(Key(i)));
} else {
Put(Key(i), "val");
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush());
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
// Change the window size to disable delete triggered compaction
@@ -408,14 +408,14 @@ TEST_P(DBTablePropertiesTest, DeletionTriggeredCompactionMarking) {
for (int i = 0; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
i < kNumKeys - kWindowSize + kNumDelsTrigger) {
Delete(Key(i));
ASSERT_OK(Delete(Key(i)));
} else {
Put(Key(i), "val");
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush());
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(1, NumTableFilesAtLevel(0));
ASSERT_LT(0, opts.statistics->getTickerCount(COMPACT_WRITE_BYTES_MARKED));
ASSERT_LT(0, opts.statistics->getTickerCount(COMPACT_READ_BYTES_MARKED));
@@ -438,8 +438,8 @@ TEST_P(DBTablePropertiesTest, RatioBasedDeletionTriggeredCompactionMarking) {
// Add an L2 file to prevent tombstones from dropping due to obsolescence
// during flush
Put(Key(0), "val");
Flush();
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Flush());
MoveFilesToLevel(2);
auto* listener = new DeletionTriggeredCompactionTestListener();
+3 -2
View File
@@ -179,7 +179,7 @@ TEST_F(DBTestTailingIterator, TailingIteratorTrimSeekToNext) {
if (i % 100 == 99) {
ASSERT_OK(Flush(1));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
if (i == 299) {
file_iters_deleted = true;
}
@@ -411,7 +411,7 @@ TEST_F(DBTestTailingIterator, TailingIteratorUpperBound) {
it->Next();
// Not valid since "21" is over the upper bound.
ASSERT_FALSE(it->Valid());
ASSERT_OK(it->status());
// This keeps track of the number of times NeedToSeekImmutable() was true.
int immutable_seeks = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
@@ -424,6 +424,7 @@ TEST_F(DBTestTailingIterator, TailingIteratorUpperBound) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_FALSE(it->Valid());
ASSERT_OK(it->status());
ASSERT_EQ(0, immutable_seeks);
}
-45
View File
@@ -1305,51 +1305,6 @@ TEST_F(DBTest, DISABLED_RepeatedWritesToSameKey) {
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, SparseMerge) {
do {
Options options = CurrentOptions();
options.compression = kNoCompression;
CreateAndReopenWithCF({"pikachu"}, options);
FillLevels("A", "Z", 1);
// Suppose there is:
// small amount of data with prefix A
// large amount of data with prefix B
// small amount of data with prefix C
// and that recent updates have made small changes to all three prefixes.
// Check that we do not do a compaction that merges all of B in one shot.
const std::string value(1000, 'x');
Put(1, "A", "va");
// Write approximately 100MB of "B" values
for (int i = 0; i < 100000; i++) {
char key[100];
snprintf(key, sizeof(key), "B%010d", i);
Put(1, key, value);
}
Put(1, "C", "vc");
ASSERT_OK(Flush(1));
dbfull()->TEST_CompactRange(0, nullptr, nullptr, handles_[1]);
// Make sparse update
Put(1, "A", "va2");
Put(1, "B100", "bvalue2");
Put(1, "C", "vc2");
ASSERT_OK(Flush(1));
// Compactions should not cause us to create a situation where
// a file overlaps too much data at the next level.
ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(handles_[1]),
20 * 1048576);
dbfull()->TEST_CompactRange(0, nullptr, nullptr);
ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(handles_[1]),
20 * 1048576);
dbfull()->TEST_CompactRange(1, nullptr, nullptr);
ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(handles_[1]),
20 * 1048576);
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
static bool Between(uint64_t val, uint64_t low, uint64_t high) {
bool result = (val >= low) && (val <= high);
+8 -7
View File
@@ -236,7 +236,7 @@ bool DBTestBase::ChangeWalOptions() {
auto options = CurrentOptions();
Destroy(options);
options.create_if_missing = true;
TryReopen(options);
Reopen(options);
return true;
} else if (option_config_ == kDBLogDir) {
option_config_ = kWalDirAndMmapReads;
@@ -244,14 +244,14 @@ bool DBTestBase::ChangeWalOptions() {
auto options = CurrentOptions();
Destroy(options);
options.create_if_missing = true;
TryReopen(options);
Reopen(options);
return true;
} else if (option_config_ == kWalDirAndMmapReads) {
option_config_ = kRecycleLogFiles;
Destroy(last_options_);
auto options = CurrentOptions();
Destroy(options);
TryReopen(options);
Reopen(options);
return true;
} else {
return false;
@@ -1129,11 +1129,12 @@ std::string DBTestBase::FilesPerLevel(int cf) {
size_t DBTestBase::CountFiles() {
std::vector<std::string> files;
env_->GetChildren(dbname_, &files);
EXPECT_OK(env_->GetChildren(dbname_, &files));
std::vector<std::string> logfiles;
if (dbname_ != last_options_.wal_dir) {
env_->GetChildren(last_options_.wal_dir, &logfiles);
Status s = env_->GetChildren(last_options_.wal_dir, &logfiles);
EXPECT_TRUE(s.ok() || s.IsNotFound());
}
return files.size() + logfiles.size();
@@ -1266,8 +1267,8 @@ void DBTestBase::GenerateNewRandomFile(Random* rnd, bool nowait) {
}
ASSERT_OK(Put("key" + rnd->RandomString(7), rnd->RandomString(200)));
if (!nowait) {
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
}
+25 -25
View File
@@ -996,7 +996,7 @@ TEST_P(DBTestUniversalCompaction, UniversalCompactionStopStyleSimilarSize) {
// First, clean up memtable before inserting new data. This will generate
// a level-0 file, with size around 0.4 (according to previously written
// data amount).
dbfull()->Flush(FlushOptions());
ASSERT_OK(dbfull()->Flush(FlushOptions()));
for (int num = 0; num < options.level0_file_num_compaction_trigger - 3;
num++) {
// Write 110KB (11 values, each 10K)
@@ -1781,7 +1781,7 @@ TEST_P(DBTestUniversalCompaction, FinalSortedRunCompactFilesConflict) {
dbfull()->TEST_write_controler().GetCompactionPressureToken();
ASSERT_OK(Put("key", "val"));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_EQ(NumTableFilesAtLevel(num_levels_ - 1), 1);
ColumnFamilyMetaData cf_meta;
@@ -1807,7 +1807,7 @@ TEST_P(DBTestUniversalCompaction, FinalSortedRunCompactFilesConflict) {
"DBTestUniversalCompaction:FinalSortedRunCompactFilesConflict:0");
for (int i = 0; i < 2; ++i) {
ASSERT_OK(Put("key", "val"));
Flush();
ASSERT_OK(Flush());
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
@@ -1911,7 +1911,7 @@ TEST_F(DBTestUniversalCompaction2, BasicL0toL1) {
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
// MoveFilesToLevel(6);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
@@ -1923,7 +1923,7 @@ TEST_F(DBTestUniversalCompaction2, BasicL0toL1) {
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -1954,7 +1954,7 @@ TEST_F(DBTestUniversalCompaction2, SingleLevel) {
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 1999; i < kNumKeys; ++i) {
if (i >= kNumKeys - kWindowSize &&
@@ -1964,7 +1964,7 @@ TEST_F(DBTestUniversalCompaction2, SingleLevel) {
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush()(;
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(1, NumTableFilesAtLevel(0));
@@ -1992,19 +1992,19 @@ TEST_F(DBTestUniversalCompaction2, MultipleLevels) {
for (i = 0; i < 500; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 500; i < 1000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 1000; i < 1500; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 1500; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -2013,15 +2013,15 @@ TEST_F(DBTestUniversalCompaction2, MultipleLevels) {
for (i = 1999; i < 2333; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 2333; i < 2666; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 2666; i < 2999; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -2031,7 +2031,7 @@ TEST_F(DBTestUniversalCompaction2, MultipleLevels) {
for (i = 1900; i < 2100; ++i) {
ASSERT_OK(Delete(Key(i)));
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -2064,19 +2064,19 @@ TEST_F(DBTestUniversalCompaction2, OverlappingL0) {
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 2000; i < 3000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 3500; i < 4000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
for (i = 2900; i < 3100; ++i) {
ASSERT_OK(Delete(Key(i)));
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(2, NumTableFilesAtLevel(0));
@@ -2106,7 +2106,7 @@ TEST_F(DBTestUniversalCompaction2, IngestBehind) {
for (i = 0; i < 2000; ++i) {
ASSERT_OK(Put(Key(i), "val"));
}
Flush();
ASSERT_OK(Flush());
// MoveFilesToLevel(6);
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
@@ -2118,7 +2118,7 @@ TEST_F(DBTestUniversalCompaction2, IngestBehind) {
ASSERT_OK(Put(Key(i), "val"));
}
}
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, NumTableFilesAtLevel(0));
@@ -2184,7 +2184,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
// Case 1: Oldest flushed file excceeds periodic compaction threshold.
ASSERT_OK(Put("foo", "bar"));
Flush();
ASSERT_OK(Flush());
ASSERT_EQ(0, periodic_compactions);
// Move clock forward so that the flushed file would qualify periodic
// compaction.
@@ -2192,7 +2192,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
// Another flush would trigger compaction the oldest file.
ASSERT_OK(Put("foo", "bar2"));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(1, periodic_compactions);
@@ -2203,7 +2203,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
periodic_compactions = 0;
// A flush doesn't trigger a periodic compaction when threshold not hit
ASSERT_OK(Put("foo", "bar2"));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(0, periodic_compactions);
@@ -2211,7 +2211,7 @@ TEST_F(DBTestUniversalCompaction2, PeriodicCompaction) {
// a compaction
ASSERT_OK(Put("foo", "bar2"));
env_->MockSleepForSeconds(48 * 60 * 60 + 100);
Flush();
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(1, periodic_compactions);
ASSERT_EQ(0, start_level);
+37 -27
View File
@@ -60,14 +60,15 @@ TEST_P(DBWriteTest, WriteStallRemoveNoSlowdownWrite) {
std::string key = "foo" + std::to_string(a);
WriteOptions wo;
wo.no_slowdown = false;
dbfull()->Put(wo, key, "bar");
ASSERT_OK(dbfull()->Put(wo, key, "bar"));
};
std::function<void()> write_no_slowdown_func = [&]() {
int a = thread_num.fetch_add(1);
std::string key = "foo" + std::to_string(a);
WriteOptions wo;
wo.no_slowdown = true;
dbfull()->Put(wo, key, "bar");
Status s = dbfull()->Put(wo, key, "bar");
ASSERT_TRUE(s.ok() || s.IsIncomplete());
};
std::function<void(void*)> unblock_main_thread_func = [&](void*) {
mutex.Lock();
@@ -77,13 +78,13 @@ TEST_P(DBWriteTest, WriteStallRemoveNoSlowdownWrite) {
};
// Create 3 L0 files and schedule 4th without waiting
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WriteThread::JoinBatchGroup:Start", unblock_main_thread_func);
@@ -104,7 +105,7 @@ TEST_P(DBWriteTest, WriteStallRemoveNoSlowdownWrite) {
// write_thread
FlushOptions fopt;
fopt.wait = false;
dbfull()->Flush(fopt);
ASSERT_OK(dbfull()->Flush(fopt));
// Create a mix of slowdown/no_slowdown write threads
mutex.Lock();
@@ -145,7 +146,7 @@ TEST_P(DBWriteTest, WriteStallRemoveNoSlowdownWrite) {
mutex.Unlock();
TEST_SYNC_POINT("DBWriteTest::WriteStallRemoveNoSlowdownWrite:1");
dbfull()->TEST_WaitForFlushMemTable(nullptr);
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(nullptr));
// This would have triggered a write stall. Unblock the write group leader
TEST_SYNC_POINT("DBWriteTest::WriteStallRemoveNoSlowdownWrite:2");
// The leader is going to create missing newer links. When the leader
@@ -178,14 +179,15 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
std::string key = "foo" + std::to_string(a);
WriteOptions wo;
wo.no_slowdown = false;
dbfull()->Put(wo, key, "bar");
ASSERT_OK(dbfull()->Put(wo, key, "bar"));
};
std::function<void()> write_no_slowdown_func = [&]() {
int a = thread_num.fetch_add(1);
std::string key = "foo" + std::to_string(a);
WriteOptions wo;
wo.no_slowdown = true;
dbfull()->Put(wo, key, "bar");
Status s = dbfull()->Put(wo, key, "bar");
ASSERT_TRUE(s.ok() || s.IsIncomplete());
};
std::function<void(void *)> unblock_main_thread_func = [&](void *) {
mutex.Lock();
@@ -195,13 +197,13 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
};
// Create 3 L0 files and schedule 4th without waiting
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
Flush();
Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar");
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo" + std::to_string(thread_num.fetch_add(1)), "bar"));
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WriteThread::JoinBatchGroup:Start", unblock_main_thread_func);
@@ -222,7 +224,7 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
// write_thread
FlushOptions fopt;
fopt.wait = false;
dbfull()->Flush(fopt);
ASSERT_OK(dbfull()->Flush(fopt));
// Create a mix of slowdown/no_slowdown write threads
mutex.Lock();
@@ -243,7 +245,7 @@ TEST_P(DBWriteTest, WriteThreadHangOnWriteStall) {
mutex.Unlock();
TEST_SYNC_POINT("DBWriteTest::WriteThreadHangOnWriteStall:1");
dbfull()->TEST_WaitForFlushMemTable(nullptr);
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(nullptr));
// This would have triggered a write stall. Unblock the write group leader
TEST_SYNC_POINT("DBWriteTest::WriteThreadHangOnWriteStall:2");
// The leader is going to create missing newer links. When the leader finishes,
@@ -307,6 +309,11 @@ TEST_P(DBWriteTest, IOErrorOnWALWritePropagateToWriteThreadFollower) {
threads[i].join();
}
ASSERT_EQ(1, leader_count);
// The Failed PUT operations can cause a BG error to be set.
// Mark it as Checked for the ASSERT_STATUS_CHECKED
dbfull()->Resume().PermitUncheckedError();
// Close before mock_env destruct.
Close();
}
@@ -351,7 +358,9 @@ TEST_P(DBWriteTest, IOErrorOnWALWriteTriggersReadOnlyMode) {
}
*/
if (!options.manual_wal_flush) {
ASSERT_FALSE(res.ok());
ASSERT_NOK(res);
} else {
ASSERT_OK(res);
}
}
// Close before mock_env destruct.
@@ -423,13 +432,14 @@ TEST_P(DBWriteTest, ConcurrentlyDisabledWAL) {
ROCKSDB_NAMESPACE::WriteOptions write_option_default;
std::string no_wal_key = no_wal_key_prefix + std::to_string(t) +
"_" + std::to_string(i);
this->Put(no_wal_key, no_wal_value, write_option_disable);
ASSERT_OK(
this->Put(no_wal_key, no_wal_value, write_option_disable));
std::string wal_key =
wal_key_prefix + std::to_string(i) + "_" + std::to_string(i);
this->Put(wal_key, wal_value, write_option_default);
dbfull()->SyncWAL();
ASSERT_OK(this->Put(wal_key, wal_value, write_option_default));
ASSERT_OK(dbfull()->SyncWAL());
}
return 0;
return;
});
}
for (auto& t: threads) {
+25 -16
View File
@@ -40,7 +40,7 @@ class DeleteFileTest : public DBTestBase {
wal_dir_(dbname_ + "/wal_files") {}
void SetOptions(Options* options) {
assert(options);
ASSERT_NE(options, nullptr);
options->delete_obsolete_files_period_micros = 0; // always do full purge
options->enable_thread_tracking = true;
options->write_buffer_size = 1024 * 1024 * 1000;
@@ -105,7 +105,7 @@ class DeleteFileTest : public DBTestBase {
void CheckFileTypeCounts(const std::string& dir, int required_log,
int required_sst, int required_manifest) {
std::vector<std::string> filenames;
env_->GetChildren(dir, &filenames);
ASSERT_OK(env_->GetChildren(dir, &filenames));
int log_cnt = 0, sst_cnt = 0, manifest_cnt = 0;
for (auto file : filenames) {
@@ -180,7 +180,8 @@ TEST_F(DeleteFileTest, AddKeysAndQueryLevels) {
ASSERT_TRUE(status.IsInvalidArgument());
// Lowest level file deletion should succeed.
ASSERT_OK(db_->DeleteFile(level2file));
status = db_->DeleteFile(level2file);
ASSERT_OK(status);
}
TEST_F(DeleteFileTest, PurgeObsoleteFilesTest) {
@@ -201,7 +202,7 @@ TEST_F(DeleteFileTest, PurgeObsoleteFilesTest) {
compact_options.change_level = true;
compact_options.target_level = 2;
Slice first_slice(first), last_slice(last);
db_->CompactRange(compact_options, &first_slice, &last_slice);
ASSERT_OK(db_->CompactRange(compact_options, &first_slice, &last_slice));
// 1 sst after compaction
CheckFileTypeCounts(dbname_, 0, 1, 1);
@@ -210,7 +211,9 @@ TEST_F(DeleteFileTest, PurgeObsoleteFilesTest) {
Iterator *itr = nullptr;
CreateTwoLevels();
itr = db_->NewIterator(ReadOptions());
db_->CompactRange(compact_options, &first_slice, &last_slice);
ASSERT_OK(itr->status());
ASSERT_OK(db_->CompactRange(compact_options, &first_slice, &last_slice));
ASSERT_OK(itr->status());
// 3 sst after compaction with live iterator
CheckFileTypeCounts(dbname_, 0, 3, 1);
delete itr;
@@ -237,7 +240,8 @@ TEST_F(DeleteFileTest, BackgroundPurgeIteratorTest) {
ReadOptions read_options;
read_options.background_purge_on_iterator_cleanup = true;
itr = db_->NewIterator(read_options);
db_->CompactRange(compact_options, &first_slice, &last_slice);
ASSERT_OK(itr->status());
ASSERT_OK(db_->CompactRange(compact_options, &first_slice, &last_slice));
// 3 sst after compaction with live iterator
CheckFileTypeCounts(dbname_, 0, 3, 1);
test::SleepingBackgroundTask sleeping_task_before;
@@ -344,11 +348,12 @@ TEST_F(DeleteFileTest, BackgroundPurgeCopyOptions) {
ReadOptions read_options;
read_options.background_purge_on_iterator_cleanup = true;
itr = db_->NewIterator(read_options);
ASSERT_OK(itr->status());
// ReadOptions is deleted, but iterator cleanup function should not be
// affected
}
db_->CompactRange(compact_options, &first_slice, &last_slice);
ASSERT_OK(db_->CompactRange(compact_options, &first_slice, &last_slice));
// 3 sst after compaction with live iterator
CheckFileTypeCounts(dbname_, 0, 3, 1);
delete itr;
@@ -382,9 +387,11 @@ TEST_F(DeleteFileTest, BackgroundPurgeTestMultipleJobs) {
ReadOptions read_options;
read_options.background_purge_on_iterator_cleanup = true;
Iterator* itr1 = db_->NewIterator(read_options);
ASSERT_OK(itr1->status());
CreateTwoLevels();
Iterator* itr2 = db_->NewIterator(read_options);
db_->CompactRange(compact_options, &first_slice, &last_slice);
ASSERT_OK(itr2->status());
ASSERT_OK(db_->CompactRange(compact_options, &first_slice, &last_slice));
// 5 sst files after 2 compactions with 2 live iterators
CheckFileTypeCounts(dbname_, 0, 5, 1);
@@ -417,6 +424,7 @@ TEST_F(DeleteFileTest, DeleteFileWithIterator) {
CreateTwoLevels();
ReadOptions read_options;
Iterator* it = db_->NewIterator(read_options);
ASSERT_OK(it->status());
std::vector<LiveFileMetaData> metadata;
db_->GetLiveFilesMetaData(&metadata);
@@ -432,7 +440,7 @@ TEST_F(DeleteFileTest, DeleteFileWithIterator) {
Status status = db_->DeleteFile(level2file);
fprintf(stdout, "Deletion status %s: %s\n",
level2file.c_str(), status.ToString().c_str());
ASSERT_TRUE(status.ok());
ASSERT_OK(status);
it->SeekToFirst();
int numKeysIterated = 0;
while(it->Valid()) {
@@ -452,7 +460,7 @@ TEST_F(DeleteFileTest, DeleteLogFiles) {
AddKeys(10, 0);
VectorLogPtr logfiles;
db_->GetSortedWalFiles(logfiles);
ASSERT_OK(db_->GetSortedWalFiles(logfiles));
ASSERT_GT(logfiles.size(), 0UL);
// Take the last log file which is expected to be alive and try to delete it
// Should not succeed because live logs are not allowed to be deleted
@@ -461,7 +469,7 @@ TEST_F(DeleteFileTest, DeleteLogFiles) {
ASSERT_OK(env_->FileExists(wal_dir_ + "/" + alive_log->PathName()));
fprintf(stdout, "Deleting alive log file %s\n",
alive_log->PathName().c_str());
ASSERT_TRUE(!db_->DeleteFile(alive_log->PathName()).ok());
ASSERT_NOK(db_->DeleteFile(alive_log->PathName()));
ASSERT_OK(env_->FileExists(wal_dir_ + "/" + alive_log->PathName()));
logfiles.clear();
@@ -469,10 +477,10 @@ TEST_F(DeleteFileTest, DeleteLogFiles) {
// Call Flush again to flush out memtable and move alive log to archived log
// and try to delete the archived log file
FlushOptions fopts;
db_->Flush(fopts);
ASSERT_OK(db_->Flush(fopts));
AddKeys(10, 0);
db_->Flush(fopts);
db_->GetSortedWalFiles(logfiles);
ASSERT_OK(db_->Flush(fopts));
ASSERT_OK(db_->GetSortedWalFiles(logfiles));
ASSERT_GT(logfiles.size(), 0UL);
std::unique_ptr<LogFile> archived_log = std::move(logfiles.front());
ASSERT_EQ(archived_log->Type(), kArchivedLogFile);
@@ -480,8 +488,8 @@ TEST_F(DeleteFileTest, DeleteLogFiles) {
fprintf(stdout, "Deleting archived log file %s\n",
archived_log->PathName().c_str());
ASSERT_OK(db_->DeleteFile(archived_log->PathName()));
ASSERT_EQ(Status::NotFound(),
env_->FileExists(wal_dir_ + "/" + archived_log->PathName()));
ASSERT_TRUE(
env_->FileExists(wal_dir_ + "/" + archived_log->PathName()).IsNotFound());
}
TEST_F(DeleteFileTest, DeleteNonDefaultColumnFamily) {
@@ -520,6 +528,7 @@ TEST_F(DeleteFileTest, DeleteNonDefaultColumnFamily) {
{
std::unique_ptr<Iterator> itr(db_->NewIterator(ReadOptions(), handles_[1]));
ASSERT_OK(itr->status());
int count = 0;
for (itr->SeekToFirst(); itr->Valid(); itr->Next()) {
ASSERT_OK(itr->status());
+77 -93
View File
@@ -173,7 +173,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWriteError) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
Reopen(options);
ASSERT_EQ("val", Get(Key(0)));
@@ -271,7 +271,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError1) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
@@ -314,7 +314,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError2) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
@@ -357,7 +357,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableError3) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ("val2", Get(Key(2)));
ASSERT_OK(Put(Key(3), "val3", wo));
@@ -399,7 +399,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteError) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -442,7 +442,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteRetryableError) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -487,7 +487,7 @@ TEST_F(DBErrorHandlingFSTest, ManifestWriteNoWALRetryableError) {
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -535,7 +535,7 @@ TEST_F(DBErrorHandlingFSTest, DoubleManifestWriteError) {
// A successful Resume() will create a new manifest file
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -568,7 +568,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteError) {
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
// Wait for flush of 2nd L0 file before starting compaction
@@ -598,7 +598,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteError) {
// This Flush will trigger a compaction, which will fail when appending to
// the manifest
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
TEST_SYNC_POINT("CompactionManifestWriteError:0");
// Clear all errors so when the compaction is retried, it will succeed
@@ -609,7 +609,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteError) {
s = dbfull()->TEST_WaitForCompact();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -642,7 +642,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
@@ -668,7 +668,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
TEST_SYNC_POINT("CompactionManifestWriteError:0");
TEST_SYNC_POINT("CompactionManifestWriteError:1");
@@ -680,7 +680,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionManifestWriteRetryableError) {
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
new_manifest = GetManifestNameFromLiveFiles();
ASSERT_NE(new_manifest, old_manifest);
@@ -706,7 +706,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteError) {
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
listener->OverrideBGError(
Status(Status::NoSpace(), Status::Severity::kHardError));
@@ -723,14 +723,14 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteError) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kHardError);
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
Destroy(options);
}
@@ -752,7 +752,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
@@ -766,7 +766,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
@@ -775,7 +775,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableError) {
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
Destroy(options);
}
@@ -790,7 +790,7 @@ TEST_F(DBErrorHandlingFSTest, CorruptionError) {
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::FlushMemTable:FlushMemTableFinished",
@@ -804,7 +804,7 @@ TEST_F(DBErrorHandlingFSTest, CorruptionError) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(),
@@ -812,7 +812,7 @@ TEST_F(DBErrorHandlingFSTest, CorruptionError) {
fault_fs_->SetFilesystemActive(true);
s = dbfull()->Resume();
ASSERT_NE(s, Status::OK());
ASSERT_NOK(s);
Destroy(options);
}
@@ -844,7 +844,7 @@ TEST_F(DBErrorHandlingFSTest, AutoRecoverFlushError) {
ASSERT_EQ(listener->WaitForRecovery(5000000), true);
s = Put(Key(1), "val");
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
Reopen(options);
ASSERT_EQ("val", Get(Key(0)));
@@ -874,7 +874,7 @@ TEST_F(DBErrorHandlingFSTest, FailRecoverFlushError) {
// We should be able to shutdown the database while auto recovery is going
// on in the background
Close();
DestroyDB(dbname_, options);
DestroyDB(dbname_, options).PermitUncheckedError();
}
TEST_F(DBErrorHandlingFSTest, WALWriteError) {
@@ -904,7 +904,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
{
@@ -960,7 +960,6 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
options.listeners.emplace_back(listener);
options.paranoid_checks = true;
options.max_bgerror_resume_count = 0;
Status s;
Random rnd(301);
DestroyAndReopen(options);
@@ -978,7 +977,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
// For the second batch, the first 2 file Append are successful, then the
@@ -1001,8 +1000,8 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wopts;
wopts.sync = true;
s = dbfull()->Write(wopts, &batch);
ASSERT_EQ(true, s.IsIOError());
Status s = dbfull()->Write(wopts, &batch);
ASSERT_TRUE(s.IsIOError());
}
fault_fs_->SetFilesystemActive(true);
SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -1018,8 +1017,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
}
// Resume and write a new batch, should be in the WAL
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(dbfull()->Resume());
{
WriteBatch batch;
@@ -1029,7 +1027,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
Reopen(options);
@@ -1055,7 +1053,6 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
options.create_if_missing = true;
options.writable_file_max_buffer_size = 32768;
options.listeners.emplace_back(listener);
Status s;
Random rnd(301);
listener->EnableAutoRecovery();
@@ -1072,7 +1069,7 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
{
@@ -1095,8 +1092,8 @@ TEST_F(DBErrorHandlingFSTest, MultiCFWALWriteError) {
SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wopts;
wopts.sync = true;
s = dbfull()->Write(wopts, &batch);
ASSERT_EQ(s, s.NoSpace());
Status s = dbfull()->Write(wopts, &batch);
ASSERT_TRUE(s.IsNoSpace());
}
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
@@ -1164,9 +1161,8 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
listener[i]->InjectFileCreationError(fault_fs[i], 3,
IOStatus::NoSpace("Out of space"));
snprintf(buf, sizeof(buf), "_%d", i);
DestroyDB(dbname_ + std::string(buf), options[i]);
ASSERT_EQ(DB::Open(options[i], dbname_ + std::string(buf), &dbptr),
Status::OK());
ASSERT_OK(DestroyDB(dbname_ + std::string(buf), options[i]));
ASSERT_OK(DB::Open(options[i], dbname_ + std::string(buf), &dbptr));
db.emplace_back(dbptr);
}
@@ -1179,8 +1175,8 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(db[i]->Write(wopts, &batch), Status::OK());
ASSERT_EQ(db[i]->Flush(FlushOptions()), Status::OK());
ASSERT_OK(db[i]->Write(wopts, &batch));
ASSERT_OK(db[i]->Flush(FlushOptions()));
}
def_env->SetFilesystemActive(false, Status::NoSpace("Out of space"));
@@ -1194,8 +1190,8 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(db[i]->Write(wopts, &batch), Status::OK());
ASSERT_EQ(db[i]->Flush(FlushOptions()), Status::OK());
ASSERT_OK(db[i]->Write(wopts, &batch));
ASSERT_OK(db[i]->Flush(FlushOptions()));
}
for (auto i = 0; i < kNumDbInstances; ++i) {
@@ -1208,8 +1204,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
for (auto i = 0; i < kNumDbInstances; ++i) {
std::string prop;
ASSERT_EQ(listener[i]->WaitForRecovery(5000000), true);
ASSERT_EQ(static_cast<DBImpl*>(db[i])->TEST_WaitForCompact(true),
Status::OK());
ASSERT_OK(static_cast<DBImpl*>(db[i])->TEST_WaitForCompact(true));
EXPECT_TRUE(db[i]->GetProperty(
"rocksdb.num-files-at-level" + NumberToString(0), &prop));
EXPECT_EQ(atoi(prop.c_str()), 0);
@@ -1226,7 +1221,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBCompactionError) {
if (getenv("KEEP_DB")) {
printf("DB is still at %s%s\n", dbname_.c_str(), buf);
} else {
Status s = DestroyDB(dbname_ + std::string(buf), options[i]);
ASSERT_OK(DestroyDB(dbname_ + std::string(buf), options[i]));
}
}
options.clear();
@@ -1281,9 +1276,8 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
break;
}
snprintf(buf, sizeof(buf), "_%d", i);
DestroyDB(dbname_ + std::string(buf), options[i]);
ASSERT_EQ(DB::Open(options[i], dbname_ + std::string(buf), &dbptr),
Status::OK());
ASSERT_OK(DestroyDB(dbname_ + std::string(buf), options[i]));
ASSERT_OK(DB::Open(options[i], dbname_ + std::string(buf), &dbptr));
db.emplace_back(dbptr);
}
@@ -1296,8 +1290,8 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(db[i]->Write(wopts, &batch), Status::OK());
ASSERT_EQ(db[i]->Flush(FlushOptions()), Status::OK());
ASSERT_OK(db[i]->Write(wopts, &batch));
ASSERT_OK(db[i]->Flush(FlushOptions()));
}
def_env->SetFilesystemActive(false, Status::NoSpace("Out of space"));
@@ -1311,11 +1305,11 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(db[i]->Write(wopts, &batch), Status::OK());
ASSERT_OK(db[i]->Write(wopts, &batch));
if (i != 1) {
ASSERT_EQ(db[i]->Flush(FlushOptions()), Status::OK());
ASSERT_OK(db[i]->Flush(FlushOptions()));
} else {
ASSERT_EQ(db[i]->Flush(FlushOptions()), Status::NoSpace());
ASSERT_TRUE(db[i]->Flush(FlushOptions()).IsNoSpace());
}
}
@@ -1329,7 +1323,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
ASSERT_EQ(s.severity(), Status::Severity::kHardError);
break;
case 2:
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
break;
}
fault_fs[i]->SetFilesystemActive(true);
@@ -1342,8 +1336,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
ASSERT_EQ(listener[i]->WaitForRecovery(5000000), true);
}
if (i == 1) {
ASSERT_EQ(static_cast<DBImpl*>(db[i])->TEST_WaitForCompact(true),
Status::OK());
ASSERT_OK(static_cast<DBImpl*>(db[i])->TEST_WaitForCompact(true));
}
EXPECT_TRUE(db[i]->GetProperty(
"rocksdb.num-files-at-level" + NumberToString(0), &prop));
@@ -1361,7 +1354,7 @@ TEST_F(DBErrorHandlingFSTest, MultiDBVariousErrors) {
if (getenv("KEEP_DB")) {
printf("DB is still at %s%s\n", dbname_.c_str(), buf);
} else {
DestroyDB(dbname_ + std::string(buf), options[i]);
EXPECT_OK(DestroyDB(dbname_ + std::string(buf), options[i]));
}
}
options.clear();
@@ -1418,12 +1411,10 @@ TEST_F(DBErrorHandlingFSTest, FLushWritNoWALRetryableeErrorAutoRecover1) {
ASSERT_EQ("val2", Get(Key(2)));
// call auto resume
s = dbfull()->Resume();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(dbfull()->Resume());
ASSERT_OK(Put(Key(3), "val3", wo));
s = Flush();
// After resume is successful, the flush should be ok.
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
ASSERT_EQ("val3", Get(Key(3)));
Destroy(options);
}
@@ -1513,8 +1504,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover1) {
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1554,8 +1544,7 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover2) {
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1602,10 +1591,9 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover3) {
// resume manually here.
s = dbfull()->Resume();
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1656,19 +1644,16 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover4) {
// Even the FS is recoverd, due to the Fatal Error in bg_error_ the resume
// and flush will all fail.
ASSERT_EQ("val1", Get(Key(1)));
s = dbfull()->Resume();
ASSERT_NE(s, Status::OK());
ASSERT_NOK(dbfull()->Resume());
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_NE(s, Status::OK());
ASSERT_NOK(Flush());
ASSERT_EQ("NOT_FOUND", Get(Key(2)));
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1708,7 +1693,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
// The first resume will cause recovery_error and its severity is the
// Fatal error
s = dbfull()->Close();
ASSERT_NE(s, Status::OK());
ASSERT_NOK(s);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
fault_fs_->SetFilesystemActive(true);
@@ -1717,7 +1702,7 @@ TEST_F(DBErrorHandlingFSTest, DISABLED_FLushWritRetryableeErrorAutoRecover5) {
ASSERT_NE("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1768,14 +1753,14 @@ TEST_F(DBErrorHandlingFSTest, FLushWritRetryableeErrorAutoRecover6) {
// The first resume will cause recovery_error and its severity is the
// Fatal error
s = dbfull()->Close();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
SyncPoint::GetInstance()->DisableProcessing();
Reopen(options);
ASSERT_EQ("val1", Get(Key(1)));
ASSERT_OK(Put(Key(2), "val2"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
ASSERT_EQ("val2", Get(Key(2)));
Destroy(options);
@@ -1912,8 +1897,7 @@ TEST_F(DBErrorHandlingFSTest,
ASSERT_OK(Put(Key(0), "val"));
ASSERT_OK(Put(Key(2), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(Flush());
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
@@ -1950,7 +1934,7 @@ TEST_F(DBErrorHandlingFSTest,
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:0");
TEST_SYNC_POINT("CompactionManifestWriteErrorAR:1");
@@ -2000,7 +1984,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableErrorAutoRecover) {
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
listener->OverrideBGError(Status(error_msg, Status::Severity::kHardError));
listener->EnableAutoRecovery(false);
@@ -2025,7 +2009,7 @@ TEST_F(DBErrorHandlingFSTest, CompactionWriteRetryableErrorAutoRecover) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kSoftError);
@@ -2065,7 +2049,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
// For the second batch, the first 2 file Append are successful, then the
@@ -2122,7 +2106,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover1) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
Reopen(options);
@@ -2166,7 +2150,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
// For the second batch, the first 2 file Append are successful, then the
@@ -2223,7 +2207,7 @@ TEST_F(DBErrorHandlingFSTest, WALWriteRetryableErrorAutoRecover2) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
Reopen(options);
@@ -2285,7 +2269,7 @@ TEST_P(DBErrorHandlingFencingTest, ManifestWriteFenced) {
old_manifest = GetManifestNameFromLiveFiles();
ASSERT_OK(Put(Key(0), "val"));
Flush();
ASSERT_OK(Flush());
ASSERT_OK(Put(Key(1), "val"));
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::LogAndApply:WriteManifest", [&](void*) {
@@ -2318,7 +2302,7 @@ TEST_P(DBErrorHandlingFencingTest, CompactionWriteFenced) {
ASSERT_OK(Put(Key(0), "va;"));
ASSERT_OK(Put(Key(2), "va;"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
listener->EnableAutoRecovery(true);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
@@ -2332,7 +2316,7 @@ TEST_P(DBErrorHandlingFencingTest, CompactionWriteFenced) {
ASSERT_OK(Put(Key(1), "val"));
s = Flush();
ASSERT_EQ(s, Status::OK());
ASSERT_OK(s);
s = dbfull()->TEST_WaitForCompact();
ASSERT_EQ(s.severity(), ROCKSDB_NAMESPACE::Status::Severity::kFatalError);
@@ -2368,7 +2352,7 @@ TEST_P(DBErrorHandlingFencingTest, WALWriteFenced) {
WriteOptions wopts;
wopts.sync = true;
ASSERT_EQ(dbfull()->Write(wopts, &batch), Status::OK());
ASSERT_OK(dbfull()->Write(wopts, &batch));
};
{
+3 -2
View File
@@ -201,7 +201,8 @@ Status ExternalSstFileIngestionJob::Prepare(
requested_checksum_func_name, &generated_checksum,
&generated_checksum_func_name,
ingestion_options_.verify_checksums_readahead_size,
db_options_.allow_mmap_reads, io_tracer_);
db_options_.allow_mmap_reads, io_tracer_,
db_options_.rate_limiter.get());
if (!io_s.ok()) {
status = io_s;
ROCKS_LOG_WARN(db_options_.info_log,
@@ -856,7 +857,7 @@ IOStatus ExternalSstFileIngestionJob::GenerateChecksumForIngestedFile(
db_options_.file_checksum_gen_factory.get(), requested_checksum_func_name,
&file_checksum, &file_checksum_func_name,
ingestion_options_.verify_checksums_readahead_size,
db_options_.allow_mmap_reads, io_tracer_);
db_options_.allow_mmap_reads, io_tracer_, db_options_.rate_limiter.get());
if (!io_s.ok()) {
return io_s;
}
+92 -135
View File
@@ -47,8 +47,8 @@ class ExternSSTFileLinkFailFallbackTest
: DBTestBase("/external_sst_file_test", /*env_do_fsync=*/true),
test_env_(new ExternalSSTTestEnv(env_, true)) {
sst_files_dir_ = dbname_ + "/sst_files/";
DestroyDir(env_, sst_files_dir_);
env_->CreateDir(sst_files_dir_);
EXPECT_EQ(DestroyDir(env_, sst_files_dir_), Status::OK());
EXPECT_EQ(env_->CreateDir(sst_files_dir_), Status::OK());
options_ = CurrentOptions();
options_.disable_auto_compactions = true;
options_.env = test_env_;
@@ -79,8 +79,8 @@ class ExternalSSTFileTest
}
void DestroyAndRecreateExternalSSTFilesDir() {
DestroyDir(env_, sst_files_dir_);
env_->CreateDir(sst_files_dir_);
ASSERT_OK(DestroyDir(env_, sst_files_dir_));
ASSERT_OK(env_->CreateDir(sst_files_dir_));
}
Status GenerateOneExternalFile(
@@ -116,7 +116,7 @@ class ExternalSSTFileTest
for (const auto& entry : data) {
s = sst_file_writer.Put(entry.first, entry.second);
if (!s.ok()) {
sst_file_writer.Finish();
sst_file_writer.Finish().PermitUncheckedError();
return s;
}
}
@@ -171,7 +171,7 @@ class ExternalSSTFileTest
for (auto& entry : data) {
s = sst_file_writer.Put(entry.first, entry.second);
if (!s.ok()) {
sst_file_writer.Finish();
sst_file_writer.Finish().PermitUncheckedError();
return s;
}
}
@@ -213,11 +213,10 @@ class ExternalSSTFileTest
size_t num_cfs = column_families.size();
assert(ifos.size() == num_cfs);
assert(data.size() == num_cfs);
Status s;
std::vector<IngestExternalFileArg> args(num_cfs);
for (size_t i = 0; i != num_cfs; ++i) {
std::string external_file_path;
s = GenerateOneExternalFile(
Status s = GenerateOneExternalFile(
options, column_families[i], data[i], file_id, sort_data,
&external_file_path,
true_data.size() == num_cfs ? &true_data[i] : nullptr);
@@ -230,8 +229,7 @@ class ExternalSSTFileTest
args[i].external_files.push_back(external_file_path);
args[i].options = ifos[i];
}
s = db_->IngestExternalFiles(args);
return s;
return db_->IngestExternalFiles(args);
}
Status GenerateAndAddExternalFile(
@@ -282,7 +280,9 @@ class ExternalSSTFileTest
return db_->IngestExternalFile(files, opts);
}
~ExternalSSTFileTest() override { DestroyDir(env_, sst_files_dir_); }
~ExternalSSTFileTest() override {
DestroyDir(env_, sst_files_dir_).PermitUncheckedError();
}
protected:
int last_file_id_ = 0;
@@ -305,8 +305,7 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file1_info;
Status s = sst_file_writer.Finish(&file1_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file1_info));
// Current file size should be non-zero after success write.
ASSERT_GT(sst_file_writer.FileSize(), 0);
@@ -319,8 +318,7 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_EQ(file1_info.smallest_range_del_key, "");
ASSERT_EQ(file1_info.largest_range_del_key, "");
// sst_file_writer already finished, cannot add this value
s = sst_file_writer.Put(Key(100), "bad_val");
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(sst_file_writer.Put(Key(100), "bad_val"));
// file2.sst (100 => 199)
std::string file2 = sst_files_dir_ + "file2.sst";
@@ -329,11 +327,9 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
// Cannot add this key because it's not after last added key
s = sst_file_writer.Put(Key(99), "bad_val");
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(sst_file_writer.Put(Key(99), "bad_val"));
ExternalSstFileInfo file2_info;
s = sst_file_writer.Finish(&file2_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file2_info));
ASSERT_EQ(file2_info.file_path, file2);
ASSERT_EQ(file2_info.num_entries, 100);
ASSERT_EQ(file2_info.smallest_key, Key(100));
@@ -347,9 +343,8 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val_overlap"));
}
ExternalSstFileInfo file3_info;
s = sst_file_writer.Finish(&file3_info);
ASSERT_OK(sst_file_writer.Finish(&file3_info));
ASSERT_TRUE(s.ok()) << s.ToString();
// Current file size should be non-zero after success finish.
ASSERT_GT(sst_file_writer.FileSize(), 0);
ASSERT_EQ(file3_info.file_path, file3);
@@ -365,8 +360,7 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val_overlap"));
}
ExternalSstFileInfo file4_info;
s = sst_file_writer.Finish(&file4_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file4_info));
ASSERT_EQ(file4_info.file_path, file4);
ASSERT_EQ(file4_info.num_entries, 10);
ASSERT_EQ(file4_info.smallest_key, Key(30));
@@ -379,8 +373,7 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file5_info;
s = sst_file_writer.Finish(&file5_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file5_info));
ASSERT_EQ(file5_info.file_path, file5);
ASSERT_EQ(file5_info.num_entries, 100);
ASSERT_EQ(file5_info.smallest_key, Key(400));
@@ -389,10 +382,9 @@ TEST_F(ExternalSSTFileTest, Basic) {
// file6.sst (delete 400 => 500)
std::string file6 = sst_files_dir_ + "file6.sst";
ASSERT_OK(sst_file_writer.Open(file6));
sst_file_writer.DeleteRange(Key(400), Key(500));
ASSERT_OK(sst_file_writer.DeleteRange(Key(400), Key(500)));
ExternalSstFileInfo file6_info;
s = sst_file_writer.Finish(&file6_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file6_info));
ASSERT_EQ(file6_info.file_path, file6);
ASSERT_EQ(file6_info.num_entries, 0);
ASSERT_EQ(file6_info.smallest_key, "");
@@ -404,17 +396,16 @@ TEST_F(ExternalSSTFileTest, Basic) {
// file7.sst (delete 500 => 570, put 520 => 599 divisible by 2)
std::string file7 = sst_files_dir_ + "file7.sst";
ASSERT_OK(sst_file_writer.Open(file7));
sst_file_writer.DeleteRange(Key(500), Key(550));
ASSERT_OK(sst_file_writer.DeleteRange(Key(500), Key(550)));
for (int k = 520; k < 560; k += 2) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
sst_file_writer.DeleteRange(Key(525), Key(575));
ASSERT_OK(sst_file_writer.DeleteRange(Key(525), Key(575)));
for (int k = 560; k < 600; k += 2) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file7_info;
s = sst_file_writer.Finish(&file7_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file7_info));
ASSERT_EQ(file7_info.file_path, file7);
ASSERT_EQ(file7_info.num_entries, 40);
ASSERT_EQ(file7_info.smallest_key, Key(520));
@@ -426,10 +417,9 @@ TEST_F(ExternalSSTFileTest, Basic) {
// file8.sst (delete 600 => 700)
std::string file8 = sst_files_dir_ + "file8.sst";
ASSERT_OK(sst_file_writer.Open(file8));
sst_file_writer.DeleteRange(Key(600), Key(700));
ASSERT_OK(sst_file_writer.DeleteRange(Key(600), Key(700)));
ExternalSstFileInfo file8_info;
s = sst_file_writer.Finish(&file8_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file8_info));
ASSERT_EQ(file8_info.file_path, file8);
ASSERT_EQ(file8_info.num_entries, 0);
ASSERT_EQ(file8_info.smallest_key, "");
@@ -441,13 +431,11 @@ TEST_F(ExternalSSTFileTest, Basic) {
// Cannot create an empty sst file
std::string file_empty = sst_files_dir_ + "file_empty.sst";
ExternalSstFileInfo file_empty_info;
s = sst_file_writer.Finish(&file_empty_info);
ASSERT_NOK(s);
ASSERT_NOK(sst_file_writer.Finish(&file_empty_info));
DestroyAndReopen(options);
// Add file using file path
s = DeprecatedAddFile({file1});
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(DeprecatedAddFile({file1}));
ASSERT_EQ(db_->GetLatestSequenceNumber(), 0U);
for (int k = 0; k < 100; k++) {
ASSERT_EQ(Get(Key(k)), Key(k) + "_val");
@@ -468,12 +456,10 @@ TEST_F(ExternalSSTFileTest, Basic) {
}
// This file has overlapping values with the existing data
s = DeprecatedAddFile({file3});
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile({file3}));
// This file has overlapping values with the existing data
s = DeprecatedAddFile({file4});
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile({file4}));
// Overwrite values of keys divisible by 5
for (int k = 0; k < 200; k += 5) {
@@ -485,8 +471,7 @@ TEST_F(ExternalSSTFileTest, Basic) {
ASSERT_OK(DeprecatedAddFile({file5}));
// This file has overlapping values with the existing data
s = DeprecatedAddFile({file6});
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile({file6}));
// Key range of file7 (500 => 598) don't overlap with any keys in DB
ASSERT_OK(DeprecatedAddFile({file7}));
@@ -614,15 +599,13 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file1_info;
Status s = sst_file_writer.Finish(&file1_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file1_info));
ASSERT_EQ(file1_info.file_path, file1);
ASSERT_EQ(file1_info.num_entries, 100);
ASSERT_EQ(file1_info.smallest_key, Key(0));
ASSERT_EQ(file1_info.largest_key, Key(99));
// sst_file_writer already finished, cannot add this value
s = sst_file_writer.Put(Key(100), "bad_val");
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(sst_file_writer.Put(Key(100), "bad_val"));
// file2.sst (100 => 199)
std::string file2 = sst_files_dir_ + "file2.sst";
@@ -631,11 +614,9 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
// Cannot add this key because it's not after last added key
s = sst_file_writer.Put(Key(99), "bad_val");
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(sst_file_writer.Put(Key(99), "bad_val"));
ExternalSstFileInfo file2_info;
s = sst_file_writer.Finish(&file2_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file2_info));
ASSERT_EQ(file2_info.file_path, file2);
ASSERT_EQ(file2_info.num_entries, 100);
ASSERT_EQ(file2_info.smallest_key, Key(100));
@@ -649,8 +630,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val_overlap"));
}
ExternalSstFileInfo file3_info;
s = sst_file_writer.Finish(&file3_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file3_info));
ASSERT_EQ(file3_info.file_path, file3);
ASSERT_EQ(file3_info.num_entries, 5);
ASSERT_EQ(file3_info.smallest_key, Key(195));
@@ -664,8 +644,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val_overlap"));
}
ExternalSstFileInfo file4_info;
s = sst_file_writer.Finish(&file4_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file4_info));
ASSERT_EQ(file4_info.file_path, file4);
ASSERT_EQ(file4_info.num_entries, 10);
ASSERT_EQ(file4_info.smallest_key, Key(30));
@@ -678,8 +657,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file5_info;
s = sst_file_writer.Finish(&file5_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file5_info));
ASSERT_EQ(file5_info.file_path, file5);
ASSERT_EQ(file5_info.num_entries, 100);
ASSERT_EQ(file5_info.smallest_key, Key(200));
@@ -691,8 +669,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.DeleteRange(Key(0), Key(75)));
ASSERT_OK(sst_file_writer.DeleteRange(Key(25), Key(100)));
ExternalSstFileInfo file6_info;
s = sst_file_writer.Finish(&file6_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file6_info));
ASSERT_EQ(file6_info.file_path, file6);
ASSERT_EQ(file6_info.num_entries, 0);
ASSERT_EQ(file6_info.smallest_key, "");
@@ -706,8 +683,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
ASSERT_OK(sst_file_writer.Open(file7));
ASSERT_OK(sst_file_writer.DeleteRange(Key(99), Key(201)));
ExternalSstFileInfo file7_info;
s = sst_file_writer.Finish(&file7_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file7_info));
ASSERT_EQ(file7_info.file_path, file7);
ASSERT_EQ(file7_info.num_entries, 0);
ASSERT_EQ(file7_info.smallest_key, "");
@@ -727,17 +703,13 @@ TEST_F(ExternalSSTFileTest, AddList) {
DestroyAndReopen(options);
// These lists of files have key ranges that overlap with each other
s = DeprecatedAddFile(file_list1);
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile(file_list1));
// Both of the following overlap on the range deletion tombstone.
s = DeprecatedAddFile(file_list4);
ASSERT_FALSE(s.ok()) << s.ToString();
s = DeprecatedAddFile(file_list5);
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile(file_list4));
ASSERT_NOK(DeprecatedAddFile(file_list5));
// Add files using file path list
s = DeprecatedAddFile(file_list0);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(DeprecatedAddFile(file_list0));
ASSERT_EQ(db_->GetLatestSequenceNumber(), 0U);
for (int k = 0; k < 200; k++) {
ASSERT_EQ(Get(Key(k)), Key(k) + "_val");
@@ -778,8 +750,7 @@ TEST_F(ExternalSSTFileTest, AddList) {
}
// This file list has overlapping values with the existing data
s = DeprecatedAddFile(file_list3);
ASSERT_FALSE(s.ok()) << s.ToString();
ASSERT_NOK(DeprecatedAddFile(file_list3));
// Overwrite values of keys divisible by 5
for (int k = 0; k < 200; k += 5) {
@@ -847,16 +818,14 @@ TEST_F(ExternalSSTFileTest, AddListAtomicity) {
for (int k = i * 100; k < (i + 1) * 100; k++) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
Status s = sst_file_writer.Finish(&files_info[i]);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&files_info[i]));
ASSERT_EQ(files_info[i].file_path, files[i]);
ASSERT_EQ(files_info[i].num_entries, 100);
ASSERT_EQ(files_info[i].smallest_key, Key(i * 100));
ASSERT_EQ(files_info[i].largest_key, Key((i + 1) * 100 - 1));
}
files.push_back(sst_files_dir_ + "file" + std::to_string(n) + ".sst");
auto s = DeprecatedAddFile(files);
ASSERT_NOK(s) << s.ToString();
ASSERT_NOK(DeprecatedAddFile(files));
for (int k = 0; k < n * 100; k++) {
ASSERT_EQ("NOT_FOUND", Get(Key(k)));
}
@@ -878,17 +847,14 @@ TEST_F(ExternalSSTFileTest, PurgeObsoleteFilesBug) {
// file1.sst (0 => 500)
std::string sst_file_path = sst_files_dir_ + "file1.sst";
Status s = sst_file_writer.Open(sst_file_path);
ASSERT_OK(s);
ASSERT_OK(sst_file_writer.Open(sst_file_path));
for (int i = 0; i < 500; i++) {
std::string k = Key(i);
s = sst_file_writer.Put(k, k + "_val");
ASSERT_OK(s);
ASSERT_OK(sst_file_writer.Put(k, k + "_val"));
}
ExternalSstFileInfo sst_file_info;
s = sst_file_writer.Finish(&sst_file_info);
ASSERT_OK(s);
ASSERT_OK(sst_file_writer.Finish(&sst_file_info));
options.delete_obsolete_files_period_micros = 0;
options.disable_auto_compactions = true;
@@ -900,12 +866,11 @@ TEST_F(ExternalSSTFileTest, PurgeObsoleteFilesBug) {
ASSERT_OK(Flush());
ASSERT_OK(Put("aaa", "xxx"));
ASSERT_OK(Flush());
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
s = DeprecatedAddFile({sst_file_path});
ASSERT_OK(s);
ASSERT_OK(DeprecatedAddFile({sst_file_path}));
for (int i = 0; i < 500; i++) {
std::string k = Key(i);
@@ -928,8 +893,7 @@ TEST_F(ExternalSSTFileTest, SkipSnapshot) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file1_info;
Status s = sst_file_writer.Finish(&file1_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file1_info));
ASSERT_EQ(file1_info.file_path, file1);
ASSERT_EQ(file1_info.num_entries, 100);
ASSERT_EQ(file1_info.smallest_key, Key(0));
@@ -942,8 +906,7 @@ TEST_F(ExternalSSTFileTest, SkipSnapshot) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file2_info;
s = sst_file_writer.Finish(&file2_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file2_info));
ASSERT_EQ(file2_info.file_path, file2);
ASSERT_EQ(file2_info.num_entries, 200);
ASSERT_EQ(file2_info.smallest_key, Key(100));
@@ -972,8 +935,7 @@ TEST_F(ExternalSSTFileTest, SkipSnapshot) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k) + "_val"));
}
ExternalSstFileInfo file3_info;
s = sst_file_writer.Finish(&file3_info);
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish(&file3_info));
ASSERT_EQ(file3_info.file_path, file3);
ASSERT_EQ(file3_info.num_entries, 100);
ASSERT_EQ(file3_info.smallest_key, Key(300));
@@ -1019,8 +981,7 @@ TEST_F(ExternalSSTFileTest, MultiThreaded) {
ASSERT_OK(sst_file_writer.Put(Key(k), Key(k)));
}
Status s = sst_file_writer.Finish();
ASSERT_TRUE(s.ok()) << s.ToString();
ASSERT_OK(sst_file_writer.Finish());
};
// Write num_files files in parallel
std::vector<port::Thread> sst_writer_threads;
@@ -1082,8 +1043,7 @@ TEST_F(ExternalSSTFileTest, MultiThreaded) {
// Overwrite values of keys divisible by 100
for (int k = 0; k < num_files * keys_per_file; k += 100) {
std::string key = Key(k);
Status s = Put(key, key + "_new");
ASSERT_TRUE(s.ok());
ASSERT_OK(Put(key, key + "_new"));
}
for (int i = 0; i < 2; i++) {
@@ -1167,7 +1127,8 @@ TEST_F(ExternalSSTFileTest, OverlappingRanges) {
// Generate the file containing the range
std::string file_name = sst_files_dir_ + env_->GenerateUniqueId();
ASSERT_OK(sst_file_writer.Open(file_name));
s = sst_file_writer.Open(file_name);
ASSERT_OK(s);
for (int k = range_start; k <= range_end; k++) {
s = sst_file_writer.Put(Key(k), range_val);
ASSERT_OK(s);
@@ -1212,10 +1173,10 @@ TEST_F(ExternalSSTFileTest, OverlappingRanges) {
// Flush / Compact the DB
if (i && i % 50 == 0) {
Flush();
ASSERT_OK(Flush());
}
if (i && i % 75 == 0) {
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
}
}
@@ -1293,7 +1254,7 @@ TEST_P(ExternalSSTFileTest, PickedLevel) {
// Hold compaction from finishing
TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevel:2");
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
EXPECT_EQ(FilesPerLevel(), "1,1,1,2");
size_t kcnt = 0;
@@ -1380,7 +1341,7 @@ TEST_F(ExternalSSTFileTest, PickedLevelBug) {
ASSERT_OK(bg_addfile_status);
ASSERT_OK(bg_compact_status);
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
int total_keys = 0;
Iterator* iter = db_->NewIterator(ReadOptions());
@@ -1417,7 +1378,7 @@ TEST_F(ExternalSSTFileTest, IngestNonExistingFile) {
// After full compaction, there should be only 1 file.
std::vector<std::string> files;
env_->GetChildren(dbname_, &files);
ASSERT_OK(env_->GetChildren(dbname_, &files));
int num_sst_files = 0;
for (auto& f : files) {
uint64_t number;
@@ -1539,7 +1500,7 @@ TEST_F(ExternalSSTFileTest, PickedLevelDynamic) {
TEST_SYNC_POINT("ExternalSSTFileTest::PickedLevelDynamic:2");
// Output of the compaction will go to L3
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
EXPECT_EQ(FilesPerLevel(), "1,0,0,2");
Close();
@@ -1681,7 +1642,7 @@ TEST_F(ExternalSSTFileTest, AddFileTrivialMoveBug) {
cro.exclusive_manual_compaction = false;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -1733,9 +1694,9 @@ TEST_F(ExternalSSTFileTest, WithUnorderedWrite) {
Options options = CurrentOptions();
options.unordered_write = true;
DestroyAndReopen(options);
Put("foo", "v1");
ASSERT_OK(Put("foo", "v1"));
SyncPoint::GetInstance()->EnableProcessing();
port::Thread writer([&]() { Put("bar", "v2"); });
port::Thread writer([&]() { ASSERT_OK(Put("bar", "v2")); });
TEST_SYNC_POINT("ExternalSSTFileTest::WithUnorderedWrite:WaitWriteWAL");
ASSERT_OK(GenerateAndAddExternalFile(options, {{"bar", "v3"}}, -1,
@@ -1784,7 +1745,7 @@ TEST_P(ExternalSSTFileTest, IngestFileWithGlobalSeqnoRandomized) {
}
size_t kcnt = 0;
VerifyDBFromMap(true_data, &kcnt, false);
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
VerifyDBFromMap(true_data, &kcnt, false);
}
}
@@ -1868,8 +1829,8 @@ TEST_P(ExternalSSTFileTest, IngestFileWithGlobalSeqnoMemtableFlush) {
ASSERT_OK(Put(Key(k), "memtable"));
true_data[Key(k)] = "memtable";
}
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_GE(entries_in_memtable, 1);
bool write_global_seqno = std::get<0>(GetParam());
@@ -1878,40 +1839,40 @@ TEST_P(ExternalSSTFileTest, IngestFileWithGlobalSeqnoMemtableFlush) {
ASSERT_OK(GenerateAndAddExternalFile(
options, {90, 100, 110}, -1, true, write_global_seqno,
verify_checksums_before_ingest, false, false, &true_data));
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_GE(entries_in_memtable, 1);
// This file will flush the memtable
ASSERT_OK(GenerateAndAddExternalFile(
options, {19, 20, 21}, -1, true, write_global_seqno,
verify_checksums_before_ingest, false, false, &true_data));
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_EQ(entries_in_memtable, 0);
for (int k : {200, 201, 205, 206}) {
ASSERT_OK(Put(Key(k), "memtable"));
true_data[Key(k)] = "memtable";
}
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_GE(entries_in_memtable, 1);
// No need for flush, this file keys fit between the memtable keys
ASSERT_OK(GenerateAndAddExternalFile(
options, {202, 203, 204}, -1, true, write_global_seqno,
verify_checksums_before_ingest, false, false, &true_data));
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_GE(entries_in_memtable, 1);
// This file will flush the memtable
ASSERT_OK(GenerateAndAddExternalFile(
options, {206, 207}, -1, true, write_global_seqno,
verify_checksums_before_ingest, false, false, &true_data));
db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable);
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kNumEntriesActiveMemTable,
&entries_in_memtable));
ASSERT_EQ(entries_in_memtable, 0);
size_t kcnt = 0;
@@ -2309,7 +2270,7 @@ TEST_P(ExternalSSTFileTest, IngestBehind) {
ASSERT_OK(Put(Key(i), "memtable"));
true_data[Key(i)] = "memtable";
}
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
// Universal picker should go at second from the bottom level
ASSERT_EQ("0,1", FilesPerLevel());
ASSERT_OK(GenerateAndAddExternalFile(
@@ -2323,7 +2284,7 @@ TEST_P(ExternalSSTFileTest, IngestBehind) {
verify_checksums_before_ingest, true /*ingest_behind*/,
false /*sort_data*/, &true_data));
ASSERT_EQ("0,1,1", FilesPerLevel());
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
// bottom level should be empty
ASSERT_EQ("0,1", FilesPerLevel());
@@ -2471,9 +2432,8 @@ TEST_P(ExternalSSTFileTest, IngestFilesIntoMultipleColumnFamilies_Success) {
// Resize the true_data vector upon construction to avoid re-alloc
std::vector<std::map<std::string, std::string>> true_data(
column_families.size());
Status s = GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data);
ASSERT_OK(s);
ASSERT_OK(GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data));
Close();
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "pikachu", "eevee"},
options);
@@ -2654,9 +2614,8 @@ TEST_P(ExternalSSTFileTest, IngestFilesIntoMultipleColumnFamilies_PrepareFail) {
std::vector<std::map<std::string, std::string>> true_data(
column_families.size());
port::Thread ingest_thread([&]() {
Status s = GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data);
ASSERT_NOK(s);
ASSERT_NOK(GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data));
});
TEST_SYNC_POINT(
"ExternalSSTFileTest::IngestFilesIntoMultipleColumnFamilies_PrepareFail:"
@@ -2724,9 +2683,8 @@ TEST_P(ExternalSSTFileTest, IngestFilesIntoMultipleColumnFamilies_CommitFail) {
std::vector<std::map<std::string, std::string>> true_data(
column_families.size());
port::Thread ingest_thread([&]() {
Status s = GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data);
ASSERT_NOK(s);
ASSERT_NOK(GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data));
});
TEST_SYNC_POINT(
"ExternalSSTFileTest::IngestFilesIntoMultipleColumnFamilies_CommitFail:"
@@ -2799,9 +2757,8 @@ TEST_P(ExternalSSTFileTest,
std::vector<std::map<std::string, std::string>> true_data(
column_families.size());
port::Thread ingest_thread([&]() {
Status s = GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data);
ASSERT_NOK(s);
ASSERT_NOK(GenerateAndAddExternalFiles(options, column_families, ifos, data,
-1, true, true_data));
});
TEST_SYNC_POINT(
"ExternalSSTFileTest::IngestFilesIntoMultipleColumnFamilies_"
@@ -2812,7 +2769,7 @@ TEST_P(ExternalSSTFileTest,
"PartialManifestWriteFail:1");
ingest_thread.join();
fault_injection_env->DropUnsyncedFileData();
ASSERT_OK(fault_injection_env->DropUnsyncedFileData());
fault_injection_env->SetFilesystemActive(true);
Close();
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "pikachu", "eevee"},
@@ -2847,7 +2804,7 @@ TEST_P(ExternalSSTFileTest, IngestFilesTriggerFlushingWithTwoWriteQueue) {
// sure that it won't enter the 2nd writer queue for the second time.
std::vector<std::pair<std::string, std::string>> data;
data.push_back(std::make_pair("1001", "v2"));
GenerateAndAddExternalFile(options, data);
ASSERT_OK(GenerateAndAddExternalFile(options, data, -1, true));
}
TEST_P(ExternalSSTFileTest, DeltaEncodingWhileGlobalSeqnoPresent) {
+9 -9
View File
@@ -192,7 +192,7 @@ class FaultInjectionTest
for (int i = start_idx; i < start_idx + num_vals; i++) {
Slice key = Key(i, &key_space);
batch.Clear();
batch.Put(key, Value(i, &value_space));
ASSERT_OK(batch.Put(key, Value(i, &value_space)));
ASSERT_OK(db_->Write(write_options, &batch));
}
}
@@ -272,12 +272,12 @@ class FaultInjectionTest
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ASSERT_OK(db_->Delete(WriteOptions(), iter->key()));
}
ASSERT_OK(iter->status());
delete iter;
FlushOptions flush_options;
flush_options.wait = true;
db_->Flush(flush_options);
ASSERT_OK(db_->Flush(flush_options));
}
// rnd cannot be null for kResetDropRandomUnsyncedData
@@ -310,7 +310,7 @@ class FaultInjectionTest
Build(write_options, 0, num_pre_sync);
if (sync_use_compact_) {
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
}
write_options.sync = false;
Build(write_options, num_pre_sync, num_post_sync);
@@ -342,7 +342,7 @@ class FaultInjectionTest
}
void WaitCompactionFinish() {
static_cast<DBImpl*>(db_->GetRootDB())->TEST_WaitForCompact();
ASSERT_OK(static_cast<DBImpl*>(db_->GetRootDB())->TEST_WaitForCompact());
ASSERT_OK(db_->Put(WriteOptions(), "", ""));
}
};
@@ -409,7 +409,7 @@ TEST_P(FaultInjectionTest, WriteOptionSyncTest) {
write_options.sync = true;
ASSERT_OK(
db_->Put(write_options, Key(2, &key_space), Value(2, &value_space)));
db_->FlushWAL(false);
ASSERT_OK(db_->FlushWAL(false));
env_->SetFilesystemActive(false);
NoWriteTestReopenWithFault(kResetDropAndDeleteUnsynced);
@@ -450,7 +450,7 @@ TEST_P(FaultInjectionTest, UninstalledCompaction) {
Build(WriteOptions(), 0, kNumKeys);
FlushOptions flush_options;
flush_options.wait = true;
db_->Flush(flush_options);
ASSERT_OK(db_->Flush(flush_options));
ASSERT_OK(db_->Put(WriteOptions(), "", ""));
TEST_SYNC_POINT("FaultInjectionTest::FaultTest:0");
TEST_SYNC_POINT("FaultInjectionTest::FaultTest:1");
@@ -521,9 +521,9 @@ TEST_P(FaultInjectionTest, WriteBatchWalTerminationTest) {
wo.sync = true;
wo.disableWAL = false;
WriteBatch batch;
batch.Put("cats", "dogs");
ASSERT_OK(batch.Put("cats", "dogs"));
batch.MarkWalTerminationPoint();
batch.Put("boys", "girls");
ASSERT_OK(batch.Put("boys", "girls"));
ASSERT_OK(db_->Write(wo, &batch));
env_->SetFilesystemActive(false);
+2 -1
View File
@@ -52,7 +52,7 @@ class FlushJobTestBase : public testing::Test {
}
void NewDB() {
SetIdentityFile(env_, dbname_);
ASSERT_OK(SetIdentityFile(env_, dbname_));
VersionEdit new_db;
new_db.SetLogNumber(0);
@@ -85,6 +85,7 @@ class FlushJobTestBase : public testing::Test {
std::string record;
new_db.EncodeTo(&record);
s = log.AddRecord(record);
ASSERT_OK(s);
for (const auto& e : new_cfs) {
record.clear();
+7 -5
View File
@@ -47,11 +47,7 @@ class ForwardLevelIterator : public InternalIterator {
pinned_iters_mgr_(nullptr),
prefix_extractor_(prefix_extractor),
allow_unprepared_value_(allow_unprepared_value) {
/*
NOTE needed for ASSERT_STATUS_CHECKED
in MergeOperatorPinningTest/MergeOperatorPinningTest.TailingIterator
*/
status_.PermitUncheckedError();
status_.PermitUncheckedError(); // Allow uninitialized status through
}
~ForwardLevelIterator() override {
@@ -244,6 +240,12 @@ ForwardIterator::ForwardIterator(DBImpl* db, const ReadOptions& read_options,
if (sv_) {
RebuildIterators(false);
}
// immutable_status_ is a local aggregation of the
// status of the immutable Iterators.
// We have to PermitUncheckedError in case it is never
// used, otherwise it will fail ASSERT_STATUS_CHECKED.
immutable_status_.PermitUncheckedError();
}
ForwardIterator::~ForwardIterator() {
+16 -16
View File
@@ -104,9 +104,9 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFiles) {
ASSERT_NE(import_cfh_, nullptr);
std::string value;
db_->Get(ReadOptions(), import_cfh_, "K1", &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, "K1", &value));
ASSERT_EQ(value, "V1");
db_->Get(ReadOptions(), import_cfh_, "K2", &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, "K2", &value));
ASSERT_EQ(value, "V2");
ASSERT_OK(db_->DropColumnFamily(import_cfh_));
ASSERT_OK(db_->DestroyColumnFamilyHandle(import_cfh_));
@@ -125,9 +125,9 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFiles) {
ASSERT_NE(import_cfh_, nullptr);
std::string value;
db_->Get(ReadOptions(), import_cfh_, "K3", &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, "K3", &value));
ASSERT_EQ(value, "V1");
db_->Get(ReadOptions(), import_cfh_, "K4", &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, "K4", &value));
ASSERT_EQ(value, "V2");
}
}
@@ -214,7 +214,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
for (int i = 0; i < 100; i++) {
std::string value;
db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value));
if (i % 16 == 0) {
ASSERT_EQ(value, Key(i) + "_overwrite4");
} else if (i % 4 == 0) {
@@ -235,7 +235,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
ASSERT_OK(db_->Flush(FlushOptions(), import_cfh_));
for (int i = 0; i < 100; i++) {
std::string value;
db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value));
if (i % 5 == 0) {
ASSERT_EQ(value, Key(i) + "_overwrite5");
} else if (i % 16 == 0) {
@@ -254,7 +254,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
db_->CompactRange(CompactRangeOptions(), import_cfh_, nullptr, nullptr));
for (int i = 0; i < 100; i++) {
std::string value;
db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value));
if (i % 5 == 0) {
ASSERT_EQ(value, Key(i) + "_overwrite5");
} else if (i % 16 == 0) {
@@ -318,12 +318,12 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
std::string value1, value2;
for (int i = 0; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value1));
ASSERT_EQ(Get(1, Key(i)), value1);
}
for (int i = 0; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2));
ASSERT_EQ(Get(1, Key(i)), value2);
}
@@ -340,16 +340,16 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1).IsNotFound());
}
for (int i = 25; i < 50; ++i) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value1));
ASSERT_EQ(Key(i) + "_overwrite3", value1);
}
for (int i = 50; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value1));
ASSERT_EQ(Key(i) + "_overwrite2", value1);
}
for (int i = 0; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2));
ASSERT_EQ(Get(1, Key(i)), value2);
}
@@ -363,16 +363,16 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1).IsNotFound());
}
for (int i = 25; i < 50; ++i) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value1));
ASSERT_EQ(Key(i) + "_overwrite3", value1);
}
for (int i = 50; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, Key(i), &value1));
ASSERT_EQ(Key(i) + "_overwrite2", value1);
}
for (int i = 0; i < 100; ++i) {
db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
ASSERT_OK(db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2));
ASSERT_EQ(Get(1, Key(i)), value2);
}
}
@@ -424,7 +424,7 @@ TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
for (int i = 0; i < 100; ++i) {
std::string value;
db_copy->Get(ReadOptions(), cfh, Key(i), &value);
ASSERT_OK(db_copy->Get(ReadOptions(), cfh, Key(i), &value));
ASSERT_EQ(Get(1, Key(i)), value);
}
ASSERT_OK(db_copy->DropColumnFamily(cfh));
+30 -23
View File
@@ -192,10 +192,10 @@ TEST_F(EventListenerTest, OnSingleDBCompactionTest) {
ASSERT_OK(Put(7, "popovich", std::string(90000, 'p')));
for (int i = 1; i < 8; ++i) {
ASSERT_OK(Flush(i));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), handles_[i],
nullptr, nullptr));
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
ASSERT_EQ(listener->compacted_dbs_.size(), cf_names.size());
@@ -211,6 +211,10 @@ class TestFlushListener : public EventListener {
: slowdown_count(0), stop_count(0), db_closed(), env_(env), test_(test) {
db_closed = false;
}
virtual ~TestFlushListener() {
prev_fc_info_.status.PermitUncheckedError(); // Ignore the status
}
void OnTableFileCreated(
const TableFileCreationInfo& info) override {
// remember the info for later checking the FlushJobInfo.
@@ -333,7 +337,7 @@ TEST_F(EventListenerTest, OnSingleDBFlushTest) {
ASSERT_OK(Put(7, "popovich", std::string(90000, 'p')));
for (int i = 1; i < 8; ++i) {
ASSERT_OK(Flush(i));
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_EQ(listener->flushed_dbs_.size(), i);
ASSERT_EQ(listener->flushed_column_family_names_.size(), i);
}
@@ -417,7 +421,7 @@ TEST_F(EventListenerTest, MultiDBMultiListeners) {
ASSERT_OK(DB::Open(options, dbname_ + ToString(d), &db));
for (size_t c = 0; c < cf_names.size(); ++c) {
ColumnFamilyHandle* handle;
db->CreateColumnFamily(cf_opts, cf_names[c], &handle);
ASSERT_OK(db->CreateColumnFamily(cf_opts, cf_names[c], &handle));
handles.push_back(handle);
}
@@ -435,7 +439,8 @@ TEST_F(EventListenerTest, MultiDBMultiListeners) {
for (size_t c = 0; c < cf_names.size(); ++c) {
for (int d = 0; d < kNumDBs; ++d) {
ASSERT_OK(dbs[d]->Flush(FlushOptions(), vec_handles[d][c]));
static_cast_with_check<DBImpl>(dbs[d])->TEST_WaitForFlushMemTable();
ASSERT_OK(
static_cast_with_check<DBImpl>(dbs[d])->TEST_WaitForFlushMemTable());
}
}
@@ -494,10 +499,10 @@ TEST_F(EventListenerTest, DisableBGCompaction) {
// keep writing until writes are forced to stop.
for (int i = 0; static_cast<int>(cf_meta.file_count) < kSlowdownTrigger * 10;
++i) {
Put(1, ToString(i), std::string(10000, 'x'), WriteOptions());
ASSERT_OK(Put(1, ToString(i), std::string(10000, 'x'), WriteOptions()));
FlushOptions fo;
fo.allow_write_stall = true;
db_->Flush(fo, handles_[1]);
ASSERT_OK(db_->Flush(fo, handles_[1]));
db_->GetColumnFamilyMetaData(handles_[1], &cf_meta);
}
ASSERT_GE(listener->slowdown_count, kSlowdownTrigger * 9);
@@ -534,7 +539,7 @@ TEST_F(EventListenerTest, CompactionReasonLevel) {
for (int i = 0; i < 4; i++) {
GenerateNewRandomFile(&rnd);
}
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_EQ(listener->compaction_reasons_.size(), 1);
ASSERT_EQ(listener->compaction_reasons_[0],
@@ -551,14 +556,14 @@ TEST_F(EventListenerTest, CompactionReasonLevel) {
}
// Do a trivial move from L0 -> L1
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
options.max_bytes_for_level_base = 1;
Close();
listener->compaction_reasons_.clear();
Reopen(options);
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GT(listener->compaction_reasons_.size(), 1);
for (auto compaction_reason : listener->compaction_reasons_) {
@@ -570,7 +575,7 @@ TEST_F(EventListenerTest, CompactionReasonLevel) {
listener->compaction_reasons_.clear();
Reopen(options);
Put("key", "value");
ASSERT_OK(Put("key", "value"));
CompactRangeOptions cro;
cro.bottommost_level_compaction = BottommostLevelCompaction::kForceOptimized;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
@@ -604,7 +609,7 @@ TEST_F(EventListenerTest, CompactionReasonUniversal) {
for (int i = 0; i < 8; i++) {
GenerateNewRandomFile(&rnd);
}
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GT(listener->compaction_reasons_.size(), 0);
for (auto compaction_reason : listener->compaction_reasons_) {
@@ -622,7 +627,7 @@ TEST_F(EventListenerTest, CompactionReasonUniversal) {
for (int i = 0; i < 8; i++) {
GenerateNewRandomFile(&rnd);
}
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GT(listener->compaction_reasons_.size(), 0);
for (auto compaction_reason : listener->compaction_reasons_) {
@@ -634,7 +639,7 @@ TEST_F(EventListenerTest, CompactionReasonUniversal) {
listener->compaction_reasons_.clear();
Reopen(options);
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_GT(listener->compaction_reasons_.size(), 0);
for (auto compaction_reason : listener->compaction_reasons_) {
@@ -663,7 +668,7 @@ TEST_F(EventListenerTest, CompactionReasonFIFO) {
for (int i = 0; i < 4; i++) {
GenerateNewRandomFile(&rnd);
}
dbfull()->TEST_WaitForCompact();
ASSERT_OK(dbfull()->TEST_WaitForCompact());
ASSERT_GT(listener->compaction_reasons_.size(), 0);
for (auto compaction_reason : listener->compaction_reasons_) {
@@ -783,7 +788,7 @@ TEST_F(EventListenerTest, TableFileCreationListenersTest) {
ASSERT_OK(Put("foo", "aaa"));
ASSERT_OK(Put("bar", "bbb"));
ASSERT_OK(Flush());
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
listener->CheckAndResetCounters(1, 1, 0, 0, 0, 0);
ASSERT_OK(Put("foo", "aaa1"));
ASSERT_OK(Put("bar", "bbb1"));
@@ -796,21 +801,23 @@ TEST_F(EventListenerTest, TableFileCreationListenersTest) {
ASSERT_OK(Put("foo", "aaa2"));
ASSERT_OK(Put("bar", "bbb2"));
ASSERT_OK(Flush());
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
listener->CheckAndResetCounters(1, 1, 0, 0, 0, 0);
const Slice kRangeStart = "a";
const Slice kRangeEnd = "z";
dbfull()->CompactRange(CompactRangeOptions(), &kRangeStart, &kRangeEnd);
dbfull()->TEST_WaitForCompact();
ASSERT_OK(
dbfull()->CompactRange(CompactRangeOptions(), &kRangeStart, &kRangeEnd));
ASSERT_OK(dbfull()->TEST_WaitForCompact());
listener->CheckAndResetCounters(0, 0, 0, 1, 1, 0);
ASSERT_OK(Put("foo", "aaa3"));
ASSERT_OK(Put("bar", "bbb3"));
ASSERT_OK(Flush());
test_env->SetStatus(Status::NotSupported("not supported"));
dbfull()->CompactRange(CompactRangeOptions(), &kRangeStart, &kRangeEnd);
dbfull()->TEST_WaitForCompact();
ASSERT_NOK(
dbfull()->CompactRange(CompactRangeOptions(), &kRangeStart, &kRangeEnd));
ASSERT_NOK(dbfull()->TEST_WaitForCompact());
listener->CheckAndResetCounters(1, 1, 0, 1, 1, 1);
Close();
}
@@ -1076,8 +1083,8 @@ TEST_F(EventListenerTest, OnFileOperationTest) {
}
DestroyAndReopen(options);
ASSERT_OK(Put("foo", "aaa"));
dbfull()->Flush(FlushOptions());
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_OK(dbfull()->Flush(FlushOptions()));
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
ASSERT_GE(listener->file_writes_.load(),
listener->file_writes_success_.load());
ASSERT_GT(listener->file_writes_.load(), 0);
+8 -6
View File
@@ -191,7 +191,7 @@ class LogTest : public ::testing::TestWithParam<std::tuple<int, bool>> {
Slice* get_reader_contents() { return &reader_contents_; }
void Write(const std::string& msg) {
writer_.AddRecord(Slice(msg));
ASSERT_OK(writer_.AddRecord(Slice(msg)));
}
size_t WrittenBytes() const {
@@ -689,8 +689,8 @@ TEST_P(LogTest, Recycle) {
new test::OverwritingStringSink(get_reader_contents()),
"" /* don't care */));
Writer recycle_writer(std::move(dest_holder), 123, true);
recycle_writer.AddRecord(Slice("foooo"));
recycle_writer.AddRecord(Slice("bar"));
ASSERT_OK(recycle_writer.AddRecord(Slice("foooo")));
ASSERT_OK(recycle_writer.AddRecord(Slice("bar")));
ASSERT_GE(get_reader_contents()->size(), log::kBlockSize * 2);
ASSERT_EQ("foooo", Read());
ASSERT_EQ("bar", Read());
@@ -782,11 +782,13 @@ class RetriableLogTest : public ::testing::TestWithParam<int> {
return file->contents_;
}
void Encode(const std::string& msg) { log_writer_->AddRecord(Slice(msg)); }
void Encode(const std::string& msg) {
ASSERT_OK(log_writer_->AddRecord(Slice(msg)));
}
void Write(const Slice& data) {
writer_->Append(data);
writer_->Sync(true);
ASSERT_OK(writer_->Append(data));
ASSERT_OK(writer_->Sync(true));
}
bool TryRead(std::string* result) {
+1 -1
View File
@@ -33,7 +33,7 @@ Writer::Writer(std::unique_ptr<WritableFileWriter>&& dest, uint64_t log_number,
Writer::~Writer() {
if (dest_) {
WriteBuffer();
WriteBuffer().PermitUncheckedError();
}
}
+16 -16
View File
@@ -100,13 +100,13 @@ TEST_F(ManualCompactionTest, CompactTouchesAllKeys) {
options.compaction_filter = new DestroyAllCompactionFilter();
ASSERT_OK(DB::Open(options, dbname_, &db));
db->Put(WriteOptions(), Slice("key1"), Slice("destroy"));
db->Put(WriteOptions(), Slice("key2"), Slice("destroy"));
db->Put(WriteOptions(), Slice("key3"), Slice("value3"));
db->Put(WriteOptions(), Slice("key4"), Slice("destroy"));
ASSERT_OK(db->Put(WriteOptions(), Slice("key1"), Slice("destroy")));
ASSERT_OK(db->Put(WriteOptions(), Slice("key2"), Slice("destroy")));
ASSERT_OK(db->Put(WriteOptions(), Slice("key3"), Slice("value3")));
ASSERT_OK(db->Put(WriteOptions(), Slice("key4"), Slice("destroy")));
Slice key4("key4");
db->CompactRange(CompactRangeOptions(), nullptr, &key4);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, &key4));
Iterator* itr = db->NewIterator(ReadOptions());
itr->SeekToFirst();
ASSERT_TRUE(itr->Valid());
@@ -135,21 +135,21 @@ TEST_F(ManualCompactionTest, Test) {
// create first key range
WriteBatch batch;
for (int i = 0; i < kNumKeys; i++) {
batch.Put(Key1(i), "value for range 1 key");
ASSERT_OK(batch.Put(Key1(i), "value for range 1 key"));
}
ASSERT_OK(db->Write(WriteOptions(), &batch));
// create second key range
batch.Clear();
for (int i = 0; i < kNumKeys; i++) {
batch.Put(Key2(i), "value for range 2 key");
ASSERT_OK(batch.Put(Key2(i), "value for range 2 key"));
}
ASSERT_OK(db->Write(WriteOptions(), &batch));
// delete second key range
batch.Clear();
for (int i = 0; i < kNumKeys; i++) {
batch.Delete(Key2(i));
ASSERT_OK(batch.Delete(Key2(i)));
}
ASSERT_OK(db->Write(WriteOptions(), &batch));
@@ -160,7 +160,7 @@ TEST_F(ManualCompactionTest, Test) {
Slice greatest(end_key.data(), end_key.size());
// commenting out the line below causes the example to work correctly
db->CompactRange(CompactRangeOptions(), &least, &greatest);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &least, &greatest));
// count the keys
Iterator* iter = db->NewIterator(ReadOptions());
@@ -205,7 +205,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
Slice start("5");
Slice end("7");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, &end);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, &end));
ASSERT_EQ(0, filter->NumKeys());
}
@@ -215,7 +215,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
Slice start("3");
Slice end("7");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, &end);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, &end));
ASSERT_EQ(2, filter->NumKeys());
ASSERT_EQ(0, filter->KeyLevel("4"));
ASSERT_EQ(0, filter->KeyLevel("8"));
@@ -227,7 +227,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
// no file has keys in range (-inf, 0]
Slice end("0");
filter->Reset();
db->CompactRange(CompactRangeOptions(), nullptr, &end);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, &end));
ASSERT_EQ(0, filter->NumKeys());
}
@@ -237,7 +237,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
// no file has keys in range [9, inf)
Slice start("9");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, nullptr);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, nullptr));
ASSERT_EQ(0, filter->NumKeys());
}
@@ -248,7 +248,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
Slice start("2");
Slice end("2");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, &end);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, &end));
ASSERT_EQ(1, filter->NumKeys());
ASSERT_EQ(0, filter->KeyLevel("2"));
}
@@ -260,7 +260,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
Slice start("2");
Slice end("5");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, &end);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, &end));
ASSERT_EQ(3, filter->NumKeys());
ASSERT_EQ(1, filter->KeyLevel("2"));
ASSERT_EQ(1, filter->KeyLevel("4"));
@@ -273,7 +273,7 @@ TEST_F(ManualCompactionTest, SkipLevel) {
// [0, inf) overlaps all files
Slice start("0");
filter->Reset();
db->CompactRange(CompactRangeOptions(), &start, nullptr);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), &start, nullptr));
ASSERT_EQ(4, filter->NumKeys());
// 1 is first compacted to L1 and then further compacted into [2, 4, 8],
// so finally the logged level for 1 is L1.
+3 -3
View File
@@ -30,14 +30,14 @@ class MemTableListTest : public testing::Test {
MemTableListTest() : db(nullptr), file_number(1) {
dbname = test::PerThreadDBPath("memtable_list_test");
options.create_if_missing = true;
DestroyDB(dbname, options);
EXPECT_OK(DestroyDB(dbname, options));
}
// Create a test db if not yet created
void CreateDB() {
if (db == nullptr) {
options.create_if_missing = true;
DestroyDB(dbname, options);
EXPECT_OK(DestroyDB(dbname, options));
// Open DB only with default column family
ColumnFamilyOptions cf_options;
std::vector<ColumnFamilyDescriptor> cf_descs;
@@ -78,7 +78,7 @@ class MemTableListTest : public testing::Test {
handles.clear();
delete db;
db = nullptr;
DestroyDB(dbname, options, cf_descs);
EXPECT_OK(DestroyDB(dbname, options, cf_descs));
}
}
+41 -42
View File
@@ -80,8 +80,8 @@ std::shared_ptr<DB> OpenDb(const std::string& dbname, const bool ttl = false,
options.create_if_missing = true;
options.merge_operator = std::make_shared<CountMergeOperator>();
options.max_successive_merges = max_successive_merges;
EXPECT_OK(DestroyDB(dbname, Options()));
Status s;
DestroyDB(dbname, Options());
// DBWithTTL is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
if (ttl) {
@@ -95,10 +95,8 @@ std::shared_ptr<DB> OpenDb(const std::string& dbname, const bool ttl = false,
assert(!ttl);
s = DB::Open(options, dbname, &db);
#endif // !ROCKSDB_LITE
if (!s.ok()) {
std::cerr << s.ToString() << std::endl;
assert(false);
}
EXPECT_OK(s);
assert(s.ok());
return std::shared_ptr<DB>(db);
}
@@ -258,21 +256,25 @@ void testCounters(Counters& counters, DB* db, bool test_compaction) {
counters.assert_set("a", 1);
if (test_compaction) db->Flush(o);
if (test_compaction) {
ASSERT_OK(db->Flush(o));
}
assert(counters.assert_get("a") == 1);
ASSERT_EQ(counters.assert_get("a"), 1);
counters.assert_remove("b");
// defaut value is 0 if non-existent
assert(counters.assert_get("b") == 0);
ASSERT_EQ(counters.assert_get("b"), 0);
counters.assert_add("a", 2);
if (test_compaction) db->Flush(o);
if (test_compaction) {
ASSERT_OK(db->Flush(o));
}
// 1+2 = 3
assert(counters.assert_get("a")== 3);
ASSERT_EQ(counters.assert_get("a"), 3);
dumpDb(db);
@@ -282,19 +284,19 @@ void testCounters(Counters& counters, DB* db, bool test_compaction) {
counters.assert_add("b", i);
sum += i;
}
assert(counters.assert_get("b") == sum);
ASSERT_EQ(counters.assert_get("b"), sum);
dumpDb(db);
if (test_compaction) {
db->Flush(o);
ASSERT_OK(db->Flush(o));
db->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, nullptr));
dumpDb(db);
assert(counters.assert_get("a")== 3);
assert(counters.assert_get("b") == sum);
ASSERT_EQ(counters.assert_get("a"), 3);
ASSERT_EQ(counters.assert_get("b"), sum);
}
}
@@ -400,14 +402,14 @@ void testSuccessiveMerge(Counters& counters, size_t max_num_merges,
sum += i;
if (i % (max_num_merges + 1) == 0) {
assert(num_merge_operator_calls == max_num_merges + 1);
ASSERT_EQ(num_merge_operator_calls, max_num_merges + 1);
} else {
assert(num_merge_operator_calls == 0);
ASSERT_EQ(num_merge_operator_calls, 0);
}
resetNumMergeOperatorCalls();
assert(counters.assert_get("z") == sum);
assert(num_merge_operator_calls == i % (max_num_merges + 1));
ASSERT_EQ(counters.assert_get("z"), sum);
ASSERT_EQ(num_merge_operator_calls, i % (max_num_merges + 1));
}
}
@@ -424,8 +426,8 @@ void testPartialMerge(Counters* counters, DB* db, size_t max_merge,
counters->assert_add("b", i);
tmp_sum += i;
}
db->Flush(o);
db->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db->Flush(o));
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_EQ(tmp_sum, counters->assert_get("b"));
if (count > max_merge) {
// in this case, FullMerge should be called instead.
@@ -438,20 +440,20 @@ void testPartialMerge(Counters* counters, DB* db, size_t max_merge,
// Test case 2: partial merge should not be called when a put is found.
resetNumPartialMergeCalls();
tmp_sum = 0;
db->Put(ROCKSDB_NAMESPACE::WriteOptions(), "c", "10");
ASSERT_OK(db->Put(ROCKSDB_NAMESPACE::WriteOptions(), "c", "10"));
for (size_t i = 1; i <= count; i++) {
counters->assert_add("c", i);
tmp_sum += i;
}
db->Flush(o);
db->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db->Flush(o));
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_EQ(tmp_sum, counters->assert_get("c"));
ASSERT_EQ(num_partial_merge_calls, 0U);
}
void testSingleBatchSuccessiveMerge(DB* db, size_t max_num_merges,
size_t num_merges) {
assert(num_merges > max_num_merges);
ASSERT_GT(num_merges, max_num_merges);
Slice key("BatchSuccessiveMerge");
uint64_t merge_value = 1;
@@ -462,15 +464,12 @@ void testSingleBatchSuccessiveMerge(DB* db, size_t max_num_merges,
// Create the batch
WriteBatch batch;
for (size_t i = 0; i < num_merges; ++i) {
batch.Merge(key, merge_value_slice);
ASSERT_OK(batch.Merge(key, merge_value_slice));
}
// Apply to memtable and count the number of merges
resetNumMergeOperatorCalls();
{
Status s = db->Write(WriteOptions(), &batch);
assert(s.ok());
}
ASSERT_OK(db->Write(WriteOptions(), &batch));
ASSERT_EQ(
num_merge_operator_calls,
static_cast<size_t>(num_merges - (num_merges % (max_num_merges + 1))));
@@ -478,10 +477,7 @@ void testSingleBatchSuccessiveMerge(DB* db, size_t max_num_merges,
// Get the value
resetNumMergeOperatorCalls();
std::string get_value_str;
{
Status s = db->Get(ReadOptions(), key, &get_value_str);
assert(s.ok());
}
ASSERT_OK(db->Get(ReadOptions(), key, &get_value_str));
assert(get_value_str.size() == sizeof(uint64_t));
uint64_t get_value = DecodeFixed64(&get_value_str[0]);
ASSERT_EQ(get_value, num_merges * merge_value);
@@ -505,7 +501,7 @@ void runTest(const std::string& dbname, const bool use_ttl = false) {
}
}
DestroyDB(dbname, Options());
ASSERT_OK(DestroyDB(dbname, Options()));
{
size_t max_merge = 5;
@@ -514,7 +510,8 @@ void runTest(const std::string& dbname, const bool use_ttl = false) {
testCounters(counters, db.get(), use_compression);
testSuccessiveMerge(counters, max_merge, max_merge * 2);
testSingleBatchSuccessiveMerge(db.get(), 5, 7);
DestroyDB(dbname, Options());
ASSERT_OK(db->Close());
ASSERT_OK(DestroyDB(dbname, Options()));
}
{
@@ -525,14 +522,16 @@ void runTest(const std::string& dbname, const bool use_ttl = false) {
auto db = OpenDb(dbname, use_ttl, max_merge);
MergeBasedCounters counters(db, 0);
testPartialMerge(&counters, db.get(), max_merge, min_merge, count);
DestroyDB(dbname, Options());
ASSERT_OK(db->Close());
ASSERT_OK(DestroyDB(dbname, Options()));
}
{
auto db = OpenDb(dbname, use_ttl, max_merge);
MergeBasedCounters counters(db, 0);
testPartialMerge(&counters, db.get(), max_merge, min_merge,
min_merge * 10);
DestroyDB(dbname, Options());
ASSERT_OK(db->Close());
ASSERT_OK(DestroyDB(dbname, Options()));
}
}
@@ -543,15 +542,15 @@ void runTest(const std::string& dbname, const bool use_ttl = false) {
counters.add("test-key", 1);
counters.add("test-key", 1);
counters.add("test-key", 1);
db->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(db->CompactRange(CompactRangeOptions(), nullptr, nullptr));
}
DB* reopen_db;
ASSERT_OK(DB::Open(Options(), dbname, &reopen_db));
std::string value;
ASSERT_TRUE(!(reopen_db->Get(ReadOptions(), "test-key", &value).ok()));
ASSERT_NOK(reopen_db->Get(ReadOptions(), "test-key", &value));
delete reopen_db;
DestroyDB(dbname, Options());
ASSERT_OK(DestroyDB(dbname, Options()));
}
/* Temporary remove this test
@@ -591,7 +590,7 @@ TEST_F(MergeTest, MergeWithCompactionAndFlush) {
testCountersWithFlushAndCompaction(counters, db.get());
}
}
DestroyDB(dbname, Options());
ASSERT_OK(DestroyDB(dbname, Options()));
}
#endif // !ROCKSDB_LITE
+1 -1
View File
@@ -61,7 +61,7 @@ class ObsoleteFilesTest : public DBTestBase {
void CheckFileTypeCounts(const std::string& dir, int required_log,
int required_sst, int required_manifest) {
std::vector<std::string> filenames;
env_->GetChildren(dir, &filenames);
ASSERT_OK(env_->GetChildren(dir, &filenames));
int log_cnt = 0;
int sst_cnt = 0;
+1 -1
View File
@@ -65,7 +65,7 @@ TEST_F(OptionsFileTest, NumberOfOptionsFiles) {
const int kReopenCount = 20;
Options opt;
opt.create_if_missing = true;
DestroyDB(dbname_, opt);
ASSERT_OK(DestroyDB(dbname_, opt));
std::unordered_set<std::string> filename_history;
DB* db;
for (int i = 0; i < kReopenCount; ++i) {
+25 -18
View File
@@ -76,12 +76,12 @@ TEST_F(PerfContextTest, SeekIntoDeletion) {
std::string key = "k" + ToString(i);
std::string value = "v" + ToString(i);
db->Put(write_options, key, value);
ASSERT_OK(db->Put(write_options, key, value));
}
for (int i = 0; i < FLAGS_total_keys -1 ; ++i) {
std::string key = "k" + ToString(i);
db->Delete(write_options, key);
ASSERT_OK(db->Delete(write_options, key));
}
HistogramImpl hist_get;
@@ -116,10 +116,9 @@ TEST_F(PerfContextTest, SeekIntoDeletion) {
auto elapsed_nanos = timer.ElapsedNanos();
if (FLAGS_verbose) {
std::cout << "SeekToFirst uesr key comparison: \n"
<< hist_seek_to_first.ToString()
<< "ikey skipped: " << get_perf_context()->internal_key_skipped_count
<< "\n"
std::cout << "SeekToFirst user key comparison: \n"
<< hist_seek_to_first.ToString() << "ikey skipped: "
<< get_perf_context()->internal_key_skipped_count << "\n"
<< "idelete skipped: "
<< get_perf_context()->internal_delete_skipped_count << "\n"
<< "elapsed: " << elapsed_nanos << "\n";
@@ -156,7 +155,7 @@ TEST_F(PerfContextTest, SeekIntoDeletion) {
}
if (FLAGS_verbose) {
std::cout << "Seek uesr key comparison: \n" << hist_seek.ToString();
std::cout << "Seek user key comparison: \n" << hist_seek.ToString();
}
}
@@ -270,7 +269,7 @@ void ProfileQueries(bool enabled_time = false) {
std::vector<std::string> values;
get_perf_context()->Reset();
db->Put(write_options, key, value);
ASSERT_OK(db->Put(write_options, key, value));
if (++num_mutex_waited > 3) {
#ifndef NDEBUG
ThreadStatusUtil::TEST_SetStateDelay(ThreadStatus::STATE_MUTEX_WAIT, 0U);
@@ -314,7 +313,10 @@ void ProfileQueries(bool enabled_time = false) {
hist_get.Add(get_perf_context()->user_key_comparison_count);
get_perf_context()->Reset();
db->MultiGet(read_options, multiget_keys, &values);
auto statuses = db->MultiGet(read_options, multiget_keys, &values);
for (const auto& s : statuses) {
ASSERT_OK(s);
}
hist_mget_snapshot.Add(get_perf_context()->get_snapshot_time);
hist_mget_memtable.Add(get_perf_context()->get_from_memtable_time);
hist_mget_files.Add(get_perf_context()->get_from_output_files_time);
@@ -324,9 +326,10 @@ void ProfileQueries(bool enabled_time = false) {
}
if (FLAGS_verbose) {
std::cout << "Put uesr key comparison: \n" << hist_put.ToString()
<< "Get uesr key comparison: \n" << hist_get.ToString()
<< "MultiGet uesr key comparison: \n" << hist_get.ToString();
std::cout << "Put user key comparison: \n"
<< hist_put.ToString() << "Get user key comparison: \n"
<< hist_get.ToString() << "MultiGet user key comparison: \n"
<< hist_get.ToString();
std::cout << "Put(): Pre and Post Process Time: \n"
<< hist_write_pre_post.ToString() << " Writing WAL time: \n"
<< hist_write_wal_time.ToString() << "\n"
@@ -428,7 +431,10 @@ void ProfileQueries(bool enabled_time = false) {
hist_get.Add(get_perf_context()->user_key_comparison_count);
get_perf_context()->Reset();
db->MultiGet(read_options, multiget_keys, &values);
auto statuses = db->MultiGet(read_options, multiget_keys, &values);
for (const auto& s : statuses) {
ASSERT_OK(s);
}
hist_mget_snapshot.Add(get_perf_context()->get_snapshot_time);
hist_mget_memtable.Add(get_perf_context()->get_from_memtable_time);
hist_mget_files.Add(get_perf_context()->get_from_output_files_time);
@@ -438,8 +444,9 @@ void ProfileQueries(bool enabled_time = false) {
}
if (FLAGS_verbose) {
std::cout << "ReadOnly Get uesr key comparison: \n" << hist_get.ToString()
<< "ReadOnly MultiGet uesr key comparison: \n"
std::cout << "ReadOnly Get user key comparison: \n"
<< hist_get.ToString()
<< "ReadOnly MultiGet user key comparison: \n"
<< hist_mget.ToString();
std::cout << "ReadOnly Get(): Time to get snapshot: \n"
@@ -539,7 +546,7 @@ TEST_F(PerfContextTest, SeekKeyComparison) {
get_perf_context()->Reset();
timer.Start();
db->Put(write_options, key, value);
ASSERT_OK(db->Put(write_options, key, value));
auto put_time = timer.ElapsedNanos();
hist_put_time.Add(put_time);
hist_wal_time.Add(get_perf_context()->write_wal_time);
@@ -573,7 +580,7 @@ TEST_F(PerfContextTest, SeekKeyComparison) {
iter->Next();
hist_next.Add(get_perf_context()->user_key_comparison_count);
}
ASSERT_OK(iter->status());
if (FLAGS_verbose) {
std::cout << "Seek:\n" << hist_seek.ToString() << "Next:\n"
<< hist_next.ToString();
@@ -835,7 +842,7 @@ TEST_F(PerfContextTest, CPUTimer) {
std::string value = "v" + i_str;
max_str = max_str > i_str ? max_str : i_str;
db->Put(write_options, key, value);
ASSERT_OK(db->Put(write_options, key, value));
}
std::string last_key = "k" + max_str;
std::string last_value = "v" + max_str;
+2 -2
View File
@@ -185,7 +185,7 @@ TEST_F(PeriodicWorkSchedulerTest, MultiInstances) {
ASSERT_EQ(expected_run, pst_st_counter);
for (int i = half; i < kInstanceNum; i++) {
dbs[i]->Close();
ASSERT_OK(dbs[i]->Close());
delete dbs[i];
}
}
@@ -217,7 +217,7 @@ TEST_F(PeriodicWorkSchedulerTest, MultiEnv) {
ASSERT_EQ(dbi->TEST_GetPeriodicWorkScheduler(),
dbfull()->TEST_GetPeriodicWorkScheduler());
db->Close();
ASSERT_OK(db->Close());
delete db;
Close();
}
+8 -8
View File
@@ -311,7 +311,7 @@ TEST(SamePrefixTest, InDomainTest) {
ASSERT_OK(db->Put(write_options, "HHKB pro2", "Mar 24, 2006"));
ASSERT_OK(db->Put(write_options, "HHKB pro2 Type-S", "June 29, 2011"));
ASSERT_OK(db->Put(write_options, "Realforce 87u", "idk"));
db->Flush(FlushOptions());
ASSERT_OK(db->Flush(FlushOptions()));
std::string result;
auto db_iter = db->NewIterator(ReadOptions());
@@ -331,7 +331,7 @@ TEST(SamePrefixTest, InDomainTest) {
ASSERT_OK(db->Put(write_options, "pikachu", "1"));
ASSERT_OK(db->Put(write_options, "Meowth", "1"));
ASSERT_OK(db->Put(write_options, "Mewtwo", "idk"));
db->Flush(FlushOptions());
ASSERT_OK(db->Flush(FlushOptions()));
std::string result;
auto db_iter = db->NewIterator(ReadOptions());
@@ -351,7 +351,7 @@ TEST_F(PrefixTest, TestResult) {
std::cout << "*** Mem table: " << options.memtable_factory->Name()
<< " number of buckets: " << num_buckets
<< std::endl;
DestroyDB(kDbName, Options());
ASSERT_OK(DestroyDB(kDbName, Options()));
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -528,7 +528,7 @@ TEST_F(PrefixTest, PrefixValid) {
while (NextOptions(num_buckets)) {
std::cout << "*** Mem table: " << options.memtable_factory->Name()
<< " number of buckets: " << num_buckets << std::endl;
DestroyDB(kDbName, Options());
ASSERT_OK(DestroyDB(kDbName, Options()));
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -543,7 +543,7 @@ TEST_F(PrefixTest, PrefixValid) {
PutKey(db.get(), write_options, 12345, 8, v18);
PutKey(db.get(), write_options, 12345, 9, v19);
PutKey(db.get(), write_options, 12346, 8, v16);
db->Flush(FlushOptions());
ASSERT_OK(db->Flush(FlushOptions()));
TestKey test_key(12346, 8);
std::string s;
ASSERT_OK(db->Delete(write_options, TestKeyToSlice(s, test_key)));
@@ -581,7 +581,7 @@ TEST_F(PrefixTest, DynamicPrefixIterator) {
while (NextOptions(FLAGS_bucket_count)) {
std::cout << "*** Mem table: " << options.memtable_factory->Name()
<< std::endl;
DestroyDB(kDbName, Options());
ASSERT_OK(DestroyDB(kDbName, Options()));
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -689,7 +689,7 @@ TEST_F(PrefixTest, PrefixSeekModePrev) {
for (size_t m = 1; m < 100; m++) {
std::cout << "[" + std::to_string(m) + "]" + "*** Mem table: "
<< options.memtable_factory->Name() << std::endl;
DestroyDB(kDbName, Options());
ASSERT_OK(DestroyDB(kDbName, Options()));
auto db = OpenDb();
WriteOptions write_options;
ReadOptions read_options;
@@ -714,7 +714,7 @@ TEST_F(PrefixTest, PrefixSeekModePrev) {
}
}
if (i < 2) {
db->Flush(FlushOptions());
ASSERT_OK(db->Flush(FlushOptions()));
}
}
+3 -2
View File
@@ -34,7 +34,7 @@ TransactionLogIteratorImpl::TransactionLogIteratorImpl(
io_tracer_(io_tracer) {
assert(files_ != nullptr);
assert(versions_ != nullptr);
current_status_.PermitUncheckedError(); // Clear on start
reporter_.env = options_->env;
reporter_.info_log = options_->info_log.get();
SeekToStartSequence(); // Seek till starting sequence
@@ -225,7 +225,8 @@ bool TransactionLogIteratorImpl::IsBatchExpected(
void TransactionLogIteratorImpl::UpdateCurrentWriteBatch(const Slice& record) {
std::unique_ptr<WriteBatch> batch(new WriteBatch());
WriteBatchInternal::SetContents(batch.get(), record);
Status s = WriteBatchInternal::SetContents(batch.get(), record);
s.PermitUncheckedError(); // TODO: What should we do with this error?
SequenceNumber expected_seq = current_last_seq_ + 1;
// If the iterator has started, then confirm that we get continuous batches
+3 -3
View File
@@ -1231,7 +1231,7 @@ TEST_F(VersionSetTest, WalEditsNotAppliedToVersion) {
[&](void* arg) { versions.push_back(reinterpret_cast<Version*>(arg)); });
SyncPoint::GetInstance()->EnableProcessing();
LogAndApplyToDefaultCF(edits);
ASSERT_OK(LogAndApplyToDefaultCF(edits));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -1267,7 +1267,7 @@ TEST_F(VersionSetTest, NonWalEditsAppliedToVersion) {
[&](void* arg) { versions.push_back(reinterpret_cast<Version*>(arg)); });
SyncPoint::GetInstance()->EnableProcessing();
LogAndApplyToDefaultCF(edits);
ASSERT_OK(LogAndApplyToDefaultCF(edits));
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -1674,7 +1674,7 @@ TEST_F(VersionSetTest, AtomicGroupWithWalEdits) {
edits.back()->MarkAtomicGroup(--remaining);
ASSERT_EQ(remaining, 0);
Status s = LogAndApplyToDefaultCF(edits);
ASSERT_OK(LogAndApplyToDefaultCF(edits));
// Recover a new VersionSet, the min log number and the last WAL should be
// kept.
+7 -3
View File
@@ -243,9 +243,13 @@ void WalManager::PurgeObsoleteWALFiles() {
size_t files_del_num = log_files_num - files_keep_num;
VectorLogPtr archived_logs;
GetSortedWalsOfType(archival_dir, archived_logs, kArchivedLogFile);
if (files_del_num > archived_logs.size()) {
s = GetSortedWalsOfType(archival_dir, archived_logs, kArchivedLogFile);
if (!s.ok()) {
ROCKS_LOG_WARN(db_options_.info_log,
"Unable to get archived WALs from: %s: %s",
archival_dir.c_str(), s.ToString().c_str());
files_del_num = 0;
} else if (files_del_num > archived_logs.size()) {
ROCKS_LOG_WARN(db_options_.info_log,
"Trying to delete more archived log files than "
"exist. Deleting all");
+9 -7
View File
@@ -69,9 +69,10 @@ class WalManagerTest : public testing::Test {
assert(current_log_writer_.get() != nullptr);
uint64_t seq = versions_->LastSequence() + 1;
WriteBatch batch;
batch.Put(key, value);
ASSERT_OK(batch.Put(key, value));
WriteBatchInternal::SetSequence(&batch, seq);
current_log_writer_->AddRecord(WriteBatchInternal::Contents(&batch));
ASSERT_OK(
current_log_writer_->AddRecord(WriteBatchInternal::Contents(&batch)));
versions_->SetLastAllocatedSequence(seq);
versions_->SetLastPublishedSequence(seq);
versions_->SetLastSequence(seq);
@@ -140,9 +141,9 @@ TEST_F(WalManagerTest, ReadFirstRecordCache) {
log::Writer writer(std::move(file_writer), 1,
db_options_.recycle_log_file_num > 0);
WriteBatch batch;
batch.Put("foo", "bar");
ASSERT_OK(batch.Put("foo", "bar"));
WriteBatchInternal::SetSequence(&batch, 10);
writer.AddRecord(WriteBatchInternal::Contents(&batch));
ASSERT_OK(writer.AddRecord(WriteBatchInternal::Contents(&batch)));
// TODO(icanadi) move SpecialEnv outside of db_test, so we can reuse it here.
// Waiting for lei to finish with db_test
@@ -167,14 +168,14 @@ namespace {
uint64_t GetLogDirSize(std::string dir_path, Env* env) {
uint64_t dir_size = 0;
std::vector<std::string> files;
env->GetChildren(dir_path, &files);
EXPECT_OK(env->GetChildren(dir_path, &files));
for (auto& f : files) {
uint64_t number;
FileType type;
if (ParseFileName(f, &number, &type) && type == kWalFile) {
std::string const file_path = dir_path + "/" + f;
uint64_t file_size;
env->GetFileSize(file_path, &file_size);
EXPECT_OK(env->GetFileSize(file_path, &file_size));
dir_size += file_size;
}
}
@@ -184,9 +185,9 @@ std::vector<std::uint64_t> ListSpecificFiles(
Env* env, const std::string& path, const FileType expected_file_type) {
std::vector<std::string> files;
std::vector<uint64_t> file_numbers;
env->GetChildren(path, &files);
uint64_t number;
FileType type;
EXPECT_OK(env->GetChildren(path, &files));
for (size_t i = 0; i < files.size(); ++i) {
if (ParseFileName(files[i], &number, &type)) {
if (type == expected_file_type) {
@@ -209,6 +210,7 @@ int CountRecords(TransactionLogIterator* iter) {
EXPECT_OK(iter->status());
iter->Next();
}
EXPECT_OK(iter->status());
return count;
}
} // namespace
+6 -6
View File
@@ -111,7 +111,7 @@ TEST_P(WriteCallbackPTest, WriteWithCallbackTest) {
void Put(const string& key, const string& val) {
kvs_.push_back(std::make_pair(key, val));
write_batch_.Put(key, val);
ASSERT_OK(write_batch_.Put(key, val));
}
void Clear() {
@@ -319,7 +319,7 @@ TEST_P(WriteCallbackPTest, WriteWithCallbackTest) {
DBImpl* db_impl_;
} publish_seq_callback(db_impl);
// seq_per_batch_ requires a natural batch separator or Noop
WriteBatchInternal::InsertNoop(&write_op.write_batch_);
ASSERT_OK(WriteBatchInternal::InsertNoop(&write_op.write_batch_));
const size_t ONE_BATCH = 1;
s = db_impl->WriteImpl(woptions, &write_op.write_batch_,
&write_op.callback_, nullptr, 0, false, nullptr,
@@ -396,8 +396,8 @@ TEST_F(WriteCallbackTest, WriteCallBackTest) {
WriteBatch wb;
wb.Put("a", "value.a");
wb.Delete("x");
ASSERT_OK(wb.Put("a", "value.a"));
ASSERT_OK(wb.Delete("x"));
// Test a simple Write
s = db->Write(write_options, &wb);
@@ -411,7 +411,7 @@ TEST_F(WriteCallbackTest, WriteCallBackTest) {
WriteCallbackTestWriteCallback1 callback1;
WriteBatch wb2;
wb2.Put("a", "value.a2");
ASSERT_OK(wb2.Put("a", "value.a2"));
s = db_impl->WriteWithCallback(write_options, &wb2, &callback1);
ASSERT_OK(s);
@@ -425,7 +425,7 @@ TEST_F(WriteCallbackTest, WriteCallBackTest) {
WriteCallbackTestWriteCallback2 callback2;
WriteBatch wb3;
wb3.Put("a", "value.a3");
ASSERT_OK(wb3.Put("a", "value.a3"));
s = db_impl->WriteWithCallback(write_options, &wb3, &callback2);
ASSERT_NOK(s);
+2 -2
View File
@@ -1505,11 +1505,11 @@ Status StressTest::TestCheckpoint(ThreadState* thread,
}
}
}
delete checkpoint;
checkpoint = nullptr;
std::vector<ColumnFamilyHandle*> cf_handles;
DB* checkpoint_db = nullptr;
if (s.ok()) {
delete checkpoint;
checkpoint = nullptr;
Options options(options_);
options.listeners.clear();
std::vector<ColumnFamilyDescriptor> cf_descs;
+1 -7
View File
@@ -69,10 +69,4 @@ if (s.ok()) s = db->Delete(rocksdb::WriteOptions(), key1);
## Further documentation
These are just simple examples of how RocksDB is used. The full documentation is currently on the [GitHub wiki](https://github.com/facebook/rocksdb/wiki).
Here are some specific details about the RocksDB implementation:
- [Architecture Guide](https://github.com/facebook/rocksdb/wiki/Rocksdb-Architecture-Guide)
- [Format of an immutable Table file](https://github.com/facebook/rocksdb/wiki/Rocksdb-Table-Format)
- [Format of a log file](https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-File-Format)
These are just simple examples of how RocksDB is used. The full documentation is currently on the [GitHub wiki](https://github.com/facebook/rocksdb/wiki).
+1 -1
View File
@@ -325,7 +325,7 @@ TEST_P(EnvMoreTestWithParam, MakeDir) {
ASSERT_OK(env_->CreateDir(test_dir_ + "/j"));
ASSERT_OK(env_->FileExists(test_dir_ + "/j"));
std::vector<std::string> children;
env_->GetChildren(test_dir_, &children);
ASSERT_OK(env_->GetChildren(test_dir_, &children));
ASSERT_EQ(1U, children.size());
// fail because file already exists
ASSERT_TRUE(!env_->CreateDir(test_dir_ + "/j").ok());
+1 -1
View File
@@ -2166,7 +2166,7 @@ TEST_P(EnvFSTestWithParam, OptionsTest) {
ASSERT_OK(db->Close());
delete db;
DestroyDB(dbname, opts);
ASSERT_OK(DestroyDB(dbname, opts));
dbname = dbname2_;
}
+11 -7
View File
@@ -56,14 +56,13 @@ DeleteScheduler::~DeleteScheduler() {
Status DeleteScheduler::DeleteFile(const std::string& file_path,
const std::string& dir_to_sync,
const bool force_bg) {
Status s;
if (rate_bytes_per_sec_.load() <= 0 || (!force_bg &&
total_trash_size_.load() >
sst_file_manager_->GetTotalSize() * max_trash_db_ratio_.load())) {
// Rate limiting is disabled or trash size makes up more than
// max_trash_db_ratio_ (default 25%) of the total DB size
TEST_SYNC_POINT("DeleteScheduler::DeleteFile");
s = fs_->DeleteFile(file_path, IOOptions(), nullptr);
Status s = fs_->DeleteFile(file_path, IOOptions(), nullptr);
if (s.ok()) {
s = sst_file_manager_->OnDeleteFile(file_path);
ROCKS_LOG_INFO(info_log_,
@@ -79,7 +78,7 @@ Status DeleteScheduler::DeleteFile(const std::string& file_path,
// Move file to trash
std::string trash_file;
s = MarkAsTrash(file_path, &trash_file);
Status s = MarkAsTrash(file_path, &trash_file);
ROCKS_LOG_INFO(info_log_, "Mark file: %s as trash -- %s", trash_file.c_str(),
s.ToString().c_str());
@@ -99,7 +98,10 @@ Status DeleteScheduler::DeleteFile(const std::string& file_path,
// Update the total trash size
uint64_t trash_file_size = 0;
fs_->GetFileSize(trash_file, IOOptions(), &trash_file_size, nullptr);
Status ignored =
fs_->GetFileSize(trash_file, IOOptions(), &trash_file_size, nullptr);
ignored.PermitUncheckedError(); //**TODO: What should we do if we failed to
// get the file size?
total_trash_size_.fetch_add(trash_file_size);
// Add file to delete queue
@@ -169,17 +171,17 @@ Status DeleteScheduler::MarkAsTrash(const std::string& file_path,
return Status::InvalidArgument("file_path is corrupted");
}
Status s;
if (DeleteScheduler::IsTrashFile(file_path)) {
// This is already a trash file
*trash_file = file_path;
return s;
return Status::OK();
}
*trash_file = file_path + kTrashExtension;
// TODO(tec) : Implement Env::RenameFileIfNotExist and remove
// file_move_mu mutex.
int cnt = 0;
Status s;
InstrumentedMutexLock l(&file_move_mu_);
while (true) {
s = fs_->FileExists(*trash_file, IOOptions(), nullptr);
@@ -197,7 +199,9 @@ Status DeleteScheduler::MarkAsTrash(const std::string& file_path,
cnt++;
}
if (s.ok()) {
sst_file_manager_->OnMoveFile(file_path, *trash_file);
//**TODO: What should we do if this returns an error?
sst_file_manager_->OnMoveFile(file_path, *trash_file)
.PermitUncheckedError();
}
return s;
}
+5 -1
View File
@@ -109,7 +109,8 @@ Status FilePrefetchBuffer::Prefetch(const IOOptions& opts,
bool FilePrefetchBuffer::TryReadFromCache(const IOOptions& opts,
uint64_t offset, size_t n,
Slice* result, bool for_compaction) {
Slice* result, Status* status,
bool for_compaction) {
if (track_min_offset_ && offset < min_offset_read_) {
min_offset_read_ = static_cast<size_t>(offset);
}
@@ -134,6 +135,9 @@ bool FilePrefetchBuffer::TryReadFromCache(const IOOptions& opts,
for_compaction);
}
if (!s.ok()) {
if (status) {
*status = s;
}
#ifndef NDEBUG
IGNORE_STATUS_IF_ERROR(s);
#endif
+1 -1
View File
@@ -75,7 +75,7 @@ class FilePrefetchBuffer {
// result : output buffer to put the data into.
// for_compaction : if cache read is done for compaction read.
bool TryReadFromCache(const IOOptions& opts, uint64_t offset, size_t n,
Slice* result, bool for_compaction = false);
Slice* result, Status* s, bool for_compaction = false);
// The minimum `offset` ever passed to TryReadFromCache(). This will nly be
// tracked if track_min_offset = true.
+4 -3
View File
@@ -132,7 +132,7 @@ IOStatus GenerateOneFileChecksum(
const std::string& requested_checksum_func_name, std::string* file_checksum,
std::string* file_checksum_func_name,
size_t verify_checksums_readahead_size, bool allow_mmap_reads,
std::shared_ptr<IOTracer>& io_tracer) {
std::shared_ptr<IOTracer>& io_tracer, RateLimiter* rate_limiter) {
if (checksum_factory == nullptr) {
return IOStatus::InvalidArgument("Checksum factory is invalid");
}
@@ -173,7 +173,8 @@ IOStatus GenerateOneFileChecksum(
return io_s;
}
reader.reset(new RandomAccessFileReader(std::move(r_file), file_path,
nullptr /*Env*/, io_tracer));
nullptr /*Env*/, io_tracer, nullptr,
0, nullptr, rate_limiter));
}
// Found that 256 KB readahead size provides the best performance, based on
@@ -194,7 +195,7 @@ IOStatus GenerateOneFileChecksum(
size_t bytes_to_read =
static_cast<size_t>(std::min(uint64_t{readahead_size}, size));
if (!prefetch_buffer.TryReadFromCache(opts, offset, bytes_to_read, &slice,
false)) {
nullptr, false)) {
return IOStatus::Corruption("file read failed");
}
if (slice.size() == 0) {
+1 -1
View File
@@ -39,7 +39,7 @@ extern IOStatus GenerateOneFileChecksum(
const std::string& requested_checksum_func_name, std::string* file_checksum,
std::string* file_checksum_func_name,
size_t verify_checksums_readahead_size, bool allow_mmap_reads,
std::shared_ptr<IOTracer>& io_tracer);
std::shared_ptr<IOTracer>& io_tracer, RateLimiter* rate_limiter = nullptr);
inline IOStatus PrepareIOFromReadOptions(const ReadOptions& ro, Env* env,
IOOptions& opts) {
+4 -4
View File
@@ -109,21 +109,21 @@ TEST_P(PrefetchTest, Basic) {
// create first key range
WriteBatch batch;
for (int i = 0; i < kNumKeys; i++) {
batch.Put(BuildKey(i), "value for range 1 key");
ASSERT_OK(batch.Put(BuildKey(i), "value for range 1 key"));
}
ASSERT_OK(db_->Write(WriteOptions(), &batch));
// create second key range
batch.Clear();
for (int i = 0; i < kNumKeys; i++) {
batch.Put(BuildKey(i, "key2"), "value for range 2 key");
ASSERT_OK(batch.Put(BuildKey(i, "key2"), "value for range 2 key"));
}
ASSERT_OK(db_->Write(WriteOptions(), &batch));
// delete second key range
batch.Clear();
for (int i = 0; i < kNumKeys; i++) {
batch.Delete(BuildKey(i, "key2"));
ASSERT_OK(batch.Delete(BuildKey(i, "key2")));
}
ASSERT_OK(db_->Write(WriteOptions(), &batch));
@@ -134,7 +134,7 @@ TEST_P(PrefetchTest, Basic) {
Slice greatest(end_key.data(), end_key.size());
// commenting out the line below causes the example to work correctly
db_->CompactRange(CompactRangeOptions(), &least, &greatest);
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), &least, &greatest));
if (support_prefetch && !use_direct_io) {
// If underline file system supports prefetch, and directIO is not enabled
+11 -3
View File
@@ -221,11 +221,19 @@ Status RandomAccessFileReader::MultiRead(const IOOptions& opts,
aligned_reqs.reserve(num_reqs);
// Align and merge the read requests.
size_t alignment = file_->GetRequiredBufferAlignment();
aligned_reqs.push_back(Align(read_reqs[0], alignment));
for (size_t i = 1; i < num_reqs; i++) {
for (size_t i = 0; i < num_reqs; i++) {
const auto& r = Align(read_reqs[i], alignment);
if (!TryMerge(&aligned_reqs.back(), r)) {
if (i == 0) {
// head
aligned_reqs.push_back(r);
} else if (!TryMerge(&aligned_reqs.back(), r)) {
// head + n
aligned_reqs.push_back(r);
} else {
// unused
r.status.PermitUncheckedError();
}
}
TEST_SYNC_POINT_CALLBACK("RandomAccessFileReader::MultiRead:AlignedReqs",
+48 -4
View File
@@ -145,6 +145,7 @@ TEST_F(RandomAccessFileReaderTest, MultiReadDirectIO) {
// Reads the first page internally.
ASSERT_EQ(aligned_reqs.size(), 1);
const FSReadRequest& aligned_r = aligned_reqs[0];
ASSERT_OK(aligned_r.status);
ASSERT_EQ(aligned_r.offset, 0);
ASSERT_EQ(aligned_r.len, page_size);
}
@@ -189,6 +190,7 @@ TEST_F(RandomAccessFileReaderTest, MultiReadDirectIO) {
// Reads the first two pages in one request internally.
ASSERT_EQ(aligned_reqs.size(), 1);
const FSReadRequest& aligned_r = aligned_reqs[0];
ASSERT_OK(aligned_r.status);
ASSERT_EQ(aligned_r.offset, 0);
ASSERT_EQ(aligned_r.len, 2 * page_size);
}
@@ -233,6 +235,7 @@ TEST_F(RandomAccessFileReaderTest, MultiReadDirectIO) {
// Reads the first 3 pages in one request internally.
ASSERT_EQ(aligned_reqs.size(), 1);
const FSReadRequest& aligned_r = aligned_reqs[0];
ASSERT_OK(aligned_r.status);
ASSERT_EQ(aligned_r.offset, 0);
ASSERT_EQ(aligned_r.len, 3 * page_size);
}
@@ -270,8 +273,10 @@ TEST_F(RandomAccessFileReaderTest, MultiReadDirectIO) {
ASSERT_EQ(aligned_reqs.size(), 2);
const FSReadRequest& aligned_r0 = aligned_reqs[0];
const FSReadRequest& aligned_r1 = aligned_reqs[1];
ASSERT_OK(aligned_r0.status);
ASSERT_EQ(aligned_r0.offset, 0);
ASSERT_EQ(aligned_r0.len, page_size);
ASSERT_OK(aligned_r1.status);
ASSERT_EQ(aligned_r1.offset, 2 * page_size);
ASSERT_EQ(aligned_r1.len, page_size);
}
@@ -287,8 +292,11 @@ TEST(FSReadRequest, Align) {
r.offset = 2000;
r.len = 2000;
r.scratch = nullptr;
ASSERT_OK(r.status);
FSReadRequest aligned_r = Align(r, 1024);
ASSERT_OK(r.status);
ASSERT_OK(aligned_r.status);
ASSERT_EQ(aligned_r.offset, 1024);
ASSERT_EQ(aligned_r.len, 3072);
}
@@ -303,14 +311,20 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 15;
src.len = 10;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
if (reverse) {
std::swap(dest, src);
}
ASSERT_FALSE(TryMerge(&dest, src));
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -320,16 +334,22 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 10;
src.len = 10;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
if (reverse) {
std::swap(dest, src);
}
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 20);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -339,16 +359,22 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 5;
src.len = 10;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
if (reverse) {
std::swap(dest, src);
}
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 15);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -358,16 +384,22 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 5;
src.len = 5;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
if (reverse) {
std::swap(dest, src);
}
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 10);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -377,16 +409,20 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 5;
src.len = 1;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 10);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -396,16 +432,20 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 0;
src.len = 10;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 10);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
{
@@ -415,16 +455,20 @@ TEST(FSReadRequest, TryMerge) {
dest.offset = 0;
dest.len = 10;
dest.scratch = nullptr;
ASSERT_OK(dest.status);
FSReadRequest src;
src.offset = 0;
src.len = 5;
src.scratch = nullptr;
ASSERT_OK(src.status);
if (reverse) std::swap(dest, src);
ASSERT_TRUE(TryMerge(&dest, src));
ASSERT_EQ(dest.offset, 0);
ASSERT_EQ(dest.len, 10);
ASSERT_OK(dest.status);
ASSERT_OK(src.status);
}
}
}
+1 -1
View File
@@ -1981,7 +1981,7 @@ extern ROCKSDB_LIBRARY_API char* rocksdb_transaction_get_for_update(
const char* key, size_t klen, size_t* vlen, unsigned char exclusive,
char** errptr);
char* rocksdb_transaction_get_for_update_cf(
extern ROCKSDB_LIBRARY_API char* rocksdb_transaction_get_for_update_cf(
rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
rocksdb_column_family_handle_t* column_family, const char* key, size_t klen,
size_t* vlen, unsigned char exclusive, char** errptr);
+5 -1
View File
@@ -86,7 +86,11 @@ struct RangeDeadlockPath {
// RocksDB
class RangeLockManagerHandle : public LockManagerHandle {
public:
// Total amount of lock memory to use (per column family)
// Set total amount of lock memory to use.
//
// @return 0 Ok
// @return EDOM Failed to set because currently using more memory than
// specified
virtual int SetMaxLockMemory(size_t max_lock_memory) = 0;
virtual size_t GetMaxLockMemory() = 0;
+3 -3
View File
@@ -39,7 +39,7 @@ class EnvLogger : public Logger {
~EnvLogger() {
if (!closed_) {
closed_ = true;
CloseHelper();
CloseHelper().PermitUncheckedError();
}
}
@@ -48,7 +48,7 @@ class EnvLogger : public Logger {
mutex_.AssertHeld();
if (flush_pending_) {
flush_pending_ = false;
file_.Flush();
file_.Flush().PermitUncheckedError();
}
last_flush_micros_ = env_->NowMicros();
}
@@ -134,7 +134,7 @@ class EnvLogger : public Logger {
assert(p <= limit);
mutex_.Lock();
// We will ignore any error returned by Append().
file_.Append(Slice(base, p - base));
file_.Append(Slice(base, p - base)).PermitUncheckedError();
flush_pending_ = true;
const uint64_t now_micros = env_->NowMicros();
if (now_micros - last_flush_micros_ >= flush_every_seconds_ * 1000000) {
+1 -1
View File
@@ -553,7 +553,7 @@ TEST_F(StatsHistoryTest, PersistentStatsReadOnly) {
// Reopen and flush memtable.
ASSERT_OK(TryReopen(options));
Flush();
ASSERT_OK(Flush());
Close();
// Now check keys in read only mode.
ASSERT_OK(ReadOnlyReopen(options));
+3
View File
@@ -267,6 +267,8 @@ LIB_SOURCES = \
utilities/transactions/lock/range/range_tree/lib/standalone_port.cc \
utilities/transactions/lock/range/range_tree/lib/util/dbt.cc \
utilities/transactions/lock/range/range_tree/lib/util/memarena.cc \
utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc \
utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc \
utilities/transactions/optimistic_transaction.cc \
utilities/transactions/optimistic_transaction_db_impl.cc \
utilities/transactions/pessimistic_transaction.cc \
@@ -536,6 +538,7 @@ TEST_MAIN_SOURCES = \
utilities/simulator_cache/sim_cache_test.cc \
utilities/table_properties_collectors/compact_on_deletion_collector_test.cc \
utilities/transactions/optimistic_transaction_test.cc \
utilities/transactions/lock/range/range_locking_test.cc \
utilities/transactions/transaction_test.cc \
utilities/transactions/lock/point/point_lock_manager_test.cc \
utilities/transactions/write_prepared_transaction_test.cc \
@@ -1768,6 +1768,9 @@ void BlockBasedTable::RetrieveMultipleBlocks(
size_t& req_idx = req_idx_for_block[valid_batch_idx];
size_t& req_offset = req_offset_for_block[valid_batch_idx];
valid_batch_idx++;
if (mget_iter->get_context) {
++(mget_iter->get_context->get_context_stats_.num_data_read);
}
FSReadRequest& req = read_reqs[req_idx];
Status s = req.status;
if (s.ok()) {
@@ -579,13 +579,13 @@ void TestBoundary(InternalKey& ik1, std::string& v1, InternalKey& ik2,
0 /*uniq_id*/, ioptions.allow_mmap_reads)));
const bool kSkipFilters = true;
const bool kImmortal = true;
ioptions.table_factory->NewTableReader(
ASSERT_OK(ioptions.table_factory->NewTableReader(
TableReaderOptions(ioptions, moptions.prefix_extractor.get(), soptions,
internal_comparator, !kSkipFilters, !kImmortal,
level_),
std::move(file_reader),
test::GetStringSinkFromLegacyWriter(file_writer.get())->contents().size(),
&table_reader);
&table_reader));
// Search using Get()
ReadOptions ro;
+2
View File
@@ -63,6 +63,8 @@ class XXH3pFilterBitsBuilder : public BuiltinFilterBitsBuilder {
}
protected:
static constexpr uint32_t kMetadataLen = 5;
// For delegating between XXH3pFilterBitsBuilders
void SwapEntriesWith(XXH3pFilterBitsBuilder* other) {
std::swap(hash_entries_, other->hash_entries_);
+4 -1
View File
@@ -63,13 +63,16 @@ inline bool BlockFetcher::TryGetFromPrefetchBuffer() {
Status s = PrepareIOFromReadOptions(read_options_, file_->env(), opts);
if (s.ok() && prefetch_buffer_->TryReadFromCache(
opts, handle_.offset(), block_size_with_trailer_, &slice_,
for_compaction_)) {
&s, for_compaction_)) {
CheckBlockChecksum();
if (!status_.ok()) {
return true;
}
got_from_prefetch_buffer_ = true;
used_buf_ = const_cast<char*>(slice_.data());
} else if (!s.ok()) {
status_ = s;
return true;
}
}
return got_from_prefetch_buffer_;
+2 -1
View File
@@ -82,6 +82,8 @@ CuckooTableBuilder::CuckooTableBuilder(
properties_.column_family_name = column_family_name;
properties_.db_id = db_id;
properties_.db_session_id = db_session_id;
status_.PermitUncheckedError();
io_status_.PermitUncheckedError();
}
void CuckooTableBuilder::Add(const Slice& key, const Slice& value) {
@@ -250,7 +252,6 @@ Status CuckooTableBuilder::Finish() {
assert(!closed_);
closed_ = true;
std::vector<CuckooBucket> buckets;
Status s;
std::string unused_bucket;
if (num_entries_ > 0) {
// Calculate the real hash size if module hash is enabled.
+3 -2
View File
@@ -307,8 +307,9 @@ Status ReadFooterFromFile(const IOOptions& opts, RandomAccessFileReader* file,
// for iterator, TryReadFromCache might do a readahead. Revisit to see if we
// need to pass a timeout at that point
if (prefetch_buffer == nullptr ||
!prefetch_buffer->TryReadFromCache(
IOOptions(), read_offset, Footer::kMaxEncodedLength, &footer_input)) {
!prefetch_buffer->TryReadFromCache(IOOptions(), read_offset,
Footer::kMaxEncodedLength,
&footer_input, nullptr)) {
if (file->use_direct_io()) {
s = file->Read(opts, read_offset, Footer::kMaxEncodedLength,
&footer_input, nullptr, &internal_buf);
+24 -13
View File
@@ -235,7 +235,11 @@ Status MockTableFactory::NewTableReader(
std::unique_ptr<RandomAccessFileReader>&& file, uint64_t /*file_size*/,
std::unique_ptr<TableReader>* table_reader,
bool /*prefetch_index_and_filter_in_cache*/) const {
uint32_t id = GetIDFromFile(file.get());
uint32_t id;
Status s = GetIDFromFile(file.get(), &id);
if (!s.ok()) {
return s;
}
MutexLock lock_guard(&file_system_.mutex);
@@ -252,7 +256,9 @@ Status MockTableFactory::NewTableReader(
TableBuilder* MockTableFactory::NewTableBuilder(
const TableBuilderOptions& /*table_builder_options*/,
uint32_t /*column_family_id*/, WritableFileWriter* file) const {
uint32_t id = GetAndWriteNextID(file);
uint32_t id;
Status s = GetAndWriteNextID(file, &id);
assert(s.ok());
return new MockTableBuilder(id, &file_system_, corrupt_mode_);
}
@@ -268,25 +274,30 @@ Status MockTableFactory::CreateMockTable(Env* env, const std::string& fname,
WritableFileWriter file_writer(NewLegacyWritableFileWrapper(std::move(file)),
fname, EnvOptions());
uint32_t id = GetAndWriteNextID(&file_writer);
file_system_.files.insert({id, std::move(file_contents)});
return Status::OK();
uint32_t id;
s = GetAndWriteNextID(&file_writer, &id);
if (s.ok()) {
file_system_.files.insert({id, std::move(file_contents)});
}
return s;
}
uint32_t MockTableFactory::GetAndWriteNextID(WritableFileWriter* file) const {
uint32_t next_id = next_id_.fetch_add(1);
Status MockTableFactory::GetAndWriteNextID(WritableFileWriter* file,
uint32_t* next_id) const {
*next_id = next_id_.fetch_add(1);
char buf[4];
EncodeFixed32(buf, next_id);
file->Append(Slice(buf, 4));
return next_id;
EncodeFixed32(buf, *next_id);
return file->Append(Slice(buf, 4));
}
uint32_t MockTableFactory::GetIDFromFile(RandomAccessFileReader* file) const {
Status MockTableFactory::GetIDFromFile(RandomAccessFileReader* file,
uint32_t* id) const {
char buf[4];
Slice result;
file->Read(IOOptions(), 0, 4, &result, buf, nullptr);
Status s = file->Read(IOOptions(), 0, 4, &result, buf, nullptr);
assert(result.size() == 4);
return DecodeFixed32(buf);
*id = DecodeFixed32(buf);
return s;
}
void MockTableFactory::AssertSingleFile(const KVVector& file_contents) {
+2 -2
View File
@@ -77,8 +77,8 @@ class MockTableFactory : public TableFactory {
void AssertLatestFile(const KVVector& file_contents);
private:
uint32_t GetAndWriteNextID(WritableFileWriter* file) const;
uint32_t GetIDFromFile(RandomAccessFileReader* file) const;
Status GetAndWriteNextID(WritableFileWriter* file, uint32_t* id) const;
Status GetIDFromFile(RandomAccessFileReader* file, uint32_t* id) const;
mutable MockTableFileSystem file_system_;
mutable std::atomic<uint32_t> next_id_;
+4 -2
View File
@@ -177,8 +177,10 @@ Status SstFileDumper::VerifyChecksum() {
Status SstFileDumper::DumpTable(const std::string& out_filename) {
std::unique_ptr<WritableFile> out_file;
Env* env = options_.env;
env->NewWritableFile(out_filename, &out_file, soptions_);
Status s = table_reader_->DumpTable(out_file.get());
Status s = env->NewWritableFile(out_filename, &out_file, soptions_);
if (s.ok()) {
s = table_reader_->DumpTable(out_file.get());
}
if (!s.ok()) {
// close the file before return error, ignore the close error if there's any
out_file->Close().PermitUncheckedError();
+18 -18
View File
@@ -377,10 +377,10 @@ class TableConstructor : public Constructor {
} else {
builder->Add(kv.first, kv.second);
}
EXPECT_TRUE(builder->status().ok());
EXPECT_OK(builder->status());
}
Status s = builder->Finish();
file_writer_->Flush();
EXPECT_OK(file_writer_->Flush());
EXPECT_TRUE(s.ok()) << s.ToString();
EXPECT_EQ(TEST_GetSink()->contents().size(), builder->FileSize());
@@ -1270,15 +1270,15 @@ class FileChecksumTestHelper {
EXPECT_TRUE(table_builder_->status().ok());
}
Status s = table_builder_->Finish();
file_writer_->Flush();
EXPECT_TRUE(s.ok());
EXPECT_OK(file_writer_->Flush());
EXPECT_OK(s);
EXPECT_EQ(sink_->contents().size(), table_builder_->FileSize());
return s;
}
std::string GetFileChecksum() {
file_writer_->Close();
EXPECT_OK(file_writer_->Close());
return table_builder_->GetFileChecksum();
}
@@ -3323,7 +3323,7 @@ TEST_P(BlockBasedTableTest, NoFileChecksum) {
f.GetFileWriter()));
ASSERT_OK(f.ResetTableBuilder(std::move(builder)));
f.AddKVtoKVMap(1000);
f.WriteKVAndFlushTable();
ASSERT_OK(f.WriteKVAndFlushTable());
ASSERT_STREQ(f.GetFileChecksumFuncName(), kUnknownFileChecksumFuncName);
ASSERT_STREQ(f.GetFileChecksum().c_str(), kUnknownFileChecksum);
}
@@ -3362,7 +3362,7 @@ TEST_P(BlockBasedTableTest, Crc32cFileChecksum) {
f.GetFileWriter()));
ASSERT_OK(f.ResetTableBuilder(std::move(builder)));
f.AddKVtoKVMap(1000);
f.WriteKVAndFlushTable();
ASSERT_OK(f.WriteKVAndFlushTable());
ASSERT_STREQ(f.GetFileChecksumFuncName(), "FileChecksumCrc32c");
std::unique_ptr<FileChecksumGenerator> checksum_crc32c_gen2 =
@@ -3418,7 +3418,7 @@ TEST_F(PlainTableTest, BasicPlainTableProperties) {
builder->Add(key, value);
}
ASSERT_OK(builder->Finish());
file_writer->Flush();
ASSERT_OK(file_writer->Flush());
test::StringSink* ss =
ROCKSDB_NAMESPACE::test::GetStringSinkFromLegacyWriter(file_writer.get());
@@ -3468,7 +3468,7 @@ TEST_F(PlainTableTest, NoFileChecksum) {
f.GetFileWriter()));
ASSERT_OK(f.ResetTableBuilder(std::move(builder)));
f.AddKVtoKVMap(1000);
f.WriteKVAndFlushTable();
ASSERT_OK(f.WriteKVAndFlushTable());
ASSERT_STREQ(f.GetFileChecksumFuncName(), kUnknownFileChecksumFuncName);
EXPECT_EQ(f.GetFileChecksum(), kUnknownFileChecksum);
}
@@ -3510,7 +3510,7 @@ TEST_F(PlainTableTest, Crc32cFileChecksum) {
f.GetFileWriter()));
ASSERT_OK(f.ResetTableBuilder(std::move(builder)));
f.AddKVtoKVMap(1000);
f.WriteKVAndFlushTable();
ASSERT_OK(f.WriteKVAndFlushTable());
ASSERT_STREQ(f.GetFileChecksumFuncName(), "FileChecksumCrc32c");
std::unique_ptr<FileChecksumGenerator> checksum_crc32c_gen2 =
@@ -4022,7 +4022,7 @@ TEST_F(PrefixTest, PrefixAndWholeKeyTest) {
const std::string kDBPath = test::PerThreadDBPath("table_prefix_test");
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
DestroyDB(kDBPath, options);
ASSERT_OK(DestroyDB(kDBPath, options));
ROCKSDB_NAMESPACE::DB* db;
ASSERT_OK(ROCKSDB_NAMESPACE::DB::Open(options, kDBPath, &db));
@@ -4081,7 +4081,7 @@ TEST_P(BlockBasedTableTest, DISABLED_TableWithGlobalSeqno) {
builder->Add(ik.Encode(), value);
}
ASSERT_OK(builder->Finish());
file_writer->Flush();
ASSERT_OK(file_writer->Flush());
test::RandomRWStringSink ss_rw(sink);
uint32_t version;
@@ -4265,7 +4265,7 @@ TEST_P(BlockBasedTableTest, BlockAlignTest) {
builder->Add(ik.Encode(), value);
}
ASSERT_OK(builder->Finish());
file_writer->Flush();
ASSERT_OK(file_writer->Flush());
test::RandomRWStringSink ss_rw(sink);
std::unique_ptr<RandomAccessFileReader> file_reader(
@@ -4360,7 +4360,7 @@ TEST_P(BlockBasedTableTest, PropertiesBlockRestartPointTest) {
builder->Add(ik.Encode(), value);
}
ASSERT_OK(builder->Finish());
file_writer->Flush();
ASSERT_OK(file_writer->Flush());
test::RandomRWStringSink ss_rw(sink);
std::unique_ptr<RandomAccessFileReader> file_reader(
@@ -4511,7 +4511,7 @@ TEST_P(BlockBasedTableTest, BadOptions) {
const std::string kDBPath =
test::PerThreadDBPath("block_based_table_bad_options_test");
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
DestroyDB(kDBPath, options);
ASSERT_OK(DestroyDB(kDBPath, options));
ROCKSDB_NAMESPACE::DB* db;
ASSERT_NOK(ROCKSDB_NAMESPACE::DB::Open(options, kDBPath, &db));
@@ -4559,9 +4559,9 @@ TEST_F(BBTTailPrefetchTest, FilePrefetchBufferMinOffset) {
TailPrefetchStats tpstats;
FilePrefetchBuffer buffer(nullptr, 0, 0, false, true);
IOOptions opts;
buffer.TryReadFromCache(opts, 500, 10, nullptr);
buffer.TryReadFromCache(opts, 480, 10, nullptr);
buffer.TryReadFromCache(opts, 490, 10, nullptr);
buffer.TryReadFromCache(opts, 500, 10, nullptr, nullptr);
buffer.TryReadFromCache(opts, 480, 10, nullptr, nullptr);
buffer.TryReadFromCache(opts, 490, 10, nullptr, nullptr);
ASSERT_EQ(480, buffer.min_offset_read());
}
@@ -44,7 +44,7 @@ const size_t kArgBufferSize = 100000;
class BlockCacheTracerTest : public testing::Test {
public:
BlockCacheTracerTest() {
test_path_ = test::PerThreadDBPath("block_cache_tracer_test");
test_path_ = test::PerThreadDBPath("block_cache_trace_analyzer_test");
env_ = ROCKSDB_NAMESPACE::Env::Default();
EXPECT_OK(env_->CreateDir(test_path_));
trace_file_path_ = test_path_ + "/block_cache_trace";
+1 -1
View File
@@ -5220,7 +5220,7 @@ class Benchmark {
}
Duration duration(FLAGS_duration, reads_);
while (!duration.Done(1)) {
while (!duration.Done(entries_per_batch_)) {
DB* db = SelectDB(thread);
if (FLAGS_multiread_stride) {
int64_t key = GetRandomKey(&thread->rand);
+38 -61
View File
@@ -27,13 +27,20 @@ int main() {
// --num_inserts = the num of inserts the first thread should perform.
// --wal_ttl = the wal ttl for the run.
DEFINE_uint64(num_inserts, 1000,
"the num of inserts the first thread should"
" perform.");
DEFINE_uint64(wal_ttl_seconds, 1000, "the wal ttl for the run(in seconds)");
DEFINE_uint64(wal_size_limit_MB, 10,
"the wal size limit for the run"
"(in MB)");
using namespace ROCKSDB_NAMESPACE;
using GFLAGS_NAMESPACE::ParseCommandLineFlags;
using GFLAGS_NAMESPACE::SetUsageMessage;
struct DataPumpThread {
size_t no_records;
DB* db; // Assumption DB is Open'ed already.
};
@@ -41,8 +48,8 @@ static void DataPumpThreadBody(void* arg) {
DataPumpThread* t = reinterpret_cast<DataPumpThread*>(arg);
DB* db = t->db;
Random rnd(301);
size_t i = 0;
while (i++ < t->no_records) {
uint64_t i = 0;
while (i++ < FLAGS_num_inserts) {
if (!db->Put(WriteOptions(), Slice(rnd.RandomString(500)),
Slice(rnd.RandomString(500)))
.ok()) {
@@ -52,45 +59,6 @@ static void DataPumpThreadBody(void* arg) {
}
}
struct ReplicationThread {
std::atomic<bool> stop;
DB* db;
volatile size_t no_read;
};
static void ReplicationThreadBody(void* arg) {
ReplicationThread* t = reinterpret_cast<ReplicationThread*>(arg);
DB* db = t->db;
std::unique_ptr<TransactionLogIterator> iter;
SequenceNumber currentSeqNum = 1;
while (!t->stop.load(std::memory_order_acquire)) {
iter.reset();
Status s;
while (!db->GetUpdatesSince(currentSeqNum, &iter).ok()) {
if (t->stop.load(std::memory_order_acquire)) {
return;
}
}
fprintf(stderr, "Refreshing iterator\n");
for (; iter->Valid(); iter->Next(), t->no_read++, currentSeqNum++) {
BatchResult res = iter->GetBatch();
if (res.sequence != currentSeqNum) {
fprintf(stderr, "Missed a seq no. b/w %ld and %ld\n",
(long)currentSeqNum, (long)res.sequence);
exit(1);
}
}
}
}
DEFINE_uint64(num_inserts, 1000,
"the num of inserts the first thread should"
" perform.");
DEFINE_uint64(wal_ttl_seconds, 1000, "the wal ttl for the run(in seconds)");
DEFINE_uint64(wal_size_limit_MB, 10,
"the wal size limit for the run"
"(in MB)");
int main(int argc, const char** argv) {
SetUsageMessage(
std::string("\nUSAGE:\n") + std::string(argv[0]) +
@@ -117,29 +85,38 @@ int main(int argc, const char** argv) {
}
DataPumpThread dataPump;
dataPump.no_records = FLAGS_num_inserts;
dataPump.db = db;
env->StartThread(DataPumpThreadBody, &dataPump);
ReplicationThread replThread;
replThread.db = db;
replThread.no_read = 0;
replThread.stop.store(false, std::memory_order_release);
env->StartThread(ReplicationThreadBody, &replThread);
while (replThread.no_read < FLAGS_num_inserts)
;
replThread.stop.store(true, std::memory_order_release);
if (replThread.no_read < dataPump.no_records) {
// no. read should be => than inserted.
fprintf(stderr,
"No. of Record's written and read not same\nRead : %" ROCKSDB_PRIszt
" Written : %" ROCKSDB_PRIszt "\n",
replThread.no_read, dataPump.no_records);
exit(1);
std::unique_ptr<TransactionLogIterator> iter;
SequenceNumber currentSeqNum = 1;
uint64_t num_read = 0;
for (;;) {
iter.reset();
// Continue to probe a bit more after all received
size_t probes = 0;
while (!db->GetUpdatesSince(currentSeqNum, &iter).ok()) {
probes++;
if (probes > 100 && num_read >= FLAGS_num_inserts) {
if (num_read > FLAGS_num_inserts) {
fprintf(stderr, "Too many updates read: %ld expected: %ld\n",
(long)num_read, (long)FLAGS_num_inserts);
exit(1);
}
fprintf(stderr, "Successful!\n");
return 0;
}
}
fprintf(stderr, "Refreshing iterator\n");
for (; iter->Valid(); iter->Next(), num_read++, currentSeqNum++) {
BatchResult res = iter->GetBatch();
if (res.sequence != currentSeqNum) {
fprintf(stderr, "Missed a seq no. b/w %ld and %ld\n",
(long)currentSeqNum, (long)res.sequence);
exit(1);
}
}
}
fprintf(stderr, "Successful!\n");
exit(0);
}
#endif // GFLAGS
+8 -4
View File
@@ -2016,12 +2016,16 @@ void ReduceDBLevelsCommand::DoCommand() {
assert(db_ != nullptr);
// Compact the whole DB to put all files to the highest level.
fprintf(stdout, "Compacting the db...\n");
db_->CompactRange(CompactRangeOptions(), GetCfHandle(), nullptr, nullptr);
st =
db_->CompactRange(CompactRangeOptions(), GetCfHandle(), nullptr, nullptr);
CloseDB();
EnvOptions soptions;
st = VersionSet::ReduceNumberOfLevels(db_path_, &options_, soptions,
new_levels_);
if (st.ok()) {
EnvOptions soptions;
st = VersionSet::ReduceNumberOfLevels(db_path_, &options_, soptions,
new_levels_);
}
if (!st.ok()) {
exec_state_ = LDBCommandExecuteResult::Failed(st.ToString());
return;
+2 -2
View File
@@ -107,7 +107,7 @@ bool ReduceLevelTest::ReduceLevels(int target_level) {
TEST_F(ReduceLevelTest, Last_Level) {
ASSERT_OK(OpenDB(true, 4));
ASSERT_OK(Put("aaaa", "11111"));
Flush();
ASSERT_OK(Flush());
MoveL0FileToLevel(3);
ASSERT_EQ(FilesOnLevel(3), 1);
CloseDB();
@@ -126,7 +126,7 @@ TEST_F(ReduceLevelTest, Last_Level) {
TEST_F(ReduceLevelTest, Top_Level) {
ASSERT_OK(OpenDB(true, 5));
ASSERT_OK(Put("aaaa", "11111"));
Flush();
ASSERT_OK(Flush());
ASSERT_EQ(FilesOnLevel(0), 1);
CloseDB();
+8 -2
View File
@@ -142,6 +142,7 @@ function main {
run_db_bench "deleterandom" $((NUM_KEYS / 10 / $NUM_THREADS))
run_db_bench "seekrandom"
run_db_bench "seekrandomwhilewriting"
run_db_bench "multireadrandom"
fi
cleanup_test_directory $TEST_ROOT_DIR
@@ -192,6 +193,8 @@ function init_arguments {
DELETE_TEST_PATH=${DELETE_TEST_PATH:-0}
SEEK_NEXTS=${SEEK_NEXTS:-10}
SEED=${SEED:-$( date +%s )}
MULTIREAD_BATCH_SIZE=${MULTIREAD_BATCH_SIZE:-128}
MULTIREAD_STRIDE=${MULTIREAD_STRIDE:-12}
}
# $1 --- benchmark name
@@ -240,7 +243,10 @@ function run_db_bench {
--max_background_compactions=$MAX_BACKGROUND_COMPACTIONS \
--num_high_pri_threads=$NUM_HIGH_PRI_THREADS \
--num_low_pri_threads=$NUM_LOW_PRI_THREADS \
--seed=$SEED) 2>&1"
--seed=$SEED \
--multiread_batched=true \
--batch_size=$MULTIREAD_BATCH_SIZE \
--multiread_stride=$MULTIREAD_STRIDE) 2>&1"
ps_cmd="ps aux"
if ! [ -z "$REMOTE_USER_AT_HOST" ]; then
echo "Running benchmark remotely on $REMOTE_USER_AT_HOST"
@@ -377,7 +383,7 @@ function build_db_bench_and_ldb {
make clean
exit_on_error $?
DEBUG_LEVEL=0 PORTABLE=1 make db_bench ldb -j32
DEBUG_LEVEL=0 make db_bench ldb -j32
exit_on_error $?
}
+1 -1
View File
@@ -126,7 +126,7 @@ class SSTDumpToolTest : public testing::Test {
tb->Add(MakeKey(i), MakeValue(i));
}
ASSERT_OK(tb->Finish());
file_writer->Close();
ASSERT_OK(file_writer->Close());
}
protected:
+1 -1
View File
@@ -85,7 +85,7 @@ class TraceAnalyzerTest : public testing::Test {
delete single_iter;
std::this_thread::sleep_for (std::chrono::seconds(1));
db_->Get(ro, "g", &value);
db_->Get(ro, "g", &value).PermitUncheckedError();
ASSERT_OK(db_->EndTrace());
+7 -6
View File
@@ -52,6 +52,7 @@ class BlockCacheTracerTest : public testing::Test {
return TableReaderCaller::kUserIterator;
}
assert(false);
return TableReaderCaller::kMaxBlockCacheLookupCaller;
}
void WriteBlockAccess(BlockCacheTraceWriter* writer, uint32_t from_key_id,
@@ -200,8 +201,8 @@ TEST_F(BlockCacheTracerTest, AtomicWrite) {
BlockCacheTraceReader reader(std::move(trace_reader));
BlockCacheTraceHeader header;
ASSERT_OK(reader.ReadHeader(&header));
ASSERT_EQ(kMajorVersion, header.rocksdb_major_version);
ASSERT_EQ(kMinorVersion, header.rocksdb_minor_version);
ASSERT_EQ(kMajorVersion, static_cast<int>(header.rocksdb_major_version));
ASSERT_EQ(kMinorVersion, static_cast<int>(header.rocksdb_minor_version));
VerifyAccess(&reader, 0, TraceType::kBlockTraceDataBlock, 1);
ASSERT_NOK(reader.ReadAccess(&record));
}
@@ -244,8 +245,8 @@ TEST_F(BlockCacheTracerTest, AtomicNoWriteAfterEndTrace) {
BlockCacheTraceReader reader(std::move(trace_reader));
BlockCacheTraceHeader header;
ASSERT_OK(reader.ReadHeader(&header));
ASSERT_EQ(kMajorVersion, header.rocksdb_major_version);
ASSERT_EQ(kMinorVersion, header.rocksdb_minor_version);
ASSERT_EQ(kMajorVersion, static_cast<int>(header.rocksdb_major_version));
ASSERT_EQ(kMinorVersion, static_cast<int>(header.rocksdb_minor_version));
VerifyAccess(&reader, 0, TraceType::kBlockTraceDataBlock, 1);
ASSERT_NOK(reader.ReadAccess(&record));
}
@@ -307,8 +308,8 @@ TEST_F(BlockCacheTracerTest, MixedBlocks) {
BlockCacheTraceReader reader(std::move(trace_reader));
BlockCacheTraceHeader header;
ASSERT_OK(reader.ReadHeader(&header));
ASSERT_EQ(kMajorVersion, header.rocksdb_major_version);
ASSERT_EQ(kMinorVersion, header.rocksdb_minor_version);
ASSERT_EQ(kMajorVersion, static_cast<int>(header.rocksdb_major_version));
ASSERT_EQ(kMinorVersion, static_cast<int>(header.rocksdb_minor_version));
// Read blocks.
VerifyAccess(&reader, 0, TraceType::kBlockTraceUncompressionDictBlock, 10);
VerifyAccess(&reader, 10, TraceType::kBlockTraceDataBlock, 10);
+5 -4
View File
@@ -215,14 +215,15 @@ void IOTracer::EndIOTrace() {
tracing_enabled = false;
}
Status IOTracer::WriteIOOp(const IOTraceRecord& record) {
// TODO: Return status and handle that in file_system_tracer.h
void IOTracer::WriteIOOp(const IOTraceRecord& record) {
if (!writer_.load()) {
return Status::OK();
return;
}
InstrumentedMutexLock lock_guard(&trace_writer_mutex_);
if (!writer_.load()) {
return Status::OK();
return;
}
return writer_.load()->WriteIOOp(record);
writer_.load()->WriteIOOp(record).PermitUncheckedError();
}
} // namespace ROCKSDB_NAMESPACE
+1 -1
View File
@@ -158,7 +158,7 @@ class IOTracer {
TSAN_SUPPRESSION bool is_tracing_enabled() const { return tracing_enabled; }
Status WriteIOOp(const IOTraceRecord& record);
void WriteIOOp(const IOTraceRecord& record);
private:
TraceOptions trace_options_;
+5 -5
View File
@@ -58,7 +58,7 @@ class IOTracerTest : public testing::Test {
record.file_name = kDummyFile + std::to_string(i);
record.len = i;
record.offset = i + 20;
ASSERT_OK(writer->WriteIOOp(record));
EXPECT_OK(writer->WriteIOOp(record));
}
}
@@ -91,7 +91,7 @@ TEST_F(IOTracerTest, AtomicWrite) {
&trace_writer));
IOTracer writer;
ASSERT_OK(writer.StartIOTrace(env_, trace_opt, std::move(trace_writer)));
ASSERT_OK(writer.WriteIOOp(record));
writer.WriteIOOp(record);
ASSERT_OK(env_->FileExists(trace_file_path_));
}
{
@@ -124,7 +124,7 @@ TEST_F(IOTracerTest, AtomicWriteBeforeStartTrace) {
IOTracer writer;
// The record should not be written to the trace_file since StartIOTrace is
// not called.
ASSERT_OK(writer.WriteIOOp(record));
writer.WriteIOOp(record);
ASSERT_OK(env_->FileExists(trace_file_path_));
}
{
@@ -149,11 +149,11 @@ TEST_F(IOTracerTest, AtomicNoWriteAfterEndTrace) {
&trace_writer));
IOTracer writer;
ASSERT_OK(writer.StartIOTrace(env_, trace_opt, std::move(trace_writer)));
ASSERT_OK(writer.WriteIOOp(record));
writer.WriteIOOp(record);
writer.EndIOTrace();
// Write the record again. This time the record should not be written since
// EndIOTrace is called.
ASSERT_OK(writer.WriteIOOp(record));
writer.WriteIOOp(record);
ASSERT_OK(env_->FileExists(trace_file_path_));
}
{
+4 -4
View File
@@ -171,16 +171,16 @@ TEST_F(WritableFileWriterTest, IncrementalBuffer) {
for (int i = 0; i < 20; i++) {
uint32_t num = r.Skewed(16) * 100 + r.Uniform(100);
std::string random_string = r.RandomString(num);
writer->Append(Slice(random_string.c_str(), num));
ASSERT_OK(writer->Append(Slice(random_string.c_str(), num)));
target.append(random_string.c_str(), num);
// In some attempts, flush in a chance of 1/10.
if (!no_flush && r.Uniform(10) == 0) {
writer->Flush();
ASSERT_OK(writer->Flush());
}
}
writer->Flush();
writer->Close();
ASSERT_OK(writer->Flush());
ASSERT_OK(writer->Close());
ASSERT_EQ(target.size(), actual.size());
ASSERT_EQ(target, actual);
}
+2 -1
View File
@@ -111,7 +111,8 @@ void GenericRateLimiter::Request(int64_t bytes, const Env::IOPriority pri,
std::chrono::microseconds now(NowMicrosMonotonic(env_));
if (now - tuned_time_ >=
kRefillsPerTune * std::chrono::microseconds(refill_period_us_)) {
Tune();
Status s = Tune();
s.PermitUncheckedError(); //**TODO: What to do on error?
}
}
+7 -7
View File
@@ -145,7 +145,7 @@ TEST_F(ThreadListTest, SimpleColumnFamilyInfoTest) {
std::vector<ThreadStatus> thread_list;
// Verify the number of running threads in each pool.
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
int running_count[ThreadStatus::NUM_THREAD_TYPES] = {0};
for (auto thread_status : thread_list) {
if (thread_status.cf_name == "pikachu" &&
@@ -166,7 +166,7 @@ TEST_F(ThreadListTest, SimpleColumnFamilyInfoTest) {
running_task.WaitUntilDone();
// Verify none of the threads are running
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
for (int i = 0; i < ThreadStatus::NUM_THREAD_TYPES; ++i) {
running_count[i] = 0;
@@ -281,7 +281,7 @@ TEST_F(ThreadListTest, SimpleEventTest) {
int state_counts[ThreadStatus::NUM_STATE_TYPES] = {0};
std::vector<ThreadStatus> thread_list;
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
UpdateStatusCounts(thread_list, operation_counts, state_counts);
VerifyAndResetCounts(correct_operation_counts, operation_counts,
ThreadStatus::NUM_OP_TYPES);
@@ -293,7 +293,7 @@ TEST_F(ThreadListTest, SimpleEventTest) {
UpdateCount(correct_operation_counts, ThreadStatus::OP_COMPACTION,
ThreadStatus::OP_UNKNOWN, kCompactionWaitTasks);
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
UpdateStatusCounts(thread_list, operation_counts, state_counts);
VerifyAndResetCounts(correct_operation_counts, operation_counts,
ThreadStatus::NUM_OP_TYPES);
@@ -305,7 +305,7 @@ TEST_F(ThreadListTest, SimpleEventTest) {
UpdateCount(correct_operation_counts, ThreadStatus::OP_FLUSH,
ThreadStatus::OP_UNKNOWN, kFlushWriteTasks);
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
UpdateStatusCounts(thread_list, operation_counts, state_counts);
VerifyAndResetCounts(correct_operation_counts, operation_counts,
ThreadStatus::NUM_OP_TYPES);
@@ -317,7 +317,7 @@ TEST_F(ThreadListTest, SimpleEventTest) {
UpdateCount(correct_operation_counts, ThreadStatus::OP_COMPACTION,
ThreadStatus::OP_UNKNOWN, kCompactionWriteTasks);
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
UpdateStatusCounts(thread_list, operation_counts, state_counts);
VerifyAndResetCounts(correct_operation_counts, operation_counts,
ThreadStatus::NUM_OP_TYPES);
@@ -329,7 +329,7 @@ TEST_F(ThreadListTest, SimpleEventTest) {
UpdateCount(correct_operation_counts, ThreadStatus::OP_COMPACTION,
ThreadStatus::OP_UNKNOWN, kCompactionReadTasks);
env->GetThreadList(&thread_list);
ASSERT_OK(env->GetThreadList(&thread_list));
UpdateStatusCounts(thread_list, operation_counts, state_counts);
VerifyAndResetCounts(correct_operation_counts, operation_counts,
ThreadStatus::NUM_OP_TYPES);
Loaded 100 of 153 files, more files were not shown because too many files have changed in this diff. Show more