Compare commits

..
132 Commits
Author SHA1 Message Date
Peter Dillinger dc58bf9f04 HISTORY.md update for bzip upgrade (#6767)
Summary:
See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767

Reviewed By: riversand963

Differential Revision: D21283307

Pulled By: pdillinger

fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8
2020-04-28 13:50:06 -07:00
Adam Retter 27e593fbe1 Update RocksJava static version of bzip2 (#6714)
Summary:
Updates the version of bzip2 used for RocksJava static builds.

Please, can we also get this cherry-picked to:

1. 6.7.fb
2. 6.8.fb
3. 6.9.fb
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714

Reviewed By: cheng-chang

Differential Revision: D21067233

Pulled By: pdillinger

fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e
2020-04-16 16:01:15 -07:00
Peter Dillinger d7b6149563 Use an Amazon S3 bucket for downloading deps (#6526)
Summary:
After we had a lot of failures with maven.org downloads, we
wanted an alternative location for downloading binary dependencies.
Hosting them through github would have been good in terms of
organizational and network dependencies, but that approach seems to be
awkward (fake releases, so would need a 'rocksdb-deps' repo) and
strangely complicated for Facebook policy on open source repositories.

This commit moves the downloads (that are not officially hosted by
others on github) from my personal rocksdb fork to an S3 bucket owned
by the Facebook RocksDB AWS account. Facebook employees can access
this through an internal tool, and we should be able to grant permission
to outside collaborators.

Assuming this works out, I will back-port to older branches to stabilize
their CI testing as well.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6526

Test Plan: CI

Differential Revision: D20430130

Pulled By: pdillinger

fbshipit-source-id: df52394a65e0a57942db3039bdaade8a4d520cb2
2020-04-16 16:01:08 -07:00
Adam Retter 6731fef82b Update to latest Snappy to fix compilation issue on latest MacOS XCode (#6496)
Summary:
* **macOS version:** 10.15.2 (Catalina)
* **XCode/Clang version:** Apple clang version 11.0.0 (clang-1100.0.33.16)

Before this bugfix the error generated is:

```
In file included from ./util/compression.h:23:
./snappy-1.1.7/snappy.h:76:59: error: unknown type name 'string'; did you mean 'std::string'?
  size_t Compress(const char* input, size_t input_length, string* output);
                                                          ^~~~~~
                                                          std::string
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here
typedef basic_string<char, char_traits<char>, allocator<char> > string;
                                                                ^
In file included from db/builder.cc:10:
In file included from ./db/builder.h:12:
In file included from ./db/range_tombstone_fragmenter.h:15:
In file included from ./db/pinned_iterators_manager.h:12:
In file included from ./table/internal_iterator.h:13:
In file included from ./table/format.h:25:
In file included from ./options/cf_options.h:14:
In file included from ./util/compression.h:23:
./snappy-1.1.7/snappy.h:85:19: error: unknown type name 'string'; did you mean 'std::string'?
                  string* uncompressed);
                  ^~~~~~
                  std::string
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here
typedef basic_string<char, char_traits<char>, allocator<char> > string;
                                                                ^
2 errors generated.
make: *** [jls/db/builder.o] Error 1
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6496

Differential Revision: D20389254

Pulled By: pdillinger

fbshipit-source-id: 2864245c8d0dba7b2ab81294241a62f2adf02e20
2020-04-15 10:05:11 -07:00
sdong 3482d4dc2a cmake: add option WITH_CORE_TOOLS to exclude tools except ldb and sst_dump (#6506)
Summary:
ldb and sst_dump are most important tools and they don't dependend on gflags. In cmake, we don't have an way to only build these two tools and exclude other tools. This is inconvenient if the environment has a problem with gflags. Add such an option WITH_CORE_TOOLS.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6506

Test Plan: cmake and build with WITH_TOOLS and without.

Differential Revision: D20473029

fbshipit-source-id: 3d730fd14bbae6eeeae7f9cc9aec50a4e488ad72
2020-04-15 10:04:56 -07:00
Yanqin Jin d75da26396 Update HISTORY and version 2020-03-30 21:44:30 -07:00
Yanqin Jin 00894a289f Use flush time for the props.creation_time for FIFO compaction (#6612)
Summary:
For FIFO compaction, we use flush time instead of oldest key time as the
creation time. This is to prevent FIFO compaction dropping files whose oldest
key time is older than TTL but which has newer keys than TTL.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6612

Test Plan: make check

Reviewed By: siying

Differential Revision: D20748217

Pulled By: riversand963

fbshipit-source-id: 3f7b00a847020760537cdddd12f6fe039e5bc663
2020-03-30 21:10:29 -07:00
sdong bba5e7bc21 Fix data race of GetCreationTimeOfOldestFile() (#6473)
Summary:
When DBImpl::GetCreationTimeOfOldestFile() calls Version::GetCreationTimeOfOldestFile(), the version is not directly or indirectly referenced, so an event like compaction can race with the operation and cause DBImpl::GetCreationTimeOfOldestFile() to access delocated data. This was caught by an ASAN run:

==268==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000b7d198 at pc 0x000018332913 bp 0x7f391510d310 sp 0x7f391510d308
READ of size 8 at 0x612000b7d198 thread T845 (store_load-33)
SCARINESS: 51 (8-byte-read-heap-use-after-free)
    #0 0x18332912 in rocksdb::Version::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/version_set.cc:1488
    https://github.com/facebook/rocksdb/issues/1 0x1803ddaa in rocksdb::DBImpl::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/db_impl/db_impl.cc:4499
    https://github.com/facebook/rocksdb/issues/2 0xe24ca09 in rocksdb::StackableDB::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/utilities/stackable_db.h:392
    ......
0x612000b7d198 is located 216 bytes inside of 296-byte region [0x612000b7d0c0,0x612000b7d1e8)
freed by thread T28 here:
    ......
    https://github.com/facebook/rocksdb/issues/5 0x1832c73f in std::vector<rocksdb::FileMetaData*, std::allocator<rocksdb::FileMetaData*> >::~vector() third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/stl_vector.h:435
    https://github.com/facebook/rocksdb/issues/6 0x1832c73f in rocksdb::VersionStorageInfo::~VersionStorageInfo() rocksdb/src/db/version_set.cc:734
    https://github.com/facebook/rocksdb/issues/7 0x1832cf42 in rocksdb::Version::~Version() rocksdb/src/db/version_set.cc:758
    https://github.com/facebook/rocksdb/issues/8 0x9d1bb5 in rocksdb::Version::Unref() rocksdb/src/db/version_set.cc:2869
    https://github.com/facebook/rocksdb/issues/9 0x183e7631 in rocksdb::Compaction::~Compaction() rocksdb/src/db/compaction/compaction.cc:275
    https://github.com/facebook/rocksdb/issues/10 0x9e6de6 in std::default_delete<rocksdb::Compaction>::operator()(rocksdb::Compaction*) const third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78
    https://github.com/facebook/rocksdb/issues/11 0x9e6de6 in std::unique_ptr<rocksdb::Compaction, std::default_delete<rocksdb::Compaction> >::reset(rocksdb::Compaction*) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:376
    https://github.com/facebook/rocksdb/issues/12 0x9e6de6 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2826
    https://github.com/facebook/rocksdb/issues/13 0x9ac3b8 in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2320
    https://github.com/facebook/rocksdb/issues/14 0x9abff7 in rocksdb::DBImpl::BGWorkCompaction(void*) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2096
    ......

Fix the issue by reference the super version and use the referenced version from it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6473

Test Plan: Run ASAN for all existing tests.

Differential Revision: D20196416

fbshipit-source-id: 5f4a7918110fc7b8dd7841932d376bc9d1e59d6f
2020-03-03 11:12:05 -08:00
sdong cfaaaaddef Buck config: Re-enable liburing under Linux (#6451)
Summary:
The known bug of liburing has been fixed. Now we can re-enable liburing under Linux
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6451

Test Plan: Watch internal CI

Differential Revision: D20079009

fbshipit-source-id: 04a6f53a900ff721f9a62a188cf906771b5d68d2
2020-02-24 18:24:51 -08:00
Cheng Chang 5a868c18cb Update release version to 6.8 (#6450)
Summary:
Update release version to 6.8
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6450

Test Plan: no code change

Differential Revision: D20071889

Pulled By: cheng-chang

fbshipit-source-id: 91450aae09b201926469ff32f59ed436366f3b74
2020-02-24 12:17:36 -08:00
Peter Dillinger 43dde332cb Share kPageSize (and other small tweaks) (#6443)
Summary:
Make kPageSize extern const size_t (used in draft https://github.com/facebook/rocksdb/issues/6427)
Make kLitteEndian constexpr bool
Clarify a couple of comments
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6443

Test Plan: make check, CI

Differential Revision: D20044558

Pulled By: pdillinger

fbshipit-source-id: e0c5cc13229c82726280dc0ddcba4078346b8418
2020-02-22 08:01:36 -08:00
sdong 942eaba091 Handle io_uring partial results (#6441)
Summary:
The logic that handles io_uring partial results was wrong. Fix the logic by putting it into a queue and continue reading.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6441

Test Plan: Make sure this patch fixes the application test case where the bug was discovered; in env_test, add a unit test that simulates partial results and make sure the results are still correct.

Differential Revision: D20018616

fbshipit-source-id: 5398a7e34d74c26d52aa69dfd604e93e95d99c62
2020-02-21 16:57:37 -08:00
Yanqin Jin 890d87fadc Some minor fix-ups (#6440)
Summary:
Cleanup some code without any real change in functionality.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6440

Differential Revision: D20015891

Pulled By: riversand963

fbshipit-source-id: 33e18754b0f002006a6d4805e9aaf84c0c8ad25a
2020-02-21 15:09:56 -08:00
Peter Dillinger ab65278b1f Misc filter_bench improvements (#6444)
Summary:
Useful in validating/testing internal fragmentation changes (https://github.com/facebook/rocksdb/issues/6427)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6444

Test Plan: manual (no changes to production code)

Differential Revision: D20040076

Pulled By: pdillinger

fbshipit-source-id: 32d26f363d2a9ab9f5bebd281dcebd9915ae340e
2020-02-21 13:31:57 -08:00
Zaiyang Li fcec56e86c Add function to set row cache on rocksdb_options_t (#6442)
Summary:
Adding a C API function to set `row_cache` on `rocksdb_options_t` as this functionality is missing.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6442

Differential Revision: D20036813

Pulled By: riversand963

fbshipit-source-id: c1fa95ea343345fbc1e57961d0d048e0e79be373
2020-02-21 11:12:42 -08:00
sdong d75ce0a8ae Mention rocksdb_namespace.h in HISTORY.md (#6439)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6439

Differential Revision: D20012442

fbshipit-source-id: a7c9569826eac39cd7ea69c90f08a21dd4caa335
2020-02-20 14:55:19 -08:00
Yanqin Jin 362b8d4393 Fix MANIFEST name assignment (#6426)
Summary:
Currently, a new MANIFEST file is assigned a new file number when 1) no
MANIFEST is open, or 2) current MANIFEST file size exceeds a threshold. This is
not sufficient. There are cases when the caller explicitly specifies that a new
MANIFEST be created. For example, if user sets options.write_dbid_to_manifest = true,
and there are WAL files, then RocksDB will run into an issue during recovery.
`DBImpl::Recover()` will call `LogAndApply()` to write dbid. At this point, the db being
recovered creates a new MANIFEST, say, MANIFEST-000003. Since there are WALs,
`DBImpl::RecoverLogFiles` will be called. Towards the end of this function, we call
`LogAndApply(new_descriptor_log=true)`, which explicitly creates a new MANIFEST.
However, the manifest_file_number is wrong before this fix. Consequently, RocksDB
opens an existing, non-empty file for append, effectively truncating the file to zero.
If a crash occurs, then there will be data loss.

Test Plan (devserver):
make check
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6426

Test Plan: make check

Differential Revision: D19951866

Pulled By: riversand963

fbshipit-source-id: 4b1b9fc28d4fe2ac12764b388ef9e61f05e766da
2020-02-20 14:30:58 -08:00
sdong fdf882ded2 Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
Summary:
When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433

Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag.

Differential Revision: D19977691

fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e
2020-02-20 12:09:57 -08:00
Gaurav Singh 4e33f1e1dc simplify user_access_only expression (#6360)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6360

Differential Revision: D19698918

Pulled By: riversand963

fbshipit-source-id: d20ecca541376cccd32fc7afb504ea90021860ee
2020-02-20 10:27:56 -08:00
Remington Brasga a993cc3a62 Fixed typo in benchmark.sh (#6434)
Summary:
TB =  1024 * GB
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6434

Differential Revision: D19978339

Pulled By: zhichao-cao

fbshipit-source-id: 5a89890110b23f0ebda4a95223f66da6736321ac
2020-02-19 17:08:02 -08:00
Yanqin Jin 5a297516e1 Ignore .vs/.vscode dir used by visual studio/ vscode (#6428)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6428

Differential Revision: D19959176

Pulled By: riversand963

fbshipit-source-id: 5b863944663462b246bb82883fa683f75ab33fc1
2020-02-18 16:10:15 -08:00
Andrew Kryczka c6abe30ee3 Fix concurrent full purge and WAL recycling (#5900)
Summary:
We were removing the file from `log_recycle_files_` before renaming it
with `ReuseWritableFile()`. Since `ReuseWritableFile()` occurs outside
the DB mutex, it was possible for a concurrent full purge to sneak in
and delete the file before it could be renamed. Consequently, `SwitchMemtable()`
would fail and the DB would enter read-only mode.

The fix is to hold the old file number in `log_recycle_files_` until
after the file has been renamed. Full purge uses that list to decide
which files to keep, so it can no longer delete a file pending recycling.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5900

Test Plan: new unit test

Differential Revision: D19771719

Pulled By: ajkr

fbshipit-source-id: 094346349ca3fb499712e62de03905acc30b5ce8
2020-02-18 13:54:13 -08:00
Andrew Kryczka 0f9dcb88b2 Return NotSupported from WriteBatchWithIndex::DeleteRange (#5393)
Summary:
As discovered in https://github.com/facebook/rocksdb/issues/5260 and https://github.com/facebook/rocksdb/issues/5392, reads on the indexed batch do not account for range tombstones. So, return `Status::NotSupported` from `WriteBatchWithIndex::DeleteRange` until we properly support it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5393

Test Plan: added unit test

Differential Revision: D19912360

Pulled By: ajkr

fbshipit-source-id: 0bbfc978ea015d64516ca708fce2429abba524cb
2020-02-18 11:18:25 -08:00
acelyc111 3a3457575d Fix compile error when LZ4 is up to r123 (#6412)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6412

Differential Revision: D19914063

Pulled By: ajkr

fbshipit-source-id: 4e401e665d4b449d24c4cdec35a4585eeda95996
2020-02-14 15:55:23 -08:00
Manuel Ung dc23c125c3 WriteUnPrepared: Untracked keys (#6404)
Summary:
For write unprepared, some applications may bypass the transaction api, and write keys directly into the write batch. However, since they are not tracked, rollbacks (both for savepoint and transaction) are not aware that these keys have to be rolled back.

The fix is to track them in `WriteUnpreparedTxn::untracked_keys_`. This is populated whenever we flush unprepared batches into the DB.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6404

Differential Revision: D19842023

Pulled By: lth

fbshipit-source-id: a9edfc643d5c905fc89da9a9a9094d30c9b70108
2020-02-14 11:31:39 -08:00
Cheng Chang 152f8a8ffe Remove unnecessary computation of index (#6406)
Summary:
`index` can be replaced by  `iter`, saving the computation of `index++`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6406

Test Plan: make check

Differential Revision: D19905056

Pulled By: cheng-chang

fbshipit-source-id: add4638959c0d2e4e77a11f3fa04ffabaf0de790
2020-02-14 08:26:23 -08:00
Cheng Chang 4034e289ad Fail fast in paranoid mode when LoadTableHandlers fail during recovering (#6368)
Summary:
Previously, when recovering version set, LoadTableHandlers failures are ignored.
If paranoid_checks is true, this failure should not be ignored, otherwise, the opened db might be in an inconsistent state.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6368

Test Plan: make check

Differential Revision: D19713459

Pulled By: cheng-chang

fbshipit-source-id: 68cb94f4f2cc43f8b024b14755193cd45cfcad55
2020-02-14 08:17:10 -08:00
wolfkdy 29e24434fe refine code (#6420)
Summary:
I create a new branch from the branch new upsteram/master and "git merge --squash".
Maybe it will fix everything.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420

Differential Revision: D19897152

Pulled By: zhichao-cao

fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136
2020-02-13 18:55:02 -08:00
Manuel Ung 908b1ee64e WriteUnPrepared: Fix assertion during recovery (#6419)
Summary:
During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`.

To fix, reset unprepared_batch_ to false after `MarkEndPrepare`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419

Differential Revision: D19896148

Pulled By: lth

fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3
2020-02-13 18:52:05 -08:00
Manuel Ung fb571509a7 WriteUnPrepared: Enable WAL during crash recovery (#6418)
Summary:
Unfortunately, it seems like mysqld reuses xids across machine restarts. When that happens, we could have something like the following happening:

```
BEGIN_PREPARE(unprepared) Put(a) END_PREPARE(xid = 1)
-- crash and recover with Put(a) rolled back as it was not prepared
BEGIN_PREPARE(prepared) Put(b) END_PREPARE(xid = 1)
COMMIT(xid = 1)
-- crash and recover with both a, b
```

To solve this, we will have to log the rollback batch into the WAL during recovery.

WritePrepared already logs the rollback batch into the WAL, if a rollback happens after prepare, so there is no problem there.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6418

Differential Revision: D19896151

Pulled By: lth

fbshipit-source-id: 2ff65ddc5fe75efd57736fed4b7cd7a109d26609
2020-02-13 18:44:39 -08:00
sdong ac8e89a443 Should flush and sync WAL when writing it in DB::Open() (#6417)
Summary:
A recent fix related to 2pc https://github.com/facebook/rocksdb/pull/6313/ writes something to WAL, but does not flush or sync. This causes assertion failure "impl->TEST_WALBufferIsEmpty()" if manual_wal_flush = true. We should fsync the entry to make sure a second power reset can recover.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6417

Test Plan: Add manual_wal_flush=true case in TransactionTest.DoubleCrashInRecovery and fix a bug in the test so that the bug can be reproduced. It passes with the fix.

Differential Revision: D19894537

fbshipit-source-id: f1e84e49e2269f583c6019743118292cd8b6598e
2020-02-13 18:41:04 -08:00
Cheng Chang 46516778dd Fix flaky test DecreaseNumBgThreads (#6393)
Summary:
The DecreaseNumBgThreads test keeps failing on Windows in AppVeyor.
It fails because it depends on a timed wait for the tasks to be dequeued from the threadpool's internal queue, but within the specified time, the task might have not been scheduled onto the newly created threads.
https://github.com/facebook/rocksdb/pull/6232 tries to fix this by waiting for longer time to let the threads scheduled.
This PR tries to fix this by replacing the timed wait with a synchronization on the task's internal conditional variable.
When the number of threads increases, instead of guessing the time needed for the task to be scheduled, it directly blocks on the conditional variable until the task starts running.
But when thread number is reduced, it still does a timed wait, but this does not lead to the flakiness now, will try to remove these timed waits in a future PR.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6393

Test Plan: Wait to see whether AppVeyor tests pass.

Differential Revision: D19890928

Pulled By: cheng-chang

fbshipit-source-id: 4e56e4addf625c98c0876e62d9d57a6f0a156f76
2020-02-13 17:27:18 -08:00
sdong acfee40af5 Remove IO URING compiler flags (#6415)
Summary:
Since IO Uring feature is not stable. Remove it from buck configuration.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6415

Test Plan: See internal build pass

Differential Revision: D19892988

fbshipit-source-id: 7fc01efc2af5ed707fb8e4e4674223aeb83cd5ea
2020-02-13 16:51:10 -08:00
Huisheng Liu 5138764eb5 Fix destroydb (#6308)
Summary:
It's observed on Windows DestroyDB failed to remove the log file because the logger is still alive in sst file manager and holding a handle to the log file. This fix makes sure the logger is released before attempt to clear the database directory.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6308

Differential Revision: D19818829

Pulled By: riversand963

fbshipit-source-id: 54c3e6859aadaaba4a49b3e851b73dc35ec7dc6a
2020-02-13 11:21:27 -08:00
sdong df3f33dd05 Fix db_bench LITE build recently broken (#6411)
Summary:
A recent change https://github.com/facebook/rocksdb/pull/6386 broke LITE build in a trivial way. Fix it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6411

Test Plan: Run "LITE=1 make all"

Differential Revision: D19871765

fbshipit-source-id: 74f0ad3f8a9d666fbde0da7fd29ba1547a811f77
2020-02-13 10:52:50 -08:00
Cheng Chang a676001f95 Revert usage of Defer. (#6410)
Summary:
Seems like this caused the following test failure on AppVeyor:
DBTest2.CrashInRecoveryMultipleCF
c:\projects\rocksdb\db\db_test_util.cc(107): error: DestroyDB(dbname_, options)
IO error: Failed to delete: C:\projects\rocksdb\db_tests\\testrocksdb-3112//db_test2_10791409581227174103/000013.sst: Access is denied.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6410

Test Plan: Wait to see whether the AppVeyor test passes.

Differential Revision: D19879872

Pulled By: cheng-chang

fbshipit-source-id: 59a9c55ca88566e9210c0b715ecc45a4fd9afe26
2020-02-13 10:31:32 -08:00
sdong 6e97d4de00 By default turn IO Uring off. (#6405)
Summary:
We realized bugs related to IO Uring. Turn it off by default.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6405

Test Plan: Manually run build_tools/build_detect_platform and observe outputs.

Differential Revision: D19862792

fbshipit-source-id: 5d5e8e2762997b72a145ae59389ef3d7e4ccd060
2020-02-12 18:01:49 -08:00
Burton Li e64508917b db_bench supports for generating random variable sized value. (#6386)
Summary:
1. `db_bench` now supports `value_size_distribution_type`, `value_size_min`, `value_size_max` options for generating random variable sized value.
2. Added `blob_db_compression_type` option for BlobDB to enable blob compression.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6386

Differential Revision: D19859406

Pulled By: zhichao-cao

fbshipit-source-id: ace52674090023fde15d832392110bf288a8e215
2020-02-12 14:47:03 -08:00
anand76 3e49249d30 Ensure all MultiGet IO errors are propagated to user (#6403)
Summary:
Unrevert the previous fix to propagate error status, and an additional fix to not treat a memtable lookup MergeInProgress status as an error.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6403

Test Plan:
Unit tests
Tried running stress tests but couldn't repro the stress failure

Differential Revision: D19846721

Pulled By: anand1976

fbshipit-source-id: 7db10cccbdc863d9b559497f0a46b608d2488ca4
2020-02-11 17:27:22 -08:00
Tomas Kolda e412a426d6 JNI direct buffer support for basic operations (#2283)
Summary:
It is very useful to support direct ByteBuffers in Java. It allows to have zero memory copy and some serializers are using that directly so one do not need to create byte[] array for it.

This change also contains some fixes for Windows JNI build.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/2283

Differential Revision: D19834971

Pulled By: pdillinger

fbshipit-source-id: 44173aa02afc9836c5498c592fd1ea95b6086e8e
2020-02-11 14:48:30 -08:00
Yanqin Jin 28aa09dcce Pass info_log_level to the inner logger of AutoRollLogger (#6388)
Summary:
Before this fix, the info_log_level passed from CreateLoggerFromOptions() will
be ignored by AutoRollLogger::logger_. This PR fixes it by setting the info log
level of logger_ during ResetLogger().

Test plan (dev server):
```
COMPILE_WITH_TSAN=1 make all && make check
make all && make check
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6388

Differential Revision: D19828045

Pulled By: riversand963

fbshipit-source-id: e1ac7de3a2090bee53b6c667f71a11f1774163e6
2020-02-10 22:26:08 -08:00
anand76 35ed530d2c Revert "Check KeyContext status in MultiGet (#6387)" (#6401)
Summary:
This reverts commit d70011bccc. The commit is causing some stress test failure due to unexpected Status::MergeInProgress() return for some keys.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6401

Differential Revision: D19826623

Pulled By: anand1976

fbshipit-source-id: edd634cede9cb7bdd2cb8f46e662ea709b16d2f1
2020-02-10 22:23:36 -08:00
Cheng Chang d3ba398bbd Update unit tests for PinnableSlice (#6399)
Summary:
1. remove AssertEmpty because calling methods on moved objects is discouraged.
2. add a test to assert that the internal buffer is moved instead of being copied.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6399

Test Plan:
make slice_test && ./slice_test
USE_CLANG=1 make analyze

Differential Revision: D19825372

Pulled By: cheng-chang

fbshipit-source-id: 2e26f8ce5ec3edbfce067db045e80bd433e704f4
2020-02-10 18:13:27 -08:00
Cheng Chang dafb568052 Add utility class Defer (#6382)
Summary:
Add a utility class `Defer` to defer the execution of a function until the Defer object goes out of scope.
Used in VersionSet:: ProcessManifestWrites as an example.
The inline comments for class `Defer` have more details.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6382

Test Plan: `make defer_test version_set_test && ./defer_test && ./version_set_test`

Differential Revision: D19797538

Pulled By: cheng-chang

fbshipit-source-id: b1a9b7306e4fd4f48ec2ab55783caa561a315f0f
2020-02-10 17:59:47 -08:00
Levi Tamasi cbf5f3be43 Do not move VersionEdit into AtomicGroupReadBuffer (#6400)
Summary:
https://github.com/facebook/rocksdb/pull/6383 surfaced an issue with
`VersionSet`/`ReactiveVersionSet` and `AtomicGroupReadBuffer::AddEdit`
(which was added in https://github.com/facebook/rocksdb/pull/5411):
`AddEdit` moves the `VersionEdit` passed to it into `replay_buffer_`,
however, the client `VersionSet` classes keep using it afterwards. This
*seemed to* work before the refactoring but it really did not: since
`VersionEdit` used to have a user-declared destructor, no move
constructor/move assignment operator was generated, and the `move` in
`AddEdit` was really a copy. The patch makes the copy explicit. Note: it
should be possible to rework this logic so that we can get away
with the move but for now, this should fix the issue.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6400

Test Plan:
`make check`
`make analyze`

Differential Revision: D19824466

Pulled By: ltamasi

fbshipit-source-id: f38033967daf2a39c78dcd6e12978bafe37632b4
2020-02-10 17:15:42 -08:00
Zhichao Cao 4369f2c7bb Checksum for each SST file and stores in MANIFEST (#6216)
Summary:
In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata.

Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string).  A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216

Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check.

Differential Revision: D19171461

Pulled By: zhichao-cao

fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
2020-02-10 15:52:52 -08:00
sdong 594e815e32 Make clang analyze happy with options_test (#6398)
Summary:
clang analysis shows following warning:

options/options_test.cc:1554:24: warning: The left operand of '-' is a garbage value
            (file_size - 1) / readahead_size + 1);
             ~~~~~~~~~ ^

Explicitly initialize file_size and add an assertion to make clang analysis happy.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6398

Test Plan: Run "make analysis" and see the warning goes away.

Differential Revision: D19819662

fbshipit-source-id: 1589ea91c0c8f78242538f01448e4ad0e5fbc219
2020-02-10 15:50:25 -08:00
sdong b2bc1da561 Try to fix some analysis failures
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6384

Test Plan: Wait and see the analysis result.

Differential Revision: D19781072

fbshipit-source-id: 75e7cb6ee619ebd289841eaabea03dd075c09d3b
2020-02-10 13:37:14 -08:00
Yutian Li 2e0159ec9e Add error status for no_slowdown & low priority write (#6396)
Summary:
When `no_slowdown` is enabled, it returns `Status::Incomplete("Write stall")` if a stall would occur. This patch adds descriptive text for when `no_slowdown` and `low_pri` are enabled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6396

Differential Revision: D19808978

Pulled By: cheng-chang

fbshipit-source-id: a53b0d25ed414c821a086531e0222027f925e627
2020-02-10 12:33:16 -08:00
Kefu Chai debc4ef18b utilities/env_librados: copy use bufferlist::iterator (#6395)
Summary:
to adapt the change in ceph upstream where the bufferlist::copy() method
was removed in
https://github.com/ceph/ceph/commit/c724369010a753bd44e11a534d1f42156c4fc12d

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6395

Differential Revision: D19816815

Pulled By: zhichao-cao

fbshipit-source-id: 9210767b91af0ecdcf5dfaa3e70edcaeea55135f
2020-02-10 11:31:16 -08:00
blackyblack 84b41a6969 Only add gtest target when building with tests (#6377)
Summary:
External CMAKE projects cannot add own gtest target due to CMP0002 policy. We should not add gtest target unless we build with tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6377

Differential Revision: D19801708

fbshipit-source-id: 662484683e8737e10397ddb0d17705da5243c4f9
2020-02-07 17:16:44 -08:00
anand76 d70011bccc Check KeyContext status in MultiGet (#6387)
Summary:
Currently, any IO errors and checksum mismatches while reading data
blocks, are being ignored by the batched MultiGet. Its only looking at
the GetContext state. Fix that.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6387

Test Plan: Add unit tests

Differential Revision: D19799819

Pulled By: anand1976

fbshipit-source-id: 46133dccbb04e64067b9fe6cda73e282203db969
2020-02-07 16:48:16 -08:00
Robert Yang 4e457278fa db/write_thread.cc: Initialize state (#6275)
Summary:
Fixed an error when compiled with -Og:
db/write_thread.cc:183:14: error: 'state' may be used uninitialized in this function [-Werror=maybe-uninitialized]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6275

Differential Revision: D19381755

fbshipit-source-id: a90bf3cd4a7248d9d71219e918fc6253deb97e3c
2020-02-07 15:20:38 -08:00
sdong 876c2dbff4 Allow readahead when reading option files. (#6372)
Summary:
Right, when reading from option files, no readahead is used and 8KB buffer is used. It might introduce high latency if the file system provide high latency and doesn't do readahead. Instead, introduce a readahead to the file. When calling inside DB, infer the value from options.log_readahead. Otherwise, a default 512KB readahead size is used.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6372

Test Plan: Add --log_readahead_size in db_bench. Run it with several options and observe read size from option files using strace.

Differential Revision: D19727739

fbshipit-source-id: e6d8053b0a64259abc087f1f388b9cd66fa8a583
2020-02-07 15:18:26 -08:00
Cheng Chang b42fa1497f Support move semantics for PinnableSlice (#6374)
Summary:
It's logically correct for PinnableSlice to support move semantics to transfer ownership of the pinned memory region. This PR adds both move constructor and move assignment to PinnableSlice.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6374

Test Plan:
A set of unit tests for the move semantics are added in slice_test.
So `make slice_test && ./slice_test`.

Differential Revision: D19739254

Pulled By: cheng-chang

fbshipit-source-id: f898bd811bb05b2d87384ec58b645e9915e8e0b1
2020-02-07 14:26:26 -08:00
Chad Austin 25fbdc5a31 Fix Buck build on macOS (#6378)
Summary:
liburing is a Linux-specific dependency, so make sure it's configured in the Linux-only Buck rules.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6378

Test Plan:
```
~/fbcode $ cp internal_repo_rocksdb/repo/TARGETS rocksdb/src
~/fbcode $ buck build mode/mac eden
```

Reviewed By: chadaustin

Differential Revision: D19760039

Pulled By: riversand963

fbshipit-source-id: 2abfce81c8b17965ef76012262cd117708e0294f
2020-02-07 14:20:12 -08:00
Levi Tamasi 752c87af78 Clean up VersionEdit a bit (#6383)
Summary:
This is a bunch of small improvements to `VersionEdit`. Namely, the patch

* Makes the names and order of variables, methods, and code chunks related
  to the various information elements more consistent, and adds missing
  getters for the sake of completeness.
* Initializes previously uninitialized stack variables.
* Marks all getters const to improve const correctness.
* Adds in-class initializers and removes the default ctor that would
  create an object with uninitialized built-in fields and call `Clear`
  afterwards.
* Adds a new type alias for new files and changes the existing `typedef`
  for deleted files into a type alias as well.
* Makes the helper method `DecodeNewFile4From` private.
* Switches from long-winded iterator syntax to range based loops in a
  couple of places.
* Fixes a couple of assignments where an integer 0 was assigned to
  boolean members.
* Fixes a getter which used to return a `const std::string` instead of
the intended `const std::string&`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383

Test Plan: make check

Differential Revision: D19780537

Pulled By: ltamasi

fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0
2020-02-07 13:27:06 -08:00
Cheng Chang 5f478b9f75 Remove outdated comment (#6379)
Summary:
Since the logic for handling IDENTITY file is now inside `NewDB`, the comment above `NewDB` is no longer relevant.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6379

Test Plan: not needed

Differential Revision: D19795440

Pulled By: cheng-chang

fbshipit-source-id: 0b1cca87ac6d92474701c46aa4c8d4d708bfa19b
2020-02-07 13:18:43 -08:00
Levi Tamasi 1b4be4cac9 BlobDB: ignore trivially moved files when updating the SST<->blob file mapping (#6381)
Summary:
BlobDB keeps track of the mapping between SSTs and blob files using
the `OnFlushCompleted` and `OnCompactionCompleted` callbacks of
the `EventListener` interface: upon receiving a flush notification, a link
is added between the newly flushed SST and the corresponding blob file;
for compactions, links are removed for the inputs and added for the outputs.
The earlier code performed this link deletion and addition even for
trivially moved files; the new code walks through the two lists together
(in a fashion that's similar to merge sort) and skips such files.
This should mitigate https://github.com/facebook/rocksdb/issues/6338,
wherein an assertion is triggered with the earlier code when a compaction
notification for a trivial move precedes the flush notification for the
moved SST.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6381

Test Plan: make check

Differential Revision: D19773729

Pulled By: ltamasi

fbshipit-source-id: ae0f273ded061110dd9334e8fb99b0d7786650b0
2020-02-07 12:50:57 -08:00
Cheng Chang 107a7ca930 Remove inappropriate comments (#6371)
Summary:
The comments are for iterators, not Cleanable.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6371

Test Plan: no need

Differential Revision: D19727527

Pulled By: cheng-chang

fbshipit-source-id: c74aeffa27ea0ce15a36ff6f9694826712cd1c70
2020-02-07 12:35:24 -08:00
Cheng Chang 0a74e1b958 Add status checks during DB::Open (#6380)
Summary:
Several statuses were not checked during DB::Open.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6380

Test Plan: make check

Differential Revision: D19780237

Pulled By: cheng-chang

fbshipit-source-id: c8d189d20344bd1607890dd1449345bda2ef96b9
2020-02-07 12:32:09 -08:00
Yanqin Jin f361cedf06 Atomic flush rollback once on failure (#6385)
Summary:
Before this fix, atomic flush codepath may hit an assertion failure on a specific failure case.
If all flush jobs within an atomic flush succeed (they do not write to MANIFEST), but batch writing version edits to MANIFEST fails, then `cfd->imm()->RollbackMemTableFlush()` will be called twice, and the second invocation hits assertion failure `assert(m->flush_in_progress_)` since the first invocation resets the variable `flush_in_progress_` to false already.

Test plan (dev server):
```
./db_flush_test --gtest_filter=DBAtomicFlushTest/DBAtomicFlushTest.RollbackAfterFailToInstallResults
make check
```
Both must succeed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6385

Differential Revision: D19782943

Pulled By: riversand963

fbshipit-source-id: 84e1592625e729d1b70fdc8479959387a74cb121
2020-02-07 10:52:10 -08:00
atul c6f75516b7 Fixing the documentation of the function (#4803)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6354

Differential Revision: D19725459

Pulled By: riversand963

fbshipit-source-id: fded24576251bfa4b289399f0909f1fe43426e28
2020-02-06 10:30:44 -08:00
Cheng Chang f5f79f01a2 Be able to read compatible leveldb sst files (#6370)
Summary:
In `DBSSTTest.SSTsWithLdbSuffixHandling`, some sst files are renamed to ldb files, the original intention of the test is to test that the ldb files can be loaded along with the sst files.

The original test checks this by `ASSERT_NE("NOT_FOUND", Get(Key(k)))`, but the problem is `Get(Key(k))` returns IO error due to path not found instead of NOT_FOUND, so the success of ASSERT_NE does not mean the key can be retrieved.

This PR updates the test to make sure Get(Key(k)) returns the original value.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6370

Test Plan: make db_sst_test && ./db_sst_test

Differential Revision: D19726278

Pulled By: cheng-chang

fbshipit-source-id: 993127f56457b315e669af4eeb92d6f956b7a4b7
2020-02-06 10:15:44 -08:00
sdong 24c9dce825 Remove include math.h (#6373)
Summary:
We see some odd errors complaining math. However, it doesn't seem that it is needed to be included. Remove the include of math.h. Just removing it from db_bench doesn't seem to break anything. Replacing sqrt from std::sqrt seems to work for histogram.cc
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6373

Test Plan: Watch Travis and appveyor to run.

Differential Revision: D19730068

fbshipit-source-id: d3ad41defcdd9f51c2da1a3673fb258f5dfacf47
2020-02-05 21:00:49 -08:00
Mike Kolupaev 1ed7d9b1b5 Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363)
Summary:
Before this PR it calls GetFileSize() once for each sst file in the DB. This can take a long time if there are be tens of thousands of sst files (e.g. in thousands of column families), and even longer if Env is talking to some remote service rather than local filesystem. This PR makes DB::Open() use sst file sizes that are already known from manifest (typically almost all files in the DB) and only call GetFileSize() for non-sst or obsolete files. Note that GetFileSize() is also called and checked against manifest in CheckConsistency(), so the calls in SstFileManagerImpl were completely redundant.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6363

Test Plan: deployed to a test cluster, looked at a dump of Env calls (from a custom instrumented Env) - no more thousands of GetFileSize()s.

Differential Revision: D19702509

Pulled By: al13n321

fbshipit-source-id: 99f8110620cb2e9d0c092dfcdbb11f3af4ff8b73
2020-02-04 13:41:53 -08:00
sdong 3a073234da Consolidate ReadFileToString() (#6366)
Summary:
It's a minor refactoring. We have two ReadFileToString() but they are very similar. Make the one with Env argument calls the one with FS argument instead.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6366

Test Plan: Run all existing tests

Differential Revision: D19712332

fbshipit-source-id: 5ae6fabf6355938690d95cda52afd1f39e0a7823
2020-02-04 11:39:23 -08:00
sdong 69c8614815 Avoid to get manifest file size when recovering from it. (#6369)
Summary:
Right now RocksDB gets manifest file size before recovering from it. The information is available in LogReader. Use it instead to prevent one file system call.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6369

Test Plan: Run all existing tests

Differential Revision: D19714872

fbshipit-source-id: 0144be324d403c99e3da875ea2feccc8f64e883d
2020-02-04 11:39:23 -08:00
Mike Kolupaev 637e64b9ac Add an option to prevent DB::Open() from querying sizes of all sst files (#6353)
Summary:
When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open.

If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes.

We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353

Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble.

Differential Revision: D19656425

Pulled By: al13n321

fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82
2020-02-04 01:27:26 -08:00
anand76 7330ec0ff1 Fix a test failure in error_handler_test (#6367)
Summary:
Fix an intermittent failure in
DBErrorHandlingTest.CompactionManifestWriteError due to a race between
background error recovery and the main test thread calling
TEST_WaitForCompact().
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6367

Test Plan: Run the test using gtest_parallel

Differential Revision: D19713802

Pulled By: anand1976

fbshipit-source-id: 29e35dc26e0984fe8334c083e059f4fa1f335d68
2020-02-03 18:16:52 -08:00
sdong f195d8d523 Use ReadFileToString() to get content from IDENTITY file (#6365)
Summary:
Right now when reading IDENTITY file, we use a very similar logic as ReadFileToString() while it does an extra file size check, which may be expensive in some file systems. There is no reason to duplicate the logic. Use ReadFileToString() instead.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6365

Test Plan: RUn all existing tests.

Differential Revision: D19709399

fbshipit-source-id: 3bac31f3b2471f98a0d2694278b41e9cd34040fe
2020-02-03 17:40:49 -08:00
sdong 36c504be17 Avoid create directory for every column families (#6358)
Summary:
A relatively recent regression causes for every CF, create and open directory is called for the DB directory, unless CF has a private directory. This doesn't scale well with large number of column families.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6358

Test Plan: Run all existing tests and see it pass. strace with db_bench --num_column_families and observe it doesn't open directory for number of column families.

Differential Revision: D19675141

fbshipit-source-id: da01d9216f1dae3f03d4064fbd88ce71245bd9be
2020-02-03 14:13:39 -08:00
Huisheng Liu eb4d6af5ae Error handler test fix (#6266)
Summary:
MultiDBCompactionError fails when it verifies the number of files on level 0 and level 1 without waiting for compaction to finish.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6266

Differential Revision: D19701639

Pulled By: riversand963

fbshipit-source-id: e96d511bcde705075f073e0b550cebcd2ecfccdc
2020-02-03 13:32:53 -08:00
Adam Retter 7242dae7fe Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.

**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.

Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).

In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.

In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.

 ---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.

With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).

These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.

 ---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.

```
ComparatorBenchmarks.put                                                native_bytewise  thrpt   25  124483.795 ± 2032.443  ops/s
ComparatorBenchmarks.put                                        native_reverse_bytewise  thrpt   25  114414.536 ± 3486.156  ops/s
ComparatorBenchmarks.put              java_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   17228.250 ± 1288.546  ops/s
ComparatorBenchmarks.put          java_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   16035.865 ± 1248.099  ops/s
ComparatorBenchmarks.put                java_bytewise_non-direct_reused-64_thread-local  thrpt   25   21571.500 ±  871.521  ops/s
ComparatorBenchmarks.put                  java_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   23613.773 ± 8465.660  ops/s
ComparatorBenchmarks.put              java_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   16768.172 ± 5618.489  ops/s
ComparatorBenchmarks.put                    java_bytewise_direct_reused-64_thread-local  thrpt   25   23921.164 ± 8734.742  ops/s
ComparatorBenchmarks.put                              java_bytewise_non-direct_no-reuse  thrpt   25   17899.684 ±  839.679  ops/s
ComparatorBenchmarks.put                                  java_bytewise_direct_no-reuse  thrpt   25   22148.316 ± 1215.527  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   11311.126 ±  820.602  ops/s
ComparatorBenchmarks.put  java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   11421.311 ±  807.210  ops/s
ComparatorBenchmarks.put        java_reverse_bytewise_non-direct_reused-64_thread-local  thrpt   25   11554.005 ±  960.556  ops/s
ComparatorBenchmarks.put          java_reverse_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   22960.523 ± 1673.421  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   18293.317 ± 1434.601  ops/s
ComparatorBenchmarks.put            java_reverse_bytewise_direct_reused-64_thread-local  thrpt   25   24479.361 ± 2157.306  ops/s
ComparatorBenchmarks.put                      java_reverse_bytewise_non-direct_no-reuse  thrpt   25    7942.286 ±  626.170  ops/s
ComparatorBenchmarks.put                          java_reverse_bytewise_direct_no-reuse  thrpt   25   11781.955 ± 1019.843  ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252

Differential Revision: D19331064

Pulled By: pdillinger

fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 12:30:13 -08:00
sdong 800d24ddc5 Fix DBTest2.ChangePrefixExtractor LITE build (#6356)
Summary:
DBTest2.ChangePrefixExtractor fails in LITE build because LITE build doesn't support adaptive build. Fix it by removing the stats check but only check correctness.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6356

Test Plan: Run the test with both of LITE and non-LITE build.

Differential Revision: D19669537

fbshipit-source-id: 6d7dd6c8a79f18e80ca1636864b9c71922030d8e
2020-01-31 15:44:14 -08:00
Maysam Yabandeh 01ab882ba3 Fix release warning for unused bg_canceled
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6357

Differential Revision: D19670931

Pulled By: maysamyabandeh

fbshipit-source-id: d528c4c7f9450f1f38b9d2a36e0d5d0865b39be9
2020-01-31 15:09:10 -08:00
sdong ec496347bc Add a unit test for prefix extractor changes (#6323)
Summary:
Add a unit test for prefix extractor change, including a check that fails due to a bug.
Also comment out the partitioned filter case which will fail the test too.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6323

Test Plan: Run the test and it passes (and fails if the SeekForPrev() part is uncommented)

Differential Revision: D19509744

fbshipit-source-id: 678202ca97b5503e9de73b54b90de9e5ba822b72
2020-01-31 11:02:03 -08:00
Maysam Yabandeh 2243030bc5 Cancel bg jobs before deleting WritePrepared DB in stress tests (#6355)
Summary:
Background jobs in WritePrepared DB might access the db via a snapshot checker callback. The stress tests therefore should cancel background jobs before deleting the db in ::Reopen.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6355

Differential Revision: D19664132

Pulled By: maysamyabandeh

fbshipit-source-id: 6060a830e8aad0015c10448286ad37c8a346ac01
2020-01-31 10:29:11 -08:00
Maysam Yabandeh 3316d29221 Disable recycle_log_file_num when it is incompatible with recovery mode (#6351)
Summary:
Non-zero recycle_log_file_num is incompatible with kPointInTimeRecovery and kAbsoluteConsistency recovery modes. Currently SanitizeOptions changes the recovery mode to kTolerateCorruptedTailRecords, while to resolve this option conflict it makes more sense to compromise recycle_log_file_num, which is a performance feature, instead of wal_recovery_mode, which is a safety feature.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6351

Differential Revision: D19648931

Pulled By: maysamyabandeh

fbshipit-source-id: dd0bf78349edc007518a00c4d63931fd69294ad7
2020-01-31 07:28:30 -08:00
Yanqin Jin f2fbc5d668 Shorten certain test names to avoid infra failure (#6352)
Summary:
Unit test names, together with other components,  are used to create log files
during some internal testing. Overly long names cause infra failure due to file
names being too long.

Look for internal tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6352

Differential Revision: D19649307

Pulled By: riversand963

fbshipit-source-id: 6f29de096e33c0eaa87d9c8702f810eda50059e7
2020-01-30 23:10:24 -08:00
Burton Li c9a5e48762 fix build warnnings on MSVC (#6309)
Summary:
Fix build warnings on MSVC. siying
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6309

Differential Revision: D19455012

Pulled By: ltamasi

fbshipit-source-id: 940739f2c92de60e47cc2bed8dd7f921459545a9
2020-01-30 16:07:26 -08:00
Peter Dillinger 90c71aa5d9 Don't download from (unreliable) maven.org (#6348)
Summary:
I set up a mirror of our Java deps on github so we can download
them through github URLs rather than maven.org, which is proving
terribly unreliable from Travis builds.

Also sanitized calls to curl, so they are easier to read and
appropriately fail on download failure.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6348

Test Plan: CI

Differential Revision: D19633621

Pulled By: pdillinger

fbshipit-source-id: 7eb3f730953db2ead758dc94039c040f406790f3
2020-01-30 11:02:08 -08:00
anand76 fb05b5a652 Force a new manifest file if append to current one fails (#6331)
Summary:
Fix for issue https://github.com/facebook/rocksdb/issues/6316

When an append/sync of the manifest file fails due to an IO error such
as NoSpace, we don't always put the DB in read-only mode. This is true
for flush and compactions, as well as foreground operatons such as column family
add/drop, CompactFiles etc. Subsequent changes to the DB will be
recorded in the same manifest file, which would have a corrupted record
in the middle due to the previous failure. On next DB::Open(), it will
fail to process the full manifest and data will be lost.

To fix this, we reset VersionSet::descriptor_log_ on append/sync
failure, which will force a new manifest file to be written on the next
append.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6331

Test Plan: Add new unit tests in error_handler_test.cc

Differential Revision: D19632951

Pulled By: anand1976

fbshipit-source-id: 68d527cb6e59a94cbbbf9f5a17a7f464381d51e3
2020-01-30 10:56:29 -08:00
Levi Tamasi 9e3ace42a4 Add statistics for BlobDB GC (#6296)
Summary:
The patch adds statistics support to the new BlobDB garbage collection implementation;
namely, it adds support for the following (pre-existing) tickers:

`BLOB_DB_GC_NUM_FILES`: the number of blob files obsoleted by the GC logic.
`BLOB_DB_GC_NUM_NEW_FILES`: the number of new blob files generated by the GC logic.
`BLOB_DB_GC_FAILURES`: the number of failed GC passes (where a GC pass is
equivalent to a (sub)compaction).
`BLOB_DB_GC_NUM_KEYS_RELOCATED`: the number of blobs relocated to new blob
files by the GC logic.
`BLOB_DB_GC_BYTES_RELOCATED`: the total size of blobs relocated to new blob files.

The tickers `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`,
`BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, and
`BLOB_DB_GC_MICROS` are not relevant for the new GC logic, and are thus marked
deprecated.

The patch also adds a couple of log messages that log the number and total size of
blobs encountered and relocated during a GC pass, as well as the number of blob
files created and obsoleted.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6296

Test Plan: Extended unit tests and used the BlobDB mode of `db_bench`.

Differential Revision: D19402513

Pulled By: ltamasi

fbshipit-source-id: d53d2bfbf4928a1db1e9346c67ebb9007b8932ec
2020-01-29 16:46:16 -08:00
sdong 71874c5aaf Fix LITE build with DBTest2.AutoPrefixMode1 (#6346)
Summary:
DBTest2.AutoPrefixMode1 doesn't pass because auto prefix mode is not supported there.
Fix it by disabling the test.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6346

Test Plan: Run DBTest2.AutoPrefixMode1 in lite mode

Differential Revision: D19627486

fbshipit-source-id: fbde75260aeecb7e6fc406e09c19a71a95aa5f08
2020-01-29 16:43:42 -08:00
Peter Dillinger 23dcf2759d Upload DB dir for all crash tests (#6344)
Summary:
Difficult to root cause crash test failures without archiving
db dir. Now all crash test configurations should save the db dir.

Also exit with error code on bad command.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6344

Test Plan:
Hmm, how about this:

    for TARGET in stress_crash asan_crash ubsan_crash tsan_crash; do EMAIL=email ONCALL=oncall TRIGGER=all SUBSCRIBER=sub build_tools/rocksdb-lego-determinator $TARGET > tmp && node -c tmp && grep -q Upload tmp || echo Bad; done

Differential Revision: D19625605

Pulled By: pdillinger

fbshipit-source-id: cb84aa93ee80b4534f4c61b90f0e0f99a41155d5
2020-01-29 15:59:07 -08:00
sdong 02ac6c9a3c Fix db_bloom_filter_test clang LITE build (#6340)
Summary:
db_bloom_filter_test break with clang LITE build with following message:

db/db_bloom_filter_test.cc:23:29: error: unused variable 'kPlainTable' [-Werror,-Wunused-const-variable]
static constexpr PseudoMode kPlainTable = -1;
                            ^

Fix it by moving the declaration out of LITE build
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6340

Test Plan:
USE_CLANG=1 LITE=1 make db_bloom_filter_test
and without LITE=1

Differential Revision: D19609834

fbshipit-source-id: 0e88f5c6759238a94f9880d84c785ac18e7cdd7e
2020-01-29 12:57:48 -08:00
Maysam Yabandeh 2f973ca96e Double Crash in kPointInTimeRecovery with TransactionDB (#6313)
Summary:
In WritePrepared there could be gap in sequence numbers. This breaks the trick we use in kPointInTimeRecovery which assume the first seq in the log right after the corrupted log is one larger than the last seq we read from the logs. To let this trick keep working, we add a dummy entry with the expected sequence to the first log right after recovery.
Also in WriteCommitted, if the log right after the corrupted log is empty, since it has no sequence number to let the sequential trick work, it is assumed as unexpected behavior. This is however expected to happen if we close the db after recovering from a corruption and before writing anything new to it. To remedy that, we apply the same technique by writing a dummy entry to the log that is created after the corrupted log.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6313

Differential Revision: D19458291

Pulled By: maysamyabandeh

fbshipit-source-id: 09bc49e574690085df45b034ca863ff315937e2d
2020-01-29 11:40:55 -08:00
Adam Retter a07a9dc904 Reduce the need to re-download dependencies (#6318)
Summary:
Both changes are related to RocksJava:

1. Allow dependencies that are already present on the host system due to Maven to be reused in Docker builds.

2. Extend the `make clean-not-downloaded` target to RocksJava, so that libraries needed as dependencies for the test suite are not deleted and re-downloaded unnecessarily.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6318

Differential Revision: D19608742

Pulled By: pdillinger

fbshipit-source-id: 25e25649e3e3212b537ac4512b40e2e53dc02ae7
2020-01-29 08:01:56 -08:00
sdong 8f2bee6747 Add ReadOptions.auto_prefix_mode (#6314)
Summary:
Add a new option ReadOptions.auto_prefix_mode. When set to true, iterator should return the same result as total order seek, but may choose to do prefix seek internally, based on iterator upper bounds. Also fix two previous bugs when handling prefix extrator changes: (1) reverse iterator should not rely on upper bound to determine prefix. Fix it with skipping prefix check. (2) block-based filter is not handled properly.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6314

Test Plan: (1) add a unit test; (2) add the check to stress test and run see whether it can pass at least one run.

Differential Revision: D19458717

fbshipit-source-id: 51c1bcc5cdd826c2469af201979a39600e779bce
2020-01-28 14:44:05 -08:00
Siying Dong 431fb6c0ba Add Google Group to Issue Template
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6339

Differential Revision: D19608457

fbshipit-source-id: 2adea28b1bd20b85ccafca1aa567030115220ea6
2020-01-28 14:40:37 -08:00
Sagar Vemuri 4f6c86226c Use the same oldest ancestor time in table properties and manifest
Summary:
./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions passed 96 / 100 times.
```
With the fix: all runs (tried 100, 1000, 10000) succeed.
```
$ TEST_TMPDIR=/dev/shm ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.LevelTtlCascadingCompactions --repeat=1000
[1000/1000] DBCompactionTest.LevelTtlCascadingCompactions (1895 ms)
```

Test Plan:
Build:
```
COMPILE_WITH_TSAN=1 make db_compaction_test -j100
```
Without the fix: a few runs out of 100 fail:
```
$ TEST_TMPDIR=/dev/shm KEEP_DB=1 ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.LevelTtlCascadingCompactions --repeat=100
...
...
Note: Google Test filter = DBCompactionTest.LevelTtlCascadingCompactions
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DBCompactionTest
[ RUN      ] DBCompactionTest.LevelTtlCascadingCompactions
db/db_compaction_test.cc:3687: Failure
Expected equality of these values:
  oldest_time
    Which is: 1580155869
  level_to_files[6][0].oldest_ancester_time
    Which is: 1580155870
DB is still at /dev/shm//db_compaction_test_6337001442947696266
[  FAILED  ] DBCompactionTest.LevelTtlCascadingCompactions (1432 ms)
[----------] 1 test from DBCompactionTest (1432 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1433 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DBCompactionTest.LevelTtlCascadingCompactions

 1 FAILED TEST
[80/100] DBCompactionTest.LevelTtlCascadingCompactions returned/aborted with exit code 1 (1489 ms)
[100/100] DBCompactionTest.LevelTtlCascadingCompactions (1522 ms)
FAILED TESTS (4/100):
    1419 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/90)
    1434 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/84)
    1457 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/82)
    1489 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/74)

Differential Revision: D19587040

Pulled By: sagar0

fbshipit-source-id: 11191ae9940837643bff47ebe18b299b4be3d950
2020-01-27 19:58:53 -08:00
sdong 7aa66c704f Move HISTORY.md entry of hash index fix from 6.7 to unreleased (#6337)
Summary:
Commits related to hash index fix have been reverted in 6.7.fb branch. Update HISTORY.md to keep it in sync.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6337

Differential Revision: D19593717

fbshipit-source-id: 466178dc6205c9e41ccced41bf281a0952bdc2ca
2020-01-27 17:45:42 -08:00
Andrew Kryczka 5b33cfa1e3 fix WriteBufferManager flush log message (#6335)
Summary:
It chooses the oldest memtable, not the largest one. This is an
important difference for users whose CFs receive non-uniform write
rates.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6335

Differential Revision: D19588865

Pulled By: maysamyabandeh

fbshipit-source-id: 62ad4325b0182f5f27858584cd73fd5978fb2cec
2020-01-27 15:49:22 -08:00
sdong f10f135938 Fix regression bug of hash index with iterator total order seek (#6328)
Summary:
https://github.com/facebook/rocksdb/pull/6028 introduces a bug for hash index in SST files. If a table reader is created when total order seek is used, prefix_extractor might be passed into table reader as null. While later when prefix seek is used, the same table reader used, hash index is checked but prefix extractor is null and the program would crash.
Fix the issue by fixing http://github.com/facebook/rocksdb/pull/6028 in the way that prefix_extractor is preserved but ReadOptions.total_order_seek is checked

Also, a null pointer check is added so that a bug like this won't cause segfault in the future.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6328

Test Plan: Add a unit test that would fail without the fix. Stress test that reproduces the crash would pass.

Differential Revision: D19586751

fbshipit-source-id: 8de77690167ddf5a77a01e167cf89430b1bfba42
2020-01-27 15:44:54 -08:00
Peter Dillinger 986df37135 Clean up PartitionedFilterBlockBuilder (#6299)
Summary:
Remove the redundant PartitionedFilterBlockBuilder::num_added_ and ::NumAdded since the parent class, FullFilterBlockBuilder, already provides them.
Also rename filters_in_partition_ and filters_per_partition_ to keys_added_to_partition_ and keys_per_partition_ to improve readability.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6299

Test Plan: make check

Differential Revision: D19413278

Pulled By: pdillinger

fbshipit-source-id: 04926ee7874477d659cb2b6ae03f2d995fb747e5
2020-01-27 13:15:14 -08:00
Fosco Marotto bd698e4f55 Update version for next release, 6.7.0 (#6320)
Summary:
Adjusted history for 6.6.1 and 6.6.2, switched master version to 6.7.0.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6320

Differential Revision: D19499272

Pulled By: gfosco

fbshipit-source-id: 2bafb2456951f231e411e9c03aaa4c044f497684
2020-01-24 15:36:32 -08:00
Maysam Yabandeh c4bc30e12d Implement PinnableSlice::remove_prefix (#6330)
Summary:
The function was left unimplemented. Although we currently don't have a use for that it was declared with an assert(0) to prevent mistakenly using the remove_prefix of the parent class. The function body  with only assert(0) however causes issues with some compiler's warning levels. The patch implements the function to avoid the warning.
It also piggybacks some minor code warning for unnecessary semicolons after the function definition.s
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6330

Differential Revision: D19559062

Pulled By: maysamyabandeh

fbshipit-source-id: 3a022484f688c9abd4556e5412bcc2628ab96a00
2020-01-24 13:04:53 -08:00
Levi Tamasi f34782a67d Fix the "records dropped" statistics (#6325)
Summary:
The earlier code used two conflicting definitions for the number of
input records going into a compaction, one based on the
`rocksdb.num.entries` table property and one based on
`CompactionIterationStats`. The first one is correct and in line
with how output records are counted, while the second one incorrectly
ignores input records in various cases when the `CompactionIterator`
advances or reseeks the input iterator (this can happen, amongst other
cases, when dealing with `SingleDelete`s, regular `Delete`s, `Merge`s,
and compaction filters). This can result in the code undercounting the
input records and computing an incorrect value for "records dropped"
during the compaction. The patch fixes this by switching over to the
correct (table property based) input record count for "records dropped".
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6325

Test Plan: Tested using `make check` and `db_bench`.

Differential Revision: D19525491

Pulled By: ltamasi

fbshipit-source-id: 4340b0b2f41546db8e356db70ca02199e48fa636
2020-01-23 15:27:22 -08:00
anand76 0672a6db64 Fix queue manipulation in WriteThread::BeginWriteStall() (#6322)
Summary:
When there is a write stall, the active write group leader calls ```BeginWriteStall()``` to walk the queue of writers and remove any with the ```no_slowdown``` option set. There was a bug in the code which updated the back pointer but not the forward pointer (```link_newer```), corrupting the list and causing some threads to wait forever. This PR fixes it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6322

Test Plan: Add a unit test in db_write_test

Differential Revision: D19538313

Pulled By: anand1976

fbshipit-source-id: 6fbed819e594913f435886606f5d36f74f235c3a
2020-01-23 14:01:28 -08:00
Maysam Yabandeh 967a2d953f Revert "crash_test to enable block-based table hash index (#6310)" (#6327)
Summary:
This reverts commit 8e309b35bb.
The stress tests are failing . Revert it until we figure the root cause.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6327

Differential Revision: D19537657

Pulled By: maysamyabandeh

fbshipit-source-id: bf34a5dd720825957729e136e9a5a729a240e61a
2020-01-23 09:09:17 -08:00
Maysam Yabandeh cb1142e00d Set index_block_restart_interval of kHashSearch to 1 in stress test (#6324)
Summary:
kHashSearch is incompatible with larger than 1 values for index_block_restart_interval. Setting it to 1 in stress tests would avoid confusion about the test parameters.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6324

Differential Revision: D19525669

Pulled By: maysamyabandeh

fbshipit-source-id: fbf3a797e0ebcebb4d32eba3728cf3583906fc8a
2020-01-22 16:33:21 -08:00
matthewvon e6e8b9e871 Correct pragma once problem with Bazel on Windows (#6321)
Summary:
This is a simple edit to have two #include file paths be consistent within range_del_aggregator.{h,cc} with everywhere else.

The impact of this inconsistency is that it actual breaks a Bazel based build on the Windows platform. The same pragma once failure occurs with both Windows Visual C++ 2019 and clang for Windows 9.0. Bazel's "sandboxing" of the builds causes both compilers to not properly recognize "rocksdb/types.h" and "include/rocksdb/types.h" to be the same file (also comparator.h). My guess is that the backslash versus forward slash mixing within path names is the underlying issue.

But, everything builds fine once the include paths in these two source files are consistent with the rest of the repository.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6321

Differential Revision: D19506585

Pulled By: ltamasi

fbshipit-source-id: 294c346607edc433ab99eaabc9c880ee7426817a
2020-01-21 16:12:43 -08:00
Levi Tamasi d305f13e21 Make DBCompactionTest.SkipStatsUpdateTest more robust (#6306)
Summary:
Currently, this test case tries to infer whether
`VersionStorageInfo::UpdateAccumulatedStats` was called during open by
checking the number of files opened against an arbitrary threshold (10).
This makes the test brittle and results in sporadic failures. The patch
changes the test case to use sync points to directly test whether
`UpdateAccumulatedStats` was called.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6306

Test Plan: `make check`

Differential Revision: D19439544

Pulled By: ltamasi

fbshipit-source-id: ceb7adf578222636a0f51740872d0278cd1a914f
2020-01-21 12:55:55 -08:00
sdong 8e309b35bb crash_test to enable block-based table hash index (#6310)
Summary:
Block-based table has index has been disabled in crash test due to bugs. We fixed a bug and re-enable it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6310

Test Plan: Finish one round of "crash_test_with_atomic_flush" test successfully while exclusively running has index. Another run also ran for several hours without failure.

Differential Revision: D19455856

fbshipit-source-id: 1192752d2c1e81ed7e5c5c7a9481c841582d5274
2020-01-21 12:27:30 -08:00
Peter Dillinger 8aa99fc71e Warn on excessive keys for legacy Bloom filter with 32-bit hash (#6317)
Summary:
With many millions of keys, the old Bloom filter implementation
for the block-based table (format_version <= 4) would have excessive FP
rate due to the limitations of feeding the Bloom filter with a 32-bit hash.
This change computes an estimated inflated FP rate due to this effect
and warns in the log whenever an SST filter is constructed (almost
certainly a "full" not "partitioned" filter) that exceeds 1.5x FP rate
due to this effect. The detailed condition is only checked if 3 million
keys or more have been added to a filter, as this should be a lower
bound for common bits/key settings (< 20).

Recommended remedies include smaller SST file size, using
format_version >= 5 (for new Bloom filter), or using partitioned
filters.

This does not change behavior other than generating warnings for some
constructed filters using the old implementation.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6317

Test Plan:
Example with warning, 15M keys @ 15 bits / key: (working_mem_size_mb is just to stop after building one filter if it's large)

    $ ./filter_bench -quick -impl=0 -working_mem_size_mb=1 -bits_per_key=15 -average_keys_per_filter=15000000 2>&1 | grep 'FP rate'
    [WARN] [/block_based/filter_policy.cc:292] Using legacy SST/BBT Bloom filter with excessive key count (15.0M @ 15bpk), causing estimated 1.8x higher filter FP rate. Consider using new Bloom with format_version>=5, smaller SST file size, or partitioned filters.
    Predicted FP rate %: 0.766702
    Average FP rate %: 0.66846

Example without warning (150K keys):

    $ ./filter_bench -quick -impl=0 -working_mem_size_mb=1 -bits_per_key=15 -average_keys_per_filter=150000 2>&1 | grep 'FP rate'
    Predicted FP rate %: 0.422857
    Average FP rate %: 0.379301
    $

With more samples at 15 bits/key:
  150K keys -> no warning; actual: 0.379% FP rate (baseline)
  1M keys -> no warning; actual: 0.396% FP rate, 1.045x
  9M keys -> no warning; actual: 0.563% FP rate, 1.485x
  10M keys -> warning (1.5x); actual: 0.564% FP rate, 1.488x
  15M keys -> warning (1.8x); actual: 0.668% FP rate, 1.76x
  25M keys -> warning (2.4x); actual: 0.880% FP rate, 2.32x

At 10 bits/key:
  150K keys -> no warning; actual: 1.17% FP rate (baseline)
  1M keys -> no warning; actual: 1.16% FP rate
  10M keys -> no warning; actual: 1.32% FP rate, 1.13x
  25M keys -> no warning; actual: 1.63% FP rate, 1.39x
  35M keys -> warning (1.6x); actual: 1.81% FP rate, 1.55x

At 5 bits/key:
  150K keys -> no warning; actual: 9.32% FP rate (baseline)
  25M keys -> no warning; actual: 9.62% FP rate, 1.03x
  200M keys -> no warning; actual: 12.2% FP rate, 1.31x
  250M keys -> warning (1.5x); actual: 12.8% FP rate, 1.37x
  300M keys -> warning (1.6x); actual: 13.4% FP rate, 1.43x

The reason for the modest inaccuracy at low bits/key is that the assumption of independence between a collision between 32-hash values feeding the filter and an FP in the filter is not quite true for implementations using "simple" logic to compute indices from the stock hash result. There's math on this in my dissertation, but I don't think it's worth the effort just for these extreme cases (> 100 million keys and low-ish bits/key).

Differential Revision: D19471715

Pulled By: pdillinger

fbshipit-source-id: f80c96893a09bf1152630ff0b964e5cdd7e35c68
2020-01-20 21:31:47 -08:00
Peter Dillinger 4b86fe1123 Log warning for high bits/key in legacy Bloom filter (#6312)
Summary:
Help users that would benefit most from new Bloom filter
implementation by logging a warning that recommends the using
format_version >= 5.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6312

Test Plan:
$ (for BPK in 10 13 14 19 20 50; do ./filter_bench -quick -impl=0 -bits_per_key=$BPK -m_queries=1 2>&1; done) | grep 'its/key'
    Bits/key actual: 10.0647
    Bits/key actual: 13.0593
    [WARN] [/block_based/filter_policy.cc:546] Using legacy Bloom filter with high (14) bits/key. Significant filter space and/or accuracy improvement is available with format_verion>=5.
    Bits/key actual: 14.0581
    [WARN] [/block_based/filter_policy.cc:546] Using legacy Bloom filter with high (19) bits/key. Significant filter space and/or accuracy improvement is available with format_verion>=5.
    Bits/key actual: 19.0542
    [WARN] [/block_based/filter_policy.cc:546] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_verion>=5.
    Bits/key actual: 20.0584
    [WARN] [/block_based/filter_policy.cc:546] Using legacy Bloom filter with high (50) bits/key. Dramatic filter space and/or accuracy improvement is available with format_verion>=5.
    Bits/key actual: 50.0577

Differential Revision: D19457191

Pulled By: pdillinger

fbshipit-source-id: 073d94cde5c70e03a160f953e1100c15ea83eda4
2020-01-17 19:37:35 -08:00
chenyou-fdu 931876e86e Separate enable-WAL and disable-WAL writer to avoid unwanted data in log files (#6290)
Summary:
When we do concurrently writes, and different write operations will have WAL enable or disable.
But the data from write operation with WAL disabled will still be logged into log files, which will lead to extra disk write/sync since we do not want any guarantee for these part of data.

Detail can be found in https://github.com/facebook/rocksdb/issues/6280. This PR avoid mixing the two types in a write group. The advantage is simpler reasoning about the write group content
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6290

Differential Revision: D19448598

Pulled By: maysamyabandeh

fbshipit-source-id: 3d990a0f79a78ea1bfc90773f6ebafc1884c20de
2020-01-17 15:54:55 -08:00
Matt Bell 7e5b04d04f Expose atomic flush option in C API (#6307)
Summary:
This PR adds a `rocksdb_options_set_atomic_flush` function to the C API.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6307

Differential Revision: D19451313

Pulled By: ltamasi

fbshipit-source-id: 750495642ef55b1ea7e13477f85c38cd6574849c
2020-01-17 12:57:48 -08:00
sdong 6b64aed4c0 Fix bug which causes crash_test to always run on sync mode (#6304)
Summary:
A previous change meant to make db_stress to run on sync=1 mode for 1/20 of the time in crash_test, but a bug caused to to always run on sync=1 mode. Fix it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6304

Test Plan: Start and kill "python -u tools/db_crashtest.py --simple whitebox" multiple times and observe that most times sync=0 is used while some times sync=1 is used.

Differential Revision: D19433000

fbshipit-source-id: 7a0adba39b17a1b3acbbd791bb0cdb743b91fa95
2020-01-17 01:46:48 -08:00
sdong d87cffaea4 Fix another bug caused by recent hash index fix (#6305)
Summary:
Recent bug fix related to hash index introduced a new bug: hash index can return NotFound but it is not handled by BlockBasedTable::Get(). The end result is that Get() stops being executed too early. Fix it by ignoring NotFound code in Get().
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6305

Test Plan: A problematic DB used to return NotFound incorrectly, and now able to return correct result. Will try to construct a unit test too.0

Differential Revision: D19438925

fbshipit-source-id: e751afa8c13728d56511cfeb1bc811ecb99f3217
2020-01-17 01:41:04 -08:00
Levi Tamasi 73f65b457e Adjust thread pool sizes when setting max_background_jobs dynamically (#6300)
Summary:
https://github.com/facebook/rocksdb/pull/2205 introduced a new
configuration option called `max_background_jobs`, superseding the
earlier options `max_background_flushes` and
`max_background_compactions`. However, unlike
`max_background_compactions`, setting `max_background_jobs` dynamically
through the `SetDBOptions` interface does not adjust the size of the
thread pools (see https://github.com/facebook/rocksdb/issues/6298). The
patch fixes this.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6300

Test Plan: Extended unit test.

Differential Revision: D19430899

Pulled By: ltamasi

fbshipit-source-id: 704006605b3c13c3d1b997ccc0831ee369721074
2020-01-16 14:35:10 -08:00
Cheng Chang 86623a7153 Update example of optimistic transaction (#6074)
Summary:
Add asserts to show the intentions of result explicitly.
Add examples to show the effect of optimistic transaction more clearly.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6074

Test Plan: `cd examples && make optimistic_transaction_example && ./optimistic_transaction_example`

Differential Revision: D18964309

Pulled By: cheng-chang

fbshipit-source-id: a524616ed9981edf2fd37ae61c5ed18c5cf25f55
2020-01-16 14:04:44 -08:00
sdong f8b5ef85ec Fix a bug caused by recent fix of Prefix Hash (#6302)
Summary:
Recent fix to Prefix Hash https://github.com/facebook/rocksdb/pull/6292 caused a bug that the newly created NotFound status in hash index is never reset. This causes reseek or implict reseek to return wrong results sometimes.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6302

Test Plan:
Add a unit test that would fail. Not fix.
crash test with hash test would fail in several seconds. With the fix, it will run about several minutes before failing with another failure.

Differential Revision: D19424572

fbshipit-source-id: c5276f36a95fd0e2837e30190476d2fe21ed8566
2020-01-16 10:47:20 -08:00
Levi Tamasi b7f1b3e51c Access Maven Central over HTTPS (#6301)
Summary:
As of 1/15/2020, Maven Central does not support plain HTTP. Because of
this, our Travis and AppVeyor builds have started failing during the
assertj download step. This patch will hopefully fix these issues.

See https://blog.sonatype.com/central-repository-moving-to-https
for more info.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6301

Test Plan:
Will monitor the builds. ("I don't always test my changes but when I do,
I do it in production.")

Differential Revision: D19422923

Pulled By: ltamasi

fbshipit-source-id: 76f9a8564a5b66ddc721d705f9cbfc736bf7a97d
2020-01-15 17:54:53 -08:00
sdong d2b4d42d4b Fix kHashSearch bug with SeekForPrev (#6297)
Summary:
When prefix is enabled the expected behavior when the prefix of the target does not exist is for Seek is to seek to any key larger than target and SeekToPrev to any key less than the target.
Currently. the prefix index (kHashSearch) returns OK status but sets Invalid() to indicate two cases: a prefix of the searched key does not exist, ii) the key is beyond the range of the keys in SST file. The SeekForPrev implementation in BlockBasedTable thus does not have enough information to know when it should set the index key to first (to return a key smaller than target). The patch fixes that by returning NotFound status for cases that the prefix does not exist. SeekForPrev in BlockBasedTable accordingly SeekToFirst instead of SeekToLast on the index iterator.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6297

Test Plan: SeekForPrev of non-exsiting prefix is added to block_test.cc, and a test case is added in db_test2, which fails without the fix.

Differential Revision: D19404695

fbshipit-source-id: cafbbf95f8f60ff9ede9ccc99d25bfa1cf6fcdc3
2020-01-15 14:28:39 -08:00
Levi Tamasi 1dd7873e08 Remove earlier partial BlobDB GC implementation (#6278)
Summary:
In addition to removing the earlier partially implemented garbage collection
logic from the BlobDB codebase, the patch also removes the test cases (as well as
the related sync points, as appropriate) that were only relevant for the old
implementation, and reworks the remaining ones so they use the new GC logic.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6278

Test Plan: `make check`

Differential Revision: D19335226

Pulled By: ltamasi

fbshipit-source-id: 0cc1794bc9892feda1426ed5522a318f3cb1b692
2020-01-14 15:08:44 -08:00
sdong 76c117b24b Fix LITE test build broken by recent commit (#6295)
Summary:
A recent commit adds a unit test that uses a function not available in LITE build. Fix it by avoiding the call
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6295

Test Plan: Run the test in LITE build and see it passes.

Differential Revision: D19395678

fbshipit-source-id: 37b42835bae02511630d80f7cafb1179401bc033
2020-01-14 13:17:04 -08:00
Maysam Yabandeh d4b7fbf0d5 kHashSearch incompatible with index_block_restart_interval>1 (#6294)
Summary:
kHashSearch index type is incompatible with index_block_restart_interval larger than 1. The patch asserts that and also resets index_block_restart_interval value if it is incompatible with kHashSearch.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6294

Differential Revision: D19394229

Pulled By: maysamyabandeh

fbshipit-source-id: 8a12712ab25e81094a7f71ecd43f773dd4fb6acd
2020-01-14 11:21:27 -08:00
sdong 894c6d21af Bug when multiple files at one level contains the same smallest key (#6285)
Summary:
The fractional cascading index is not correctly generated when two files at the same level contains the same smallest or largest user key.
The result would be that it would hit an assertion in debug mode and lower level files might be skipped.
This might cause wrong results when the same user keys are of merge operands and Get() is called using the exact user key. In that case, the lower files would need to further checked.
The fix is to fix the fractional cascading index.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6285

Test Plan: Add a unit test which would cause the assertion which would be fixed.

Differential Revision: D19358426

fbshipit-source-id: 39b2b1558075fd95e99491d462a67f9f2298c48e
2020-01-13 16:27:42 -08:00
Qinfan Wu 6733be033e More const pointers in C API (#6283)
Summary:
This makes it easier to call the functions from Rust as otherwise they require mutable types.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6283

Differential Revision: D19349991

Pulled By: wqfish

fbshipit-source-id: e8da7a75efe8cd97757baef8ca844a054f2519b4
2020-01-10 19:27:09 -08:00
Sagar Vemuri cfa585611d Consider all compaction input files to compute the oldest ancestor time (#6279)
Summary:
Look at all compaction input files to compute the oldest ancestor time.

In https://github.com/facebook/rocksdb/issues/5992 we changed how creation_time (aka oldest-ancestor-time) table property of compaction output files is computed from max(creation-time-of-all-compaction-inputs) to min(creation-time-of-all-inputs). This exposed a bug where, during compaction, the creation_time:s of only the L0 compaction inputs were being looked at, and all other input levels were being ignored. This PR fixes the issue.
Some TTL compactions when using Level-Style compactions might not have run due to this bug.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6279

Test Plan: Enhanced the unit tests to validate that the correct time is propagated to the compaction outputs.

Differential Revision: D19337812

Pulled By: sagar0

fbshipit-source-id: edf8a72f11e405e93032ff5f45590816debe0bb4
2020-01-10 19:02:42 -08:00
Maysam Yabandeh eff5e076f5 unordered_write incompatible with max_successive_merges (#6284)
Summary:
unordered_write is incompatible with non-zero max_successive_merges. Although we check this at runtime, we currently don't prevent the user from setting this combination in options. This has led to stress tests to fail with this combination is tried in ::SetOptions.
The patch fixes that and also reverts the changes performed by https://github.com/facebook/rocksdb/pull/6254, in which max_successive_merges was mistakenly declared incompatible with unordered_write.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6284

Differential Revision: D19356115

Pulled By: maysamyabandeh

fbshipit-source-id: f06dadec777622bd75f267361c022735cf8cecb6
2020-01-10 16:53:19 -08:00
anand76 687119aeaf Variable key length in db_stress (#6273)
Summary:
Undo https://github.com/facebook/rocksdb/issues/6243 and fix the crash test failures.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6273

Test Plan: Run make ubsan_crash_test

Differential Revision: D19331472

Pulled By: anand1976

fbshipit-source-id: 30aa4a36c1b0f77a97159d82bbfd1cd767878e28
2020-01-09 21:27:18 -08:00
Yanqin Jin 6a9989381f Fix compilation under LITE (#6277)
Summary:
Fix compilation under LITE by putting `#ifndef ROCKSDB_LITE` around a code block.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6277

Differential Revision: D19334157

Pulled By: riversand963

fbshipit-source-id: 947111ed68aa550f5ea424b216c1442a8af9e32b
2020-01-09 15:57:39 -08:00
sdong 39410bcb3d Fix some shadow warning (#6242)
Summary:
Some shadow warning shows up when using gcc 4.8. An example:

./utilities/blob_db/blob_compaction_filter.h: In constructor ‘rocksdb::blob_db::BlobIndexCompactionFilterFactoryBase::BlobIndexCompactionFilterFactoryBase(rocksdb::blob_db::lobDBImpl*, rocksdb::Env*, rocksdb::Statistics*)’:
./utilities/blob_db/blob_compaction_filter.h:121:7: error: declaration of ‘blob_db_impl’ shadows a member of 'this' [-Werror=shadow]
       : blob_db_impl_(blob_db_impl), env_(_env), statistics_(_statistics) {}
       ^

Fix them.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6242

Test Plan: Build and see the warnings go away.

Differential Revision: D19217789

fbshipit-source-id: 8ef631941f23dab47a388e060adec24b72efd65e
2020-01-08 18:20:13 -08:00
Yanqin Jin cfd9732f65 Remove inaccurate code comment (#6274)
Summary:
Remove a comment.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6274

Differential Revision: D19323151

Pulled By: riversand963

fbshipit-source-id: d0d804d6882edcd94e35544ef45578b32ff1caae
2020-01-08 17:51:42 -08:00
Huisheng Liu e5b476f551 Update file indexer to take timestamp into consideration (#6205)
Summary:
Exclude timestamp in key comparison during boundary calculation to avoid key versions being excluded.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6205

Differential Revision: D19166765

Pulled By: riversand963

fbshipit-source-id: bbe08816fef8de349a83ebd59a595ad844021f24
2020-01-08 16:31:23 -08:00
Amber1990Zhang 941bd15aed add user nebula (#6271)
Summary:
As title. add a new user [**Nebula Graph**](https://github.com/vesoft-inc/nebula) to the user doc.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6271

Differential Revision: D19319345

fbshipit-source-id: 52a54372cecc701c34da4ea6b1cf27f3b7498efb
2020-01-08 13:46:43 -08:00
sdong 1244abef66 Stress Test: relax prefix iterator check condition (#6269)
Summary:
Right now, when validating prefix iterator, if control iterator is invalidate but prefix iterator shows value, we determine it as a test failure. However, this fails to consider the case where a file or memtable containing a tombstone is filtered out by a prefix bloom filter. The fix is to relax the check in this case. If we are out of prefix range, then ignore the check.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6269

Test Plan: Run crash_test for a short while and it still passes.

Differential Revision: D19317594

fbshipit-source-id: b964a1cdc1df5efe439d4b32f8023e1fbc8598c1
2020-01-08 13:32:06 -08:00
Maysam Yabandeh f4a378be3e Print out non-ok DB::Open status in db_stress (#6272)
Summary:
The crash test is failing with non-ok status after TransactionDB::Open. This patch adds more debugging information.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6272

Differential Revision: D19314527

Pulled By: maysamyabandeh

fbshipit-source-id: d45ecb0f2144e052fb4b5fdd483150440991a3b4
2020-01-08 12:10:55 -08:00
Adam Retter 6477075f2c JMH microbenchmarks for RocksJava (#6241)
Summary:
This is the start of some JMH microbenchmarks for RocksJava.

Such benchmarks can help us decide on performance improvements of the Java API.

At the moment, I have only added benchmarks for various Comparator options, as that is one of the first areas where I want to improve performance. I plan to expand this to many more tests.

Details of how to compile and run the benchmarks are in the `README.md`.

A run of these on a XEON 3.5 GHz 4vCPU (QEMU Virtual CPU version 2.5+) / 8GB RAM KVM with Ubuntu 18.04, OpenJDK 1.8.0_232, and gcc 8.3.0 produced the following:

```
# Run complete. Total time: 01:43:17

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                                         (comparatorName)   Mode  Cnt       Score       Error  Units
ComparatorBenchmarks.put                           native_bytewise thrpt   25   122373.920 ±  2200.538  ops/s
ComparatorBenchmarks.put              java_bytewise_adaptive_mutex thrpt   25    17388.201 ±  1444.006  ops/s
ComparatorBenchmarks.put          java_bytewise_non-adaptive_mutex thrpt   25    16887.150 ±  1632.204  ops/s
ComparatorBenchmarks.put       java_direct_bytewise_adaptive_mutex thrpt   25    15644.572 ±  1791.189  ops/s
ComparatorBenchmarks.put   java_direct_bytewise_non-adaptive_mutex thrpt   25    14869.601 ±  2252.135  ops/s
ComparatorBenchmarks.put                   native_reverse_bytewise thrpt   25   116528.735 ±  4168.797  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_adaptive_mutex thrpt   25    10651.975 ±   545.998  ops/s
ComparatorBenchmarks.put  java_reverse_bytewise_non-adaptive_mutex thrpt   25    10514.224 ±   930.069  ops/s
```

Indicating a ~7x difference between comparators implemented natively (C++) and those implemented in Java. Let's see if we can't improve on that in the near future...
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6241

Differential Revision: D19290410

Pulled By: pdillinger

fbshipit-source-id: 25d44bf3a31de265502ed0c5d8a28cf4c7cb9c0b
2020-01-07 15:46:09 -08:00
965 changed files with 20030 additions and 11396 deletions

No files matched your search

+2
View File
@@ -52,6 +52,8 @@ trace_analyzer
trace_analyzer_test
block_cache_trace_analyzer
.DS_Store
.vs
.vscode
java/out
java/target
+4 -2
View File
@@ -81,7 +81,9 @@ install:
sudo apt-get install -y mingw-w64 ;
fi
- if [[ "${JOB_NAME}" == cmake* ]] && [ "${TRAVIS_OS_NAME}" == linux ]; then
mkdir cmake-dist && curl -sfSL https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5-Linux-x86_64.tar.gz | tar --strip-components=1 -C cmake-dist -xz && export PATH=$PWD/cmake-dist/bin:$PATH;
CMAKE_DIST_URL="https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-$(uname -m).tar.bz2";
TAR_OPT="--strip-components=1 -xj";
mkdir cmake-dist && curl --silent --fail --show-error --location "${CMAKE_DIST_URL}" | tar -C cmake-dist ${TAR_OPT} && export PATH=$PWD/cmake-dist/bin:$PATH;
fi
- if [[ "${JOB_NAME}" == java_test ]]; then
java -version && echo "JAVA_HOME=${JAVA_HOME}";
@@ -103,7 +105,7 @@ script:
OPT=-DTRAVIS V=1 ROCKSDBTESTS_START=db_block_cache_test ROCKSDBTESTS_END=db_iter_test make -j4 check_some
;;
2)
OPT=-DTRAVIS V=1 make -j4 tools && OPT=-DTRAVIS V=1 ROCKSDBTESTS_START=db_iter_test ROCKSDBTESTS_END=options_file_test make -j4 check_some
OPT="-DTRAVIS -DROCKSDB_NAMESPACE=alternative_rocksdb_ns" V=1 make -j4 tools && OPT="-DTRAVIS -DROCKSDB_NAMESPACE=alternative_rocksdb_ns" V=1 ROCKSDBTESTS_START=db_iter_test ROCKSDBTESTS_END=options_file_test make -j4 check_some
;;
3)
OPT=-DTRAVIS V=1 ROCKSDBTESTS_START=options_file_test ROCKSDBTESTS_END=write_prepared_transaction_test make -j4 check_some
+15 -6
View File
@@ -665,6 +665,7 @@ set(SOURCES
util/random.cc
util/rate_limiter.cc
util/slice.cc
util/file_checksum_helper.cc
util/status.cc
util/string_util.cc
util/thread_local.cc
@@ -906,15 +907,15 @@ if(NOT WIN32 OR ROCKSDB_INSTALL_ON_WINDOWS)
)
endif()
add_subdirectory(third-party/gtest-1.8.1/fused-src/gtest)
add_library(testharness STATIC
test_util/testharness.cc)
target_link_libraries(testharness gtest)
# Tests are excluded from Release builds
CMAKE_DEPENDENT_OPTION(WITH_TESTS "build with tests" ON
"CMAKE_BUILD_TYPE STREQUAL Debug" OFF)
if(WITH_TESTS)
add_subdirectory(third-party/gtest-1.8.1/fused-src/gtest)
add_library(testharness STATIC
test_util/testharness.cc)
target_link_libraries(testharness gtest)
set(TESTS
cache/cache_test.cc
cache/lru_cache_test.cc
@@ -1025,6 +1026,7 @@ if(WITH_TESTS)
util/bloom_test.cc
util/coding_test.cc
util/crc32c_test.cc
util/defer_test.cc
util/dynamic_bloom_test.cc
util/file_reader_writer_test.cc
util/filelock_test.cc
@@ -1033,6 +1035,7 @@ if(WITH_TESTS)
util/random_test.cc
util/rate_limiter_test.cc
util/repeatable_thread_test.cc
util/slice_test.cc
util/slice_transform_test.cc
util/timer_queue_test.cc
util/thread_list_test.cc
@@ -1166,9 +1169,15 @@ if(WITH_BENCHMARK_TOOLS)
${ROCKSDB_LIB})
endif()
option(WITH_CORE_TOOLS "build with ldb and sst_dump" ON)
option(WITH_TOOLS "build with tools" ON)
if(WITH_TOOLS)
if(WITH_CORE_TOOLS OR WITH_TOOLS)
add_subdirectory(tools)
add_custom_target(core_tools
DEPENDS ${core_tool_deps})
endif()
if(WITH_TOOLS)
add_subdirectory(db_stress_tool)
add_custom_target(tools
DEPENDS ${tool_deps})
+57 -5
View File
@@ -1,5 +1,40 @@
# Rocksdb Change Log
## Unreleased
### Bug Fixes
* Upgraded version of bzip library (1.0.6 -> 1.0.8) used with RocksJava to address potential vulnerabilities if an attacker can manipulate compressed data saved and loaded by RocksDB (not normal). See issue #6703.
## 6.8.1 (03/30/2020)
### Behavior changes
* Since RocksDB 6.8.0, ttl-based FIFO compaction can drop a file whose oldest key becomes older than options.ttl while others have not. This fix reverts this and makes ttl-based FIFO compaction use the file's flush time as the criterion. This fix also requires that max_open_files = -1 and compaction_options_fifo.allow_compaction = false to function properly.
## 6.8.0 (02/24/2020)
### Java API Changes
* Major breaking changes to Java comparators, toward standardizing on ByteBuffer for performant, locale-neutral operations on keys (#6252).
* Added overloads of common API methods using direct ByteBuffers for keys and values (#2283).
### Bug Fixes
* Fix incorrect results while block-based table uses kHashSearch, together with Prev()/SeekForPrev().
* Fix a bug that prevents opening a DB after two consecutive crash with TransactionDB, where the first crash recovers from a corrupted WAL with kPointInTimeRecovery but the second cannot.
* Fixed issue #6316 that can cause a corruption of the MANIFEST file in the middle when writing to it fails due to no disk space.
* Add DBOptions::skip_checking_sst_file_sizes_on_db_open. It disables potentially expensive checking of all sst file sizes in DB::Open().
* BlobDB now ignores trivially moved files when updating the mapping between blob files and SSTs. This should mitigate issue #6338 where out of order flush/compaction notifications could trigger an assertion with the earlier code.
* Batched MultiGet() ignores IO errors while reading data blocks, causing it to potentially continue looking for a key and returning stale results.
* `WriteBatchWithIndex::DeleteRange` returns `Status::NotSupported`. Previously it returned success even though reads on the batch did not account for range tombstones. The corresponding language bindings now cannot be used. In C, that includes `rocksdb_writebatch_wi_delete_range`, `rocksdb_writebatch_wi_delete_range_cf`, `rocksdb_writebatch_wi_delete_rangev`, and `rocksdb_writebatch_wi_delete_rangev_cf`. In Java, that includes `WriteBatchWithIndex::deleteRange`.
* Assign new MANIFEST file number when caller tries to create a new MANIFEST by calling LogAndApply(..., new_descriptor_log=true). This bug can cause MANIFEST being overwritten during recovery if options.write_dbid_to_manifest = true and there are WAL file(s).
### Performance Improvements
* Perfom readahead when reading from option files. Inside DB, options.log_readahead_size will be used as the readahead size. In other cases, a default 512KB is used.
### Public API Change
* The BlobDB garbage collector now emits the statistics `BLOB_DB_GC_NUM_FILES` (number of blob files obsoleted during GC), `BLOB_DB_GC_NUM_NEW_FILES` (number of new blob files generated during GC), `BLOB_DB_GC_FAILURES` (number of failed GC passes), `BLOB_DB_GC_NUM_KEYS_RELOCATED` (number of blobs relocated during GC), and `BLOB_DB_GC_BYTES_RELOCATED` (total size of blobs relocated during GC). On the other hand, the following statistics, which are not relevant for the new GC implementation, are now deprecated: `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`, `BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, and `BLOB_DB_GC_MICROS`.
* Disable recycle_log_file_num when an inconsistent recovery modes are requested: kPointInTimeRecovery and kAbsoluteConsistency
### New Features
* Added the checksum for each SST file generated by Flush or Compaction. Added sst_file_checksum_func to Options such that user can plugin their own SST file checksum function via override the FileChecksumFunc class. If user does not set the sst_file_checksum_func, SST file checksum calculation will not be enabled. The checksum information inlcuding uint32_t checksum value and a checksum function name (string). The checksum information is stored in FileMetadata in version store and also logged to MANIFEST. A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST (stored in an unordered_map).
* `db_bench` now supports `value_size_distribution_type`, `value_size_min`, `value_size_max` options for generating random variable sized value. Added `blob_db_compression_type` option for BlobDB to enable blob compression.
* Replace RocksDB namespace "rocksdb" with flag "ROCKSDB_NAMESPACE" which if is not defined, defined as "rocksdb" in header file rocksdb_namespace.h.
## 6.7.0 (01/21/2020)
### Public API Change
* Added a rocksdb::FileSystem class in include/rocksdb/file_system.h to encapsulate file creation/read/write operations, and an option DBOptions::file_system to allow a user to pass in an instance of rocksdb::FileSystem. If its a non-null value, this will take precendence over DBOptions::env for file operations. A new API rocksdb::FileSystem::Default() returns a platform default object. The DBOptions::env option and Env::Default() API will continue to be used for threading and other OS related functions, and where DBOptions::file_system is not specified, for file operations. For storage developers who are accustomed to rocksdb::Env, the interface in rocksdb::FileSystem is new and will probably undergo some changes as more storage systems are ported to it from rocksdb::Env. As of now, no env other than Posix has been ported to the new interface.
* A new rocksdb::NewSstFileManager() API that allows the caller to pass in separate Env and FileSystem objects.
@@ -8,18 +43,35 @@
### Bug Fixes
* Fix a bug that can cause unnecessary bg thread to be scheduled(#6104).
* Fix a bug in which a snapshot read could be affected by a DeleteRange after the snapshot (#6062).
* Fix crash caused by concurrent CF iterations and drops(#6147).
* Fixed two performance issues related to memtable history trimming. First, a new SuperVersion is now created only if some memtables were actually trimmed. Second, trimming is only scheduled if there is at least one flushed memtable that is kept in memory for the purposes of transaction conflict checking.
* Fix a bug in WriteBatchWithIndex::MultiGetFromBatchAndDB, which is called by Transaction::MultiGet, that causes due to stale pointer access when the number of keys is > 32
* BlobDB no longer updates the SST to blob file mapping upon failed compactions.
* Fixed a bug where BlobDB was comparing the `ColumnFamilyHandle` pointers themselves instead of only the column family IDs when checking whether an API call uses the default column family or not.
* Fix a race condition for cfd->log_number_ between manifest switch and memtable switch (PR 6249) when number of column families is greater than 1.
* Fix a bug on fractional cascading index when multiple files at the same level contain the same smallest user key, and those user keys are for merge operands. In this case, Get() the exact key may miss some merge operands.
* Delcare kHashSearch index type feature-incompatible with index_block_restart_interval larger than 1.
* Fixed an issue where the thread pools were not resized upon setting `max_background_jobs` dynamically through the `SetDBOptions` interface.
* Fix a bug that can cause write threads to hang when a slowdown/stall happens and there is a mix of writers with WriteOptions::no_slowdown set/unset.
* Fixed an issue where an incorrect "number of input records" value was used to compute the "records dropped" statistics for compactions.
* Fix a regression bug that causes segfault when hash is used, max_open_files != -1 and total order seek is used and switched back.
### New Features
* It is now possible to enable periodic compactions for the base DB when using BlobDB.
* BlobDB now garbage collects non-TTL blobs when `enable_garbage_collection` is set to `true` in `BlobDBOptions`. Garbage collection is performed during compaction: any valid blobs located in the oldest N files (where N is the number of non-TTL blob files multiplied by the value of `BlobDBOptions::garbage_collection_cutoff`) encountered during compaction get relocated to new blob files, and old blob files are dropped once they are no longer needed. Note: we recommend enabling periodic compactions for the base DB when using this feature to deal with the case when some old blob files are kept alive by SSTs that otherwise do not get picked for compaction.
* `db_bench` now supports the `garbage_collection_cutoff` option for BlobDB.
* Introduce ReadOptions.auto_prefix_mode. When set to true, iterator will return the same result as total order seek, but may choose to use prefix seek internally based on seek key and iterator upper bound.
* MultiGet() can use IO Uring to parallelize read from the same SST file. This featuer is by default disabled. It can be enabled with environment variable ROCKSDB_USE_IO_URING.
## 6.6.2 (01/13/2020)
### Bug Fixes
* Fixed a bug where non-L0 compaction input files were not considered to compute the `creation_time` of new compaction outputs.
## 6.6.1 (01/02/2020)
### Bug Fixes
* Fix a bug in WriteBatchWithIndex::MultiGetFromBatchAndDB, which is called by Transaction::MultiGet, that causes due to stale pointer access when the number of keys is > 32
* Fixed two performance issues related to memtable history trimming. First, a new SuperVersion is now created only if some memtables were actually trimmed. Second, trimming is only scheduled if there is at least one flushed memtable that is kept in memory for the purposes of transaction conflict checking.
* BlobDB no longer updates the SST to blob file mapping upon failed compactions.
* Fix a bug in which a snapshot read through an iterator could be affected by a DeleteRange after the snapshot (#6062).
* Fixed a bug where BlobDB was comparing the `ColumnFamilyHandle` pointers themselves instead of only the column family IDs when checking whether an API call uses the default column family or not.
* Delete superversions in BackgroundCallPurge.
* Fix use-after-free and double-deleting files in BackgroundCallPurge().
## 6.6.0 (11/25/2019)
### Bug Fixes
+34 -21
View File
@@ -503,6 +503,7 @@ TESTS = \
data_block_hash_index_test \
cache_test \
corruption_test \
slice_test \
slice_transform_test \
dbformat_test \
fault_injection_test \
@@ -595,6 +596,7 @@ TESTS = \
db_secondary_test \
block_cache_tracer_test \
block_cache_trace_analyzer_test \
defer_test \
ifeq ($(USE_FOLLY_DISTRIBUTED_MUTEX),1)
TESTS += folly_synchronization_distributed_mutex_test
@@ -1128,16 +1130,21 @@ unity_test: db/db_test.o db/db_test_util.o $(TESTHARNESS) $(TOOLLIBOBJECTS) unit
rocksdb.h rocksdb.cc: build_tools/amalgamate.py Makefile $(LIB_SOURCES) unity.cc
build_tools/amalgamate.py -I. -i./include unity.cc -x include/rocksdb/c.h -H rocksdb.h -o rocksdb.cc
clean: clean-ext-libraries-all clean-rocks
clean: clean-ext-libraries-all clean-rocks clean-rocksjava
clean-not-downloaded: clean-ext-libraries-bin clean-rocks
clean-not-downloaded: clean-ext-libraries-bin clean-rocks clean-not-downloaded-rocksjava
clean-rocks:
rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(PARALLEL_TEST) $(LIBRARY) $(SHARED)
rm -rf $(CLEAN_FILES) ios-x86 ios-arm scan_build_report
$(FIND) . -name "*.[oda]" -exec rm -f {} \;
$(FIND) . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
cd java; $(MAKE) clean
clean-rocksjava:
cd java && $(MAKE) clean
clean-not-downloaded-rocksjava:
cd java && $(MAKE) clean-not-downloaded
clean-ext-libraries-all:
rm -rf bzip2* snappy* zlib* lz4* zstd*
@@ -1286,6 +1293,9 @@ corruption_test: db/corruption_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARN
crc32c_test: util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
slice_test: util/slice_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
slice_transform_test: util/slice_transform_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
@@ -1705,6 +1715,9 @@ block_cache_tracer_test: trace_replay/block_cache_tracer_test.o trace_replay/blo
block_cache_trace_analyzer_test: tools/block_cache_analyzer/block_cache_trace_analyzer_test.o tools/block_cache_analyzer/block_cache_trace_analyzer.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
defer_test: util/defer_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
#-------------------------------------------------
# make install related stuff
INSTALL_PATH ?= /usr/local
@@ -1784,11 +1797,11 @@ SHA256_CMD = sha256sum
ZLIB_VER ?= 1.2.11
ZLIB_SHA256 ?= c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1
ZLIB_DOWNLOAD_BASE ?= http://zlib.net
BZIP2_VER ?= 1.0.6
BZIP2_SHA256 ?= a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd
BZIP2_DOWNLOAD_BASE ?= https://downloads.sourceforge.net/project/bzip2
SNAPPY_VER ?= 1.1.7
SNAPPY_SHA256 ?= 3dfa02e873ff51a11ee02b9ca391807f0c8ea0529a4924afa645fbf97163f9d4
BZIP2_VER ?= 1.0.8
BZIP2_SHA256 ?= ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269
BZIP2_DOWNLOAD_BASE ?= https://sourceware.org/pub/bzip2
SNAPPY_VER ?= 1.1.8
SNAPPY_SHA256 ?= 16b677f07832a612b0836178db7f374e414f94657c138e6993cbfc5dcc58651f
SNAPPY_DOWNLOAD_BASE ?= https://github.com/google/snappy/archive
LZ4_VER ?= 1.9.2
LZ4_SHA256 ?= 658ba6191fa44c92280d4aa2c271b0f4fbc0e34d249578dd05e50e76d0e5efcc
@@ -1834,7 +1847,7 @@ endif
libz.a:
-rm -rf zlib-$(ZLIB_VER)
ifeq (,$(wildcard ./zlib-$(ZLIB_VER).tar.gz))
curl --output zlib-$(ZLIB_VER).tar.gz -L ${ZLIB_DOWNLOAD_BASE}/zlib-$(ZLIB_VER).tar.gz
curl --fail --output zlib-$(ZLIB_VER).tar.gz --location ${ZLIB_DOWNLOAD_BASE}/zlib-$(ZLIB_VER).tar.gz
endif
ZLIB_SHA256_ACTUAL=`$(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(ZLIB_SHA256)" != "$$ZLIB_SHA256_ACTUAL" ]; then \
@@ -1848,7 +1861,7 @@ endif
libbz2.a:
-rm -rf bzip2-$(BZIP2_VER)
ifeq (,$(wildcard ./bzip2-$(BZIP2_VER).tar.gz))
curl --output bzip2-$(BZIP2_VER).tar.gz -L ${CURL_SSL_OPTS} ${BZIP2_DOWNLOAD_BASE}/bzip2-$(BZIP2_VER).tar.gz
curl --fail --output bzip2-$(BZIP2_VER).tar.gz --location ${CURL_SSL_OPTS} ${BZIP2_DOWNLOAD_BASE}/bzip2-$(BZIP2_VER).tar.gz
endif
BZIP2_SHA256_ACTUAL=`$(SHA256_CMD) bzip2-$(BZIP2_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(BZIP2_SHA256)" != "$$BZIP2_SHA256_ACTUAL" ]; then \
@@ -1862,7 +1875,7 @@ endif
libsnappy.a:
-rm -rf snappy-$(SNAPPY_VER)
ifeq (,$(wildcard ./snappy-$(SNAPPY_VER).tar.gz))
curl --output snappy-$(SNAPPY_VER).tar.gz -L ${CURL_SSL_OPTS} ${SNAPPY_DOWNLOAD_BASE}/$(SNAPPY_VER).tar.gz
curl --fail --output snappy-$(SNAPPY_VER).tar.gz --location ${CURL_SSL_OPTS} ${SNAPPY_DOWNLOAD_BASE}/$(SNAPPY_VER).tar.gz
endif
SNAPPY_SHA256_ACTUAL=`$(SHA256_CMD) snappy-$(SNAPPY_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(SNAPPY_SHA256)" != "$$SNAPPY_SHA256_ACTUAL" ]; then \
@@ -1877,7 +1890,7 @@ endif
liblz4.a:
-rm -rf lz4-$(LZ4_VER)
ifeq (,$(wildcard ./lz4-$(LZ4_VER).tar.gz))
curl --output lz4-$(LZ4_VER).tar.gz -L ${CURL_SSL_OPTS} ${LZ4_DOWNLOAD_BASE}/v$(LZ4_VER).tar.gz
curl --fail --output lz4-$(LZ4_VER).tar.gz --location ${CURL_SSL_OPTS} ${LZ4_DOWNLOAD_BASE}/v$(LZ4_VER).tar.gz
endif
LZ4_SHA256_ACTUAL=`$(SHA256_CMD) lz4-$(LZ4_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(LZ4_SHA256)" != "$$LZ4_SHA256_ACTUAL" ]; then \
@@ -1891,7 +1904,7 @@ endif
libzstd.a:
-rm -rf zstd-$(ZSTD_VER)
ifeq (,$(wildcard ./zstd-$(ZSTD_VER).tar.gz))
curl --output zstd-$(ZSTD_VER).tar.gz -L ${CURL_SSL_OPTS} ${ZSTD_DOWNLOAD_BASE}/v$(ZSTD_VER).tar.gz
curl --fail --output zstd-$(ZSTD_VER).tar.gz --location ${CURL_SSL_OPTS} ${ZSTD_DOWNLOAD_BASE}/v$(ZSTD_VER).tar.gz
endif
ZSTD_SHA256_ACTUAL=`$(SHA256_CMD) zstd-$(ZSTD_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(ZSTD_SHA256)" != "$$ZSTD_SHA256_ACTUAL" ]; then \
@@ -1961,35 +1974,35 @@ rocksdbjavastaticreleasedocker: rocksdbjavastatic rocksdbjavastaticdockerx86 roc
rocksdbjavastaticdockerx86:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x86-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos6_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_x86-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos6_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerx86_64:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x64-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos6_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_x64-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos6_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerppc64le:
mkdir -p java/target
docker run --rm --name rocksdb_linux_ppc64le-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos7_ppc64le-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_ppc64le-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos7_ppc64le-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerarm64v8:
mkdir -p java/target
docker run --rm --name rocksdb_linux_arm64v8-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos7_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_arm64v8-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos7_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerx86musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x86-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_x86-musl-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerx86_64musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x64-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_x64-musl-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerppc64lemusl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_ppc64le-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_ppc64le-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_ppc64le-musl-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_ppc64le-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerarm64v8musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_arm64v8-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
docker run --rm --name rocksdb_linux_arm64v8-musl-be --attach stdin --attach stdout --attach stderr --volume $(HOME)/.m2:/root/.m2:ro --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticpublish: rocksdbjavastaticrelease rocksdbjavastaticpublishcentral
+17 -3
View File
@@ -26,14 +26,13 @@ ROCKSDB_EXTERNAL_DEPS = [
("lz4", None, "lz4"),
("zstd", None),
("tbb", None),
("liburing", None, "uring"),
("googletest", None, "gtest"),
]
ROCKSDB_OS_DEPS = [
(
"linux",
["third-party//numa:numa"],
["third-party//numa:numa", "third-party//liburing:uring"],
),
]
@@ -49,6 +48,7 @@ ROCKSDB_OS_PREPROCESSOR_FLAGS = [
"-DROCKSDB_SCHED_GETCPU_PRESENT",
"-DROCKSDB_IOURING_PRESENT",
"-DHAVE_SSE42",
"-DLIBURING",
"-DNUMA",
],
),
@@ -72,7 +72,6 @@ ROCKSDB_PREPROCESSOR_FLAGS = [
"-DZSTD_STATIC_LINKING_ONLY",
"-DGFLAGS=gflags",
"-DTBB",
"-DLIBURING",
# Added missing flags from output of build_detect_platform
"-DROCKSDB_BACKTRACE",
@@ -284,6 +283,7 @@ cpp_library(
"util/concurrent_task_limiter_impl.cc",
"util/crc32c.cc",
"util/dynamic_bloom.cc",
"util/file_checksum_helper.cc",
"util/hash.cc",
"util/murmurhash.cc",
"util/random.cc",
@@ -904,6 +904,13 @@ ROCKS_TESTS = [
[],
[],
],
[
"defer_test",
"util/defer_test.cc",
"serial",
[],
[],
],
[
"delete_scheduler_test",
"file/delete_scheduler_test.cc",
@@ -1310,6 +1317,13 @@ ROCKS_TESTS = [
[],
[],
],
[
"slice_test",
"util/slice_test.cc",
"serial",
[],
[],
],
[
"slice_transform_test",
"util/slice_transform_test.cc",
+4
View File
@@ -102,3 +102,7 @@ LzLabs is using RocksDB as a storage engine in their multi-database distributed
## Crux
[Crux](https://github.com/juxt/crux) is a document database that uses RocksDB for local [EAV](https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model) index storage to enable point-in-time bitemporal Datalog queries. The "unbundled" architecture uses Kafka to provide horizontal scalability.
## Nebula Graph
[Nebula Graph](https://github.com/vesoft-inc/nebula) is a distributed, scalable, lightning-fast, open source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.
+3 -3
View File
@@ -29,7 +29,7 @@ install:
- md %THIRDPARTY_HOME%
- echo "Building Snappy dependency..."
- cd %THIRDPARTY_HOME%
- curl -fsSL -o snappy-1.1.7.zip https://github.com/google/snappy/archive/1.1.7.zip
- curl --fail --silent --show-error --output snappy-1.1.7.zip --location https://github.com/google/snappy/archive/1.1.7.zip
- unzip snappy-1.1.7.zip
- cd snappy-1.1.7
- mkdir build
@@ -39,7 +39,7 @@ install:
- msbuild Snappy.sln /p:Configuration=Release /p:Platform=x64
- echo "Building LZ4 dependency..."
- cd %THIRDPARTY_HOME%
- curl -fsSL -o lz4-1.8.3.zip https://github.com/lz4/lz4/archive/v1.8.3.zip
- curl --fail --silent --show-error --output lz4-1.8.3.zip --location https://github.com/lz4/lz4/archive/v1.8.3.zip
- unzip lz4-1.8.3.zip
- cd lz4-1.8.3\visual\VS2010
- ps: $CMD="$Env:DEV_ENV"; & $CMD lz4.sln /upgrade
@@ -47,7 +47,7 @@ install:
- msbuild lz4.sln /p:Configuration=Release /p:Platform=x64
- echo "Building ZStd dependency..."
- cd %THIRDPARTY_HOME%
- curl -fsSL -o zstd-1.4.0.zip https://github.com/facebook/zstd/archive/v1.4.0.zip
- curl --fail --silent --show-error --output zstd-1.4.0.zip --location https://github.com/facebook/zstd/archive/v1.4.0.zip
- unzip zstd-1.4.0.zip
- cd zstd-1.4.0\build\VS2010
- ps: $CMD="$Env:DEV_ENV"; & $CMD zstd.sln /upgrade
+2 -3
View File
@@ -32,14 +32,13 @@ ROCKSDB_EXTERNAL_DEPS = [
("lz4", None, "lz4"),
("zstd", None),
("tbb", None),
("liburing", None, "uring"),
("googletest", None, "gtest"),
]
ROCKSDB_OS_DEPS = [
(
"linux",
["third-party//numa:numa"],
["third-party//numa:numa", "third-party//liburing:uring"],
),
]
@@ -55,6 +54,7 @@ ROCKSDB_OS_PREPROCESSOR_FLAGS = [
"-DROCKSDB_SCHED_GETCPU_PRESENT",
"-DROCKSDB_IOURING_PRESENT",
"-DHAVE_SSE42",
"-DLIBURING",
"-DNUMA",
],
),
@@ -78,7 +78,6 @@ ROCKSDB_PREPROCESSOR_FLAGS = [
"-DZSTD_STATIC_LINKING_ONLY",
"-DGFLAGS=gflags",
"-DTBB",
"-DLIBURING",
# Added missing flags from output of build_detect_platform
"-DROCKSDB_BACKTRACE",
+13 -11
View File
@@ -150,18 +150,20 @@ case "$TARGET_OS" in
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
fi
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
# check for liburing
$CXX $CFLAGS -x c++ - -luring -o /dev/null 2>/dev/null <<EOF
#include <liburing.h>
int main() {
struct io_uring ring;
io_uring_queue_init(1, &ring, 0);
return 0;
}
if test $ROCKSDB_USE_IO_URING; then
# check for liburing
$CXX $CFLAGS -x c++ - -luring -o /dev/null 2>/dev/null <<EOF
#include <liburing.h>
int main() {
struct io_uring ring;
io_uring_queue_init(1, &ring, 0);
return 0;
}
EOF
if [ "$?" = 0 ]; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -luring"
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_IOURING_PRESENT"
if [ "$?" = 0 ]; then
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -luring"
COMMON_FLAGS="$COMMON_FLAGS -DROCKSDB_IOURING_PRESENT"
fi
fi
if test -z "$USE_FOLLY_DISTRIBUTED_MUTEX"; then
USE_FOLLY_DISTRIBUTED_MUTEX=1
+1 -1
View File
@@ -378,7 +378,7 @@ function send_to_ods {
echo >&2 "ERROR: Key $key doesn't have a value."
return
fi
curl -s "https://www.intern.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build$git_br&key=$key&value=$value" \
curl --silent "https://www.intern.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build$git_br&key=$key&value=$value" \
--connect-timeout 60
}
+7 -2
View File
@@ -414,7 +414,8 @@ STRESS_CRASH_TEST_COMMANDS="[
'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 crash_test || $CONTRUN_NAME=crash_test $TASK_CREATION_TOOL',
'user':'root',
$PARSER
}
},
$UPLOAD_DB_DIR,
],
$REPORT
}
@@ -542,6 +543,7 @@ ASAN_CRASH_TEST_COMMANDS="[
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
@@ -634,6 +636,7 @@ UBSAN_CRASH_TEST_COMMANDS="[
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
@@ -751,6 +754,7 @@ TSAN_CRASH_TEST_COMMANDS="[
'user':'root',
$PARSER
},
$UPLOAD_DB_DIR,
],
$REPORT
}
@@ -880,7 +884,7 @@ run_regression()
# parameters: $1 -- key, $2 -- value
function send_size_to_ods {
curl -s "https://www.intern.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build&key=rocksdb.build_size.$1&value=$2" \
curl --silent "https://www.intern.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build&key=rocksdb.build_size.$1&value=$2" \
--connect-timeout 60
}
@@ -1059,5 +1063,6 @@ case $1 in
;;
*)
echo "Invalid determinator command"
exit 1
;;
esac
+4 -4
View File
@@ -45,7 +45,7 @@ DEFINE_int32(erase_percent, 10,
DEFINE_bool(use_clock_cache, false, "");
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CacheBench;
namespace {
@@ -154,7 +154,7 @@ class CacheBench {
}
bool Run() {
rocksdb::Env* env = rocksdb::Env::Default();
ROCKSDB_NAMESPACE::Env* env = ROCKSDB_NAMESPACE::Env::Default();
PrintEnv();
SharedState shared(this);
@@ -257,7 +257,7 @@ class CacheBench {
printf("----------------------------\n");
}
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
ParseCommandLineFlags(&argc, &argv, true);
@@ -267,7 +267,7 @@ int main(int argc, char** argv) {
exit(1);
}
rocksdb::CacheBench bench;
ROCKSDB_NAMESPACE::CacheBench bench;
if (FLAGS_populate_cache) {
bench.PopulateCache();
}
+2 -2
View File
@@ -20,7 +20,7 @@
#include "util/coding.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// Conversions between numeric keys/values and the types expected by Cache.
static std::string EncodeKey(int k) {
@@ -765,7 +765,7 @@ INSTANTIATE_TEST_CASE_P(CacheTestInstance, CacheTest, testing::Values(kLRU));
#endif // SUPPORT_CLOCK_CACHE
INSTANTIATE_TEST_CASE_P(CacheTestInstance, LRUCacheTest, testing::Values(kLRU));
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+4 -4
View File
@@ -11,7 +11,7 @@
#ifndef SUPPORT_CLOCK_CACHE
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
std::shared_ptr<Cache> NewClockCache(
size_t /*capacity*/, int /*num_shard_bits*/, bool /*strict_capacity_limit*/,
@@ -20,7 +20,7 @@ std::shared_ptr<Cache> NewClockCache(
return nullptr;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#else
@@ -41,7 +41,7 @@ std::shared_ptr<Cache> NewClockCache(
#include "util/autovector.h"
#include "util/mutexlock.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
@@ -756,6 +756,6 @@ std::shared_ptr<Cache> NewClockCache(
capacity, num_shard_bits, strict_capacity_limit, metadata_charge_policy);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // SUPPORT_CLOCK_CACHE
+2 -2
View File
@@ -16,7 +16,7 @@
#include "util/mutexlock.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
LRUHandleTable::LRUHandleTable() : list_(nullptr), length_(0), elems_(0) {
Resize();
@@ -571,4 +571,4 @@ std::shared_ptr<Cache> NewLRUCache(
std::move(memory_allocator), use_adaptive_mutex, metadata_charge_policy);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -16,7 +16,7 @@
#include "port/port.h"
#include "util/autovector.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// LRU cache implementation. This class is not thread-safe.
@@ -336,4 +336,4 @@ class LRUCache
int num_shards_ = 0;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -10,7 +10,7 @@
#include "port/port.h"
#include "test_util/testharness.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class LRUCacheTest : public testing::Test {
public:
@@ -190,7 +190,7 @@ TEST_F(LRUCacheTest, EntriesWithPriority) {
ValidateLRUList({"e", "f", "g", "Z", "d"}, 2);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+2 -2
View File
@@ -13,7 +13,7 @@
#include "util/mutexlock.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
ShardedCache::ShardedCache(size_t capacity, int num_shard_bits,
bool strict_capacity_limit,
@@ -159,4 +159,4 @@ int GetDefaultCacheShardBits(size_t capacity) {
return num_shard_bits;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -16,7 +16,7 @@
#include "rocksdb/cache.h"
#include "util/hash.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// Single cache shard interface.
class CacheShard {
@@ -108,4 +108,4 @@ class ShardedCache : public Cache {
extern int GetDefaultCacheShardBits(size_t capacity);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -16,7 +16,7 @@
#include "table/iterator_wrapper.h"
#include "util/user_comparator_wrapper.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
Status ArenaWrappedDBIter::GetProperty(std::string prop_name,
std::string* prop) {
@@ -103,4 +103,4 @@ ArenaWrappedDBIter* NewArenaWrappedDbIterator(
return iter;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+15 -15
View File
@@ -20,7 +20,7 @@
#include "rocksdb/iterator.h"
#include "util/autovector.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class Arena;
@@ -45,26 +45,26 @@ class ArenaWrappedDBIter : public Iterator {
// Set the internal iterator wrapped inside the DB Iterator. Usually it is
// a merging iterator.
virtual void SetIterUnderDBIter(InternalIterator* iter) {
static_cast<DBIter*>(db_iter_)->SetIter(iter);
db_iter_->SetIter(iter);
}
virtual bool Valid() const override { return db_iter_->Valid(); }
virtual void SeekToFirst() override { db_iter_->SeekToFirst(); }
virtual void SeekToLast() override { db_iter_->SeekToLast(); }
virtual void Seek(const Slice& target) override { db_iter_->Seek(target); }
virtual void SeekForPrev(const Slice& target) override {
bool Valid() const override { return db_iter_->Valid(); }
void SeekToFirst() override { db_iter_->SeekToFirst(); }
void SeekToLast() override { db_iter_->SeekToLast(); }
void Seek(const Slice& target) override { db_iter_->Seek(target); }
void SeekForPrev(const Slice& target) override {
db_iter_->SeekForPrev(target);
}
virtual void Next() override { db_iter_->Next(); }
virtual void Prev() override { db_iter_->Prev(); }
virtual Slice key() const override { return db_iter_->key(); }
virtual Slice value() const override { return db_iter_->value(); }
virtual Status status() const override { return db_iter_->status(); }
void Next() override { db_iter_->Next(); }
void Prev() override { db_iter_->Prev(); }
Slice key() const override { return db_iter_->key(); }
Slice value() const override { return db_iter_->value(); }
Status status() const override { return db_iter_->status(); }
bool IsBlob() const { return db_iter_->IsBlob(); }
virtual Status GetProperty(std::string prop_name, std::string* prop) override;
Status GetProperty(std::string prop_name, std::string* prop) override;
virtual Status Refresh() override;
Status Refresh() override;
void Init(Env* env, const ReadOptions& read_options,
const ImmutableCFOptions& cf_options,
@@ -109,4 +109,4 @@ extern ArenaWrappedDBIter* NewArenaWrappedDbIterator(
ReadCallback* read_callback, DBImpl* db_impl = nullptr,
ColumnFamilyData* cfd = nullptr, bool allow_blob = false,
bool allow_refresh = true);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -12,7 +12,7 @@
#include "util/coding.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// BlobIndex is a pointer to the blob and metadata of the blob. The index is
// stored in base DB as ValueType::kTypeBlobIndex.
@@ -175,5 +175,5 @@ class BlobIndex {
CompressionType compression_ = kNoCompression;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+9 -5
View File
@@ -37,7 +37,7 @@
#include "test_util/sync_point.h"
#include "util/stop_watch.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class TableFactory;
@@ -131,9 +131,10 @@ Status BuildTable(
file->SetIOPriority(io_priority);
file->SetWriteLifeTimeHint(write_hint);
file_writer.reset(
new WritableFileWriter(std::move(file), fname, file_options, env,
ioptions.statistics, ioptions.listeners));
file_writer.reset(new WritableFileWriter(
std::move(file), fname, file_options, env, ioptions.statistics,
ioptions.listeners, ioptions.sst_file_checksum_func));
builder = NewTableBuilder(
ioptions, mutable_cf_options, internal_comparator,
int_tbl_prop_collector_factories, column_family_id,
@@ -199,6 +200,9 @@ Status BuildTable(
if (table_properties) {
*table_properties = tp;
}
// Add the checksum information to file metadata.
meta->file_checksum = builder->GetFileChecksum();
meta->file_checksum_func_name = builder->GetFileChecksumFuncName();
}
delete builder;
@@ -256,4 +260,4 @@ Status BuildTable(
return s;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -22,7 +22,7 @@
#include "rocksdb/types.h"
#include "table/scoped_arena_iterator.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
struct Options;
struct FileMetaData;
@@ -85,4 +85,4 @@ extern Status BuildTable(
Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET,
const uint64_t file_creation_time = 0);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+144 -121
View File
@@ -46,74 +46,74 @@
#include <unordered_set>
#include <map>
using rocksdb::BytewiseComparator;
using rocksdb::Cache;
using rocksdb::ColumnFamilyDescriptor;
using rocksdb::ColumnFamilyHandle;
using rocksdb::ColumnFamilyOptions;
using rocksdb::CompactionFilter;
using rocksdb::CompactionFilterFactory;
using rocksdb::CompactionFilterContext;
using rocksdb::CompactionOptionsFIFO;
using rocksdb::Comparator;
using rocksdb::CompressionType;
using rocksdb::WALRecoveryMode;
using rocksdb::DB;
using rocksdb::DBOptions;
using rocksdb::DbPath;
using rocksdb::Env;
using rocksdb::EnvOptions;
using rocksdb::InfoLogLevel;
using rocksdb::FileLock;
using rocksdb::FilterPolicy;
using rocksdb::FlushOptions;
using rocksdb::IngestExternalFileOptions;
using rocksdb::Iterator;
using rocksdb::Logger;
using rocksdb::MergeOperator;
using rocksdb::MergeOperators;
using rocksdb::NewBloomFilterPolicy;
using rocksdb::NewLRUCache;
using rocksdb::Options;
using rocksdb::BlockBasedTableOptions;
using rocksdb::CuckooTableOptions;
using rocksdb::RandomAccessFile;
using rocksdb::Range;
using rocksdb::ReadOptions;
using rocksdb::SequentialFile;
using rocksdb::Slice;
using rocksdb::SliceParts;
using rocksdb::SliceTransform;
using rocksdb::Snapshot;
using rocksdb::SstFileWriter;
using rocksdb::Status;
using rocksdb::WritableFile;
using rocksdb::WriteBatch;
using rocksdb::WriteBatchWithIndex;
using rocksdb::WriteOptions;
using rocksdb::LiveFileMetaData;
using rocksdb::BackupEngine;
using rocksdb::BackupableDBOptions;
using rocksdb::BackupInfo;
using rocksdb::BackupID;
using rocksdb::RestoreOptions;
using rocksdb::CompactRangeOptions;
using rocksdb::BottommostLevelCompaction;
using rocksdb::RateLimiter;
using rocksdb::NewGenericRateLimiter;
using rocksdb::PinnableSlice;
using rocksdb::TransactionDBOptions;
using rocksdb::TransactionDB;
using rocksdb::TransactionOptions;
using rocksdb::OptimisticTransactionDB;
using rocksdb::OptimisticTransactionOptions;
using rocksdb::Transaction;
using rocksdb::Checkpoint;
using rocksdb::TransactionLogIterator;
using rocksdb::BatchResult;
using rocksdb::PerfLevel;
using rocksdb::PerfContext;
using rocksdb::MemoryUtil;
using ROCKSDB_NAMESPACE::BackupableDBOptions;
using ROCKSDB_NAMESPACE::BackupEngine;
using ROCKSDB_NAMESPACE::BackupID;
using ROCKSDB_NAMESPACE::BackupInfo;
using ROCKSDB_NAMESPACE::BatchResult;
using ROCKSDB_NAMESPACE::BlockBasedTableOptions;
using ROCKSDB_NAMESPACE::BottommostLevelCompaction;
using ROCKSDB_NAMESPACE::BytewiseComparator;
using ROCKSDB_NAMESPACE::Cache;
using ROCKSDB_NAMESPACE::Checkpoint;
using ROCKSDB_NAMESPACE::ColumnFamilyDescriptor;
using ROCKSDB_NAMESPACE::ColumnFamilyHandle;
using ROCKSDB_NAMESPACE::ColumnFamilyOptions;
using ROCKSDB_NAMESPACE::CompactionFilter;
using ROCKSDB_NAMESPACE::CompactionFilterContext;
using ROCKSDB_NAMESPACE::CompactionFilterFactory;
using ROCKSDB_NAMESPACE::CompactionOptionsFIFO;
using ROCKSDB_NAMESPACE::CompactRangeOptions;
using ROCKSDB_NAMESPACE::Comparator;
using ROCKSDB_NAMESPACE::CompressionType;
using ROCKSDB_NAMESPACE::CuckooTableOptions;
using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::DBOptions;
using ROCKSDB_NAMESPACE::DbPath;
using ROCKSDB_NAMESPACE::Env;
using ROCKSDB_NAMESPACE::EnvOptions;
using ROCKSDB_NAMESPACE::FileLock;
using ROCKSDB_NAMESPACE::FilterPolicy;
using ROCKSDB_NAMESPACE::FlushOptions;
using ROCKSDB_NAMESPACE::InfoLogLevel;
using ROCKSDB_NAMESPACE::IngestExternalFileOptions;
using ROCKSDB_NAMESPACE::Iterator;
using ROCKSDB_NAMESPACE::LiveFileMetaData;
using ROCKSDB_NAMESPACE::Logger;
using ROCKSDB_NAMESPACE::MemoryUtil;
using ROCKSDB_NAMESPACE::MergeOperator;
using ROCKSDB_NAMESPACE::MergeOperators;
using ROCKSDB_NAMESPACE::NewBloomFilterPolicy;
using ROCKSDB_NAMESPACE::NewGenericRateLimiter;
using ROCKSDB_NAMESPACE::NewLRUCache;
using ROCKSDB_NAMESPACE::OptimisticTransactionDB;
using ROCKSDB_NAMESPACE::OptimisticTransactionOptions;
using ROCKSDB_NAMESPACE::Options;
using ROCKSDB_NAMESPACE::PerfContext;
using ROCKSDB_NAMESPACE::PerfLevel;
using ROCKSDB_NAMESPACE::PinnableSlice;
using ROCKSDB_NAMESPACE::RandomAccessFile;
using ROCKSDB_NAMESPACE::Range;
using ROCKSDB_NAMESPACE::RateLimiter;
using ROCKSDB_NAMESPACE::ReadOptions;
using ROCKSDB_NAMESPACE::RestoreOptions;
using ROCKSDB_NAMESPACE::SequentialFile;
using ROCKSDB_NAMESPACE::Slice;
using ROCKSDB_NAMESPACE::SliceParts;
using ROCKSDB_NAMESPACE::SliceTransform;
using ROCKSDB_NAMESPACE::Snapshot;
using ROCKSDB_NAMESPACE::SstFileWriter;
using ROCKSDB_NAMESPACE::Status;
using ROCKSDB_NAMESPACE::Transaction;
using ROCKSDB_NAMESPACE::TransactionDB;
using ROCKSDB_NAMESPACE::TransactionDBOptions;
using ROCKSDB_NAMESPACE::TransactionLogIterator;
using ROCKSDB_NAMESPACE::TransactionOptions;
using ROCKSDB_NAMESPACE::WALRecoveryMode;
using ROCKSDB_NAMESPACE::WritableFile;
using ROCKSDB_NAMESPACE::WriteBatch;
using ROCKSDB_NAMESPACE::WriteBatchWithIndex;
using ROCKSDB_NAMESPACE::WriteOptions;
using std::shared_ptr;
using std::vector;
@@ -452,7 +452,7 @@ struct rocksdb_slicetransform_t : public SliceTransform {
};
struct rocksdb_universal_compaction_options_t {
rocksdb::CompactionOptionsUniversal *rep;
ROCKSDB_NAMESPACE::CompactionOptionsUniversal* rep;
};
static bool SaveError(char** errptr, const Status& s) {
@@ -494,8 +494,9 @@ rocksdb_t* rocksdb_open_with_ttl(
const char* name,
int ttl,
char** errptr) {
rocksdb::DBWithTTL* db;
if (SaveError(errptr, rocksdb::DBWithTTL::Open(options->rep, std::string(name), &db, ttl))) {
ROCKSDB_NAMESPACE::DBWithTTL* db;
if (SaveError(errptr, ROCKSDB_NAMESPACE::DBWithTTL::Open(
options->rep, std::string(name), &db, ttl))) {
return nullptr;
}
rocksdb_t* result = new rocksdb_t;
@@ -664,17 +665,15 @@ void rocksdb_close(rocksdb_t* db) {
}
void rocksdb_options_set_uint64add_merge_operator(rocksdb_options_t* opt) {
opt->rep.merge_operator = rocksdb::MergeOperators::CreateUInt64AddOperator();
opt->rep.merge_operator =
ROCKSDB_NAMESPACE::MergeOperators::CreateUInt64AddOperator();
}
rocksdb_t* rocksdb_open_column_families(
const rocksdb_options_t* db_options,
const char* name,
int num_column_families,
const char** column_family_names,
const rocksdb_options_t** column_family_options,
rocksdb_column_family_handle_t** column_family_handles,
char** errptr) {
const rocksdb_options_t* db_options, const char* name,
int num_column_families, const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i < num_column_families; i++) {
column_families.push_back(ColumnFamilyDescriptor(
@@ -700,14 +699,11 @@ rocksdb_t* rocksdb_open_column_families(
}
rocksdb_t* rocksdb_open_for_read_only_column_families(
const rocksdb_options_t* db_options,
const char* name,
int num_column_families,
const char** column_family_names,
const rocksdb_options_t** column_family_options,
const rocksdb_options_t* db_options, const char* name,
int num_column_families, const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles,
unsigned char error_if_log_file_exist,
char** errptr) {
unsigned char error_if_log_file_exist, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i < num_column_families; i++) {
column_families.push_back(ColumnFamilyDescriptor(
@@ -735,8 +731,8 @@ rocksdb_t* rocksdb_open_for_read_only_column_families(
rocksdb_t* rocksdb_open_as_secondary_column_families(
const rocksdb_options_t* db_options, const char* name,
const char* secondary_path, int num_column_families,
const char** column_family_names,
const rocksdb_options_t** column_family_options,
const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i != num_column_families; ++i) {
@@ -2084,7 +2080,7 @@ void rocksdb_options_set_block_based_table_factory(
rocksdb_block_based_table_options_t* table_options) {
if (table_options) {
opt->rep.table_factory.reset(
rocksdb::NewBlockBasedTableFactory(table_options->rep));
ROCKSDB_NAMESPACE::NewBlockBasedTableFactory(table_options->rep));
}
}
@@ -2128,7 +2124,7 @@ void rocksdb_options_set_cuckoo_table_factory(
rocksdb_cuckoo_table_options_t* table_options) {
if (table_options) {
opt->rep.table_factory.reset(
rocksdb::NewCuckooTableFactory(table_options->rep));
ROCKSDB_NAMESPACE::NewCuckooTableFactory(table_options->rep));
}
}
@@ -2318,7 +2314,7 @@ void rocksdb_options_set_max_bytes_for_level_multiplier_additional(
}
void rocksdb_options_enable_statistics(rocksdb_options_t* opt) {
opt->rep.statistics = rocksdb::CreateDBStatistics();
opt->rep.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
}
void rocksdb_options_set_skip_stats_update_on_db_open(rocksdb_options_t* opt,
@@ -2326,6 +2322,11 @@ void rocksdb_options_set_skip_stats_update_on_db_open(rocksdb_options_t* opt,
opt->rep.skip_stats_update_on_db_open = val;
}
void rocksdb_options_set_skip_checking_sst_file_sizes_on_db_open(
rocksdb_options_t* opt, unsigned char val) {
opt->rep.skip_checking_sst_file_sizes_on_db_open = val;
}
void rocksdb_options_set_num_levels(rocksdb_options_t* opt, int n) {
opt->rep.num_levels = n;
}
@@ -2469,16 +2470,20 @@ void rocksdb_options_set_access_hint_on_compaction_start(
rocksdb_options_t* opt, int v) {
switch(v) {
case 0:
opt->rep.access_hint_on_compaction_start = rocksdb::Options::NONE;
opt->rep.access_hint_on_compaction_start =
ROCKSDB_NAMESPACE::Options::NONE;
break;
case 1:
opt->rep.access_hint_on_compaction_start = rocksdb::Options::NORMAL;
opt->rep.access_hint_on_compaction_start =
ROCKSDB_NAMESPACE::Options::NORMAL;
break;
case 2:
opt->rep.access_hint_on_compaction_start = rocksdb::Options::SEQUENTIAL;
opt->rep.access_hint_on_compaction_start =
ROCKSDB_NAMESPACE::Options::SEQUENTIAL;
break;
case 3:
opt->rep.access_hint_on_compaction_start = rocksdb::Options::WILLNEED;
opt->rep.access_hint_on_compaction_start =
ROCKSDB_NAMESPACE::Options::WILLNEED;
break;
}
}
@@ -2644,7 +2649,7 @@ void rocksdb_options_prepare_for_bulk_load(rocksdb_options_t* opt) {
}
void rocksdb_options_set_memtable_vector_rep(rocksdb_options_t *opt) {
opt->rep.memtable_factory.reset(new rocksdb::VectorRepFactory);
opt->rep.memtable_factory.reset(new ROCKSDB_NAMESPACE::VectorRepFactory);
}
void rocksdb_options_set_memtable_prefix_bloom_size_ratio(
@@ -2660,26 +2665,29 @@ void rocksdb_options_set_memtable_huge_page_size(rocksdb_options_t* opt,
void rocksdb_options_set_hash_skip_list_rep(
rocksdb_options_t *opt, size_t bucket_count,
int32_t skiplist_height, int32_t skiplist_branching_factor) {
rocksdb::MemTableRepFactory* factory = rocksdb::NewHashSkipListRepFactory(
bucket_count, skiplist_height, skiplist_branching_factor);
ROCKSDB_NAMESPACE::MemTableRepFactory* factory =
ROCKSDB_NAMESPACE::NewHashSkipListRepFactory(
bucket_count, skiplist_height, skiplist_branching_factor);
opt->rep.memtable_factory.reset(factory);
}
void rocksdb_options_set_hash_link_list_rep(
rocksdb_options_t *opt, size_t bucket_count) {
opt->rep.memtable_factory.reset(rocksdb::NewHashLinkListRepFactory(bucket_count));
opt->rep.memtable_factory.reset(
ROCKSDB_NAMESPACE::NewHashLinkListRepFactory(bucket_count));
}
void rocksdb_options_set_plain_table_factory(
rocksdb_options_t *opt, uint32_t user_key_len, int bloom_bits_per_key,
double hash_table_ratio, size_t index_sparseness) {
rocksdb::PlainTableOptions options;
ROCKSDB_NAMESPACE::PlainTableOptions options;
options.user_key_len = user_key_len;
options.bloom_bits_per_key = bloom_bits_per_key;
options.hash_table_ratio = hash_table_ratio;
options.index_sparseness = index_sparseness;
rocksdb::TableFactory* factory = rocksdb::NewPlainTableFactory(options);
ROCKSDB_NAMESPACE::TableFactory* factory =
ROCKSDB_NAMESPACE::NewPlainTableFactory(options);
opt->rep.table_factory.reset(factory);
}
@@ -2709,7 +2717,8 @@ void rocksdb_options_set_report_bg_io_stats(
}
void rocksdb_options_set_compaction_style(rocksdb_options_t *opt, int style) {
opt->rep.compaction_style = static_cast<rocksdb::CompactionStyle>(style);
opt->rep.compaction_style =
static_cast<ROCKSDB_NAMESPACE::CompactionStyle>(style);
}
void rocksdb_options_set_universal_compaction_options(rocksdb_options_t *opt, rocksdb_universal_compaction_options_t *uco) {
@@ -2723,7 +2732,7 @@ void rocksdb_options_set_fifo_compaction_options(
}
char *rocksdb_options_statistics_get_string(rocksdb_options_t *opt) {
rocksdb::Statistics *statistics = opt->rep.statistics.get();
ROCKSDB_NAMESPACE::Statistics* statistics = opt->rep.statistics.get();
if (statistics) {
return strdup(statistics->ToString().c_str());
}
@@ -2736,6 +2745,11 @@ void rocksdb_options_set_ratelimiter(rocksdb_options_t *opt, rocksdb_ratelimiter
}
}
void rocksdb_options_set_atomic_flush(rocksdb_options_t* opt,
unsigned char atomic_flush) {
opt->rep.atomic_flush = atomic_flush;
}
rocksdb_ratelimiter_t* rocksdb_ratelimiter_create(
int64_t rate_bytes_per_sec,
int64_t refill_period_us,
@@ -2751,6 +2765,12 @@ void rocksdb_ratelimiter_destroy(rocksdb_ratelimiter_t *limiter) {
delete limiter;
}
void rocksdb_options_set_row_cache(rocksdb_options_t* opt, rocksdb_cache_t* cache) {
if(cache) {
opt->rep.row_cache = cache->rep;
}
}
void rocksdb_set_perf_level(int v) {
PerfLevel level = static_cast<PerfLevel>(v);
SetPerfLevel(level);
@@ -2758,7 +2778,7 @@ void rocksdb_set_perf_level(int v) {
rocksdb_perfcontext_t* rocksdb_perfcontext_create() {
rocksdb_perfcontext_t* context = new rocksdb_perfcontext_t;
context->rep = rocksdb::get_perf_context();
context->rep = ROCKSDB_NAMESPACE::get_perf_context();
return context;
}
@@ -3060,11 +3080,12 @@ rocksdb_filterpolicy_t* rocksdb_filterpolicy_create_bloom_format(int bits_per_ke
return rep_->KeyMayMatch(key, filter);
}
// No need to override GetFilterBitsBuilder if this one is overridden
rocksdb::FilterBitsBuilder* GetBuilderWithContext(
const rocksdb::FilterBuildingContext& context) const override {
ROCKSDB_NAMESPACE::FilterBitsBuilder* GetBuilderWithContext(
const ROCKSDB_NAMESPACE::FilterBuildingContext& context)
const override {
return rep_->GetBuilderWithContext(context);
}
rocksdb::FilterBitsReader* GetFilterBitsReader(
ROCKSDB_NAMESPACE::FilterBitsReader* GetFilterBitsReader(
const Slice& contents) const override {
return rep_->GetFilterBitsReader(contents);
}
@@ -3166,7 +3187,7 @@ void rocksdb_readoptions_set_iterate_lower_bound(
void rocksdb_readoptions_set_read_tier(
rocksdb_readoptions_t* opt, int v) {
opt->rep.read_tier = static_cast<rocksdb::ReadTier>(v);
opt->rep.read_tier = static_cast<ROCKSDB_NAMESPACE::ReadTier>(v);
}
void rocksdb_readoptions_set_tailing(
@@ -3338,7 +3359,7 @@ rocksdb_env_t* rocksdb_create_default_env() {
rocksdb_env_t* rocksdb_create_mem_env() {
rocksdb_env_t* result = new rocksdb_env_t;
result->rep = rocksdb::NewMemEnv(Env::Default());
result->rep = ROCKSDB_NAMESPACE::NewMemEnv(Env::Default());
result->is_default = false;
return result;
}
@@ -3553,7 +3574,7 @@ struct Wrapper : public rocksdb_slicetransform_t {
rocksdb_slicetransform_t* rocksdb_slicetransform_create_fixed_prefix(size_t prefixLen) {
Wrapper* wrapper = new Wrapper;
wrapper->rep_ = rocksdb::NewFixedPrefixTransform(prefixLen);
wrapper->rep_ = ROCKSDB_NAMESPACE::NewFixedPrefixTransform(prefixLen);
wrapper->state_ = nullptr;
wrapper->destructor_ = &Wrapper::DoNothing;
return wrapper;
@@ -3561,7 +3582,7 @@ rocksdb_slicetransform_t* rocksdb_slicetransform_create_fixed_prefix(size_t pref
rocksdb_slicetransform_t* rocksdb_slicetransform_create_noop() {
Wrapper* wrapper = new Wrapper;
wrapper->rep_ = rocksdb::NewNoopTransform();
wrapper->rep_ = ROCKSDB_NAMESPACE::NewNoopTransform();
wrapper->state_ = nullptr;
wrapper->destructor_ = &Wrapper::DoNothing;
return wrapper;
@@ -3569,7 +3590,7 @@ rocksdb_slicetransform_t* rocksdb_slicetransform_create_noop() {
rocksdb_universal_compaction_options_t* rocksdb_universal_compaction_options_create() {
rocksdb_universal_compaction_options_t* result = new rocksdb_universal_compaction_options_t;
result->rep = new rocksdb::CompactionOptionsUniversal;
result->rep = new ROCKSDB_NAMESPACE::CompactionOptionsUniversal;
return result;
}
@@ -3600,7 +3621,8 @@ void rocksdb_universal_compaction_options_set_compression_size_percent(
void rocksdb_universal_compaction_options_set_stop_style(
rocksdb_universal_compaction_options_t* uco, int style) {
uco->rep->stop_style = static_cast<rocksdb::CompactionStopStyle>(style);
uco->rep->stop_style =
static_cast<ROCKSDB_NAMESPACE::CompactionStopStyle>(style);
}
void rocksdb_universal_compaction_options_destroy(
@@ -3630,7 +3652,7 @@ void rocksdb_options_set_min_level_to_compress(rocksdb_options_t* opt, int level
assert(level <= opt->rep.num_levels);
opt->rep.compression_per_level.resize(opt->rep.num_levels);
for (int i = 0; i < level; i++) {
opt->rep.compression_per_level[i] = rocksdb::kNoCompression;
opt->rep.compression_per_level[i] = ROCKSDB_NAMESPACE::kNoCompression;
}
for (int i = level; i < opt->rep.num_levels; i++) {
opt->rep.compression_per_level[i] = opt->rep.compression;
@@ -3837,8 +3859,8 @@ rocksdb_transactiondb_t* rocksdb_transactiondb_open(
rocksdb_transactiondb_t* rocksdb_transactiondb_open_column_families(
const rocksdb_options_t* options,
const rocksdb_transactiondb_options_t* txn_db_options, const char* name,
int num_column_families, const char** column_family_names,
const rocksdb_options_t** column_family_options,
int num_column_families, const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i < num_column_families; i++) {
@@ -4215,8 +4237,8 @@ rocksdb_optimistictransactiondb_t* rocksdb_optimistictransactiondb_open(
rocksdb_optimistictransactiondb_t*
rocksdb_optimistictransactiondb_open_column_families(
const rocksdb_options_t* db_options, const char* name,
int num_column_families, const char** column_family_names,
const rocksdb_options_t** column_family_options,
int num_column_families, const char* const* column_family_names,
const rocksdb_options_t* const* column_family_options,
rocksdb_column_family_handle_t** column_family_handles, char** errptr) {
std::vector<ColumnFamilyDescriptor> column_families;
for (int i = 0; i < num_column_families; i++) {
@@ -4332,7 +4354,8 @@ const char* rocksdb_pinnableslice_value(const rocksdb_pinnableslice_t* v,
return v->rep.data();
}
// container to keep databases and caches in order to use rocksdb::MemoryUtil
// container to keep databases and caches in order to use
// ROCKSDB_NAMESPACE::MemoryUtil
struct rocksdb_memory_consumers_t {
std::vector<rocksdb_t*> dbs;
std::unordered_set<rocksdb_cache_t*> caches;
@@ -4360,7 +4383,7 @@ void rocksdb_memory_consumers_destroy(rocksdb_memory_consumers_t* consumers) {
delete consumers;
}
// contains memory usage statistics provided by rocksdb::MemoryUtil
// contains memory usage statistics provided by ROCKSDB_NAMESPACE::MemoryUtil
struct rocksdb_memory_usage_t {
uint64_t mem_table_total;
uint64_t mem_table_unflushed;
@@ -4382,7 +4405,7 @@ rocksdb_memory_usage_t* rocksdb_approximate_memory_usage_create(
cache_set.insert(const_cast<const Cache*>(cache->rep.get()));
}
std::map<rocksdb::MemoryUtil::UsageType, uint64_t> usage_by_type;
std::map<ROCKSDB_NAMESPACE::MemoryUtil::UsageType, uint64_t> usage_by_type;
auto status = MemoryUtil::GetApproximateMemoryUsageByType(dbs, cache_set,
&usage_by_type);
-17
View File
@@ -835,23 +835,6 @@ int main(int argc, char** argv) {
rocksdb_writebatch_wi_iterate(wbi, &pos, CheckPut, CheckDel);
CheckCondition(pos == 3);
rocksdb_writebatch_wi_clear(wbi);
rocksdb_writebatch_wi_put(wbi, "bar", 3, "b", 1);
rocksdb_writebatch_wi_put(wbi, "bay", 3, "d", 1);
rocksdb_writebatch_wi_delete_range(wbi, "bar", 3, "bay", 3);
rocksdb_write_writebatch_wi(db, woptions, wbi, &err);
CheckNoError(err);
CheckGet(db, roptions, "bar", NULL);
CheckGet(db, roptions, "bay", "d");
rocksdb_writebatch_wi_clear(wbi);
const char* start_list[1] = {"bay"};
const size_t start_sizes[1] = {3};
const char* end_list[1] = {"baz"};
const size_t end_sizes[1] = {3};
rocksdb_writebatch_wi_delete_rangev(wbi, 1, start_list, start_sizes, end_list,
end_sizes);
rocksdb_write_writebatch_wi(db, woptions, wbi, &err);
CheckNoError(err);
CheckGet(db, roptions, "bay", NULL);
rocksdb_writebatch_wi_destroy(wbi);
}
+23 -14
View File
@@ -36,7 +36,7 @@
#include "util/autovector.h"
#include "util/compression.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
ColumnFamilyHandleImpl::ColumnFamilyHandleImpl(
ColumnFamilyData* column_family_data, DBImpl* db, InstrumentedMutex* mutex)
@@ -160,11 +160,6 @@ Status CheckConcurrentWritesSupported(const ColumnFamilyOptions& cf_options) {
return Status::InvalidArgument(
"Memtable doesn't concurrent writes (allow_concurrent_memtable_write)");
}
if (cf_options.max_successive_merges != 0) {
return Status::InvalidArgument(
"max_successive_merges > 0 is incompatible "
"with concurrent writes (allow_concurrent_memtable_write)");
}
return Status::OK();
}
@@ -1252,6 +1247,11 @@ Status ColumnFamilyData::ValidateOptions(
if (s.ok() && db_options.allow_concurrent_memtable_write) {
s = CheckConcurrentWritesSupported(cf_options);
}
if (s.ok() && db_options.unordered_write &&
cf_options.max_successive_merges != 0) {
s = Status::InvalidArgument(
"max_successive_merges > 0 is incompatible with unordered_write");
}
if (s.ok()) {
s = CheckCFPathsSupported(db_options, cf_options);
}
@@ -1320,17 +1320,26 @@ Env::WriteLifeTimeHint ColumnFamilyData::CalculateSSTWriteHint(int level) {
static_cast<int>(Env::WLTH_MEDIUM));
}
Status ColumnFamilyData::AddDirectories() {
Status ColumnFamilyData::AddDirectories(
std::map<std::string, std::shared_ptr<Directory>>* created_dirs) {
Status s;
assert(created_dirs != nullptr);
assert(data_dirs_.empty());
for (auto& p : ioptions_.cf_paths) {
std::unique_ptr<Directory> path_directory;
s = DBImpl::CreateAndNewDirectory(ioptions_.env, p.path, &path_directory);
if (!s.ok()) {
return s;
auto existing_dir = created_dirs->find(p.path);
if (existing_dir == created_dirs->end()) {
std::unique_ptr<Directory> path_directory;
s = DBImpl::CreateAndNewDirectory(ioptions_.env, p.path, &path_directory);
if (!s.ok()) {
return s;
}
assert(path_directory != nullptr);
data_dirs_.emplace_back(path_directory.release());
(*created_dirs)[p.path] = data_dirs_.back();
} else {
data_dirs_.emplace_back(existing_dir->second);
}
assert(path_directory != nullptr);
data_dirs_.emplace_back(path_directory.release());
}
assert(data_dirs_.size() == ioptions_.cf_paths.size());
return s;
@@ -1511,4 +1520,4 @@ const Comparator* GetColumnFamilyUserComparator(
return nullptr;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+7 -4
View File
@@ -27,7 +27,7 @@
#include "trace_replay/block_cache_tracer.h"
#include "util/thread_local.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class Version;
class VersionSet;
@@ -497,7 +497,10 @@ class ColumnFamilyData {
Env::WriteLifeTimeHint CalculateSSTWriteHint(int level);
Status AddDirectories();
// created_dirs remembers directory created, so that we don't need to call
// the same data creation operation again.
Status AddDirectories(
std::map<std::string, std::shared_ptr<Directory>>* created_dirs);
Directory* GetDataDir(size_t path_id) const;
@@ -589,7 +592,7 @@ class ColumnFamilyData {
std::atomic<uint64_t> last_memtable_id_;
// Directories corresponding to cf_paths.
std::vector<std::unique_ptr<Directory>> data_dirs_;
std::vector<std::shared_ptr<Directory>> data_dirs_;
};
// ColumnFamilySet has interesting thread-safety requirements
@@ -751,4 +754,4 @@ extern uint32_t GetColumnFamilyID(ColumnFamilyHandle* column_family);
extern const Comparator* GetColumnFamilyUserComparator(
ColumnFamilyHandle* column_family);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+89 -87
View File
@@ -30,7 +30,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
static const int kValueSize = 1000;
@@ -91,7 +91,7 @@ class ColumnFamilyTestBase : public testing::Test {
column_families.push_back(cfdescriptor);
}
Close();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Destroy(column_families);
delete env_;
}
@@ -595,14 +595,15 @@ TEST_P(ColumnFamilyTest, DontReuseColumnFamilyID) {
TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
Open();
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WriteOptionsFile:1",
"ColumnFamilyTest.CreateCFRaceWithGetAggProperty:1"},
{"ColumnFamilyTest.CreateCFRaceWithGetAggProperty:2",
"DBImpl::WriteOptionsFile:2"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread thread([&] { CreateColumnFamilies({"one"}); });
ROCKSDB_NAMESPACE::port::Thread thread(
[&] { CreateColumnFamilies({"one"}); });
TEST_SYNC_POINT("ColumnFamilyTest.CreateCFRaceWithGetAggProperty:1");
uint64_t pv;
@@ -611,7 +612,7 @@ TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
thread.join();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // !ROCKSDB_LITE
@@ -1253,13 +1254,14 @@ TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
// #endif // !ROCKSDB_LITE
class TestComparator : public Comparator {
int Compare(const rocksdb::Slice& /*a*/,
const rocksdb::Slice& /*b*/) const override {
int Compare(const ROCKSDB_NAMESPACE::Slice& /*a*/,
const ROCKSDB_NAMESPACE::Slice& /*b*/) const override {
return 0;
}
const char* Name() const override { return "Test"; }
void FindShortestSeparator(std::string* /*start*/,
const rocksdb::Slice& /*limit*/) const override {}
void FindShortestSeparator(
std::string* /*start*/,
const ROCKSDB_NAMESPACE::Slice& /*limit*/) const override {}
void FindShortSuccessor(std::string* /*key*/) const override {}
};
@@ -1427,11 +1429,11 @@ TEST_P(ColumnFamilyTest, MultipleManualCompactions) {
AssertFilesPerLevel(ToString(i + 1), 1);
}
bool cf_1_1 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::MultiManual:4", "ColumnFamilyTest::MultiManual:1"},
{"ColumnFamilyTest::MultiManual:2", "ColumnFamilyTest::MultiManual:5"},
{"ColumnFamilyTest::MultiManual:2", "ColumnFamilyTest::MultiManual:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::MultiManual:4");
@@ -1440,7 +1442,7 @@ TEST_P(ColumnFamilyTest, MultipleManualCompactions) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<port::Thread> threads;
threads.emplace_back([&] {
CompactRangeOptions compact_options;
@@ -1483,8 +1485,8 @@ TEST_P(ColumnFamilyTest, MultipleManualCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
Close();
}
@@ -1523,11 +1525,11 @@ TEST_P(ColumnFamilyTest, AutomaticAndManualCompactions) {
dbfull()->TEST_write_controler().GetCompactionPressureToken();
bool cf_1_1 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::AutoManual:4", "ColumnFamilyTest::AutoManual:1"},
{"ColumnFamilyTest::AutoManual:2", "ColumnFamilyTest::AutoManual:5"},
{"ColumnFamilyTest::AutoManual:2", "ColumnFamilyTest::AutoManual:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
cf_1_1 = false;
@@ -1535,7 +1537,7 @@ TEST_P(ColumnFamilyTest, AutomaticAndManualCompactions) {
TEST_SYNC_POINT("ColumnFamilyTest::AutoManual:3");
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// SETUP column family "one" -- universal style
for (int i = 0; i < one.level0_file_num_compaction_trigger; ++i) {
PutRandomData(1, 10, 12000, true);
@@ -1553,7 +1555,7 @@ TEST_P(ColumnFamilyTest, AutomaticAndManualCompactions) {
WaitForFlush(2);
AssertFilesPerLevel(ToString(i + 1), 2);
}
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::port::Thread threads([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = false;
ASSERT_OK(
@@ -1580,8 +1582,8 @@ TEST_P(ColumnFamilyTest, AutomaticAndManualCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(ColumnFamilyTest, ManualAndAutomaticCompactions) {
@@ -1627,11 +1629,11 @@ TEST_P(ColumnFamilyTest, ManualAndAutomaticCompactions) {
}
bool cf_1_1 = true;
bool cf_1_2 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::ManualAuto:4", "ColumnFamilyTest::ManualAuto:1"},
{"ColumnFamilyTest::ManualAuto:5", "ColumnFamilyTest::ManualAuto:2"},
{"ColumnFamilyTest::ManualAuto:2", "ColumnFamilyTest::ManualAuto:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::ManualAuto:4");
@@ -1643,8 +1645,8 @@ TEST_P(ColumnFamilyTest, ManualAndAutomaticCompactions) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::port::Thread threads([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = false;
ASSERT_OK(
@@ -1679,8 +1681,8 @@ TEST_P(ColumnFamilyTest, ManualAndAutomaticCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(ColumnFamilyTest, SameCFManualManualCompactions) {
@@ -1721,13 +1723,13 @@ TEST_P(ColumnFamilyTest, SameCFManualManualCompactions) {
}
bool cf_1_1 = true;
bool cf_1_2 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::ManualManual:4", "ColumnFamilyTest::ManualManual:2"},
{"ColumnFamilyTest::ManualManual:4", "ColumnFamilyTest::ManualManual:5"},
{"ColumnFamilyTest::ManualManual:1", "ColumnFamilyTest::ManualManual:2"},
{"ColumnFamilyTest::ManualManual:1",
"ColumnFamilyTest::ManualManual:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::ManualManual:4");
@@ -1739,8 +1741,8 @@ TEST_P(ColumnFamilyTest, SameCFManualManualCompactions) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::port::Thread threads([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = true;
ASSERT_OK(
@@ -1760,7 +1762,7 @@ TEST_P(ColumnFamilyTest, SameCFManualManualCompactions) {
1);
}
rocksdb::port::Thread threads1([&] {
ROCKSDB_NAMESPACE::port::Thread threads1([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = false;
ASSERT_OK(
@@ -1781,8 +1783,8 @@ TEST_P(ColumnFamilyTest, SameCFManualManualCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactions) {
@@ -1823,12 +1825,12 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactions) {
}
bool cf_1_1 = true;
bool cf_1_2 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::ManualAuto:4", "ColumnFamilyTest::ManualAuto:2"},
{"ColumnFamilyTest::ManualAuto:4", "ColumnFamilyTest::ManualAuto:5"},
{"ColumnFamilyTest::ManualAuto:1", "ColumnFamilyTest::ManualAuto:2"},
{"ColumnFamilyTest::ManualAuto:1", "ColumnFamilyTest::ManualAuto:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::ManualAuto:4");
@@ -1840,8 +1842,8 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactions) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::port::Thread threads([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = false;
ASSERT_OK(
@@ -1874,8 +1876,8 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactionsLevel) {
@@ -1916,14 +1918,14 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactionsLevel) {
}
bool cf_1_1 = true;
bool cf_1_2 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::ManualAuto:4", "ColumnFamilyTest::ManualAuto:2"},
{"ColumnFamilyTest::ManualAuto:4", "ColumnFamilyTest::ManualAuto:5"},
{"ColumnFamilyTest::ManualAuto:3", "ColumnFamilyTest::ManualAuto:2"},
{"LevelCompactionPicker::PickCompactionBySize:0",
"ColumnFamilyTest::ManualAuto:3"},
{"ColumnFamilyTest::ManualAuto:1", "ColumnFamilyTest::ManualAuto:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::ManualAuto:4");
@@ -1935,8 +1937,8 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactionsLevel) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::port::Thread threads([&] {
CompactRangeOptions compact_options;
compact_options.exclusive_manual_compaction = false;
ASSERT_OK(
@@ -1967,8 +1969,8 @@ TEST_P(ColumnFamilyTest, SameCFManualAutomaticCompactionsLevel) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
// In this test, we generate enough files to trigger automatic compactions.
@@ -2008,12 +2010,12 @@ TEST_P(ColumnFamilyTest, SameCFAutomaticManualCompactions) {
bool cf_1_1 = true;
bool cf_1_2 = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyTest::AutoManual:4", "ColumnFamilyTest::AutoManual:2"},
{"ColumnFamilyTest::AutoManual:4", "ColumnFamilyTest::AutoManual:5"},
{"CompactionPicker::CompactRange:Conflict",
"ColumnFamilyTest::AutoManual:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (cf_1_1) {
TEST_SYNC_POINT("ColumnFamilyTest::AutoManual:4");
@@ -2025,7 +2027,7 @@ TEST_P(ColumnFamilyTest, SameCFAutomaticManualCompactions) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// SETUP column family "one" -- universal style
for (int i = 0; i < one.level0_file_num_compaction_trigger; ++i) {
@@ -2059,8 +2061,8 @@ TEST_P(ColumnFamilyTest, SameCFAutomaticManualCompactions) {
ASSERT_NE("NOT_FOUND", Get(1, *key_iter));
key_iter++;
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
@@ -2415,7 +2417,7 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
options.write_buffer_size = 100000; // small write buffer size
Reopen({options, options});
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"VersionSet::LogAndApply::ColumnFamilyDrop:0",
"FlushJob::WriteLevel0Table"},
{"VersionSet::LogAndApply::ColumnFamilyDrop:1",
@@ -2423,7 +2425,7 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
{"FlushJob::InstallResults",
"VersionSet::LogAndApply::ColumnFamilyDrop:2"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
test::SleepingBackgroundTask sleeping_task;
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask, &sleeping_task,
@@ -2502,21 +2504,21 @@ TEST_P(ColumnFamilyTest, CreateAndDropRace) {
auto main_thread_id = std::this_thread::get_id();
rocksdb::SyncPoint::GetInstance()->SetCallBack("PersistRocksDBOptions:start",
[&](void* /*arg*/) {
auto current_thread_id = std::this_thread::get_id();
// If it's the main thread hitting this sync-point, then it
// will be blocked until some other thread update the test_stage.
if (main_thread_id == current_thread_id) {
test_stage = kMainThreadStartPersistingOptionsFile;
while (test_stage < kChildThreadFinishDroppingColumnFamily &&
!ordered_by_writethread) {
Env::Default()->SleepForMicroseconds(100);
}
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"PersistRocksDBOptions:start", [&](void* /*arg*/) {
auto current_thread_id = std::this_thread::get_id();
// If it's the main thread hitting this sync-point, then it
// will be blocked until some other thread update the test_stage.
if (main_thread_id == current_thread_id) {
test_stage = kMainThreadStartPersistingOptionsFile;
while (test_stage < kChildThreadFinishDroppingColumnFamily &&
!ordered_by_writethread) {
Env::Default()->SleepForMicroseconds(100);
}
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WriteThread::EnterUnbatched:Wait", [&](void* /*arg*/) {
// This means a thread doing DropColumnFamily() is waiting for
// other thread to finish persisting options.
@@ -2528,12 +2530,12 @@ TEST_P(ColumnFamilyTest, CreateAndDropRace) {
Open({"default", "one", "two", "three"},
{cf_opts[0], cf_opts[1], cf_opts[2], cf_opts[3]});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Start a thread that will drop the first column family
// and its comparator
rocksdb::port::Thread drop_cf_thread(DropSingleColumnFamily, this, 1,
&comparators);
ROCKSDB_NAMESPACE::port::Thread drop_cf_thread(DropSingleColumnFamily, this,
1, &comparators);
DropColumnFamilies({2});
@@ -2546,8 +2548,8 @@ TEST_P(ColumnFamilyTest, CreateAndDropRace) {
}
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
@@ -2982,10 +2984,10 @@ TEST_P(ColumnFamilyTest, FlushCloseWALFiles) {
ASSERT_OK(Put(0, "fodor", "mirko"));
ASSERT_OK(Put(1, "fodor", "mirko"));
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"DBImpl::BGWorkFlush:done", "FlushCloseWALFiles:0"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Block flush jobs from running
test::SleepingBackgroundTask sleeping_task;
@@ -3001,7 +3003,7 @@ TEST_P(ColumnFamilyTest, FlushCloseWALFiles) {
sleeping_task.WakeUp();
sleeping_task.WaitUntilDone();
TEST_SYNC_POINT("FlushCloseWALFiles:0");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ(1, env.num_open_wal_file_.load());
Reopen();
@@ -3079,14 +3081,14 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
ASSERT_OK(Put(0, "fodor", "mirko"));
ASSERT_OK(Put(1, "fodor", "mirko"));
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"ColumnFamilyTest::IteratorCloseWALFile2:0",
"DBImpl::BGWorkPurge:start"},
{"ColumnFamilyTest::IteratorCloseWALFile2:2",
"DBImpl::BackgroundCallFlush:start"},
{"DBImpl::BGWorkPurge:end", "ColumnFamilyTest::IteratorCloseWALFile2:1"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = true;
@@ -3104,7 +3106,7 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
TEST_SYNC_POINT("ColumnFamilyTest::IteratorCloseWALFile2:2");
WaitForFlush(1);
ASSERT_EQ(1, env.num_open_wal_file_.load());
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Reopen();
ASSERT_EQ("mirko", Get(0, "fodor"));
@@ -3148,14 +3150,14 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
ASSERT_OK(Put(0, "fodor", "mirko"));
ASSERT_OK(Put(1, "fodor", "mirko"));
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"ColumnFamilyTest::IteratorCloseWALFile2:0",
"DBImpl::BGWorkPurge:start"},
{"ColumnFamilyTest::IteratorCloseWALFile2:2",
"DBImpl::BackgroundCallFlush:start"},
{"DBImpl::BGWorkPurge:end", "ColumnFamilyTest::IteratorCloseWALFile2:1"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = true;
@@ -3179,7 +3181,7 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
ASSERT_EQ(1, env.delete_count_.load());
delete it;
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Reopen();
ASSERT_EQ("mirko", Get(0, "fodor"));
@@ -3199,15 +3201,15 @@ TEST_P(ColumnFamilyTest, LogSyncConflictFlush) {
Put(0, "", "");
Put(1, "foo", "bar");
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::SyncWAL:BeforeMarkLogsSynced:1",
"ColumnFamilyTest::LogSyncConflictFlush:1"},
{"ColumnFamilyTest::LogSyncConflictFlush:2",
"DBImpl::SyncWAL:BeforeMarkLogsSynced:2"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread thread([&] { db_->SyncWAL(); });
ROCKSDB_NAMESPACE::port::Thread thread([&] { db_->SyncWAL(); });
TEST_SYNC_POINT("ColumnFamilyTest::LogSyncConflictFlush:1");
Flush(1);
@@ -3218,7 +3220,7 @@ TEST_P(ColumnFamilyTest, LogSyncConflictFlush) {
thread.join();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Close();
}
#endif
@@ -3367,7 +3369,7 @@ TEST_P(ColumnFamilyTest, MultipleCFPathsTest) {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
extern "C" {
@@ -3378,7 +3380,7 @@ void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
#endif // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
RegisterCustomObjects(argc, argv);
return RUN_ALL_TESTS();
+15 -15
View File
@@ -18,7 +18,7 @@
#include "test_util/testharness.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CompactFilesTest : public testing::Test {
public:
@@ -81,11 +81,11 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
assert(s.ok());
assert(db);
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"CompactFilesImpl:0", "BackgroundCallCompaction:0"},
{"BackgroundCallCompaction:1", "CompactFilesImpl:1"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// create couple files
// Background compaction starts and waits in BackgroundCallCompaction:0
@@ -95,7 +95,7 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
db->Flush(FlushOptions());
}
rocksdb::ColumnFamilyMetaData meta;
ROCKSDB_NAMESPACE::ColumnFamilyMetaData meta;
db->GetColumnFamilyMetaData(&meta);
std::string file1;
for (auto& file : meta.levels[0].files) {
@@ -108,12 +108,12 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
// The background compaction then notices that there is an L0 compaction
// already in progress and doesn't do an L0 compaction
// Once the background compaction finishes, the compact files finishes
ASSERT_OK(
db->CompactFiles(rocksdb::CompactionOptions(), {file1, file2}, 0));
ASSERT_OK(db->CompactFiles(ROCKSDB_NAMESPACE::CompactionOptions(),
{file1, file2}, 0));
break;
}
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
delete db;
}
@@ -224,14 +224,14 @@ TEST_F(CompactFilesTest, CapturingPendingFiles) {
auto l0_files = collector->GetFlushedFiles();
EXPECT_EQ(5, l0_files.size());
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"CompactFilesImpl:2", "CompactFilesTest.CapturingPendingFiles:0"},
{"CompactFilesTest.CapturingPendingFiles:1", "CompactFilesImpl:3"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Start compacting files.
rocksdb::port::Thread compaction_thread(
ROCKSDB_NAMESPACE::port::Thread compaction_thread(
[&] { EXPECT_OK(db->CompactFiles(CompactionOptions(), l0_files, 1)); });
// In the meantime flush another file.
@@ -242,7 +242,7 @@ TEST_F(CompactFilesTest, CapturingPendingFiles) {
compaction_thread.join();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
delete db;
@@ -296,12 +296,12 @@ TEST_F(CompactFilesTest, CompactionFilterWithGetSv) {
db->Flush(FlushOptions());
// Compact all L0 files using CompactFiles
rocksdb::ColumnFamilyMetaData meta;
ROCKSDB_NAMESPACE::ColumnFamilyMetaData meta;
db->GetColumnFamilyMetaData(&meta);
for (auto& file : meta.levels[0].files) {
std::string fname = file.db_path + "/" + file.name;
ASSERT_OK(
db->CompactFiles(rocksdb::CompactionOptions(), {fname}, 0));
db->CompactFiles(ROCKSDB_NAMESPACE::CompactionOptions(), {fname}, 0));
}
@@ -347,7 +347,7 @@ TEST_F(CompactFilesTest, SentinelCompressionType) {
compaction_opts.compression = CompressionType::kDisableCompressionOption;
ASSERT_OK(db->CompactFiles(compaction_opts, l0_files, 1));
rocksdb::TablePropertiesCollection all_tables_props;
ROCKSDB_NAMESPACE::TablePropertiesCollection all_tables_props;
ASSERT_OK(db->GetPropertiesOfAllTables(&all_tables_props));
for (const auto& name_and_table_props : all_tables_props) {
ASSERT_EQ(CompressionTypeToString(CompressionType::kZlibCompression),
@@ -402,7 +402,7 @@ TEST_F(CompactFilesTest, GetCompactionJobInfo) {
delete db;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+2 -2
View File
@@ -9,7 +9,7 @@
#include "db/version_set.h"
#include "table/get_context.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
extern void MarkKeyMayExist(void* arg);
extern bool SaveValue(void* arg, const ParsedInternalKey& parsed_key,
@@ -156,5 +156,5 @@ Status CompactedDBImpl::Open(const Options& options,
return s;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+2 -2
View File
@@ -9,7 +9,7 @@
#include <vector>
#include "db/db_impl/db_impl.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CompactedDBImpl : public DBImpl {
public:
@@ -109,5 +109,5 @@ class CompactedDBImpl : public DBImpl {
const Comparator* user_comparator_;
LevelFilesBrief files_;
};
}
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+9 -7
View File
@@ -16,7 +16,7 @@
#include "test_util/sync_point.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
const uint64_t kRangeTombstoneSentinel =
PackSequenceAndType(kMaxSequenceNumber, kTypeRangeDeletion);
@@ -545,11 +545,13 @@ bool Compaction::ShouldFormSubcompactions() const {
uint64_t Compaction::MinInputFileOldestAncesterTime() const {
uint64_t min_oldest_ancester_time = port::kMaxUint64;
for (const auto& file : inputs_[0].files) {
uint64_t oldest_ancester_time = file->TryGetOldestAncesterTime();
if (oldest_ancester_time != 0) {
min_oldest_ancester_time =
std::min(min_oldest_ancester_time, oldest_ancester_time);
for (const auto& level_files : inputs_) {
for (const auto& file : level_files.files) {
uint64_t oldest_ancester_time = file->TryGetOldestAncesterTime();
if (oldest_ancester_time != 0) {
min_oldest_ancester_time =
std::min(min_oldest_ancester_time, oldest_ancester_time);
}
}
}
return min_oldest_ancester_time;
@@ -559,4 +561,4 @@ int Compaction::GetInputBaseLevel() const {
return input_vstorage_->base_level();
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -13,7 +13,7 @@
#include "options/cf_options.h"
#include "util/autovector.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// The file contains class Compaction, as well as some helper functions
// and data structures used by the class.
@@ -381,4 +381,4 @@ class Compaction {
// Return sum of sizes of all files in `files`.
extern uint64_t TotalFileSize(const std::vector<FileMetaData*>& files);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
@@ -5,6 +5,8 @@
#pragma once
#include "rocksdb/rocksdb_namespace.h"
struct CompactionIterationStats {
// Compaction statistics
+2 -2
View File
@@ -28,7 +28,7 @@
((seq) <= earliest_snapshot_ && \
(snapshot_checker_ == nullptr || LIKELY(IsInEarliestSnapshot(seq))))
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
CompactionIterator::CompactionIterator(
InternalIterator* input, const Comparator* cmp, MergeHelper* merge_helper,
@@ -771,4 +771,4 @@ bool CompactionIterator::IsInEarliestSnapshot(SequenceNumber sequence) {
return in_snapshot == SnapshotCheckerResult::kInSnapshot;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -19,7 +19,7 @@
#include "options/cf_options.h"
#include "rocksdb/compaction_filter.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CompactionIterator {
public:
@@ -237,4 +237,4 @@ class CompactionIterator {
manual_compaction_paused_->load(std::memory_order_relaxed);
}
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+4 -4
View File
@@ -14,7 +14,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// Expects no merging attempts.
class NoMergingMergeOp : public MergeOperator {
@@ -494,7 +494,7 @@ TEST_P(CompactionIteratorTest, ShuttingDownInFilter) {
compaction_proxy_->key_not_exists_beyond_output_level = true;
std::atomic<bool> seek_done{false};
rocksdb::port::Thread compaction_thread([&] {
ROCKSDB_NAMESPACE::port::Thread compaction_thread([&] {
c_iter_->SeekToFirst();
EXPECT_FALSE(c_iter_->Valid());
EXPECT_TRUE(c_iter_->status().IsShutdownInProgress());
@@ -531,7 +531,7 @@ TEST_P(CompactionIteratorTest, ShuttingDownInMerge) {
compaction_proxy_->key_not_exists_beyond_output_level = true;
std::atomic<bool> seek_done{false};
rocksdb::port::Thread compaction_thread([&] {
ROCKSDB_NAMESPACE::port::Thread compaction_thread([&] {
c_iter_->SeekToFirst();
ASSERT_FALSE(c_iter_->Valid());
ASSERT_TRUE(c_iter_->status().IsShutdownInProgress());
@@ -968,7 +968,7 @@ TEST_F(CompactionIteratorWithSnapshotCheckerTest, CompactionFilter_FullMerge) {
compaction_filter.get());
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+25 -21
View File
@@ -59,7 +59,7 @@
#include "util/stop_watch.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
const char* GetCompactionReasonString(CompactionReason compaction_reason) {
switch (compaction_reason) {
@@ -144,7 +144,6 @@ struct CompactionJob::SubcompactionState {
// State during the subcompaction
uint64_t total_bytes;
uint64_t num_input_records;
uint64_t num_output_records;
CompactionJobStats compaction_job_stats;
uint64_t approx_size;
@@ -165,7 +164,6 @@ struct CompactionJob::SubcompactionState {
builder(nullptr),
current_output_file_size(0),
total_bytes(0),
num_input_records(0),
num_output_records(0),
approx_size(size),
grandparent_index(0),
@@ -186,7 +184,6 @@ struct CompactionJob::SubcompactionState {
builder = std::move(o.builder);
current_output_file_size = std::move(o.current_output_file_size);
total_bytes = std::move(o.total_bytes);
num_input_records = std::move(o.num_input_records);
num_output_records = std::move(o.num_output_records);
compaction_job_stats = std::move(o.compaction_job_stats);
approx_size = std::move(o.approx_size);
@@ -245,13 +242,11 @@ struct CompactionJob::CompactionState {
Status status;
uint64_t total_bytes;
uint64_t num_input_records;
uint64_t num_output_records;
explicit CompactionState(Compaction* c)
: compaction(c),
total_bytes(0),
num_input_records(0),
num_output_records(0) {}
size_t NumOutputFiles() {
@@ -289,7 +284,6 @@ struct CompactionJob::CompactionState {
void CompactionJob::AggregateStatistics() {
for (SubcompactionState& sc : compact_->sub_compact_states) {
compact_->total_bytes += sc.total_bytes;
compact_->num_input_records += sc.num_input_records;
compact_->num_output_records += sc.num_output_records;
}
if (compaction_job_stats_) {
@@ -770,12 +764,12 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
auto stream = event_logger_->LogToBuffer(log_buffer_);
stream << "job" << job_id_ << "event"
<< "compaction_finished"
<< "compaction_time_micros" << compaction_stats_.micros
<< "compaction_time_cpu_micros" << compaction_stats_.cpu_micros
<< "output_level" << compact_->compaction->output_level()
<< "num_output_files" << compact_->NumOutputFiles()
<< "total_output_size" << compact_->total_bytes << "num_input_records"
<< compact_->num_input_records << "num_output_records"
<< "compaction_time_micros" << stats.micros
<< "compaction_time_cpu_micros" << stats.cpu_micros << "output_level"
<< compact_->compaction->output_level() << "num_output_files"
<< compact_->NumOutputFiles() << "total_output_size"
<< compact_->total_bytes << "num_input_records"
<< stats.num_input_records << "num_output_records"
<< compact_->num_output_records << "num_subcompactions"
<< compact_->sub_compact_states.size() << "output_compression"
<< CompressionTypeToString(compact_->compaction->output_compression());
@@ -991,7 +985,6 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
}
}
sub_compact->num_input_records = c_iter_stats.num_input_records;
sub_compact->compaction_job_stats.num_input_deletion_records =
c_iter_stats.num_input_deletion_records;
sub_compact->compaction_job_stats.num_corrupt_keys =
@@ -1303,6 +1296,11 @@ Status CompactionJob::FinishCompactionOutputFile(
}
const uint64_t current_bytes = sub_compact->builder->FileSize();
if (s.ok()) {
// Add the checksum information to file metadata.
meta->file_checksum = sub_compact->builder->GetFileChecksum();
meta->file_checksum_func_name =
sub_compact->builder->GetFileChecksumFuncName();
meta->fd.file_size = current_bytes;
}
sub_compact->current_output()->finished = true;
@@ -1515,7 +1513,8 @@ Status CompactionJob::OpenCompactionOutputFile(
sub_compact->compaction->immutable_cf_options()->listeners;
sub_compact->outfile.reset(
new WritableFileWriter(std::move(writable_file), fname, file_options_,
env_, db_options_.statistics.get(), listeners));
env_, db_options_.statistics.get(), listeners,
db_options_.sst_file_checksum_func.get()));
// If the Column family flag is to only optimize filters for hits,
// we can skip creating filters if this is the bottommost_level where
@@ -1589,6 +1588,8 @@ void CompactionJob::UpdateCompactionStats() {
}
}
uint64_t num_output_records = 0;
for (const auto& sub_compact : compact_->sub_compact_states) {
size_t num_output_files = sub_compact.outputs.size();
if (sub_compact.builder != nullptr) {
@@ -1598,13 +1599,16 @@ void CompactionJob::UpdateCompactionStats() {
}
compaction_stats_.num_output_files += static_cast<int>(num_output_files);
num_output_records += sub_compact.num_output_records;
for (const auto& out : sub_compact.outputs) {
compaction_stats_.bytes_written += out.meta.fd.file_size;
}
if (sub_compact.num_input_records > sub_compact.num_output_records) {
compaction_stats_.num_dropped_records +=
sub_compact.num_input_records - sub_compact.num_output_records;
}
}
if (compaction_stats_.num_input_records > num_output_records) {
compaction_stats_.num_dropped_records =
compaction_stats_.num_input_records - num_output_records;
}
}
@@ -1632,7 +1636,7 @@ void CompactionJob::UpdateCompactionJobStats(
// input information
compaction_job_stats_->total_input_bytes =
stats.bytes_read_non_output_levels + stats.bytes_read_output_level;
compaction_job_stats_->num_input_records = compact_->num_input_records;
compaction_job_stats_->num_input_records = stats.num_input_records;
compaction_job_stats_->num_input_files =
stats.num_input_files_in_non_output_levels +
stats.num_input_files_in_output_level;
@@ -1693,4 +1697,4 @@ void CompactionJob::LogCompaction() {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -44,7 +44,7 @@
#include "util/stop_watch.h"
#include "util/thread_local.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class Arena;
class ErrorHandler;
@@ -195,4 +195,4 @@ class CompactionJob {
Env::Priority thread_pri_;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+12 -12
View File
@@ -61,7 +61,7 @@
#if !defined(IOS_CROSS_COMPILE)
#ifndef ROCKSDB_LITE
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
static std::string RandomString(Random* rnd, int len, double ratio) {
std::string r;
@@ -110,9 +110,9 @@ class CompactionJobStatsTest : public testing::Test,
}
~CompactionJobStatsTest() override {
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->LoadDependency({});
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
Close();
Options options;
options.db_paths.emplace_back(dbname_, 0);
@@ -801,7 +801,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
stats_checker->set_verify_next_comp_io_stats(true);
std::atomic<bool> first_prepare_write(true);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::Append:BeforePrepareWrite", [&](void* /*arg*/) {
if (first_prepare_write.load()) {
options.env->SleepForMicroseconds(3);
@@ -810,7 +810,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
});
std::atomic<bool> first_flush(true);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::Flush:BeforeAppend", [&](void* /*arg*/) {
if (first_flush.load()) {
options.env->SleepForMicroseconds(3);
@@ -819,7 +819,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
});
std::atomic<bool> first_sync(true);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::SyncInternal:0", [&](void* /*arg*/) {
if (first_sync.load()) {
options.env->SleepForMicroseconds(3);
@@ -828,14 +828,14 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
});
std::atomic<bool> first_range_sync(true);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::RangeSync:0", [&](void* /*arg*/) {
if (first_range_sync.load()) {
options.env->SleepForMicroseconds(3);
first_range_sync.store(false);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Compact(1, smallest_key, largest_key);
@@ -844,7 +844,7 @@ TEST_P(CompactionJobStatsTest, CompactionJobStatsTest) {
ASSERT_TRUE(!first_flush.load());
ASSERT_TRUE(!first_sync.load());
ASSERT_TRUE(!first_range_sync.load());
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
ASSERT_EQ(stats_checker->NumberOfUnverifiedStats(), 0U);
}
@@ -1019,10 +1019,10 @@ TEST_P(CompactionJobStatsTest, UniversalCompactionTest) {
INSTANTIATE_TEST_CASE_P(CompactionJobStatsTest, CompactionJobStatsTest,
::testing::Values(1, 4));
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+4 -3
View File
@@ -29,7 +29,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
@@ -187,7 +187,8 @@ class CompactionJobTest : public testing::Test {
VersionEdit edit;
edit.AddFile(level, file_number, 0, 10, smallest_key, largest_key,
smallest_seqno, largest_seqno, false, oldest_blob_file_number,
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime,
kUnknownFileChecksum, kUnknownFileChecksumFuncName);
mutex_.Lock();
versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
@@ -1062,7 +1063,7 @@ TEST_F(CompactionJobTest, OldestBlobFileNumber) {
/* expected_oldest_blob_file_number */ 19);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+2 -2
View File
@@ -23,7 +23,7 @@
#include "util/random.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
uint64_t TotalCompensatedFileSize(const std::vector<FileMetaData*>& files) {
@@ -1128,4 +1128,4 @@ bool CompactionPicker::GetOverlappingL0Files(
return true;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -22,7 +22,7 @@
#include "rocksdb/options.h"
#include "rocksdb/status.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// The file contains an abstract class CompactionPicker, and its two
// sub-classes LevelCompactionPicker and NullCompactionPicker, as
@@ -310,4 +310,4 @@ CompressionOptions GetCompressionOptions(const ImmutableCFOptions& ioptions,
int level,
const bool enable_compression = true);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+10 -8
View File
@@ -17,7 +17,7 @@
#include "logging/log_buffer.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
uint64_t GetTotalFilesSize(const std::vector<FileMetaData*>& files) {
uint64_t total_size = 0;
@@ -70,10 +70,9 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
// avoid underflow
if (current_time > mutable_cf_options.ttl) {
for (auto ritr = level_files.rbegin(); ritr != level_files.rend(); ++ritr) {
auto f = *ritr;
if (f->fd.table_reader != nullptr &&
f->fd.table_reader->GetTableProperties() != nullptr) {
auto creation_time =
FileMetaData* f = *ritr;
if (f->fd.table_reader && f->fd.table_reader->GetTableProperties()) {
uint64_t creation_time =
f->fd.table_reader->GetTableProperties()->creation_time;
if (creation_time == 0 ||
creation_time >= (current_time - mutable_cf_options.ttl)) {
@@ -96,11 +95,14 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
}
for (const auto& f : inputs[0].files) {
uint64_t creation_time = 0;
if (f && f->fd.table_reader && f->fd.table_reader->GetTableProperties()) {
creation_time = f->fd.table_reader->GetTableProperties()->creation_time;
}
ROCKS_LOG_BUFFER(log_buffer,
"[%s] FIFO compaction: picking file %" PRIu64
" with creation time %" PRIu64 " for deletion",
cf_name.c_str(), f->fd.GetNumber(),
f->fd.table_reader->GetTableProperties()->creation_time);
cf_name.c_str(), f->fd.GetNumber(), creation_time);
}
Compaction* c = new Compaction(
@@ -238,5 +240,5 @@ Compaction* FIFOCompactionPicker::CompactRange(
return c;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+2 -2
View File
@@ -12,7 +12,7 @@
#include "db/compaction/compaction_picker.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class FIFOCompactionPicker : public CompactionPicker {
public:
FIFOCompactionPicker(const ImmutableCFOptions& ioptions,
@@ -49,5 +49,5 @@ class FIFOCompactionPicker : public CompactionPicker {
VersionStorageInfo* version,
LogBuffer* log_buffer);
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+2 -2
View File
@@ -15,7 +15,7 @@
#include "logging/log_buffer.h"
#include "test_util/sync_point.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
bool LevelCompactionPicker::NeedsCompaction(
const VersionStorageInfo* vstorage) const {
@@ -555,4 +555,4 @@ Compaction* LevelCompactionPicker::PickCompaction(
log_buffer, mutable_cf_options, ioptions_);
return builder.PickCompaction();
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -11,7 +11,7 @@
#include "db/compaction/compaction_picker.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// Picking compactions for leveled compaction. See wiki page
// https://github.com/facebook/rocksdb/wiki/Leveled-Compaction
// for description of Leveled compaction.
@@ -29,4 +29,4 @@ class LevelCompactionPicker : public CompactionPicker {
const VersionStorageInfo* vstorage) const override;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+4 -3
View File
@@ -17,7 +17,7 @@
#include "test_util/testutil.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CountingLogger : public Logger {
public:
@@ -95,7 +95,8 @@ class CompactionPickerTest : public testing::Test {
InternalKey(smallest, smallest_seq, kTypeValue),
InternalKey(largest, largest_seq, kTypeValue), smallest_seq,
largest_seq, /* marked_for_compact */ false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime,
kUnknownFileChecksum, kUnknownFileChecksumFuncName);
f->compensated_file_size =
(compensated_file_size != 0) ? compensated_file_size : file_size;
vstorage_->AddFile(level, f);
@@ -1732,7 +1733,7 @@ TEST_F(CompactionPickerTest, IntraL0ForEarliestSeqno) {
ASSERT_EQ(0, compaction->output_level());
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+2 -2
View File
@@ -23,7 +23,7 @@
#include "util/random.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
// A helper class that form universal compactions. The class is used by
// UniversalCompactionPicker::PickCompaction().
@@ -1100,6 +1100,6 @@ Compaction* UniversalCompactionBuilder::PickPeriodicCompaction() {
return c;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+2 -2
View File
@@ -12,7 +12,7 @@
#include "db/compaction/compaction_picker.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class UniversalCompactionPicker : public CompactionPicker {
public:
UniversalCompactionPicker(const ImmutableCFOptions& ioptions,
@@ -27,5 +27,5 @@ class UniversalCompactionPicker : public CompactionPicker {
virtual bool NeedsCompaction(
const VersionStorageInfo* vstorage) const override;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+13 -13
View File
@@ -18,10 +18,10 @@
using std::unique_ptr;
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
static const Comparator* comparator;
static const Comparator* kTestComparator = nullptr;
class KVIter : public Iterator {
public:
@@ -74,7 +74,7 @@ void AssertItersEqual(Iterator* iter1, Iterator* iter2) {
void DoRandomIteraratorTest(DB* db, std::vector<std::string> source_strings,
Random* rnd, int num_writes, int num_iter_ops,
int num_trigger_flush) {
stl_wrappers::KVMap map((stl_wrappers::LessOfComparator(comparator)));
stl_wrappers::KVMap map((stl_wrappers::LessOfComparator(kTestComparator)));
for (int i = 0; i < num_writes; i++) {
if (num_trigger_flush > 0 && i != 0 && i % num_trigger_flush == 0) {
@@ -263,19 +263,19 @@ class ComparatorDBTest
public:
ComparatorDBTest() : env_(Env::Default()), db_(nullptr) {
comparator = BytewiseComparator();
kTestComparator = BytewiseComparator();
dbname_ = test::PerThreadDBPath("comparator_db_test");
BlockBasedTableOptions toptions;
toptions.format_version = GetParam();
last_options_.table_factory.reset(
rocksdb::NewBlockBasedTableFactory(toptions));
ROCKSDB_NAMESPACE::NewBlockBasedTableFactory(toptions));
EXPECT_OK(DestroyDB(dbname_, last_options_));
}
~ComparatorDBTest() override {
delete db_;
EXPECT_OK(DestroyDB(dbname_, last_options_));
comparator = BytewiseComparator();
kTestComparator = BytewiseComparator();
}
DB* GetDB() { return db_; }
@@ -286,7 +286,7 @@ class ComparatorDBTest
} else {
comparator_guard.reset();
}
comparator = cmp;
kTestComparator = cmp;
last_options_.comparator = cmp;
}
@@ -334,7 +334,7 @@ TEST_P(ComparatorDBTest, SimpleSuffixReverseComparator) {
for (int rnd_seed = 301; rnd_seed < 316; rnd_seed++) {
Options* opt = GetOptions();
opt->comparator = comparator;
opt->comparator = kTestComparator;
DestroyAndReopen();
Random rnd(rnd_seed);
@@ -360,7 +360,7 @@ TEST_P(ComparatorDBTest, Uint64Comparator) {
for (int rnd_seed = 301; rnd_seed < 316; rnd_seed++) {
Options* opt = GetOptions();
opt->comparator = comparator;
opt->comparator = kTestComparator;
DestroyAndReopen();
Random rnd(rnd_seed);
Random64 rnd64(rnd_seed);
@@ -384,7 +384,7 @@ TEST_P(ComparatorDBTest, DoubleComparator) {
for (int rnd_seed = 301; rnd_seed < 316; rnd_seed++) {
Options* opt = GetOptions();
opt->comparator = comparator;
opt->comparator = kTestComparator;
DestroyAndReopen();
Random rnd(rnd_seed);
@@ -409,7 +409,7 @@ TEST_P(ComparatorDBTest, HashComparator) {
for (int rnd_seed = 301; rnd_seed < 316; rnd_seed++) {
Options* opt = GetOptions();
opt->comparator = comparator;
opt->comparator = kTestComparator;
DestroyAndReopen();
Random rnd(rnd_seed);
@@ -428,7 +428,7 @@ TEST_P(ComparatorDBTest, TwoStrComparator) {
for (int rnd_seed = 301; rnd_seed < 316; rnd_seed++) {
Options* opt = GetOptions();
opt->comparator = comparator;
opt->comparator = kTestComparator;
DestroyAndReopen();
Random rnd(rnd_seed);
@@ -652,7 +652,7 @@ TEST_P(ComparatorDBTest, SeparatorSuccessorRandomizeTest) {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+2 -2
View File
@@ -11,7 +11,7 @@
#include "db/db_impl/db_impl.h"
#include "util/cast_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
void CancelAllBackgroundWork(DB* db, bool wait) {
(static_cast_with_check<DBImpl, DB>(db->GetRootDB()))
@@ -72,6 +72,6 @@ Status VerifySstFileChecksum(const Options& options,
return s;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+13 -14
View File
@@ -33,7 +33,7 @@
#include "test_util/testutil.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
static const int kValueSize = 1000;
@@ -98,7 +98,7 @@ class CorruptionTest : public testing::Test {
void RepairDB() {
delete db_;
db_ = nullptr;
ASSERT_OK(::rocksdb::RepairDB(dbname_, options_));
ASSERT_OK(::ROCKSDB_NAMESPACE::RepairDB(dbname_, options_));
}
void Build(int n, int flush_every = 0) {
@@ -393,12 +393,16 @@ TEST_F(CorruptionTest, TableFileIndexData) {
// corrupt an index block of an entire file
Corrupt(kTableFile, -2000, 500);
Reopen();
options.paranoid_checks = false;
Reopen(&options);
dbi = reinterpret_cast<DBImpl*>(db_);
// one full file may be readable, since only one was corrupted
// the other file should be fully non-readable, since index was corrupted
Check(0, 5000);
ASSERT_NOK(dbi->VerifyChecksum());
// In paranoid mode, the db cannot be opened due to the corrupted file.
ASSERT_TRUE(TryReopen().IsCorruption());
}
TEST_F(CorruptionTest, MissingDescriptor) {
@@ -554,13 +558,7 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
ASSERT_OK(TryReopen());
CorruptFile(filename, static_cast<int>(range_del_handle.offset()), 1);
// The test case does not fail on TryReopen because failure to preload table
// handlers is not considered critical.
ASSERT_OK(TryReopen());
std::string val;
// However, it does fail on any read involving that file since that file
// cannot be opened with a corrupt range deletion meta-block.
ASSERT_TRUE(db_->Get(ReadOptions(), "a", &val).IsCorruption());
ASSERT_TRUE(TryReopen().IsCorruption());
}
TEST_F(CorruptionTest, FileSystemStateCorrupted) {
@@ -585,18 +583,19 @@ TEST_F(CorruptionTest, FileSystemStateCorrupted) {
env_.NewWritableFile(filename, &file, EnvOptions());
file->Append(Slice("corrupted sst"));
file.reset();
Status x = TryReopen(&options);
ASSERT_TRUE(x.IsCorruption());
} else { // delete the file
env_.DeleteFile(filename);
Status x = TryReopen(&options);
ASSERT_TRUE(x.IsPathNotFound());
}
Status x = TryReopen(&options);
ASSERT_TRUE(x.IsCorruption());
DestroyDB(dbname_, options_);
Reopen(&options);
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+16 -9
View File
@@ -15,7 +15,7 @@
#include "test_util/testutil.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class CuckooTableDBTest : public testing::Test {
private:
@@ -298,17 +298,25 @@ TEST_F(CuckooTableDBTest, AdaptiveTable) {
dbfull()->TEST_FlushMemTable();
// Write some keys using plain table.
std::shared_ptr<TableFactory> block_based_factory(
NewBlockBasedTableFactory());
std::shared_ptr<TableFactory> plain_table_factory(
NewPlainTableFactory());
std::shared_ptr<TableFactory> cuckoo_table_factory(
NewCuckooTableFactory());
options.create_if_missing = false;
options.table_factory.reset(NewPlainTableFactory());
options.table_factory.reset(NewAdaptiveTableFactory(
plain_table_factory, block_based_factory, plain_table_factory,
cuckoo_table_factory));
Reopen(&options);
ASSERT_OK(Put("key4", "v4"));
ASSERT_OK(Put("key1", "v5"));
dbfull()->TEST_FlushMemTable();
// Write some keys using block based table.
std::shared_ptr<TableFactory> block_based_factory(
NewBlockBasedTableFactory());
options.table_factory.reset(NewAdaptiveTableFactory(block_based_factory));
options.table_factory.reset(NewAdaptiveTableFactory(
block_based_factory, block_based_factory, plain_table_factory,
cuckoo_table_factory));
Reopen(&options);
ASSERT_OK(Put("key5", "v6"));
ASSERT_OK(Put("key2", "v7"));
@@ -320,14 +328,13 @@ TEST_F(CuckooTableDBTest, AdaptiveTable) {
ASSERT_EQ("v4", Get("key4"));
ASSERT_EQ("v6", Get("key5"));
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
if (rocksdb::port::kLittleEndian) {
if (ROCKSDB_NAMESPACE::port::kLittleEndian) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
else {
} else {
fprintf(stderr, "SKIPPED as Cuckoo table doesn't support Big Endian\n");
return 0;
}
+362 -39
View File
@@ -17,7 +17,7 @@
#include "test_util/sync_point.h"
#endif
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBBasicTest : public DBTestBase {
public:
@@ -27,8 +27,8 @@ class DBBasicTest : public DBTestBase {
TEST_F(DBBasicTest, OpenWhenOpen) {
Options options = CurrentOptions();
options.env = env_;
rocksdb::DB* db2 = nullptr;
rocksdb::Status s = DB::Open(options, dbname_, &db2);
ROCKSDB_NAMESPACE::DB* db2 = nullptr;
ROCKSDB_NAMESPACE::Status s = DB::Open(options, dbname_, &db2);
ASSERT_EQ(Status::Code::kIOError, s.code());
ASSERT_EQ(Status::SubCode::kNone, s.subcode());
@@ -1048,8 +1048,8 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCF) {
}
int get_sv_count = 0;
rocksdb::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::AfterRefSV", [&](void* /*arg*/) {
if (++get_sv_count == 2) {
// After MultiGet refs a couple of CFs, flush all CFs so MultiGet
@@ -1073,7 +1073,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCF) {
}
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<int> cfs;
std::vector<std::string> keys;
@@ -1139,12 +1139,12 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFMutex) {
int get_sv_count = 0;
int retries = 0;
bool last_try = false;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::LastTry", [&](void* /*arg*/) {
last_try = true;
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::AfterRefSV", [&](void* /*arg*/) {
if (last_try) {
return;
@@ -1160,7 +1160,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFMutex) {
}
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<int> cfs;
std::vector<std::string> keys;
@@ -1198,8 +1198,8 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFSnapshot) {
}
int get_sv_count = 0;
rocksdb::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::DBImpl* db = reinterpret_cast<DBImpl*>(db_);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MultiGet::AfterRefSV", [&](void* /*arg*/) {
if (++get_sv_count == 2) {
for (int i = 0; i < 8; ++i) {
@@ -1219,7 +1219,7 @@ TEST_P(DBMultiGetTestWithParam, MultiGetMultiCFSnapshot) {
}
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<int> cfs;
std::vector<std::string> keys;
@@ -1405,6 +1405,91 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevel) {
}
}
TEST_F(DBBasicTest, MultiGetBatchedMultiLevelMerge) {
Options options = CurrentOptions();
options.disable_auto_compactions = true;
options.merge_operator = MergeOperators::CreateStringAppendOperator();
BlockBasedTableOptions bbto;
bbto.filter_policy.reset(NewBloomFilterPolicy(10, false));
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
Reopen(options);
int num_keys = 0;
for (int i = 0; i < 128; ++i) {
ASSERT_OK(Put("key_" + std::to_string(i), "val_l2_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
num_keys = 0;
}
MoveFilesToLevel(2);
for (int i = 0; i < 128; i += 3) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l1_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
num_keys = 0;
}
MoveFilesToLevel(1);
for (int i = 0; i < 128; i += 5) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_l0_" + std::to_string(i)));
num_keys++;
if (num_keys == 8) {
Flush();
num_keys = 0;
}
}
if (num_keys > 0) {
Flush();
num_keys = 0;
}
ASSERT_EQ(0, num_keys);
for (int i = 0; i < 128; i += 9) {
ASSERT_OK(Merge("key_" + std::to_string(i), "val_mem_" + std::to_string(i)));
}
std::vector<std::string> keys;
std::vector<std::string> values;
for (int i = 32; i < 80; ++i) {
keys.push_back("key_" + std::to_string(i));
}
values = MultiGet(keys, nullptr);
ASSERT_EQ(values.size(), keys.size());
for (unsigned int j = 0; j < 48; ++j) {
int key = j + 32;
std::string value;
value.append("val_l2_" + std::to_string(key));
if (key % 3 == 0) {
value.append(",");
value.append("val_l1_" + std::to_string(key));
}
if (key % 5 == 0) {
value.append(",");
value.append("val_l0_" + std::to_string(key));
}
if (key % 9 == 0) {
value.append(",");
value.append("val_mem_" + std::to_string(key));
}
ASSERT_EQ(values[j], value);
}
}
// Test class for batched MultiGet with prefix extractor
// Param bool - If true, use partitioned filters
// If false, use full filter block
@@ -1473,7 +1558,7 @@ TEST_P(DBMultiGetRowCacheTest, MultiGetBatched) {
do {
option_config_ = kRowCache;
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
SetPerfLevel(kEnableCount);
ASSERT_OK(Put(1, "k1", "v1"));
@@ -1579,13 +1664,13 @@ TEST_F(DBBasicTest, GetAllKeyVersions) {
ASSERT_OK(Delete(std::to_string(i)));
}
std::vector<KeyVersion> key_versions;
ASSERT_OK(rocksdb::GetAllKeyVersions(db_, Slice(), Slice(),
std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_OK(ROCKSDB_NAMESPACE::GetAllKeyVersions(
db_, Slice(), Slice(), std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates, key_versions.size());
ASSERT_OK(rocksdb::GetAllKeyVersions(db_, handles_[0], Slice(), Slice(),
std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_OK(ROCKSDB_NAMESPACE::GetAllKeyVersions(
db_, handles_[0], Slice(), Slice(), std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates, key_versions.size());
// Check non-default column family
@@ -1598,9 +1683,9 @@ TEST_F(DBBasicTest, GetAllKeyVersions) {
for (size_t i = 0; i != kNumDeletes - 1; ++i) {
ASSERT_OK(Delete(1, std::to_string(i)));
}
ASSERT_OK(rocksdb::GetAllKeyVersions(db_, handles_[1], Slice(), Slice(),
std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_OK(ROCKSDB_NAMESPACE::GetAllKeyVersions(
db_, handles_[1], Slice(), Slice(), std::numeric_limits<size_t>::max(),
&key_versions));
ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates - 3, key_versions.size());
}
#endif // !ROCKSDB_LITE
@@ -2011,6 +2096,90 @@ TEST_P(DBBasicTestWithParallelIO, MultiGet) {
}
}
TEST_P(DBBasicTestWithParallelIO, MultiGetWithChecksumMismatch) {
std::vector<std::string> key_data(10);
std::vector<Slice> keys;
// We cannot resize a PinnableSlice vector, so just set initial size to
// largest we think we will need
std::vector<PinnableSlice> values(10);
std::vector<Status> statuses;
int read_count = 0;
ReadOptions ro;
ro.fill_cache = fill_cache();
SyncPoint::GetInstance()->SetCallBack(
"RetrieveMultipleBlocks:VerifyChecksum", [&](void *status) {
Status* s = static_cast<Status*>(status);
read_count++;
if (read_count == 2) {
*s = Status::Corruption();
}
});
SyncPoint::GetInstance()->EnableProcessing();
// Warm up the cache first
key_data.emplace_back(Key(0));
keys.emplace_back(Slice(key_data.back()));
key_data.emplace_back(Key(50));
keys.emplace_back(Slice(key_data.back()));
statuses.resize(keys.size());
dbfull()->MultiGet(ro, dbfull()->DefaultColumnFamily(), keys.size(),
keys.data(), values.data(), statuses.data(), true);
ASSERT_TRUE(CheckValue(0, values[0].ToString()));
//ASSERT_TRUE(CheckValue(50, values[1].ToString()));
ASSERT_EQ(statuses[0], Status::OK());
ASSERT_EQ(statuses[1], Status::Corruption());
SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBBasicTestWithParallelIO, MultiGetWithMissingFile) {
std::vector<std::string> key_data(10);
std::vector<Slice> keys;
// We cannot resize a PinnableSlice vector, so just set initial size to
// largest we think we will need
std::vector<PinnableSlice> values(10);
std::vector<Status> statuses;
ReadOptions ro;
ro.fill_cache = fill_cache();
SyncPoint::GetInstance()->SetCallBack(
"TableCache::MultiGet:FindTable", [&](void *status) {
Status* s = static_cast<Status*>(status);
*s = Status::IOError();
});
// DB open will create table readers unless we reduce the table cache
// capacity.
// SanitizeOptions will set max_open_files to minimum of 20. Table cache
// is allocated with max_open_files - 10 as capacity. So override
// max_open_files to 11 so table cache capacity will become 1. This will
// prevent file open during DB open and force the file to be opened
// during MultiGet
SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void *arg) {
int* max_open_files = (int*)arg;
*max_open_files = 11;
});
SyncPoint::GetInstance()->EnableProcessing();
Reopen(CurrentOptions());
// Warm up the cache first
key_data.emplace_back(Key(0));
keys.emplace_back(Slice(key_data.back()));
key_data.emplace_back(Key(50));
keys.emplace_back(Slice(key_data.back()));
statuses.resize(keys.size());
dbfull()->MultiGet(ro, dbfull()->DefaultColumnFamily(), keys.size(),
keys.data(), values.data(), statuses.data(), true);
ASSERT_EQ(statuses[0], Status::IOError());
ASSERT_EQ(statuses[1], Status::IOError());
SyncPoint::GetInstance()->DisableProcessing();
}
INSTANTIATE_TEST_CASE_P(
ParallelIO, DBBasicTestWithParallelIO,
// Params are as follows -
@@ -2021,23 +2190,15 @@ INSTANTIATE_TEST_CASE_P(
::testing::Combine(::testing::Bool(), ::testing::Bool(),
::testing::Bool(), ::testing::Bool()));
class DBBasicTestWithTimestampWithParam
: public DBTestBase,
public testing::WithParamInterface<bool> {
class DBBasicTestWithTimestampBase : public DBTestBase {
public:
DBBasicTestWithTimestampWithParam()
: DBTestBase("/db_basic_test_with_timestamp") {}
explicit DBBasicTestWithTimestampBase(const std::string& dbname)
: DBTestBase(dbname) {}
protected:
class TestComparator : public Comparator {
private:
const Comparator* cmp_without_ts_;
class TestComparatorBase : public Comparator {
public:
explicit TestComparator(size_t ts_sz)
: Comparator(ts_sz), cmp_without_ts_(nullptr) {
cmp_without_ts_ = BytewiseComparator();
}
explicit TestComparatorBase(size_t ts_sz) : Comparator(ts_sz) {}
const char* Name() const override { return "TestComparator"; }
@@ -2055,13 +2216,15 @@ class DBBasicTestWithTimestampWithParam
Slice(b.data() + b.size() - timestamp_size(), timestamp_size()));
}
virtual int CompareImpl(const Slice& a, const Slice& b) const = 0;
int CompareWithoutTimestamp(const Slice& a, const Slice& b) const override {
assert(a.size() >= timestamp_size());
assert(b.size() >= timestamp_size());
Slice k1 = StripTimestampFromUserKey(a, timestamp_size());
Slice k2 = StripTimestampFromUserKey(b, timestamp_size());
return cmp_without_ts_->Compare(k1, k2);
return CompareImpl(k1, k2);
}
int CompareTimestamp(const Slice& ts1, const Slice& ts2) const override {
@@ -2107,6 +2270,166 @@ class DBBasicTestWithTimestampWithParam
}
};
class DBBasicTestWithTimestamp : public DBBasicTestWithTimestampBase {
public:
DBBasicTestWithTimestamp()
: DBBasicTestWithTimestampBase("/db_basic_test_with_timestamp") {}
protected:
class TestComparator : public TestComparatorBase {
public:
const int kKeyPrefixLength =
3; // 3: length of "key" in generated keys ("key" + std::to_string(j))
explicit TestComparator(size_t ts_sz) : TestComparatorBase(ts_sz) {}
int CompareImpl(const Slice& a, const Slice& b) const override {
int n1 = atoi(
std::string(a.data() + kKeyPrefixLength, a.size() - kKeyPrefixLength)
.c_str());
int n2 = atoi(
std::string(b.data() + kKeyPrefixLength, b.size() - kKeyPrefixLength)
.c_str());
return (n1 < n2) ? -1 : (n1 > n2) ? 1 : 0;
}
};
};
#ifndef ROCKSDB_LITE
// A class which remembers the name of each flushed file.
class FlushedFileCollector : public EventListener {
public:
FlushedFileCollector() {}
~FlushedFileCollector() override {}
void OnFlushCompleted(DB* /*db*/, const FlushJobInfo& info) override {
InstrumentedMutexLock lock(&mutex_);
flushed_files_.push_back(info.file_path);
}
std::vector<std::string> GetFlushedFiles() {
std::vector<std::string> result;
{
InstrumentedMutexLock lock(&mutex_);
result = flushed_files_;
}
return result;
}
void ClearFlushedFiles() {
InstrumentedMutexLock lock(&mutex_);
flushed_files_.clear();
}
private:
std::vector<std::string> flushed_files_;
InstrumentedMutex mutex_;
};
TEST_F(DBBasicTestWithTimestamp, PutAndGetWithCompaction) {
const int kNumKeysPerFile = 8192;
const size_t kNumTimestamps = 2;
const size_t kNumKeysPerTimestamp = (kNumKeysPerFile - 1) / kNumTimestamps;
const size_t kSplitPosBase = kNumKeysPerTimestamp / 2;
Options options = CurrentOptions();
options.create_if_missing = true;
options.env = env_;
options.memtable_factory.reset(new SpecialSkipListFactory(kNumKeysPerFile));
FlushedFileCollector* collector = new FlushedFileCollector();
options.listeners.emplace_back(collector);
std::string tmp;
size_t ts_sz = EncodeTimestamp(0, 0, &tmp).size();
TestComparator test_cmp(ts_sz);
options.comparator = &test_cmp;
BlockBasedTableOptions bbto;
bbto.filter_policy.reset(NewBloomFilterPolicy(
10 /*bits_per_key*/, false /*use_block_based_builder*/));
bbto.whole_key_filtering = true;
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
size_t num_cfs = handles_.size();
ASSERT_EQ(2, num_cfs);
std::vector<std::string> write_ts_strs(kNumTimestamps);
std::vector<std::string> read_ts_strs(kNumTimestamps);
std::vector<Slice> write_ts_list;
std::vector<Slice> read_ts_list;
for (size_t i = 0; i != kNumTimestamps; ++i) {
write_ts_list.emplace_back(EncodeTimestamp(i * 2, 0, &write_ts_strs[i]));
read_ts_list.emplace_back(EncodeTimestamp(1 + i * 2, 0, &read_ts_strs[i]));
const Slice& write_ts = write_ts_list.back();
WriteOptions wopts;
wopts.timestamp = &write_ts;
for (int cf = 0; cf != static_cast<int>(num_cfs); ++cf) {
for (size_t j = 0; j != kNumKeysPerTimestamp; ++j) {
ASSERT_OK(Put(cf, "key" + std::to_string(j),
"value_" + std::to_string(j) + "_" + std::to_string(i),
wopts));
if (j == kSplitPosBase + i || j == kNumKeysPerTimestamp - 1) {
// flush all keys with the same timestamp to two sst files, split at
// incremental positions such that lowerlevel[1].smallest.userkey ==
// higherlevel[0].largest.userkey
ASSERT_OK(Flush(cf));
// compact files (2 at each level) to a lower level such that all keys
// with the same timestamp is at one level, with newer versions at
// higher levels.
CompactionOptions compact_opt;
compact_opt.compression = kNoCompression;
db_->CompactFiles(compact_opt, handles_[cf],
collector->GetFlushedFiles(),
static_cast<int>(kNumTimestamps - i));
collector->ClearFlushedFiles();
}
}
}
}
const auto& verify_db_func = [&]() {
for (size_t i = 0; i != kNumTimestamps; ++i) {
ReadOptions ropts;
ropts.timestamp = &read_ts_list[i];
for (int cf = 0; cf != static_cast<int>(num_cfs); ++cf) {
ColumnFamilyHandle* cfh = handles_[cf];
for (size_t j = 0; j != kNumKeysPerTimestamp; ++j) {
std::string value;
ASSERT_OK(db_->Get(ropts, cfh, "key" + std::to_string(j), &value));
ASSERT_EQ("value_" + std::to_string(j) + "_" + std::to_string(i),
value);
}
}
}
};
verify_db_func();
}
#endif // !ROCKSDB_LITE
class DBBasicTestWithTimestampWithParam
: public DBBasicTestWithTimestampBase,
public testing::WithParamInterface<bool> {
public:
DBBasicTestWithTimestampWithParam()
: DBBasicTestWithTimestampBase(
"/db_basic_test_with_timestamp_with_param") {}
protected:
class TestComparator : public TestComparatorBase {
private:
const Comparator* cmp_without_ts_;
public:
explicit TestComparator(size_t ts_sz)
: TestComparatorBase(ts_sz), cmp_without_ts_(nullptr) {
cmp_without_ts_ = BytewiseComparator();
}
int CompareImpl(const Slice& a, const Slice& b) const override {
return cmp_without_ts_->Compare(a, b);
}
};
};
TEST_P(DBBasicTestWithTimestampWithParam, PutAndGet) {
const int kNumKeysPerFile = 8192;
const size_t kNumTimestamps = 6;
@@ -2204,7 +2527,7 @@ TEST_P(DBBasicTestWithTimestampWithParam, PutAndGet) {
INSTANTIATE_TEST_CASE_P(Timestamp, DBBasicTestWithTimestampWithParam,
::testing::Bool());
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
extern "C" {
@@ -2215,7 +2538,7 @@ void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
#endif // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
RegisterCustomObjects(argc, argv);
return RUN_ALL_TESTS();
+3 -3
View File
@@ -23,7 +23,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// kTypeBlobIndex is a value type used by BlobDB only. The base rocksdb
// should accept the value type on write, and report not supported value
@@ -427,10 +427,10 @@ TEST_F(DBBlobIndexTest, Iterate) {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+10 -10
View File
@@ -12,7 +12,7 @@
#include "port/stack_trace.h"
#include "util/compression.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBBlockCacheTest : public DBTestBase {
private:
@@ -46,7 +46,7 @@ class DBBlockCacheTest : public DBTestBase {
options.create_if_missing = true;
options.avoid_flush_during_recovery = false;
// options.compression = kNoCompression;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.table_factory.reset(new BlockBasedTableFactory(table_options));
return options;
}
@@ -292,7 +292,7 @@ TEST_F(DBBlockCacheTest, TestWithCompressedBlockCache) {
TEST_F(DBBlockCacheTest, IndexAndFilterBlocksOfNewTableAddedToCache) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
@@ -378,7 +378,7 @@ TEST_F(DBBlockCacheTest, FillCacheAndIterateDB) {
TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
LRUCacheOptions co;
@@ -464,7 +464,7 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksCachePriority) {
for (auto priority : {Cache::Priority::LOW, Cache::Priority::HIGH}) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache.reset(new MockCache());
@@ -520,7 +520,7 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksCachePriority) {
TEST_F(DBBlockCacheTest, ParanoidFileChecks) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.level0_file_num_compaction_trigger = 2;
options.paranoid_file_checks = true;
BlockBasedTableOptions table_options;
@@ -576,7 +576,7 @@ TEST_F(DBBlockCacheTest, CompressedCache) {
for (int iter = 0; iter < 4; iter++) {
Options options = CurrentOptions();
options.write_buffer_size = 64 * 1024; // small write buffer
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
switch (iter) {
@@ -705,7 +705,7 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
options.compression_opts.max_dict_bytes = 4096;
options.create_if_missing = true;
options.num_levels = 2;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.target_file_size_base = kNumEntriesPerFile * kNumBytesPerEntry;
BlockBasedTableOptions table_options;
table_options.cache_index_and_filter_blocks = true;
@@ -752,10 +752,10 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+39 -31
View File
@@ -12,16 +12,10 @@
#include "rocksdb/perf_context.h"
#include "table/block_based/filter_policy_internal.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
namespace {
using BFP = BloomFilterPolicy;
namespace BFP2 {
// Extends BFP::Mode with option to use Plain table
using PseudoMode = int;
static constexpr PseudoMode kPlainTable = -1;
} // namespace BFP2
} // namespace
// DB tests related to bloom filter.
@@ -94,7 +88,7 @@ TEST_P(DBBloomFilterTestDefFormatVersion, KeyMayExist) {
// indexes
continue;
}
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_TRUE(!db_->KeyMayExist(ropts, handles_[1], "a", &value));
@@ -156,7 +150,7 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloomCustomPrefixExtractor) {
Options options = last_options_;
options.prefix_extractor =
std::make_shared<SliceTransformLimitedDomainGeneric>();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
get_perf_context()->EnablePerLevelPerfContext();
BlockBasedTableOptions bbto;
bbto.filter_policy.reset(NewBloomFilterPolicy(10, false));
@@ -222,7 +216,7 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloom) {
for (bool partition_filters : {true, false}) {
Options options = last_options_;
options.prefix_extractor.reset(NewFixedPrefixTransform(8));
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
get_perf_context()->EnablePerLevelPerfContext();
BlockBasedTableOptions bbto;
bbto.filter_policy.reset(NewBloomFilterPolicy(10, false));
@@ -273,7 +267,7 @@ TEST_F(DBBloomFilterTest, WholeKeyFilterProp) {
for (bool partition_filters : {true, false}) {
Options options = last_options_;
options.prefix_extractor.reset(NewFixedPrefixTransform(3));
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
get_perf_context()->EnablePerLevelPerfContext();
BlockBasedTableOptions bbto;
@@ -537,7 +531,7 @@ INSTANTIATE_TEST_CASE_P(
TEST_F(DBBloomFilterTest, BloomFilterRate) {
while (ChangeFilterOptions()) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
get_perf_context()->EnablePerLevelPerfContext();
CreateAndReopenWithCF({"pikachu"}, options);
@@ -569,7 +563,7 @@ TEST_F(DBBloomFilterTest, BloomFilterRate) {
TEST_F(DBBloomFilterTest, BloomFilterCompatibility) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.filter_policy.reset(NewBloomFilterPolicy(10, true));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@@ -613,7 +607,7 @@ TEST_F(DBBloomFilterTest, BloomFilterCompatibility) {
TEST_F(DBBloomFilterTest, BloomFilterReverseCompatibility) {
for (bool partition_filters : {true, false}) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
if (partition_filters) {
table_options.partition_filters = true;
@@ -660,17 +654,18 @@ class TestingWrappedBlockBasedFilterPolicy : public FilterPolicy {
return "TestingWrappedBlockBasedFilterPolicy";
}
void CreateFilter(const rocksdb::Slice* keys, int n,
void CreateFilter(const ROCKSDB_NAMESPACE::Slice* keys, int n,
std::string* dst) const override {
std::unique_ptr<rocksdb::Slice[]> user_keys(new rocksdb::Slice[n]);
std::unique_ptr<ROCKSDB_NAMESPACE::Slice[]> user_keys(
new ROCKSDB_NAMESPACE::Slice[n]);
for (int i = 0; i < n; ++i) {
user_keys[i] = convertKey(keys[i]);
}
return filter_->CreateFilter(user_keys.get(), n, dst);
}
bool KeyMayMatch(const rocksdb::Slice& key,
const rocksdb::Slice& filter) const override {
bool KeyMayMatch(const ROCKSDB_NAMESPACE::Slice& key,
const ROCKSDB_NAMESPACE::Slice& filter) const override {
counter_++;
return filter_->KeyMayMatch(convertKey(key), filter);
}
@@ -681,13 +676,16 @@ class TestingWrappedBlockBasedFilterPolicy : public FilterPolicy {
const FilterPolicy* filter_;
mutable uint32_t counter_;
rocksdb::Slice convertKey(const rocksdb::Slice& key) const { return key; }
ROCKSDB_NAMESPACE::Slice convertKey(
const ROCKSDB_NAMESPACE::Slice& key) const {
return key;
}
};
} // namespace
TEST_F(DBBloomFilterTest, WrappedBlockBasedFilterPolicy) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
TestingWrappedBlockBasedFilterPolicy* policy =
@@ -803,7 +801,7 @@ class TestingContextCustomFilterPolicy
TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
for (bool fifo : {true, false}) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.compaction_style =
fifo ? kCompactionStyleFIFO : kCompactionStyleLevel;
@@ -902,7 +900,7 @@ class SliceTransformLimitedDomain : public SliceTransform {
TEST_F(DBBloomFilterTest, PrefixExtractorFullFilter) {
BlockBasedTableOptions bbto;
// Full Filter Block
bbto.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));
bbto.filter_policy.reset(ROCKSDB_NAMESPACE::NewBloomFilterPolicy(10, false));
bbto.whole_key_filtering = false;
Options options = CurrentOptions();
@@ -931,7 +929,7 @@ TEST_F(DBBloomFilterTest, PrefixExtractorFullFilter) {
TEST_F(DBBloomFilterTest, PrefixExtractorBlockFilter) {
BlockBasedTableOptions bbto;
// Block Filter Block
bbto.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, true));
bbto.filter_policy.reset(ROCKSDB_NAMESPACE::NewBloomFilterPolicy(10, true));
Options options = CurrentOptions();
options.prefix_extractor = std::make_shared<SliceTransformLimitedDomain>();
@@ -970,7 +968,8 @@ TEST_F(DBBloomFilterTest, MemtableWholeKeyBloomFilter) {
Options options = CurrentOptions();
options.memtable_prefix_bloom_size_ratio =
static_cast<double>(kMemtablePrefixFilterSize) / kMemtableSize;
options.prefix_extractor.reset(rocksdb::NewFixedPrefixTransform(kPrefixLen));
options.prefix_extractor.reset(
ROCKSDB_NAMESPACE::NewFixedPrefixTransform(kPrefixLen));
options.write_buffer_size = kMemtableSize;
options.memtable_whole_key_filtering = false;
Reopen(options);
@@ -1031,6 +1030,14 @@ TEST_F(DBBloomFilterTest, MemtablePrefixBloomOutOfDomain) {
}
#ifndef ROCKSDB_LITE
namespace {
namespace BFP2 {
// Extends BFP::Mode with option to use Plain table
using PseudoMode = int;
static constexpr PseudoMode kPlainTable = -1;
} // namespace BFP2
} // namespace
class BloomStatsTestWithParam
: public DBBloomFilterTest,
public testing::WithParamInterface<std::tuple<BFP2::PseudoMode, bool>> {
@@ -1040,7 +1047,8 @@ class BloomStatsTestWithParam
partition_filters_ = std::get<1>(GetParam());
options_.create_if_missing = true;
options_.prefix_extractor.reset(rocksdb::NewFixedPrefixTransform(4));
options_.prefix_extractor.reset(
ROCKSDB_NAMESPACE::NewFixedPrefixTransform(4));
options_.memtable_prefix_bloom_size_ratio =
8.0 * 1024.0 / static_cast<double>(options_.write_buffer_size);
if (bfp_impl_ == BFP2::kPlainTable) {
@@ -1321,7 +1329,7 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
bbto.whole_key_filtering = true;
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
options.optimize_filters_for_hits = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
get_perf_context()->Reset();
get_perf_context()->EnablePerLevelPerfContext();
CreateAndReopenWithCF({"mypikachu"}, options);
@@ -1447,13 +1455,13 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
CompactRangeOptions compact_options;
compact_options.bottommost_level_compaction =
@@ -1893,10 +1901,10 @@ TEST_F(DBBloomFilterTest, DynamicBloomFilterOptions) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+5 -6
View File
@@ -10,7 +10,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
static int cfilter_count = 0;
static int cfilter_skips = 0;
@@ -47,8 +47,7 @@ class DBTestCompactionFilterWithCompactParam
#ifndef ROCKSDB_VALGRIND_RUN
INSTANTIATE_TEST_CASE_P(
DBTestCompactionFilterWithCompactOption,
DBTestCompactionFilterWithCompactParam,
CompactionFilterWithOption, DBTestCompactionFilterWithCompactParam,
::testing::Values(DBTestBase::OptionConfig::kDefault,
DBTestBase::OptionConfig::kUniversalCompaction,
DBTestBase::OptionConfig::kUniversalCompactionMultiLevel,
@@ -56,7 +55,7 @@ INSTANTIATE_TEST_CASE_P(
DBTestBase::OptionConfig::kUniversalSubcompactions));
#else
// Run fewer cases in valgrind
INSTANTIATE_TEST_CASE_P(DBTestCompactionFilterWithCompactOption,
INSTANTIATE_TEST_CASE_P(CompactionFilterWithOption,
DBTestCompactionFilterWithCompactParam,
::testing::Values(DBTestBase::OptionConfig::kDefault));
#endif // ROCKSDB_VALGRIND_RUN
@@ -864,10 +863,10 @@ TEST_F(DBTestCompactionFilter, IgnoreSnapshotsFalse) {
delete options.compaction_filter;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+181 -162
View File
@@ -18,7 +18,7 @@
#include "test_util/sync_point.h"
#include "util/concurrent_task_limiter_impl.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// SYNC_POINT is not supported in released Windows mode.
#if !defined(ROCKSDB_LITE)
@@ -400,6 +400,7 @@ TEST_F(DBCompactionTest, SkipStatsUpdateTest) {
// The test will need to be updated if the internal behavior changes.
Options options = DeletionTriggerOptions(CurrentOptions());
options.disable_auto_compactions = true;
options.env = env_;
DestroyAndReopen(options);
Random rnd(301);
@@ -410,31 +411,33 @@ TEST_F(DBCompactionTest, SkipStatsUpdateTest) {
values.push_back(RandomString(&rnd, kCDTValueSize));
ASSERT_OK(Put(Key(k), values[k]));
}
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
ASSERT_OK(Flush());
Close();
int update_acc_stats_called = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionStorageInfo::UpdateAccumulatedStats",
[&](void* /* arg */) { ++update_acc_stats_called; });
SyncPoint::GetInstance()->EnableProcessing();
// Reopen the DB with stats-update disabled
options.skip_stats_update_on_db_open = true;
options.max_open_files = 20;
env_->random_file_open_counter_.store(0);
Reopen(options);
// As stats-update is disabled, we expect a very low number of
// random file open.
// Note that this number must be changed accordingly if we change
// the number of files needed to be opened in the DB::Open process.
const int kMaxFileOpenCount = 10;
ASSERT_LT(env_->random_file_open_counter_.load(), kMaxFileOpenCount);
ASSERT_EQ(update_acc_stats_called, 0);
// Repeat the reopen process, but this time we enable
// stats-update.
options.skip_stats_update_on_db_open = false;
env_->random_file_open_counter_.store(0);
Reopen(options);
// Since we do a normal stats update on db-open, there
// will be more random open files.
ASSERT_GT(env_->random_file_open_counter_.load(), kMaxFileOpenCount);
ASSERT_GT(update_acc_stats_called, 0);
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, TestTableReaderForCompaction) {
@@ -448,7 +451,7 @@ TEST_F(DBCompactionTest, TestTableReaderForCompaction) {
int num_table_cache_lookup = 0;
int num_new_table_reader = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"TableCache::FindTable:0", [&](void* arg) {
assert(arg != nullptr);
bool no_io = *(reinterpret_cast<bool*>(arg));
@@ -457,10 +460,10 @@ TEST_F(DBCompactionTest, TestTableReaderForCompaction) {
num_table_cache_lookup++;
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"TableCache::GetTableReader:0",
[&](void* /*arg*/) { num_new_table_reader++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int k = 0; k < options.level0_file_num_compaction_trigger; ++k) {
ASSERT_OK(Put(Key(k), Key(k)));
@@ -530,7 +533,7 @@ TEST_F(DBCompactionTest, TestTableReaderForCompaction) {
ASSERT_EQ(num_table_cache_lookup + old_num_table_cache_lookup2, 3);
ASSERT_EQ(num_new_table_reader, 0);
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(DBCompactionTestWithParam, CompactionDeletionTriggerReopen) {
@@ -1020,10 +1023,10 @@ TEST_F(DBCompactionTest, RecoverDuringMemtableCompaction) {
TEST_P(DBCompactionTestWithParam, TrivialMoveOneFile) {
int32_t trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.write_buffer_size = 100000000;
@@ -1071,19 +1074,19 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveOneFile) {
}
ASSERT_EQ(trivial_move, 1);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBCompactionTestWithParam, TrivialMoveNonOverlappingFiles) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.disable_auto_compactions = true;
@@ -1170,19 +1173,19 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveNonOverlappingFiles) {
ASSERT_EQ(trivial_move, 0);
ASSERT_EQ(non_trivial_move, 1);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBCompactionTestWithParam, TrivialMoveTargetLevel) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.disable_auto_compactions = true;
@@ -1235,10 +1238,10 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveTargetLevel) {
TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
bool first = true;
@@ -1246,11 +1249,11 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
// 4 -> 1: ensure the order of two non-trivial compactions
// 5 -> 2 and 5 -> 3: ensure we do a check before two non-trivial compactions
// are installed
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBCompaction::ManualPartial:4", "DBCompaction::ManualPartial:1"},
{"DBCompaction::ManualPartial:5", "DBCompaction::ManualPartial:2"},
{"DBCompaction::ManualPartial:5", "DBCompaction::ManualPartial:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (first) {
first = false;
@@ -1261,7 +1264,7 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.write_buffer_size = 10 * 1024 * 1024;
@@ -1326,7 +1329,7 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
ASSERT_EQ(trivial_move, 6);
ASSERT_EQ(non_trivial_move, 0);
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::port::Thread threads([&] {
compact_options.change_level = false;
compact_options.exclusive_manual_compaction = false;
std::string begin_string = Key(0);
@@ -1380,18 +1383,18 @@ TEST_P(DBCompactionTestWithParam, ManualCompactionPartial) {
TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
bool first = true;
bool second = true;
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBCompaction::PartialFill:4", "DBCompaction::PartialFill:1"},
{"DBCompaction::PartialFill:2", "DBCompaction::PartialFill:3"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun", [&](void* /*arg*/) {
if (first) {
TEST_SYNC_POINT("DBCompaction::PartialFill:4");
@@ -1401,7 +1404,7 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.write_buffer_size = 10 * 1024 * 1024;
@@ -1462,7 +1465,7 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
ASSERT_EQ(trivial_move, 2);
ASSERT_EQ(non_trivial_move, 0);
rocksdb::port::Thread threads([&] {
ROCKSDB_NAMESPACE::port::Thread threads([&] {
compact_options.change_level = false;
compact_options.exclusive_manual_compaction = false;
std::string begin_string = Key(0);
@@ -1504,7 +1507,7 @@ TEST_F(DBCompactionTest, DISABLED_ManualPartialFill) {
}
TEST_F(DBCompactionTest, ManualCompactionWithUnorderedWrite) {
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WriteImpl:UnorderedWriteAfterWriteWAL",
"DBCompactionTest::ManualCompactionWithUnorderedWrite:WaitWriteWAL"},
{"DBImpl::WaitForPendingWrites:BeforeBlock",
@@ -1517,7 +1520,7 @@ TEST_F(DBCompactionTest, ManualCompactionWithUnorderedWrite) {
ASSERT_OK(Flush());
Put("bar", "v1");
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
port::Thread writer([&]() { Put("foo", "v2"); });
TEST_SYNC_POINT(
@@ -1828,13 +1831,13 @@ TEST_F(DBCompactionTest, DeleteFileRangeFileEndpointsOverlapBug) {
TEST_P(DBCompactionTestWithParam, TrivialMoveToLastLevelWithFiles) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.write_buffer_size = 100000000;
@@ -1883,7 +1886,7 @@ TEST_P(DBCompactionTestWithParam, TrivialMoveToLastLevelWithFiles) {
ASSERT_EQ(Get(Key(i)), values[i]);
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBCompactionTestWithParam, LevelCompactionThirdPath) {
@@ -2396,12 +2399,12 @@ TEST_F(DBCompactionTest, L0_CompactionBug_Issue44_b) {
TEST_F(DBCompactionTest, ManualAutoRace) {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::BGWorkCompaction", "DBCompactionTest::ManualAutoRace:1"},
{"DBImpl::RunManualCompaction:WaitScheduled",
"BackgroundCallCompaction:0"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Put(1, "foo", "");
Put(1, "bar", "");
@@ -2432,13 +2435,13 @@ TEST_F(DBCompactionTest, ManualAutoRace) {
// Eventually the cancelled compaction will be rescheduled and executed.
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("0,1", FilesPerLevel(0));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBCompactionTestWithParam, ManualCompaction) {
Options options = CurrentOptions();
options.max_subcompactions = max_subcompactions_;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
// iter - 0 with 7 levels
@@ -2486,7 +2489,7 @@ TEST_P(DBCompactionTestWithParam, ManualCompaction) {
options = CurrentOptions();
options.num_levels = 3;
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
}
@@ -2833,19 +2836,19 @@ TEST_P(DBCompactionTestWithParam, CompressLevelCompaction) {
kZlibCompression};
int matches = 0, didnt_match = 0, trivial_move = 0, non_trivial = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"Compaction::InputCompressionMatchesOutput:Matches",
[&](void* /*arg*/) { matches++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"Compaction::InputCompressionMatchesOutput:DidntMatch",
[&](void* /*arg*/) { didnt_match++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
@@ -2906,7 +2909,7 @@ TEST_P(DBCompactionTestWithParam, CompressLevelCompaction) {
ASSERT_EQ(trivial_move, 12);
ASSERT_EQ(non_trivial, 8);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
for (int i = 0; i < key_idx; i++) {
auto v = Get(Key(i));
@@ -2968,13 +2971,13 @@ TEST_F(DBCompactionTest, SuggestCompactRangeNoTwoLevel0Compactions) {
}
db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"CompactionJob::Run():Start",
"DBCompactionTest::SuggestCompactRangeNoTwoLevel0Compactions:1"},
{"DBCompactionTest::SuggestCompactRangeNoTwoLevel0Compactions:2",
"CompactionJob::Run():End"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// trigger L0 compaction
for (int num = 0; num < options.level0_file_num_compaction_trigger + 1;
@@ -3010,25 +3013,26 @@ static std::string ShortKey(int i) {
TEST_P(DBCompactionTestWithParam, ForceBottommostLevelCompaction) {
int32_t trivial_move = 0;
int32_t non_trivial_move = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:TrivialMove",
[&](void* /*arg*/) { trivial_move++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial_move++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// The key size is guaranteed to be <= 8
class ShortKeyComparator : public Comparator {
int Compare(const rocksdb::Slice& a,
const rocksdb::Slice& b) const override {
int Compare(const ROCKSDB_NAMESPACE::Slice& a,
const ROCKSDB_NAMESPACE::Slice& b) const override {
assert(a.size() <= 8);
assert(b.size() <= 8);
return BytewiseComparator()->Compare(a, b);
}
const char* Name() const override { return "ShortKeyComparator"; }
void FindShortestSeparator(std::string* start,
const rocksdb::Slice& limit) const override {
void FindShortestSeparator(
std::string* start,
const ROCKSDB_NAMESPACE::Slice& limit) const override {
return BytewiseComparator()->FindShortestSeparator(start, limit);
}
void FindShortSuccessor(std::string* key) const override {
@@ -3105,7 +3109,7 @@ TEST_P(DBCompactionTestWithParam, ForceBottommostLevelCompaction) {
ASSERT_EQ(Get(ShortKey(i)), values[i]);
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
@@ -3120,10 +3124,10 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
Random rnd(301);
std::string value(RandomString(&rnd, kValueSize));
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompactionBySize:0",
"CompactionJob::Run():Start"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// index: 0 1 2 3 4 5 6 7 8 9
// size: 1MB 1MB 1MB 1MB 1MB 2MB 1MB 1MB 1MB 1MB
@@ -3146,7 +3150,7 @@ TEST_P(DBCompactionTestWithParam, IntraL0Compaction) {
ASSERT_OK(Flush());
}
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
std::vector<std::vector<FileMetaData>> level_to_files;
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
@@ -3174,10 +3178,10 @@ TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
Random rnd(301);
std::string value(RandomString(&rnd, kValueSize));
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompactionBySize:0",
"CompactionJob::Run():Start"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// index: 0 1 2 3 4 5 6 7 8 9
// size: 1MB 1MB 1MB 1MB 1MB 1MB 1MB 1MB 1MB 1MB
@@ -3203,7 +3207,7 @@ TEST_P(DBCompactionTestWithParam, IntraL0CompactionDoesNotObsoleteDeletions) {
ASSERT_OK(Flush());
}
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
std::vector<std::vector<FileMetaData>> level_to_files;
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
@@ -3256,7 +3260,7 @@ TEST_P(DBCompactionTestWithParam, FullCompactionInBottomPriThreadPool) {
// Verify that size amplification did occur
ASSERT_EQ(NumSortedRuns(), 1);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
Env::Default()->SetBackgroundThreads(0, Env::Priority::BOTTOM);
}
@@ -3267,7 +3271,7 @@ TEST_F(DBCompactionTest, OptimizedDeletionObsoleting) {
const int kNumL0Files = 4;
Options options = CurrentOptions();
options.level0_file_num_compaction_trigger = kNumL0Files;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
// put key 1 and 3 in separate L1, L2 files.
@@ -3312,12 +3316,12 @@ TEST_F(DBCompactionTest, CompactFilesPendingL0Bug) {
options.max_background_compactions = 2;
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"LevelCompactionPicker::PickCompaction:Return",
"DBCompactionTest::CompactFilesPendingL0Bug:Picked"},
{"DBCompactionTest::CompactFilesPendingL0Bug:ManualCompacted",
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
auto schedule_multi_compaction_token =
dbfull()->TEST_write_controler().GetCompactionPressureToken();
@@ -3347,7 +3351,7 @@ TEST_F(DBCompactionTest, CompactFilesPendingL0Bug) {
->CompactFiles(CompactionOptions(), input_filenames,
0 /* output_level */));
TEST_SYNC_POINT("DBCompactionTest::CompactFilesPendingL0Bug:ManualCompacted");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, CompactFilesOverlapInL0Bug) {
@@ -3421,13 +3425,13 @@ TEST_F(DBCompactionTest, CompactBottomLevelFilesWithDeletions) {
// file without changing file count.
db_->ReleaseSnapshot(snapshot);
ASSERT_EQ(kMaxSequenceNumber, dbfull()->bottommost_files_mark_threshold_);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->compaction_reason() ==
CompactionReason::kBottommostFiles);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->TEST_WaitForCompact();
db_->GetLiveFilesMetaData(&post_release_metadata);
ASSERT_EQ(pre_release_metadata.size(), post_release_metadata.size());
@@ -3441,7 +3445,7 @@ TEST_F(DBCompactionTest, CompactBottomLevelFilesWithDeletions) {
// deletion markers/deleted keys.
ASSERT_LT(post_file.size, pre_file.size);
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
@@ -3490,16 +3494,16 @@ TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
// compacted.
ASSERT_OK(Put("a", "1"));
Flush();
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->compaction_reason() == CompactionReason::kTtl);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->TEST_WaitForCompact();
// All non-L0 files are deleted, as they contained only deleted data.
ASSERT_EQ("1", FilesPerLevel());
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
// Test dynamically changing ttl.
@@ -3537,12 +3541,12 @@ TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("1,2,0,2", FilesPerLevel());
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
ASSERT_TRUE(compaction->compaction_reason() == CompactionReason::kTtl);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Dynamically change ttl to 10 hours.
// This should trigger a ttl compaction, as 12 hours have already passed.
@@ -3550,7 +3554,7 @@ TEST_F(DBCompactionTest, LevelCompactExpiredTtlFiles) {
dbfull()->TEST_WaitForCompact();
// All non-L0 files are deleted, as they contained only deleted data.
ASSERT_EQ("1", FilesPerLevel());
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
@@ -3567,7 +3571,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
options.max_open_files = 20;
}
// RocksDB sanitize max open files to at least 20. Modify it back.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = static_cast<int*>(arg);
*max_open_files = 2;
@@ -3575,7 +3579,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
// In the case where all files are opened and doing DB restart
// forcing the oldest ancester time in manifest file to be 0 to
// simulate the case of reading from an old version.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionEdit::EncodeTo:VarintOldestAncesterTime", [&](void* arg) {
if (if_restart && if_open_all_files) {
std::string* encoded_fieled = static_cast<std::string*>(arg);
@@ -3591,7 +3595,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
DestroyAndReopen(options);
int ttl_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
@@ -3599,7 +3603,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
ttl_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Add two L6 files with key ranges: [1 .. 100], [101 .. 200].
Random rnd(301);
@@ -3607,6 +3611,15 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
// Get the first file's creation time. This will be the oldest file in the
// DB. Compactions inolving this file's descendents should keep getting
// this time.
std::vector<std::vector<FileMetaData>> level_to_files;
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
&level_to_files);
uint64_t oldest_time = level_to_files[0][0].oldest_ancester_time;
// Add 1 hour and do another flush.
env_->addon_time_.fetch_add(1 * 60 * 60);
for (int i = 101; i <= 200; ++i) {
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
@@ -3614,11 +3627,13 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
MoveFilesToLevel(6);
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
env_->addon_time_.fetch_add(1 * 60 * 60);
// Add two L4 files with key ranges: [1 .. 50], [51 .. 150].
for (int i = 1; i <= 50; ++i) {
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
Flush();
env_->addon_time_.fetch_add(1 * 60 * 60);
for (int i = 51; i <= 150; ++i) {
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
}
@@ -3626,6 +3641,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
MoveFilesToLevel(4);
ASSERT_EQ("0,0,0,0,2,0,2", FilesPerLevel());
env_->addon_time_.fetch_add(1 * 60 * 60);
// Add one L1 file with key range: [26, 75].
for (int i = 26; i <= 75; ++i) {
ASSERT_OK(Put(Key(i), RandomString(&rnd, kValueSize)));
@@ -3667,6 +3683,10 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
ASSERT_EQ("1,0,0,0,0,0,1", FilesPerLevel());
ASSERT_EQ(5, ttl_compactions);
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
&level_to_files);
ASSERT_EQ(oldest_time, level_to_files[6][0].oldest_ancester_time);
env_->addon_time_.fetch_add(25 * 60 * 60);
ASSERT_OK(Put(Key(2), "1"));
if (if_restart) {
@@ -3678,7 +3698,7 @@ TEST_F(DBCompactionTest, LevelTtlCascadingCompactions) {
ASSERT_EQ("1,0,0,0,0,0,1", FilesPerLevel());
ASSERT_GE(ttl_compactions, 6);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
}
@@ -3698,7 +3718,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
options.max_open_files = 20;
}
// RocksDB sanitize max open files to at least 20. Modify it back.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = static_cast<int*>(arg);
*max_open_files = 0;
@@ -3706,7 +3726,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
// In the case where all files are opened and doing DB restart
// forcing the file creation time in manifest file to be 0 to
// simulate the case of reading from an old version.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionEdit::EncodeTo:VarintFileCreationTime", [&](void* arg) {
if (if_restart && if_open_all_files) {
std::string* encoded_fieled = static_cast<std::string*>(arg);
@@ -3722,7 +3742,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
DestroyAndReopen(options);
int periodic_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
@@ -3730,7 +3750,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
periodic_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
@@ -3782,7 +3802,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
// + 4.
ASSERT_EQ(9, periodic_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
}
@@ -3806,7 +3826,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
int periodic_compactions = 0;
bool set_file_creation_time_to_zero = true;
bool set_creation_time_to_zero = true;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
@@ -3814,7 +3834,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
periodic_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"PropertyBlockBuilder::AddTableProperty:Start", [&](void* arg) {
TableProperties* props = reinterpret_cast<TableProperties*>(arg);
if (set_file_creation_time_to_zero) {
@@ -3824,7 +3844,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
props->creation_time = 0;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < kNumFiles; ++i) {
@@ -3857,7 +3877,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
// Make sure that all files go through periodic compaction.
ASSERT_EQ(kNumFiles, periodic_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
@@ -3877,7 +3897,7 @@ TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
int periodic_compactions = 0;
int ttl_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
@@ -3887,7 +3907,7 @@ TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
ttl_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
@@ -3938,7 +3958,7 @@ TEST_F(DBCompactionTest, LevelPeriodicAndTtlCompaction) {
ASSERT_EQ(6, periodic_compactions);
ASSERT_EQ(6, ttl_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, LevelPeriodicCompactionWithCompactionFilters) {
@@ -3991,7 +4011,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithCompactionFilters) {
dbfull()->GetOptions().periodic_compaction_seconds);
int periodic_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
@@ -3999,7 +4019,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithCompactionFilters) {
periodic_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
@@ -4023,7 +4043,7 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithCompactionFilters) {
// The two old files go through the periodic compaction process
ASSERT_EQ(2, periodic_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -4049,16 +4069,17 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByL0FileCount) {
if (i == 0) {
// ensure the auto compaction doesn't finish until manual compaction has
// had a chance to be delayed.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WaitUntilFlushWouldNotStallWrites:StallWait",
"CompactionJob::Run():End"}});
} else {
// ensure the auto-compaction doesn't finish until manual compaction has
// continued without delay.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::FlushMemTable:StallWaitDone", "CompactionJob::Run():End"}});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::FlushMemTable:StallWaitDone",
"CompactionJob::Run():End"}});
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int j = 0; j < kNumL0FilesLimit - 1; ++j) {
@@ -4077,7 +4098,7 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByL0FileCount) {
dbfull()->TEST_WaitForCompact();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(1), 0);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -4103,17 +4124,17 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByImmMemTableCount) {
if (i == 0) {
// ensure the flush doesn't finish until manual compaction has had a
// chance to be delayed.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WaitUntilFlushWouldNotStallWrites:StallWait",
"FlushJob::WriteLevel0Table"}});
} else {
// ensure the flush doesn't finish until manual compaction has continued
// without delay.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::FlushMemTable:StallWaitDone",
"FlushJob::WriteLevel0Table"}});
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int j = 0; j < kNumImmMemTableLimit - 1; ++j) {
@@ -4134,7 +4155,7 @@ TEST_F(DBCompactionTest, CompactRangeDelayedByImmMemTableCount) {
dbfull()->TEST_WaitForFlushMemTable();
ASSERT_EQ(0, NumTableFilesAtLevel(0));
ASSERT_GT(NumTableFilesAtLevel(1), 0);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -4155,12 +4176,12 @@ TEST_F(DBCompactionTest, CompactRangeShutdownWhileDelayed) {
// The calls to close CF/DB wait until the manual compaction stalls.
// The auto-compaction waits until the manual compaction finishes to ensure
// the signal comes from closing CF/DB, not from compaction making progress.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WaitUntilFlushWouldNotStallWrites:StallWait",
"DBCompactionTest::CompactRangeShutdownWhileDelayed:PreShutdown"},
{"DBCompactionTest::CompactRangeShutdownWhileDelayed:PostManual",
"CompactionJob::Run():End"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int j = 0; j < kNumL0FilesLimit - 1; ++j) {
@@ -4193,7 +4214,7 @@ TEST_F(DBCompactionTest, CompactRangeShutdownWhileDelayed) {
TEST_SYNC_POINT(
"DBCompactionTest::CompactRangeShutdownWhileDelayed:PostManual");
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -4212,13 +4233,13 @@ TEST_F(DBCompactionTest, CompactRangeSkipFlushAfterDelay) {
// The manual flush includes the memtable that was active when CompactRange
// began. So it unblocks CompactRange and precludes its flush. Throughout the
// test, stall conditions are upheld via high L0 file count.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::WaitUntilFlushWouldNotStallWrites:StallWait",
"DBCompactionTest::CompactRangeSkipFlushAfterDelay:PreFlush"},
{"DBCompactionTest::CompactRangeSkipFlushAfterDelay:PostFlush",
"DBImpl::FlushMemTable:StallWaitDone"},
{"DBImpl::FlushMemTable:StallWaitDone", "CompactionJob::Run():End"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
//used for the delayable flushes
FlushOptions flush_opts;
@@ -4248,7 +4269,7 @@ TEST_F(DBCompactionTest, CompactRangeSkipFlushAfterDelay) {
db_->GetProperty(DB::Properties::kNumEntriesActiveMemTable, &num_keys_in_memtable);
ASSERT_EQ(ToString(1), num_keys_in_memtable);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, CompactRangeFlushOverlappingMemtable) {
@@ -4490,29 +4511,29 @@ TEST_F(DBCompactionTest, CompactionLimiter) {
port::Mutex mutex;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:BeforeCompaction", [&](void* arg) {
const auto& cf_name = static_cast<ColumnFamilyData*>(arg)->GetName();
auto iter = cf_to_limiter.find(cf_name);
if (iter != cf_to_limiter.end()) {
MutexLock l(&mutex);
ASSERT_GE(iter->second->limit_tasks, ++iter->second->tasks);
iter->second->max_tasks = std::max(iter->second->max_tasks,
iter->second->limit_tasks);
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:BeforeCompaction", [&](void* arg) {
const auto& cf_name = static_cast<ColumnFamilyData*>(arg)->GetName();
auto iter = cf_to_limiter.find(cf_name);
if (iter != cf_to_limiter.end()) {
MutexLock l(&mutex);
ASSERT_GE(iter->second->limit_tasks, ++iter->second->tasks);
iter->second->max_tasks =
std::max(iter->second->max_tasks, iter->second->limit_tasks);
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:AfterCompaction", [&](void* arg) {
const auto& cf_name = static_cast<ColumnFamilyData*>(arg)->GetName();
auto iter = cf_to_limiter.find(cf_name);
if (iter != cf_to_limiter.end()) {
MutexLock l(&mutex);
ASSERT_GE(--iter->second->tasks, 0);
}
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:AfterCompaction", [&](void* arg) {
const auto& cf_name = static_cast<ColumnFamilyData*>(arg)->GetName();
auto iter = cf_to_limiter.find(cf_name);
if (iter != cf_to_limiter.end()) {
MutexLock l(&mutex);
ASSERT_GE(--iter->second->tasks, 0);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Block all compact threads in thread pool.
const size_t kTotalFlushTasks = kMaxBackgroundThreads / 4;
@@ -4850,12 +4871,10 @@ TEST_F(DBCompactionTest, CompactionDuringShutdown) {
Flush();
}
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:BeforeRun",
[&](void* /*arg*/) {
dbfull()->shutting_down_.store(true);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
[&](void* /*arg*/) { dbfull()->shutting_down_.store(true); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_OK(dbfull()->error_handler_.GetBGError());
}
@@ -4941,7 +4960,7 @@ TEST_F(DBCompactionTest, ConsistencyFailTest) {
Options options = CurrentOptions();
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"VersionBuilder::CheckConsistency", [&](void* arg) {
auto p =
reinterpret_cast<std::pair<FileMetaData**, FileMetaData**>*>(arg);
@@ -4952,7 +4971,7 @@ TEST_F(DBCompactionTest, ConsistencyFailTest) {
*(p->second) = temp;
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int k = 0; k < 2; ++k) {
ASSERT_OK(Put("foo", "bar"));
@@ -4960,7 +4979,7 @@ TEST_F(DBCompactionTest, ConsistencyFailTest) {
}
ASSERT_NOK(Put("foo", "bar"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
void IngestOneKeyValue(DBImpl* db, const std::string& key,
@@ -4968,7 +4987,7 @@ void IngestOneKeyValue(DBImpl* db, const std::string& key,
ExternalSstFileInfo info;
std::string f = test::PerThreadDBPath("sst_file" + key);
EnvOptions env;
rocksdb::SstFileWriter writer(env, options);
ROCKSDB_NAMESPACE::SstFileWriter writer(env, options);
auto s = writer.Open(f);
ASSERT_OK(s);
// ASSERT_OK(writer.Put(Key(), ""));
@@ -4995,14 +5014,14 @@ TEST_P(DBCompactionTestWithParam,
std::atomic<int> pick_intra_l0_count(0);
std::string value(RandomString(&rnd, kValueSize));
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBCompactionTestWithParam::FlushAfterIntraL0:1",
"CompactionJob::Run():Start"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FindIntraL0Compaction",
[&](void* /*arg*/) { pick_intra_l0_count.fetch_add(1); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// prevents trivial move
for (int i = 0; i < 10; ++i) {
@@ -5037,7 +5056,7 @@ TEST_P(DBCompactionTestWithParam,
ASSERT_OK(Put(Key(2), "b"));
ASSERT_EQ(10, NumTableFilesAtLevel(0));
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
std::vector<std::vector<FileMetaData>> level_to_files;
dbfull()->TEST_GetFilesMetaData(dbfull()->DefaultColumnFamily(),
&level_to_files);
@@ -5074,13 +5093,13 @@ TEST_P(DBCompactionTestWithParam,
ASSERT_EQ(0, NumTableFilesAtLevel(0));
std::atomic<int> pick_intra_l0_count(0);
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBCompactionTestWithParam::IntraL0CompactionAfterFlush:1",
"CompactionJob::Run():Start"}});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FindIntraL0Compaction",
[&](void* /*arg*/) { pick_intra_l0_count.fetch_add(1); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Make 6 L0 sst.
for (int i = 0; i < 6; ++i) {
if (i % 2 == 0) {
@@ -5118,7 +5137,7 @@ TEST_P(DBCompactionTestWithParam,
sleeping_tasks.WaitUntilDone();
TEST_SYNC_POINT("DBCompactionTestWithParam::IntraL0CompactionAfterFlush:1");
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
uint64_t error_count = 0;
db_->GetIntProperty("rocksdb.background-errors", &error_count);
@@ -5133,11 +5152,11 @@ TEST_P(DBCompactionTestWithParam,
}
#endif // !defined(ROCKSDB_LITE)
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
+20 -20
View File
@@ -16,7 +16,7 @@
#include "port/port.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBTestDynamicLevel : public DBTestBase {
public:
DBTestDynamicLevel() : DBTestBase("/db_dynamic_level_test") {}
@@ -221,10 +221,10 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
// Make sure that the compaction starts before the last bit of data is
// flushed, so that the base level isn't raised to L1.
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"CompactionJob::Run():Start", "DynamicLevelMaxBytesBase2:0"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->SetOptions({
{"disable_auto_compactions", "false"},
@@ -235,20 +235,20 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
dbfull()->TEST_WaitForCompact();
ASSERT_TRUE(db_->GetIntProperty("rocksdb.base-level", &int_prop));
ASSERT_EQ(2U, int_prop);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
// Write more data until the base level changes to L1. There will be
// a manual compaction going on at the same time.
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"CompactionJob::Run():Start", "DynamicLevelMaxBytesBase2:1"},
{"DynamicLevelMaxBytesBase2:2", "CompactionJob::Run():End"},
{"DynamicLevelMaxBytesBase2:compact_range_finish",
"FlushJob::WriteLevel0Table"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
rocksdb::port::Thread thread([this] {
ROCKSDB_NAMESPACE::port::Thread thread([this] {
TEST_SYNC_POINT("DynamicLevelMaxBytesBase2:compact_range_start");
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
TEST_SYNC_POINT("DynamicLevelMaxBytesBase2:compact_range_finish");
@@ -265,8 +265,8 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBase2) {
thread.join();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_TRUE(db_->GetIntProperty("rocksdb.base-level", &int_prop));
ASSERT_EQ(1U, int_prop);
@@ -329,16 +329,16 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesCompactRange) {
ASSERT_TRUE(db_->GetProperty("rocksdb.num-files-at-level2", &str_prop));
ASSERT_EQ("0", str_prop);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
std::set<int> output_levels;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionPicker::CompactRange:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
output_levels.insert(compaction->output_level());
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr);
ASSERT_EQ(output_levels.size(), 2);
@@ -373,10 +373,10 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBaseInc) {
DestroyAndReopen(options);
int non_trivial = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial",
[&](void* /*arg*/) { non_trivial++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
const int total_keys = 3000;
@@ -388,7 +388,7 @@ TEST_F(DBTestDynamicLevel, DynamicLevelMaxBytesBaseInc) {
}
Flush();
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ(non_trivial, 0);
@@ -457,7 +457,7 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
compaction_finished = false;
// Issue manual compaction in one thread and still verify DB state
// in main thread.
rocksdb::port::Thread t([&]() {
ROCKSDB_NAMESPACE::port::Thread t([&]() {
CompactRangeOptions compact_options;
compact_options.change_level = true;
compact_options.target_level = options.num_levels - 1;
@@ -488,13 +488,13 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
ASSERT_EQ(NumTableFilesAtLevel(1), 0);
ASSERT_EQ(NumTableFilesAtLevel(2), 0);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
+3 -3
View File
@@ -12,7 +12,7 @@
#include <iostream>
#include <string>
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBEncryptionTest : public DBTestBase {
public:
@@ -113,10 +113,10 @@ TEST_F(DBEncryptionTest, ReadEmptyFile) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+2 -2
View File
@@ -21,7 +21,7 @@
#include "test_util/sync_point.h"
#include "util/mutexlock.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
Status DBImpl::DisableFileDeletions() {
InstrumentedMutexLock l(&mutex_);
@@ -172,6 +172,6 @@ Status DBImpl::GetCurrentWalFile(std::unique_ptr<LogFile>* current_log_file) {
return wal_manager_.GetLiveWalFile(current_logfile_number, current_log_file);
}
}
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+32 -3
View File
@@ -18,7 +18,7 @@
#include "util/cast_util.h"
#include "util/mutexlock.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBFlushTest : public DBTestBase {
public:
@@ -741,15 +741,44 @@ TEST_P(DBAtomicFlushTest, CFDropRaceWithWaitForFlushMemTables) {
SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBAtomicFlushTest, RollbackAfterFailToInstallResults) {
bool atomic_flush = GetParam();
if (!atomic_flush) {
return;
}
auto fault_injection_env = std::make_shared<FaultInjectionTestEnv>(env_);
Options options = CurrentOptions();
options.env = fault_injection_env.get();
options.create_if_missing = true;
options.atomic_flush = atomic_flush;
CreateAndReopenWithCF({"pikachu"}, options);
ASSERT_EQ(2, handles_.size());
for (size_t cf = 0; cf < handles_.size(); ++cf) {
ASSERT_OK(Put(static_cast<int>(cf), "a", "value"));
}
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
SyncPoint::GetInstance()->SetCallBack(
"VersionSet::ProcessManifestWrites:BeforeWriteLastVersionEdit:0",
[&](void* /*arg*/) { fault_injection_env->SetFilesystemActive(false); });
SyncPoint::GetInstance()->EnableProcessing();
FlushOptions flush_opts;
Status s = db_->Flush(flush_opts, handles_);
ASSERT_NOK(s);
fault_injection_env->SetFilesystemActive(true);
Close();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
INSTANTIATE_TEST_CASE_P(DBFlushDirectIOTest, DBFlushDirectIOTest,
testing::Bool());
INSTANTIATE_TEST_CASE_P(DBAtomicFlushTest, DBAtomicFlushTest, testing::Bool());
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+120 -61
View File
@@ -102,7 +102,7 @@
#include "util/stop_watch.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
const std::string kDefaultColumnFamilyName("default");
const std::string kPersistentStatsColumnFamilyName(
@@ -644,7 +644,7 @@ void DBImpl::StartTimedTasks() {
stats_dump_period_sec = mutable_db_options_.stats_dump_period_sec;
if (stats_dump_period_sec > 0) {
if (!thread_dump_stats_) {
thread_dump_stats_.reset(new rocksdb::RepeatableThread(
thread_dump_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::DumpStats(); }, "dump_st", env_,
static_cast<uint64_t>(stats_dump_period_sec) * kMicrosInSecond));
}
@@ -652,7 +652,7 @@ void DBImpl::StartTimedTasks() {
stats_persist_period_sec = mutable_db_options_.stats_persist_period_sec;
if (stats_persist_period_sec > 0) {
if (!thread_persist_stats_) {
thread_persist_stats_.reset(new rocksdb::RepeatableThread(
thread_persist_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::PersistStats(); }, "pst_st", env_,
static_cast<uint64_t>(stats_persist_period_sec) * kMicrosInSecond));
}
@@ -1005,12 +1005,36 @@ Status DBImpl::SetDBOptions(
}
}
if (s.ok()) {
if (new_options.max_background_compactions >
mutable_db_options_.max_background_compactions) {
env_->IncBackgroundThreadsIfNeeded(
new_options.max_background_compactions, Env::Priority::LOW);
const BGJobLimits current_bg_job_limits =
GetBGJobLimits(immutable_db_options_.max_background_flushes,
mutable_db_options_.max_background_compactions,
mutable_db_options_.max_background_jobs,
/* parallelize_compactions */ true);
const BGJobLimits new_bg_job_limits = GetBGJobLimits(
immutable_db_options_.max_background_flushes,
new_options.max_background_compactions,
new_options.max_background_jobs, /* parallelize_compactions */ true);
const bool max_flushes_increased =
new_bg_job_limits.max_flushes > current_bg_job_limits.max_flushes;
const bool max_compactions_increased =
new_bg_job_limits.max_compactions >
current_bg_job_limits.max_compactions;
if (max_flushes_increased || max_compactions_increased) {
if (max_flushes_increased) {
env_->IncBackgroundThreadsIfNeeded(new_bg_job_limits.max_flushes,
Env::Priority::HIGH);
}
if (max_compactions_increased) {
env_->IncBackgroundThreadsIfNeeded(new_bg_job_limits.max_compactions,
Env::Priority::LOW);
}
MaybeScheduleFlushOrCompaction();
}
if (new_options.stats_dump_period_sec !=
mutable_db_options_.stats_dump_period_sec) {
if (thread_dump_stats_) {
@@ -1019,7 +1043,7 @@ Status DBImpl::SetDBOptions(
mutex_.Lock();
}
if (new_options.stats_dump_period_sec > 0) {
thread_dump_stats_.reset(new rocksdb::RepeatableThread(
thread_dump_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::DumpStats(); }, "dump_st", env_,
static_cast<uint64_t>(new_options.stats_dump_period_sec) *
kMicrosInSecond));
@@ -1035,7 +1059,7 @@ Status DBImpl::SetDBOptions(
mutex_.Lock();
}
if (new_options.stats_persist_period_sec > 0) {
thread_persist_stats_.reset(new rocksdb::RepeatableThread(
thread_persist_stats_.reset(new ROCKSDB_NAMESPACE::RepeatableThread(
[this]() { DBImpl::PersistStats(); }, "pst_st", env_,
static_cast<uint64_t>(new_options.stats_persist_period_sec) *
kMicrosInSecond));
@@ -2277,7 +2301,8 @@ Status DBImpl::CreateColumnFamilyImpl(const ColumnFamilyOptions& cf_options,
auto* cfd =
versions_->GetColumnFamilySet()->GetColumnFamily(column_family_name);
assert(cfd != nullptr);
s = cfd->AddDirectories();
std::map<std::string, std::shared_ptr<Directory>> dummy_created_dirs;
s = cfd->AddDirectories(&dummy_created_dirs);
}
if (s.ok()) {
single_column_family_mode_ = false;
@@ -2536,7 +2561,7 @@ ArenaWrappedDBIter* DBImpl::NewIteratorImpl(const ReadOptions& read_options,
env_, read_options, *cfd->ioptions(), sv->mutable_cf_options, snapshot,
sv->mutable_cf_options.max_sequential_skip_in_iterations,
sv->version_number, read_callback, this, cfd, allow_blob,
((read_options.snapshot != nullptr) ? false : allow_refresh));
read_options.snapshot != nullptr ? false : allow_refresh);
InternalIterator* internal_iter =
NewInternalIterator(read_options, cfd, sv, db_iter->GetArena(),
@@ -3293,27 +3318,67 @@ Status DBImpl::CheckConsistency() {
TEST_SYNC_POINT("DBImpl::CheckConsistency:AfterGetLiveFilesMetaData");
std::string corruption_messages;
for (const auto& md : metadata) {
// md.name has a leading "/".
std::string file_path = md.db_path + md.name;
uint64_t fsize = 0;
TEST_SYNC_POINT("DBImpl::CheckConsistency:BeforeGetFileSize");
Status s = env_->GetFileSize(file_path, &fsize);
if (!s.ok() &&
env_->GetFileSize(Rocks2LevelTableFileName(file_path), &fsize).ok()) {
s = Status::OK();
if (immutable_db_options_.skip_checking_sst_file_sizes_on_db_open) {
// Instead of calling GetFileSize() for each expected file, call
// GetChildren() for the DB directory and check that all expected files
// are listed, without checking their sizes.
// Since sst files might be in different directories, do it for each
// directory separately.
std::map<std::string, std::vector<std::string>> files_by_directory;
for (const auto& md : metadata) {
// md.name has a leading "/". Remove it.
std::string fname = md.name;
if (!fname.empty() && fname[0] == '/') {
fname = fname.substr(1);
}
files_by_directory[md.db_path].push_back(fname);
}
if (!s.ok()) {
corruption_messages +=
"Can't access " + md.name + ": " + s.ToString() + "\n";
} else if (fsize != md.size) {
corruption_messages += "Sst file size mismatch: " + file_path +
". Size recorded in manifest " +
ToString(md.size) + ", actual size " +
ToString(fsize) + "\n";
for (const auto& dir_files : files_by_directory) {
std::string directory = dir_files.first;
std::vector<std::string> existing_files;
Status s = env_->GetChildren(directory, &existing_files);
if (!s.ok()) {
corruption_messages +=
"Can't list files in " + directory + ": " + s.ToString() + "\n";
continue;
}
std::sort(existing_files.begin(), existing_files.end());
for (const std::string& fname : dir_files.second) {
if (!std::binary_search(existing_files.begin(), existing_files.end(),
fname) &&
!std::binary_search(existing_files.begin(), existing_files.end(),
Rocks2LevelTableFileName(fname))) {
corruption_messages +=
"Missing sst file " + fname + " in " + directory + "\n";
}
}
}
} else {
for (const auto& md : metadata) {
// md.name has a leading "/".
std::string file_path = md.db_path + md.name;
uint64_t fsize = 0;
TEST_SYNC_POINT("DBImpl::CheckConsistency:BeforeGetFileSize");
Status s = env_->GetFileSize(file_path, &fsize);
if (!s.ok() &&
env_->GetFileSize(Rocks2LevelTableFileName(file_path), &fsize).ok()) {
s = Status::OK();
}
if (!s.ok()) {
corruption_messages +=
"Can't access " + md.name + ": " + s.ToString() + "\n";
} else if (fsize != md.size) {
corruption_messages += "Sst file size mismatch: " + file_path +
". Size recorded in manifest " +
ToString(md.size) + ", actual size " +
ToString(fsize) + "\n";
}
}
}
if (corruption_messages.size() == 0) {
return Status::OK();
} else {
@@ -3329,31 +3394,12 @@ Status DBImpl::GetDbIdentity(std::string& identity) const {
Status DBImpl::GetDbIdentityFromIdentityFile(std::string* identity) const {
std::string idfilename = IdentityFileName(dbname_);
const FileOptions soptions;
std::unique_ptr<SequentialFileReader> id_file_reader;
Status s;
{
std::unique_ptr<FSSequentialFile> idfile;
s = fs_->NewSequentialFile(idfilename, soptions, &idfile, nullptr);
if (!s.ok()) {
return s;
}
id_file_reader.reset(
new SequentialFileReader(std::move(idfile), idfilename));
Status s = ReadFileToString(fs_.get(), idfilename, identity);
if (!s.ok()) {
return s;
}
uint64_t file_size;
s = fs_->GetFileSize(idfilename, IOOptions(), &file_size, nullptr);
if (!s.ok()) {
return s;
}
char* buffer =
reinterpret_cast<char*>(alloca(static_cast<size_t>(file_size)));
Slice id;
s = id_file_reader->Read(static_cast<size_t>(file_size), &id, buffer);
if (!s.ok()) {
return s;
}
identity->assign(id.ToString());
// If last character is '\n' remove it from identity
if (identity->size() > 0 && identity->back() == '\n') {
identity->pop_back();
@@ -3545,6 +3591,11 @@ Status DestroyDB(const std::string& dbname, const Options& options,
env->UnlockFile(lock); // Ignore error since state is already gone
env->DeleteFile(lockname);
// sst_file_manager holds a ref to the logger. Make sure the logger is
// gone before trying to remove the directory.
soptions.sst_file_manager.reset();
env->DeleteDir(dbname); // Ignore error in case dir contains other files
}
return result;
@@ -4325,8 +4376,8 @@ Status DBImpl::VerifyChecksum(const ReadOptions& read_options) {
const auto& fd = vstorage->LevelFilesBrief(i).files[j].fd;
std::string fname = TableFileName(cfd->ioptions()->cf_paths,
fd.GetNumber(), fd.GetPathId());
s = rocksdb::VerifySstFileChecksum(opts, file_options_, read_options,
fname);
s = ROCKSDB_NAMESPACE::VerifySstFileChecksum(opts, file_options_,
read_options, fname);
}
}
if (!s.ok()) {
@@ -4471,13 +4522,21 @@ Status DBImpl::GetCreationTimeOfOldestFile(uint64_t* creation_time) {
if (mutable_db_options_.max_open_files == -1) {
uint64_t oldest_time = port::kMaxUint64;
for (auto cfd : *versions_->GetColumnFamilySet()) {
uint64_t ctime;
cfd->current()->GetCreationTimeOfOldestFile(&ctime);
if (ctime < oldest_time) {
oldest_time = ctime;
}
if (oldest_time == 0) {
break;
if (!cfd->IsDropped()) {
uint64_t ctime;
{
SuperVersion* sv = GetAndRefSuperVersion(cfd);
Version* version = sv->current;
version->GetCreationTimeOfOldestFile(&ctime);
ReturnAndCleanupSuperVersion(cfd, sv);
}
if (ctime < oldest_time) {
oldest_time = ctime;
}
if (oldest_time == 0) {
break;
}
}
}
*creation_time = oldest_time;
@@ -4488,4 +4547,4 @@ Status DBImpl::GetCreationTimeOfOldestFile(uint64_t* creation_time) {
}
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+14 -9
View File
@@ -62,7 +62,7 @@
#include "util/stop_watch.h"
#include "util/thread_local.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class Arena;
class ArenaWrappedDBIter;
@@ -826,7 +826,6 @@ class DBImpl : public DB {
std::vector<ColumnFamilyHandle*>* handles, DB** dbptr,
const bool seq_per_batch, const bool batch_per_txn);
static Status CreateAndNewDirectory(Env* env, const std::string& dirname,
std::unique_ptr<Directory>* directory);
@@ -1110,10 +1109,13 @@ class DBImpl : public DB {
// Recover the descriptor from persistent storage. May do a significant
// amount of work to recover recently logged updates. Any changes to
// be made to the descriptor are added to *edit.
// recovered_seq is set to less than kMaxSequenceNumber if the log's tail is
// skipped.
virtual Status Recover(
const std::vector<ColumnFamilyDescriptor>& column_families,
bool read_only = false, bool error_if_log_file_exist = false,
bool error_if_data_exists_in_logs = false);
bool error_if_data_exists_in_logs = false,
uint64_t* recovered_seq = nullptr);
virtual bool OwnTablesAndLogs() const { return true; }
@@ -1355,8 +1357,10 @@ class DBImpl : public DB {
JobContext* job_context, LogBuffer* log_buffer, Env::Priority thread_pri);
// REQUIRES: log_numbers are sorted in ascending order
// corrupted_log_found is set to true if we recover from a corrupted log file.
Status RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
SequenceNumber* next_sequence, bool read_only);
SequenceNumber* next_sequence, bool read_only,
bool* corrupted_log_found);
// The following two methods are used to flush a memtable to
// storage. The first one is used at database RecoveryTime (when the
@@ -1379,6 +1383,7 @@ class DBImpl : public DB {
Status ThrottleLowPriWritesIfNeeded(const WriteOptions& write_options,
WriteBatch* my_batch);
// REQUIRES: mutex locked and in write thread.
Status ScheduleFlushes(WriteContext* context);
void MaybeFlushStatsCF(autovector<ColumnFamilyData*>* cfds);
@@ -1450,10 +1455,10 @@ class DBImpl : public DB {
// REQUIRES: mutex locked and in write thread.
void AssignAtomicFlushSeq(const autovector<ColumnFamilyData*>& cfds);
// REQUIRES: mutex locked
// REQUIRES: mutex locked and in write thread.
Status SwitchWAL(WriteContext* write_context);
// REQUIRES: mutex locked
// REQUIRES: mutex locked and in write thread.
Status HandleWriteBufferFull(WriteContext* write_context);
// REQUIRES: mutex locked
@@ -2010,11 +2015,11 @@ class DBImpl : public DB {
// handle for scheduling stats dumping at fixed intervals
// REQUIRES: mutex locked
std::unique_ptr<rocksdb::RepeatableThread> thread_dump_stats_;
std::unique_ptr<ROCKSDB_NAMESPACE::RepeatableThread> thread_dump_stats_;
// handle for scheduling stats snapshoting at fixed intervals
// REQUIRES: mutex locked
std::unique_ptr<rocksdb::RepeatableThread> thread_persist_stats_;
std::unique_ptr<ROCKSDB_NAMESPACE::RepeatableThread> thread_persist_stats_;
// When set, we use a separate queue for writes that dont write to memtable.
// In 2PC these are the writes at Prepare phase.
@@ -2099,4 +2104,4 @@ static void ClipToRange(T* ptr, V minvalue, V maxvalue) {
if (static_cast<V>(*ptr) < minvalue) *ptr = minvalue;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+25 -22
View File
@@ -22,7 +22,7 @@
#include "util/cast_util.h"
#include "util/concurrent_task_limiter_impl.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
bool DBImpl::EnoughRoomForCompaction(
ColumnFamilyData* cfd, const std::vector<CompactionInputFiles>& inputs,
@@ -409,7 +409,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
s = Status::OK();
}
if (s.ok() || s.IsShutdownInProgress() || s.IsColumnFamilyDropped()) {
if (s.ok() || s.IsShutdownInProgress()) {
// Sync on all distinct output directories.
for (auto dir : distinct_output_dirs) {
if (dir != nullptr) {
@@ -420,6 +420,23 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
}
}
}
} else {
// Need to undo atomic flush if something went wrong, i.e. s is not OK and
// it is not because of CF drop.
// Have to cancel the flush jobs that have NOT executed because we need to
// unref the versions.
for (int i = 0; i != num_cfs; ++i) {
if (!exec_status[i].first) {
jobs[i]->Cancel();
}
}
for (int i = 0; i != num_cfs; ++i) {
if (exec_status[i].first && exec_status[i].second.ok()) {
auto& mems = jobs[i]->GetMemTables();
cfds[i]->imm()->RollbackMemtableFlush(mems,
file_meta[i].fd.GetNumber());
}
}
}
if (s.ok()) {
@@ -526,23 +543,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
#endif // ROCKSDB_LITE
}
// Need to undo atomic flush if something went wrong, i.e. s is not OK and
// it is not because of CF drop.
if (!s.ok() && !s.IsColumnFamilyDropped()) {
// Have to cancel the flush jobs that have NOT executed because we need to
// unref the versions.
for (int i = 0; i != num_cfs; ++i) {
if (!exec_status[i].first) {
jobs[i]->Cancel();
}
}
for (int i = 0; i != num_cfs; ++i) {
if (exec_status[i].first && exec_status[i].second.ok()) {
auto& mems = jobs[i]->GetMemTables();
cfds[i]->imm()->RollbackMemtableFlush(mems,
file_meta[i].fd.GetNumber());
}
}
if (!s.ok() && !s.IsShutdownInProgress()) {
Status new_bg_error = s;
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
@@ -1257,7 +1258,8 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->oldest_blob_file_number,
f->oldest_ancester_time, f->file_creation_time);
f->oldest_ancester_time, f->file_creation_time,
f->file_checksum, f->file_checksum_func_name);
}
ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[%s] Apply version edit:\n%s", cfd->GetName().c_str(),
@@ -2668,7 +2670,8 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
f->largest, f->fd.smallest_seqno,
f->fd.largest_seqno, f->marked_for_compaction,
f->oldest_blob_file_number, f->oldest_ancester_time,
f->file_creation_time);
f->file_creation_time, f->file_checksum,
f->file_checksum_func_name);
ROCKS_LOG_BUFFER(
log_buffer,
@@ -3110,4 +3113,4 @@ void DBImpl::GetSnapshotContext(
}
*snapshot_seqs = snapshots_.GetAll(earliest_write_conflict_snapshot);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+10 -5
View File
@@ -15,7 +15,7 @@
#include "monitoring/thread_status_updater.h"
#include "util/cast_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
uint64_t DBImpl::TEST_GetLevel0TotalSize() {
InstrumentedMutexLock l(&mutex_);
return default_cf_handle_->cfd()->current()->storage_info()->NumLevelBytes(0);
@@ -24,7 +24,9 @@ uint64_t DBImpl::TEST_GetLevel0TotalSize() {
void DBImpl::TEST_SwitchWAL() {
WriteContext write_context;
InstrumentedMutexLock l(&mutex_);
void* writer = TEST_BeginWrite();
SwitchWAL(&write_context);
TEST_EndWrite(writer);
}
bool DBImpl::TEST_WALBufferIsEmpty(bool lock) {
@@ -106,15 +108,18 @@ Status DBImpl::TEST_SwitchMemtable(ColumnFamilyData* cfd) {
cfd = default_cf_handle_->cfd();
}
Status s;
void* writer = TEST_BeginWrite();
if (two_write_queues_) {
WriteThread::Writer nonmem_w;
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
Status s = SwitchMemtable(cfd, &write_context);
s = SwitchMemtable(cfd, &write_context);
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
return s;
} else {
return SwitchMemtable(cfd, &write_context);
s = SwitchMemtable(cfd, &write_context);
}
TEST_EndWrite(writer);
return s;
}
Status DBImpl::TEST_FlushMemTable(bool wait, bool allow_write_stall,
@@ -285,5 +290,5 @@ bool DBImpl::TEST_IsPersistentStatsEnabled() const {
size_t DBImpl::TEST_EstimateInMemoryStatsHistorySize() const {
return EstimateInMemoryStatsHistorySize();
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // NDEBUG
+4 -3
View File
@@ -17,7 +17,7 @@
#include "db/version_set.h"
#include "rocksdb/status.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
Status DBImpl::SuggestCompactRange(ColumnFamilyHandle* column_family,
@@ -129,7 +129,8 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->oldest_blob_file_number,
f->oldest_ancester_time, f->file_creation_time);
f->oldest_ancester_time, f->file_creation_time,
f->file_checksum, f->file_checksum_func_name);
}
status = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
@@ -147,4 +148,4 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
}
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+4 -4
View File
@@ -17,7 +17,7 @@
#include "file/sst_file_manager_impl.h"
#include "util/autovector.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
uint64_t DBImpl::MinLogNumberToKeep() {
if (allow_2pc()) {
@@ -618,9 +618,9 @@ uint64_t PrecomputeMinLogNumberToKeep(
// family being flushed (`cfd_to_flush`).
uint64_t cf_min_log_number_to_keep = 0;
for (auto& e : edit_list) {
if (e->has_log_number()) {
if (e->HasLogNumber()) {
cf_min_log_number_to_keep =
std::max(cf_min_log_number_to_keep, e->log_number());
std::max(cf_min_log_number_to_keep, e->GetLogNumber());
}
}
if (cf_min_log_number_to_keep == 0) {
@@ -664,4 +664,4 @@ uint64_t PrecomputeMinLogNumberToKeep(
return min_log_number_to_keep;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+105 -29
View File
@@ -23,7 +23,7 @@
#include "test_util/sync_point.h"
#include "util/rate_limiter.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
Options SanitizeOptions(const std::string& dbname, const Options& src) {
auto db_options = SanitizeOptions(dbname, DBOptions(src));
ImmutableDBOptions immutable_db_options(db_options);
@@ -100,12 +100,11 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
if (result.recycle_log_file_num &&
(result.wal_recovery_mode == WALRecoveryMode::kPointInTimeRecovery ||
result.wal_recovery_mode == WALRecoveryMode::kAbsoluteConsistency)) {
// kPointInTimeRecovery is indistinguishable from
// kTolerateCorruptedTailRecords in recycle mode since we define
// the "end" of the log as the first corrupt record we encounter.
// kPointInTimeRecovery is inconsistent with recycle log file feature since
// we define the "end" of the log as the first corrupt record we encounter.
// kAbsoluteConsistency doesn't make sense because even a clean
// shutdown leaves old junk at the end of the log file.
result.wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
result.recycle_log_file_num = 0;
}
if (result.wal_dir.empty()) {
@@ -173,6 +172,13 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
result.sst_file_manager = sst_file_manager;
}
#endif
if (!result.paranoid_checks) {
result.skip_checking_sst_file_sizes_on_db_open = true;
ROCKS_LOG_INFO(result.info_log,
"file size check will be skipped during open.");
}
return result;
}
@@ -345,7 +351,8 @@ Status Directories::SetDirectories(Env* env, const std::string& dbname,
Status DBImpl::Recover(
const std::vector<ColumnFamilyDescriptor>& column_families, bool read_only,
bool error_if_log_file_exist, bool error_if_data_exists_in_logs) {
bool error_if_log_file_exist, bool error_if_data_exists_in_logs,
uint64_t* recovered_seq) {
mutex_.AssertHeld();
bool is_new_db = false;
@@ -367,9 +374,6 @@ Status DBImpl::Recover(
s = env_->FileExists(current_fname);
if (s.IsNotFound()) {
if (immutable_db_options_.create_if_missing) {
// Has to be called only after Identity File creation is successful
// because DB ID is stored in Manifest if
// immutable_db_options_.write_dbid_to_manifest = true
s = NewDB();
is_new_db = true;
if (!s.ok()) {
@@ -435,27 +439,28 @@ Status DBImpl::Recover(
assert(s.IsIOError());
return s;
}
GetDbIdentityFromIdentityFile(&db_id_);
if (immutable_db_options_.write_dbid_to_manifest) {
s = GetDbIdentityFromIdentityFile(&db_id_);
if (immutable_db_options_.write_dbid_to_manifest && s.ok()) {
VersionEdit edit;
edit.SetDBId(db_id_);
Options options;
MutableCFOptions mutable_cf_options(options);
versions_->db_id_ = db_id_;
versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
s = versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
mutable_cf_options, &edit, &mutex_, nullptr,
false);
}
} else {
SetIdentityFile(env_, dbname_, db_id_);
s = SetIdentityFile(env_, dbname_, db_id_);
}
if (immutable_db_options_.paranoid_checks && s.ok()) {
s = CheckConsistency();
}
if (s.ok() && !read_only) {
std::map<std::string, std::shared_ptr<Directory>> created_dirs;
for (auto cfd : *versions_->GetColumnFamilySet()) {
s = cfd->AddDirectories();
s = cfd->AddDirectories(&created_dirs);
if (!s.ok()) {
return s;
}
@@ -466,16 +471,16 @@ Status DBImpl::Recover(
s = InitPersistStatsColumnFamily();
}
// Initial max_total_in_memory_state_ before recovery logs. Log recovery
// may check this value to decide whether to flush.
max_total_in_memory_state_ = 0;
for (auto cfd : *versions_->GetColumnFamilySet()) {
auto* mutable_cf_options = cfd->GetLatestMutableCFOptions();
max_total_in_memory_state_ += mutable_cf_options->write_buffer_size *
mutable_cf_options->max_write_buffer_number;
}
if (s.ok()) {
// Initial max_total_in_memory_state_ before recovery logs. Log recovery
// may check this value to decide whether to flush.
max_total_in_memory_state_ = 0;
for (auto cfd : *versions_->GetColumnFamilySet()) {
auto* mutable_cf_options = cfd->GetLatestMutableCFOptions();
max_total_in_memory_state_ += mutable_cf_options->write_buffer_size *
mutable_cf_options->max_write_buffer_number;
}
SequenceNumber next_sequence(kMaxSequenceNumber);
default_cf_handle_ = new ColumnFamilyHandleImpl(
versions_->GetColumnFamilySet()->GetDefault(), this, &mutex_);
@@ -541,7 +546,12 @@ Status DBImpl::Recover(
if (!logs.empty()) {
// Recover in the order in which the logs were generated
std::sort(logs.begin(), logs.end());
s = RecoverLogFiles(logs, &next_sequence, read_only);
bool corrupted_log_found = false;
s = RecoverLogFiles(logs, &next_sequence, read_only,
&corrupted_log_found);
if (corrupted_log_found && recovered_seq != nullptr) {
*recovered_seq = next_sequence;
}
if (!s.ok()) {
// Clear memtables if recovery failed
for (auto cfd : *versions_->GetColumnFamilySet()) {
@@ -671,7 +681,8 @@ Status DBImpl::InitPersistStatsColumnFamily() {
// REQUIRES: log_numbers are sorted in ascending order
Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
SequenceNumber* next_sequence, bool read_only) {
SequenceNumber* next_sequence, bool read_only,
bool* corrupted_log_found) {
struct LogReporter : public log::Reader::Reporter {
Env* env;
Logger* info_log;
@@ -973,6 +984,9 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
status = Status::OK();
stop_replay_for_corruption = true;
corrupted_log_number = log_number;
if (corrupted_log_found != nullptr) {
*corrupted_log_found = true;
}
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Point in time recovered to log #%" PRIu64
" seq #%" PRIu64,
@@ -1001,6 +1015,9 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
// the corrupted log number, which means CF contains data beyond the point of
// corruption. This could during PIT recovery when the WAL is corrupted and
// some (but not all) CFs are flushed
// Exclude the PIT case where no log is dropped after the corruption point.
// This is to cover the case for empty logs after corrupted log, in which we
// don't reset stop_replay_for_corruption.
if (stop_replay_for_corruption == true &&
(immutable_db_options_.wal_recovery_mode ==
WALRecoveryMode::kPointInTimeRecovery ||
@@ -1242,7 +1259,8 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
meta.fd.GetFileSize(), meta.smallest, meta.largest,
meta.fd.smallest_seqno, meta.fd.largest_seqno,
meta.marked_for_compaction, meta.oldest_blob_file_number,
meta.oldest_ancester_time, meta.file_creation_time);
meta.oldest_ancester_time, meta.file_creation_time,
meta.file_checksum, meta.file_checksum_func_name);
}
InternalStats::CompactionStats stats(CompactionReason::kFlush, 1);
@@ -1311,6 +1329,8 @@ Status DBImpl::CreateWAL(uint64_t log_file_num, uint64_t recycle_log_number,
recycle_log_number);
std::string old_log_fname =
LogFileName(immutable_db_options_.wal_dir, recycle_log_number);
TEST_SYNC_POINT("DBImpl::CreateWAL:BeforeReuseWritableFile1");
TEST_SYNC_POINT("DBImpl::CreateWAL:BeforeReuseWritableFile2");
s = fs_->ReuseWritableFile(log_fname, old_log_fname, opt_file_options,
&lfile, /*dbg=*/nullptr);
} else {
@@ -1396,7 +1416,8 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
impl->mutex_.Lock();
// Handles create_if_missing, error_if_exists
s = impl->Recover(column_families);
uint64_t recovered_seq(kMaxSequenceNumber);
s = impl->Recover(column_families, false, false, false, &recovered_seq);
if (s.ok()) {
uint64_t new_log_number = impl->versions_->NewFileNumber();
log::Writer* new_log = nullptr;
@@ -1454,9 +1475,36 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
if (impl->two_write_queues_) {
impl->log_write_mutex_.Unlock();
}
impl->DeleteObsoleteFiles();
s = impl->directories_.GetDbDir()->Fsync();
}
if (s.ok()) {
// In WritePrepared there could be gap in sequence numbers. This breaks
// the trick we use in kPointInTimeRecovery which assumes the first seq in
// the log right after the corrupted log is one larger than the last seq
// we read from the logs. To let this trick keep working, we add a dummy
// entry with the expected sequence to the first log right after recovery.
// In non-WritePrepared case also the new log after recovery could be
// empty, and thus missing the consecutive seq hint to distinguish
// middle-log corruption to corrupted-log-remained-after-recovery. This
// case also will be addressed by a dummy write.
if (recovered_seq != kMaxSequenceNumber) {
WriteBatch empty_batch;
WriteBatchInternal::SetSequence(&empty_batch, recovered_seq);
WriteOptions write_options;
uint64_t log_used, log_size;
log::Writer* log_writer = impl->logs_.back().writer;
s = impl->WriteToWAL(empty_batch, log_writer, &log_used, &log_size);
if (s.ok()) {
// Need to fsync, otherwise it might get lost after a power reset.
s = impl->FlushWAL(false);
if (s.ok()) {
s = log_writer->file()->Sync(impl->immutable_db_options_.use_fsync);
}
}
}
}
}
if (s.ok() && impl->immutable_db_options_.persist_stats_to_disk) {
// try to read format version but no need to fail Open() even if it fails
@@ -1512,6 +1560,27 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
if (s.ok() && sfm) {
// Notify SstFileManager about all sst files that already exist in
// db_paths[0] and cf_paths[0] when the DB is opened.
// SstFileManagerImpl needs to know sizes of the files. For files whose size
// we already know (sst files that appear in manifest - typically that's the
// vast majority of all files), we'll pass the size to SstFileManager.
// For all other files SstFileManager will query the size from filesystem.
std::vector<LiveFileMetaData> metadata;
impl->mutex_.Lock();
impl->versions_->GetLiveFilesMetaData(&metadata);
impl->mutex_.Unlock();
std::unordered_map<std::string, uint64_t> known_file_sizes;
for (const auto& md : metadata) {
std::string name = md.name;
if (!name.empty() && name[0] == '/') {
name = name.substr(1);
}
known_file_sizes[name] = md.size;
}
std::vector<std::string> paths;
paths.emplace_back(impl->immutable_db_options_.db_paths[0].path);
for (auto& cf : column_families) {
@@ -1531,7 +1600,14 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
std::string file_path = path + "/" + file_name;
if (ParseFileName(file_name, &file_number, &file_type) &&
file_type == kTableFile) {
sfm->OnAddFile(file_path);
if (known_file_sizes.count(file_name)) {
// We're assuming that each sst file name exists in at most one of
// the paths.
sfm->OnAddFile(file_path, known_file_sizes.at(file_name),
/* compaction */ false);
} else {
sfm->OnAddFile(file_path);
}
}
}
}
@@ -1572,4 +1648,4 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
}
return s;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -12,7 +12,7 @@
#include "db/merge_context.h"
#include "monitoring/perf_context_imp.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
@@ -218,4 +218,4 @@ Status DB::OpenForReadOnly(
}
#endif // !ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -11,7 +11,7 @@
#include <vector>
#include "db/db_impl/db_impl.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBImplReadOnly : public DBImpl {
public:
@@ -132,6 +132,6 @@ class DBImplReadOnly : public DBImpl {
private:
friend class DB;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+8 -3
View File
@@ -13,7 +13,7 @@
#include "monitoring/perf_context_imp.h"
#include "util/cast_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
DBImplSecondary::DBImplSecondary(const DBOptions& db_options,
@@ -29,7 +29,7 @@ DBImplSecondary::~DBImplSecondary() {}
Status DBImplSecondary::Recover(
const std::vector<ColumnFamilyDescriptor>& column_families,
bool /*readonly*/, bool /*error_if_log_file_exist*/,
bool /*error_if_data_exists_in_logs*/) {
bool /*error_if_data_exists_in_logs*/, uint64_t*) {
mutex_.AssertHeld();
JobContext job_context(0);
@@ -468,6 +468,11 @@ Status DBImplSecondary::CheckConsistency() {
// approach and just proceed.
TEST_SYNC_POINT_CALLBACK(
"DBImplSecondary::CheckConsistency:AfterFirstAttempt", &s);
if (immutable_db_options_.skip_checking_sst_file_sizes_on_db_open) {
return Status::OK();
}
std::vector<LiveFileMetaData> metadata;
versions_->GetLiveFilesMetaData(&metadata);
@@ -663,4 +668,4 @@ Status DB::OpenAsSecondary(
}
#endif // !ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+4 -3
View File
@@ -11,7 +11,7 @@
#include <vector>
#include "db/db_impl/db_impl.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// A wrapper class to hold log reader, log reporter, log status.
class LogReaderContainer {
@@ -78,7 +78,8 @@ class DBImplSecondary : public DBImpl {
// and log_readers_ to facilitate future operations.
Status Recover(const std::vector<ColumnFamilyDescriptor>& column_families,
bool read_only, bool error_if_log_file_exist,
bool error_if_data_exists_in_logs) override;
bool error_if_data_exists_in_logs,
uint64_t* = nullptr) override;
// Implementations of the DB interface
using DB::Get;
@@ -327,6 +328,6 @@ class DBImplSecondary : public DBImpl {
std::unordered_map<ColumnFamilyData*, uint64_t> cfd_to_current_log_;
};
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+12 -5
View File
@@ -15,7 +15,7 @@
#include "options/options_helper.h"
#include "test_util/sync_point.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// Convenience methods
Status DBImpl::Put(const WriteOptions& o, ColumnFamilyHandle* column_family,
const Slice& key, const Slice& val) {
@@ -1292,7 +1292,7 @@ Status DBImpl::HandleWriteBufferFull(WriteContext* write_context) {
// suboptimal but still correct.
ROCKS_LOG_INFO(
immutable_db_options_.info_log,
"Flushing column family with largest mem table size. Write buffer is "
"Flushing column family with oldest memtable entry. Write buffer is "
"using %" ROCKSDB_PRIszt " bytes out of a total of %" ROCKSDB_PRIszt ".",
write_buffer_manager_->memory_usage(),
write_buffer_manager_->buffer_size());
@@ -1458,7 +1458,7 @@ Status DBImpl::ThrottleLowPriWritesIfNeeded(const WriteOptions& write_options,
return Status::OK();
}
if (write_options.no_slowdown) {
return Status::Incomplete();
return Status::Incomplete("Low priority write stall");
} else {
assert(my_batch != nullptr);
// Rate limit those writes. The reason that we don't completely wait
@@ -1631,7 +1631,6 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
if (creating_new_log && immutable_db_options_.recycle_log_file_num &&
!log_recycle_files_.empty()) {
recycle_log_number = log_recycle_files_.front();
log_recycle_files_.pop_front();
}
uint64_t new_log_number =
creating_new_log ? versions_->NewFileNumber() : logfile_number_;
@@ -1668,6 +1667,14 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
". Immutable memtables: %d.\n",
cfd->GetName().c_str(), new_log_number, num_imm_unflushed);
mutex_.Lock();
if (recycle_log_number != 0) {
// Since renaming the file is done outside DB mutex, we need to ensure
// concurrent full purges don't delete the file while we're recycling it.
// To achieve that we hold the old log number in the recyclable list until
// after it has been renamed.
assert(log_recycle_files_.front() == recycle_log_number);
log_recycle_files_.pop_front();
}
if (s.ok() && creating_new_log) {
log_write_mutex_.Lock();
assert(new_log != nullptr);
@@ -1829,4 +1836,4 @@ Status DB::Merge(const WriteOptions& opt, ColumnFamilyHandle* column_family,
}
return Write(opt, &batch);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+3 -3
View File
@@ -13,7 +13,7 @@
#include "test_util/fault_injection_test_env.h"
#include "test_util/sync_point.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
class DBSecondaryTest : public DBTestBase {
@@ -860,10 +860,10 @@ TEST_F(DBSecondaryTest, CheckConsistencyWhenOpen) {
}
#endif //! ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+2 -2
View File
@@ -14,7 +14,7 @@
#include "file/filename.h"
#include "rocksdb/env.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
void DumpDBFileSummary(const ImmutableDBOptions& options,
const std::string& dbname) {
@@ -120,4 +120,4 @@ void DumpDBFileSummary(const ImmutableDBOptions& options,
Header(options.info_log, "Write Ahead Log file in %s: %s\n",
options.wal_dir.c_str(), wal_info.c_str());
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -8,7 +8,7 @@
#include "options/db_options.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
void DumpDBFileSummary(const ImmutableDBOptions& options,
const std::string& dbname);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+7 -7
View File
@@ -9,7 +9,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBTestInPlaceUpdate : public DBTestBase {
public:
@@ -73,7 +73,7 @@ TEST_F(DBTestInPlaceUpdate, InPlaceUpdateCallbackSmallerSize) {
options.env = env_;
options.write_buffer_size = 100000;
options.inplace_callback =
rocksdb::DBTestInPlaceUpdate::updateInPlaceSmallerSize;
ROCKSDB_NAMESPACE::DBTestInPlaceUpdate::updateInPlaceSmallerSize;
options.allow_concurrent_memtable_write = false;
Reopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
@@ -102,7 +102,7 @@ TEST_F(DBTestInPlaceUpdate, InPlaceUpdateCallbackSmallerVarintSize) {
options.env = env_;
options.write_buffer_size = 100000;
options.inplace_callback =
rocksdb::DBTestInPlaceUpdate::updateInPlaceSmallerVarintSize;
ROCKSDB_NAMESPACE::DBTestInPlaceUpdate::updateInPlaceSmallerVarintSize;
options.allow_concurrent_memtable_write = false;
Reopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
@@ -131,7 +131,7 @@ TEST_F(DBTestInPlaceUpdate, InPlaceUpdateCallbackLargeNewValue) {
options.env = env_;
options.write_buffer_size = 100000;
options.inplace_callback =
rocksdb::DBTestInPlaceUpdate::updateInPlaceLargerSize;
ROCKSDB_NAMESPACE::DBTestInPlaceUpdate::updateInPlaceLargerSize;
options.allow_concurrent_memtable_write = false;
Reopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
@@ -158,7 +158,7 @@ TEST_F(DBTestInPlaceUpdate, InPlaceUpdateCallbackNoAction) {
options.env = env_;
options.write_buffer_size = 100000;
options.inplace_callback =
rocksdb::DBTestInPlaceUpdate::updateInPlaceNoAction;
ROCKSDB_NAMESPACE::DBTestInPlaceUpdate::updateInPlaceNoAction;
options.allow_concurrent_memtable_write = false;
Reopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
@@ -168,10 +168,10 @@ TEST_F(DBTestInPlaceUpdate, InPlaceUpdateCallbackNoAction) {
ASSERT_EQ(Get(1, "key"), "NOT_FOUND");
} while (ChangeCompactOptions());
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+21 -21
View File
@@ -10,7 +10,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBIOFailureTest : public DBTestBase {
public:
@@ -270,14 +270,14 @@ TEST_F(DBIOFailureTest, FlushSstRangeSyncError) {
Status s;
std::atomic<int> range_sync_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::RangeSync", [&](void* arg) {
if (range_sync_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
*st = Status::IOError("range sync dummy error");
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
std::string rnd_str =
@@ -302,7 +302,7 @@ TEST_F(DBIOFailureTest, FlushSstRangeSyncError) {
ASSERT_NOK(Put(1, "foo2", "bar3"));
ASSERT_EQ("bar", Get(1, "foo"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_GE(1, range_sync_called.load());
ReopenWithColumnFamilies({"default", "pikachu"}, options);
@@ -350,14 +350,14 @@ TEST_F(DBIOFailureTest, CompactSstRangeSyncError) {
dbfull()->TEST_WaitForFlushMemTable(handles_[1]);
std::atomic<int> range_sync_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::RangeSync", [&](void* arg) {
if (range_sync_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
*st = Status::IOError("range sync dummy error");
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->SetOptions(handles_[1],
{
@@ -369,7 +369,7 @@ TEST_F(DBIOFailureTest, CompactSstRangeSyncError) {
ASSERT_NOK(Put(1, "foo2", "bar3"));
ASSERT_EQ("bar", Get(1, "foo"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_GE(1, range_sync_called.load());
ReopenWithColumnFamilies({"default", "pikachu"}, options);
@@ -389,7 +389,7 @@ TEST_F(DBIOFailureTest, FlushSstCloseError) {
CreateAndReopenWithCF({"pikachu"}, options);
Status s;
std::atomic<int> close_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::Close", [&](void* arg) {
if (close_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
@@ -397,7 +397,7 @@ TEST_F(DBIOFailureTest, FlushSstCloseError) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(1, "foo", "bar"));
ASSERT_OK(Put(1, "foo1", "bar1"));
@@ -409,7 +409,7 @@ TEST_F(DBIOFailureTest, FlushSstCloseError) {
ASSERT_EQ("bar2", Get(1, "foo"));
ASSERT_EQ("bar1", Get(1, "foo1"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_EQ("bar2", Get(1, "foo"));
@@ -441,7 +441,7 @@ TEST_F(DBIOFailureTest, CompactionSstCloseError) {
dbfull()->TEST_WaitForCompact();
std::atomic<int> close_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::Close", [&](void* arg) {
if (close_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
@@ -449,7 +449,7 @@ TEST_F(DBIOFailureTest, CompactionSstCloseError) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->SetOptions(handles_[1],
{
{"disable_auto_compactions", "false"},
@@ -460,7 +460,7 @@ TEST_F(DBIOFailureTest, CompactionSstCloseError) {
ASSERT_NOK(Put(1, "foo2", "bar3"));
ASSERT_EQ("bar3", Get(1, "foo"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_EQ("bar3", Get(1, "foo"));
@@ -480,7 +480,7 @@ TEST_F(DBIOFailureTest, FlushSstSyncError) {
CreateAndReopenWithCF({"pikachu"}, options);
Status s;
std::atomic<int> sync_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::Sync", [&](void* arg) {
if (sync_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
@@ -488,7 +488,7 @@ TEST_F(DBIOFailureTest, FlushSstSyncError) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put(1, "foo", "bar"));
ASSERT_OK(Put(1, "foo1", "bar1"));
@@ -500,7 +500,7 @@ TEST_F(DBIOFailureTest, FlushSstSyncError) {
ASSERT_EQ("bar2", Get(1, "foo"));
ASSERT_EQ("bar1", Get(1, "foo1"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_EQ("bar2", Get(1, "foo"));
@@ -533,7 +533,7 @@ TEST_F(DBIOFailureTest, CompactionSstSyncError) {
dbfull()->TEST_WaitForCompact();
std::atomic<int> sync_called(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SpecialEnv::SStableFile::Sync", [&](void* arg) {
if (sync_called.fetch_add(1) == 0) {
Status* st = static_cast<Status*>(arg);
@@ -541,7 +541,7 @@ TEST_F(DBIOFailureTest, CompactionSstSyncError) {
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->SetOptions(handles_[1],
{
{"disable_auto_compactions", "false"},
@@ -552,17 +552,17 @@ TEST_F(DBIOFailureTest, CompactionSstSyncError) {
ASSERT_NOK(Put(1, "foo2", "bar3"));
ASSERT_EQ("bar3", Get(1, "foo"));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ReopenWithColumnFamilies({"default", "pikachu"}, options);
ASSERT_EQ("bar3", Get(1, "foo"));
}
#endif // !(defined NDEBUG) || !defined(OS_WIN)
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+33 -49
View File
@@ -31,20 +31,7 @@
#include "util/string_util.h"
#include "util/user_comparator_wrapper.h"
namespace rocksdb {
#if 0
static void DumpInternalIter(Iterator* iter) {
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ParsedInternalKey k;
if (!ParseInternalKey(iter->key(), &k)) {
fprintf(stderr, "Corrupt '%s'\n", EscapeString(iter->key()).c_str());
} else {
fprintf(stderr, "@ '%s'\n", k.DebugString().c_str());
}
}
}
#endif
namespace ROCKSDB_NAMESPACE {
DBIter::DBIter(Env* _env, const ReadOptions& read_options,
const ImmutableCFOptions& cf_options,
@@ -73,7 +60,9 @@ DBIter::DBIter(Env* _env, const ReadOptions& read_options,
? read_options.prefix_same_as_start
: false),
pin_thru_lifetime_(read_options.pin_data),
total_order_seek_(read_options.total_order_seek),
expect_total_order_inner_iter_(prefix_extractor_ == nullptr ||
read_options.total_order_seek ||
read_options.auto_prefix_mode),
allow_blob_(allow_blob),
is_blob_(false),
arena_mode_(arena_mode),
@@ -181,7 +170,7 @@ void DBIter::Next() {
// a delete marker or a sequence number higher than sequence_
// saved_key_ MUST have a proper user_key before calling this function
//
// The prefix parameter, if not null, indicates that we need to iterator
// The prefix parameter, if not null, indicates that we need to iterate
// within the prefix, and the iterator needs to be made invalid, if no
// more entry for the prefix can be found.
bool DBIter::FindNextUserEntry(bool skipping_saved_key, const Slice* prefix) {
@@ -202,13 +191,14 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
// or equal to saved_key_. We could skip these entries either because
// sequence numbers were too high or because skipping_saved_key = true.
// What saved_key_ contains throughout this method:
// - if skipping_saved_key : saved_key_ contains the key that we need
// to skip,
// and we haven't seen any keys greater than that,
// - if num_skipped > 0 : saved_key_ contains the key that we have skipped
// num_skipped times, and we haven't seen any keys
// greater than that,
// - none of the above : saved_key_ can contain anything, it doesn't matter.
// - if skipping_saved_key : saved_key_ contains the key that we need
// to skip, and we haven't seen any keys greater
// than that,
// - if num_skipped > 0 : saved_key_ contains the key that we have skipped
// num_skipped times, and we haven't seen any keys
// greater than that,
// - none of the above : saved_key_ can contain anything, it doesn't
// matter.
uint64_t num_skipped = 0;
// For write unprepared, the target sequence number in reseek could be larger
// than the snapshot, and thus needs to be skipped again. This could result in
@@ -290,9 +280,7 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
if (start_seqnum_ > 0) {
// we are taking incremental snapshot here
// incremental snapshots aren't supported on DB with range deletes
assert(!(
(ikey_.type == kTypeBlobIndex) && (start_seqnum_ > 0)
));
assert(ikey_.type != kTypeBlobIndex);
if (ikey_.sequence >= start_seqnum_) {
saved_key_.SetInternalKey(ikey_);
valid_ = true;
@@ -323,7 +311,7 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
valid_ = false;
return false;
}
@@ -370,7 +358,7 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
// to seek to the target sequence number.
int cmp =
user_comparator_.Compare(ikey_.user_key, saved_key_.GetUserKey());
if (cmp == 0 || (skipping_saved_key && cmp <= 0)) {
if (cmp == 0 || (skipping_saved_key && cmp < 0)) {
num_skipped++;
} else {
saved_key_.SetUserKey(
@@ -389,7 +377,7 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
// reseek previously.
//
// TODO(lth): If we reseek to sequence number greater than ikey_.sequence,
// than it does not make sense to reseek as we would actually land further
// then it does not make sense to reseek as we would actually land further
// away from the desired key. There is opportunity for optimization here.
if (num_skipped > max_skip_ && !reseek_done) {
is_key_seqnum_zero_ = false;
@@ -495,7 +483,7 @@ bool DBIter::MergeValuesNewToOld() {
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
} else {
status_ =
Status::NotSupported("Blob DB does not support merge operator.");
@@ -567,7 +555,7 @@ bool DBIter::ReverseToForward() {
// When moving backwards, iter_ is positioned on _previous_ key, which may
// not exist or may have different prefix than the current key().
// If that's the case, seek iter_ to current key.
if ((prefix_extractor_ != nullptr && !total_order_seek_) || !iter_.Valid()) {
if (!expect_total_order_inner_iter() || !iter_.Valid()) {
IterKey last_key;
last_key.SetInternalKey(ParsedInternalKey(
saved_key_.GetUserKey(), kMaxSequenceNumber, kValueTypeForSeek));
@@ -603,15 +591,14 @@ bool DBIter::ReverseToBackward() {
// key, which may not exist or may have prefix different from current.
// If that's the case, seek to saved_key_.
if (current_entry_is_merged_ &&
((prefix_extractor_ != nullptr && !total_order_seek_) ||
!iter_.Valid())) {
(!expect_total_order_inner_iter() || !iter_.Valid())) {
IterKey last_key;
// Using kMaxSequenceNumber and kValueTypeForSeek
// (not kValueTypeForSeekForPrev) to seek to a key strictly smaller
// than saved_key_.
last_key.SetInternalKey(ParsedInternalKey(
saved_key_.GetUserKey(), kMaxSequenceNumber, kValueTypeForSeek));
if (prefix_extractor_ != nullptr && !total_order_seek_) {
if (!expect_total_order_inner_iter()) {
iter_.SeekForPrev(last_key.GetInternalKey());
} else {
// Some iterators may not support SeekForPrev(), so we avoid using it
@@ -797,7 +784,7 @@ bool DBIter::FindValueForCurrentKey() {
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
} else {
status_ =
Status::NotSupported("Blob DB does not support merge operator.");
@@ -820,7 +807,7 @@ bool DBIter::FindValueForCurrentKey() {
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
valid_ = false;
return false;
}
@@ -891,7 +878,7 @@ bool DBIter::FindValueForCurrentKeyUsingSeek() {
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
valid_ = false;
return false;
}
@@ -953,7 +940,7 @@ bool DBIter::FindValueForCurrentKeyUsingSeek() {
ROCKS_LOG_ERROR(logger_, "Encounter unexpected blob index.");
status_ = Status::NotSupported(
"Encounter unexpected blob index. Please open DB with "
"rocksdb::blob_db::BlobDB instead.");
"ROCKSDB_NAMESPACE::blob_db::BlobDB instead.");
} else {
status_ =
Status::NotSupported("Blob DB does not support merge operator.");
@@ -978,8 +965,8 @@ bool DBIter::FindValueForCurrentKeyUsingSeek() {
// Make sure we leave iter_ in a good state. If it's valid and we don't care
// about prefixes, that's already good enough. Otherwise it needs to be
// seeked to the current key.
if ((prefix_extractor_ != nullptr && !total_order_seek_) || !iter_.Valid()) {
if (prefix_extractor_ != nullptr && !total_order_seek_) {
if (!expect_total_order_inner_iter() || !iter_.Valid()) {
if (!expect_total_order_inner_iter()) {
iter_.SeekForPrev(last_key);
} else {
iter_.Seek(last_key);
@@ -1129,18 +1116,16 @@ void DBIter::Seek(const Slice& target) {
// Now the inner iterator is placed to the target position. From there,
// we need to find out the next key that is visible to the user.
//
ClearSavedValue();
if (prefix_same_as_start_) {
// The case where the iterator needs to be invalidated if it has exausted
// keys within the same prefix of the seek key.
assert(prefix_extractor_ != nullptr);
Slice target_prefix;
target_prefix = prefix_extractor_->Transform(target);
Slice target_prefix = prefix_extractor_->Transform(target);
FindNextUserEntry(false /* not skipping saved_key */,
&target_prefix /* prefix */);
if (valid_) {
// Remember the prefix of the seek key for the future Prev() call to
// Remember the prefix of the seek key for the future Next() call to
// check.
prefix_.SetUserKey(target_prefix);
}
@@ -1196,8 +1181,7 @@ void DBIter::SeekForPrev(const Slice& target) {
// The case where the iterator needs to be invalidated if it has exausted
// keys within the same prefix of the seek key.
assert(prefix_extractor_ != nullptr);
Slice target_prefix;
target_prefix = prefix_extractor_->Transform(target);
Slice target_prefix = prefix_extractor_->Transform(target);
PrevInternal(&target_prefix);
if (valid_) {
// Remember the prefix of the seek key for the future Prev() call to
@@ -1224,7 +1208,7 @@ void DBIter::SeekToFirst() {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, env_);
// Don't use iter_::Seek() if we set a prefix extractor
// because prefix seek will be used.
if (prefix_extractor_ != nullptr && !total_order_seek_) {
if (!expect_total_order_inner_iter()) {
max_skip_ = std::numeric_limits<uint64_t>::max();
}
status_ = Status::OK();
@@ -1277,7 +1261,7 @@ void DBIter::SeekToLast() {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, env_);
// Don't use iter_::Seek() if we set a prefix extractor
// because prefix seek will be used.
if (prefix_extractor_ != nullptr && !total_order_seek_) {
if (!expect_total_order_inner_iter()) {
max_skip_ = std::numeric_limits<uint64_t>::max();
}
status_ = Status::OK();
@@ -1323,4 +1307,4 @@ Iterator* NewDBIterator(Env* env, const ReadOptions& read_options,
return db_iter;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+19 -13
View File
@@ -20,15 +20,15 @@
#include "table/iterator_wrapper.h"
#include "util/autovector.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// This file declares the factory functions of DBIter, in its original form
// or a wrapped form with class ArenaWrappedDBIter, which is defined here.
// Class DBIter, which is declared and implemented inside db_iter.cc, is
// a iterator that converts internal keys (yielded by an InternalIterator)
// an iterator that converts internal keys (yielded by an InternalIterator)
// that were live at the specified sequence number into appropriate user
// keys.
// Each internal key is consist of a user key, a sequence number, and a value
// Each internal key consists of a user key, a sequence number, and a value
// type. DBIter deals with multiple key versions, tombstones, merge operands,
// etc, and exposes an Iterator.
// For example, DBIter may wrap following InternalIterator:
@@ -133,14 +133,12 @@ class DBIter final : public Iterator {
local_stats_.BumpGlobalStatistics(statistics_);
iter_.DeleteIter(arena_mode_);
}
virtual void SetIter(InternalIterator* iter) {
void SetIter(InternalIterator* iter) {
assert(iter_.iter() == nullptr);
iter_.Set(iter);
iter_.iter()->SetPinnedItersMgr(&pinned_iters_mgr_);
}
virtual ReadRangeDelAggregator* GetRangeDelAggregator() {
return &range_del_agg_;
}
ReadRangeDelAggregator* GetRangeDelAggregator() { return &range_del_agg_; }
bool Valid() const override { return valid_; }
Slice key() const override {
@@ -184,7 +182,7 @@ class DBIter final : public Iterator {
void SeekForPrev(const Slice& target) final override;
void SeekToFirst() final override;
void SeekToLast() final override;
Env* env() { return env_; }
Env* env() const { return env_; }
void set_sequence(uint64_t s) {
sequence_ = s;
if (read_callback_) {
@@ -202,10 +200,10 @@ class DBIter final : public Iterator {
bool ReverseToBackward();
// Set saved_key_ to the seek key to target, with proper sequence number set.
// It might get adjusted if the seek key is smaller than iterator lower bound.
void SetSavedKeyToSeekTarget(const Slice& /*target*/);
void SetSavedKeyToSeekTarget(const Slice& target);
// Set saved_key_ to the seek key to target, with proper sequence number set.
// It might get adjusted if the seek key is larger than iterator upper bound.
void SetSavedKeyToSeekForPrevTarget(const Slice& /*target*/);
void SetSavedKeyToSeekForPrevTarget(const Slice& target);
bool FindValueForCurrentKey();
bool FindValueForCurrentKeyUsingSeek();
bool FindUserKeyBeforeSavedKey();
@@ -221,7 +219,7 @@ class DBIter final : public Iterator {
// If prefix is not null, we need to set the iterator to invalid if no more
// entry can be found within the prefix.
void PrevInternal(const Slice* /*prefix*/);
void PrevInternal(const Slice* prefix);
bool TooManyInternalKeysSkipped(bool increment = true);
bool IsVisible(SequenceNumber sequence);
@@ -257,6 +255,11 @@ class DBIter final : public Iterator {
num_internal_keys_skipped_ = 0;
}
bool expect_total_order_inner_iter() {
assert(expect_total_order_inner_iter_ || prefix_extractor_ != nullptr);
return expect_total_order_inner_iter_;
}
const SliceTransform* prefix_extractor_;
Env* const env_;
Logger* logger_;
@@ -302,7 +305,9 @@ class DBIter final : public Iterator {
// Means that we will pin all data blocks we read as long the Iterator
// is not deleted, will be true if ReadOptions::pin_data is true
const bool pin_thru_lifetime_;
const bool total_order_seek_;
// Expect the inner iterator to maintain a total order.
// prefix_extractor_ must be non-NULL if the value is false.
const bool expect_total_order_inner_iter_;
bool allow_blob_;
bool is_blob_;
bool arena_mode_;
@@ -323,6 +328,7 @@ class DBIter final : public Iterator {
// if this value > 0 iterator will return internal keys
SequenceNumber start_seqnum_;
};
// Return a new iterator that converts internal keys (yielded by
// "*internal_iter") that were live at the specified `sequence` number
// into appropriate user keys.
@@ -335,4 +341,4 @@ extern Iterator* NewDBIterator(
ReadCallback* read_callback, DBImpl* db_impl = nullptr,
ColumnFamilyData* cfd = nullptr, bool allow_blob = false);
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
+2 -2
View File
@@ -29,7 +29,7 @@ bool FLAGS_verbose = false;
#endif
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBIteratorStressTest : public testing::Test {
public:
@@ -645,7 +645,7 @@ TEST_F(DBIteratorStressTest, StressTest) {
<< "\n mutated on the fly: " << num_recently_removed << std::endl;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+30 -30
View File
@@ -22,7 +22,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
static uint64_t TestGetTickerCount(const Options& options,
Tickers ticker_type) {
@@ -691,7 +691,7 @@ TEST_F(DBIteratorTest, DBIteratorEmpty) {
TEST_F(DBIteratorTest, DBIteratorUseSkipCountSkips) {
ReadOptions ro;
Options options;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.merge_operator = MergeOperators::CreateFromStringId("stringappend");
TestIterator* internal_iter = new TestIterator(BytewiseComparator());
@@ -746,7 +746,7 @@ TEST_F(DBIteratorTest, DBIteratorUseSkip) {
}
internal_iter->Finish();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
std::unique_ptr<Iterator> db_iter(NewDBIterator(
env_, ro, cf_options, mutable_cf_options, BytewiseComparator(),
internal_iter, i + 2, options.max_sequential_skip_in_iterations,
@@ -2481,7 +2481,7 @@ TEST_F(DBIteratorTest, DBIteratorTestDifferentialSnapshots) {
ReadOptions ro;
ro.iter_start_seqnum=5;
Options options;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
TestIterator* internal_iter = new TestIterator(BytewiseComparator());
for (size_t i = 0; i < 10; ++i) {
@@ -2516,7 +2516,7 @@ TEST_F(DBIteratorTest, DBIteratorTestDifferentialSnapshots) {
ReadOptions ro;
ro.iter_start_seqnum=5;
Options options;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
TestIterator* internal_iter = new TestIterator(BytewiseComparator());
for (size_t i = 0; i < 10; ++i) {
@@ -2661,10 +2661,10 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace1) {
// Test call back inserts a key in the end of the mem table after
// MergeIterator::Prev() realized the mem table iterator is at its end
// and before an SeekToLast() is called.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev",
[&](void* /*arg*/) { internal_iter2_->Add("z", kTypeValue, "7", 12u); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2683,7 +2683,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace1) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace2) {
@@ -2697,12 +2697,12 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace2) {
// Test call back inserts entries for update a key in the end of the
// mem table after MergeIterator::Prev() realized the mem tableiterator is at
// its end and before an SeekToLast() is called.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* /*arg*/) {
internal_iter2_->Add("z", kTypeValue, "7", 12u);
internal_iter2_->Add("z", kTypeValue, "7", 11u);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2721,7 +2721,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace2) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace3) {
@@ -2735,7 +2735,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace3) {
// Test call back inserts entries for update a key in the end of the
// mem table after MergeIterator::Prev() realized the mem table iterator is at
// its end and before an SeekToLast() is called.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* /*arg*/) {
internal_iter2_->Add("z", kTypeValue, "7", 16u, true);
internal_iter2_->Add("z", kTypeValue, "7", 15u, true);
@@ -2744,7 +2744,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace3) {
internal_iter2_->Add("z", kTypeValue, "7", 12u, true);
internal_iter2_->Add("z", kTypeValue, "7", 11u, true);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2763,7 +2763,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace3) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace4) {
@@ -2779,7 +2779,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace4) {
// Test call back inserts entries for update a key before "z" in
// mem table after MergeIterator::Prev() calls mem table iterator's
// Seek() and before calling Prev()
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* arg) {
IteratorWrapper* it = reinterpret_cast<IteratorWrapper*>(arg);
if (it->key().starts_with("z")) {
@@ -2791,7 +2791,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace4) {
internal_iter2_->Add("x", kTypeValue, "7", 11u, true);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2814,7 +2814,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace4) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace5) {
@@ -2830,7 +2830,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace5) {
// Test call back inserts entries for update a key before "z" in
// mem table after MergeIterator::Prev() calls mem table iterator's
// Seek() and before calling Prev()
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* arg) {
IteratorWrapper* it = reinterpret_cast<IteratorWrapper*>(arg);
if (it->key().starts_with("z")) {
@@ -2838,7 +2838,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace5) {
internal_iter2_->Add("x", kTypeValue, "7", 15u, true);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2861,7 +2861,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace5) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace6) {
@@ -2877,14 +2877,14 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace6) {
// Test call back inserts an entry for update a key before "z" in
// mem table after MergeIterator::Prev() calls mem table iterator's
// Seek() and before calling Prev()
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* arg) {
IteratorWrapper* it = reinterpret_cast<IteratorWrapper*>(arg);
if (it->key().starts_with("z")) {
internal_iter2_->Add("x", kTypeValue, "7", 16u, true);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2907,7 +2907,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace6) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace7) {
@@ -2926,7 +2926,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace7) {
// Test call back inserts entries for update a key before "z" in
// mem table after MergeIterator::Prev() calls mem table iterator's
// Seek() and before calling Prev()
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* arg) {
IteratorWrapper* it = reinterpret_cast<IteratorWrapper*>(arg);
if (it->key().starts_with("z")) {
@@ -2938,7 +2938,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace7) {
internal_iter2_->Add("x", kTypeValue, "7", 11u, true);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2961,7 +2961,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace7) {
ASSERT_EQ(db_iter_->key().ToString(), "a");
ASSERT_EQ(db_iter_->value().ToString(), "4");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace8) {
@@ -2979,7 +2979,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace8) {
// Test call back inserts two keys before "z" in mem table after
// MergeIterator::Prev() calls mem table iterator's Seek() and
// before calling Prev()
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"MergeIterator::Prev:BeforePrev", [&](void* arg) {
IteratorWrapper* it = reinterpret_cast<IteratorWrapper*>(arg);
if (it->key().starts_with("z")) {
@@ -2987,7 +2987,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace8) {
internal_iter2_->Add("y", kTypeValue, "7", 17u, true);
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
db_iter_->Prev();
ASSERT_TRUE(db_iter_->Valid());
@@ -2998,7 +2998,7 @@ TEST_F(DBIterWithMergeIterTest, InnerMergeIteratorDataRace8) {
ASSERT_EQ(db_iter_->key().ToString(), "d");
ASSERT_EQ(db_iter_->value().ToString(), "7");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3167,7 +3167,7 @@ TEST_F(DBIteratorTest, ReverseToForwardWithDisappearingKeys) {
EXPECT_LT(internal_iter->steps(), 20);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
+14 -14
View File
@@ -18,7 +18,7 @@
#include "rocksdb/perf_context.h"
#include "table/block_based/flush_block_policy.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// A dumb ReadCallback which saying every key is committed.
class DummyReadCallback : public ReadCallback {
@@ -109,7 +109,7 @@ TEST_P(DBIteratorTest, NonBlockingIteration) {
do {
ReadOptions non_blocking_opts, regular_opts;
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
non_blocking_opts.read_tier = kBlockCacheTier;
CreateAndReopenWithCF({"pikachu"}, options);
// write one kv to the database.
@@ -535,7 +535,7 @@ TEST_P(DBIteratorTest, IterReseek) {
Options options = CurrentOptions(options_override);
options.max_sequential_skip_in_iterations = 3;
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
CreateAndReopenWithCF({"pikachu"}, options);
@@ -1026,7 +1026,7 @@ TEST_P(DBIteratorTest, DBIteratorBoundMultiSeek) {
Options options = CurrentOptions();
options.env = env_;
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.prefix_extractor = nullptr;
DestroyAndReopen(options);
ASSERT_OK(Put("a", "0"));
@@ -1081,10 +1081,10 @@ TEST_P(DBIteratorTest, DBIteratorBoundOptimizationTest) {
for (auto format_version : {2, 3, 4}) {
int upper_bound_hits = 0;
Options options = CurrentOptions();
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"BlockBasedTableIterator:out_of_bound",
[&upper_bound_hits](void*) { upper_bound_hits++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
options.env = env_;
options.create_if_missing = true;
options.prefix_extractor = nullptr;
@@ -1132,7 +1132,7 @@ TEST_P(DBIteratorTest, IndexWithFirstKey) {
options.create_if_missing = true;
options.prefix_extractor = nullptr;
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
Statistics* stats = options.statistics.get();
BlockBasedTableOptions table_options;
table_options.index_type =
@@ -1223,7 +1223,7 @@ TEST_P(DBIteratorTest, IndexWithFirstKeyGet) {
options.create_if_missing = true;
options.prefix_extractor = nullptr;
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
Statistics* stats = options.statistics.get();
BlockBasedTableOptions table_options;
table_options.index_type =
@@ -2012,7 +2012,7 @@ TEST_P(DBIteratorTest, IterPrevKeyCrossingBlocksRandomized) {
TEST_P(DBIteratorTest, IteratorWithLocalStatistics) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
Random rnd(301);
@@ -2110,7 +2110,7 @@ TEST_P(DBIteratorTest, ReadAhead) {
options.env = env_;
options.disable_auto_compactions = true;
options.write_buffer_size = 4 << 20;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.block_size = 1024;
table_options.no_block_cache = true;
@@ -2188,7 +2188,7 @@ TEST_P(DBIteratorTest, DBIteratorSkipRecentDuplicatesTest) {
options.max_sequential_skip_in_iterations = 3;
options.prefix_extractor = nullptr;
options.write_buffer_size = 1 << 27; // big enough to avoid flush
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
// Insert.
@@ -2480,7 +2480,7 @@ TEST_P(DBIteratorTest, UpperBoundWithPrevReseek) {
TEST_P(DBIteratorTest, SkipStatistics) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
int skip_count = 0;
@@ -2989,10 +2989,10 @@ TEST_F(DBIteratorWithReadCallbackTest, ReadCallback) {
delete iter;
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+11 -11
View File
@@ -15,7 +15,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBTestXactLogIterator : public DBTestBase {
public:
@@ -98,14 +98,14 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorRace) {
for (int test = 0; test < LOG_ITERATOR_RACE_TEST_COUNT; ++test) {
// Setup sync point dependency to reproduce the race condition of
// a log file moved to archived dir, in the middle of GetSortedWalFiles
rocksdb::SyncPoint::GetInstance()->LoadDependency(
{ { sync_points[test][0], sync_points[test][1] },
{ sync_points[test][2], sync_points[test][3] },
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{sync_points[test][0], sync_points[test][1]},
{sync_points[test][2], sync_points[test][3]},
});
do {
rocksdb::SyncPoint::GetInstance()->ClearTrace();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Options options = OptionsForLogIterTest();
DestroyAndReopen(options);
Put("key1", DummyString(1024));
@@ -123,7 +123,7 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorRace) {
ExpectRecords(4, iter);
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// trigger async flush, and log move. Well, log move will
// wait until the GetSortedWalFiles:1 to reproduce the race
// condition
@@ -185,7 +185,7 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorCorruptedLog) {
dbfull()->Flush(FlushOptions());
dbfull()->FlushWAL(false);
// Corrupt this log to create a gap
rocksdb::VectorLogPtr wal_files;
ROCKSDB_NAMESPACE::VectorLogPtr wal_files;
ASSERT_OK(dbfull()->GetSortedWalFiles(wal_files));
const auto logfile_path = dbname_ + "/" + wal_files.front()->PathName();
if (mem_env_) {
@@ -277,13 +277,13 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBlobs) {
"Delete(0, key2)",
handler.seen);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
+7 -7
View File
@@ -13,7 +13,7 @@
#include "rocksdb/memtablerep.h"
#include "rocksdb/slice_transform.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBMemTableTest : public DBTestBase {
public:
@@ -232,8 +232,8 @@ TEST_F(DBMemTableTest, ConcurrentMergeWrite) {
value.clear();
// Write Merge concurrently
rocksdb::port::Thread write_thread1([&]() {
MemTablePostProcessInfo post_process_info1;
ROCKSDB_NAMESPACE::port::Thread write_thread1([&]() {
MemTablePostProcessInfo post_process_info1;
std::string v1;
for (int seq = 1; seq < num_ops / 2; seq++) {
PutFixed64(&v1, seq);
@@ -243,8 +243,8 @@ TEST_F(DBMemTableTest, ConcurrentMergeWrite) {
v1.clear();
}
});
rocksdb::port::Thread write_thread2([&]() {
MemTablePostProcessInfo post_process_info2;
ROCKSDB_NAMESPACE::port::Thread write_thread2([&]() {
MemTablePostProcessInfo post_process_info2;
std::string v2;
for (int seq = num_ops / 2; seq < num_ops; seq++) {
PutFixed64(&v2, seq);
@@ -331,10 +331,10 @@ TEST_F(DBMemTableTest, ColumnFamilyId) {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+3 -3
View File
@@ -17,7 +17,7 @@
#include "utilities/merge_operators/sortlist.h"
#include "utilities/merge_operators/string_append/stringappend2.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBMergeOperandTest : public DBTestBase {
public:
@@ -231,10 +231,10 @@ TEST_F(DBMergeOperandTest, GetMergeOperandsBasic) {
ASSERT_EQ(values[2], "am");
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+8 -10
View File
@@ -12,7 +12,7 @@
#include "utilities/merge_operators.h"
#include "utilities/merge_operators/string_append/stringappend2.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class TestReadCallback : public ReadCallback {
public:
@@ -150,8 +150,6 @@ TEST_F(DBMergeOperatorTest, MergeErrorOnWrite) {
options.create_if_missing = true;
options.merge_operator.reset(new TestPutOperator());
options.max_successive_merges = 3;
// allow_concurrent_memtable_write is incompatible with max_successive_merges
options.allow_concurrent_memtable_write = false;
options.env = env_;
Reopen(options);
ASSERT_OK(Merge("k1", "v1"));
@@ -420,8 +418,8 @@ TEST_P(MergeOperatorPinningTest, TailingIterator) {
delete iter;
};
rocksdb::port::Thread writer_thread(writer_func);
rocksdb::port::Thread reader_thread(reader_func);
ROCKSDB_NAMESPACE::port::Thread writer_thread(writer_func);
ROCKSDB_NAMESPACE::port::Thread reader_thread(reader_func);
writer_thread.join();
reader_thread.join();
@@ -458,19 +456,19 @@ TEST_F(DBMergeOperatorTest, TailingIteratorMemtableUnrefedBySomeoneElse) {
bool pushed_first_operand = false;
bool stepped_to_next_operand = false;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBIter::MergeValuesNewToOld:PushedFirstOperand", [&](void*) {
EXPECT_FALSE(pushed_first_operand);
pushed_first_operand = true;
db_->Flush(FlushOptions()); // Switch to SuperVersion B
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBIter::MergeValuesNewToOld:SteppedToNextOperand", [&](void*) {
EXPECT_FALSE(stepped_to_next_operand);
stepped_to_next_operand = true;
someone_else.reset(); // Unpin SuperVersion A
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ReadOptions ro;
ro.tailing = true;
@@ -659,10 +657,10 @@ TEST_P(PerConfigMergeOperatorPinningTest, Randomized) {
VerifyDBFromMap(true_data);
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+25 -23
View File
@@ -23,7 +23,7 @@
#include "test_util/testutil.h"
#include "util/random.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBOptionsTest : public DBTestBase {
public:
@@ -134,19 +134,17 @@ TEST_F(DBOptionsTest, SetBytesPerSync) {
int i = 0;
const std::string kValue(kValueSize, 'v');
ASSERT_EQ(options.bytes_per_sync, dbfull()->GetDBOptions().bytes_per_sync);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::RangeSync:0", [&](void* /*arg*/) {
counter++;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::RangeSync:0", [&](void* /*arg*/) { counter++; });
WriteOptions write_opts;
// should sync approximately 40MB/1MB ~= 40 times.
for (i = 0; i < 40; i++) {
Put(Key(i), kValue, write_opts);
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
low_bytes_per_sync = counter;
ASSERT_GT(low_bytes_per_sync, 35);
ASSERT_LT(low_bytes_per_sync, 45);
@@ -160,7 +158,7 @@ TEST_F(DBOptionsTest, SetBytesPerSync) {
for (i = 0; i < 40; i++) {
Put(Key(i), kValue, write_opts);
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
ASSERT_GT(counter, 5);
ASSERT_LT(counter, 15);
@@ -183,11 +181,9 @@ TEST_F(DBOptionsTest, SetWalBytesPerSync) {
ASSERT_EQ(512, dbfull()->GetDBOptions().wal_bytes_per_sync);
int counter = 0;
int low_bytes_per_sync = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::RangeSync:0", [&](void* /*arg*/) {
counter++;
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::RangeSync:0", [&](void* /*arg*/) { counter++; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
const std::string kValue(kValueSize, 'v');
int i = 0;
for (; i < 10; i++) {
@@ -223,7 +219,7 @@ TEST_F(DBOptionsTest, WritableFileMaxBufferSize) {
std::atomic<int> match_cnt(0);
std::atomic<int> unmatch_cnt(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"WritableFileWriter::WritableFileWriter:0", [&](void* arg) {
int value = static_cast<int>(reinterpret_cast<uintptr_t>(arg));
if (value == buffer_size) {
@@ -232,7 +228,7 @@ TEST_F(DBOptionsTest, WritableFileMaxBufferSize) {
unmatch_cnt++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
int i = 0;
for (; i < 3; i++) {
ASSERT_OK(Put("foo", ToString(i)));
@@ -423,16 +419,22 @@ TEST_F(DBOptionsTest, SetBackgroundJobs) {
std::to_string(options.max_background_jobs)}}));
}
ASSERT_EQ(options.max_background_jobs / 4,
dbfull()->TEST_BGFlushesAllowed());
const int expected_max_flushes = options.max_background_jobs / 4;
ASSERT_EQ(expected_max_flushes, dbfull()->TEST_BGFlushesAllowed());
ASSERT_EQ(1, dbfull()->TEST_BGCompactionsAllowed());
auto stop_token = dbfull()->TEST_write_controler().GetStopToken();
ASSERT_EQ(options.max_background_jobs / 4,
dbfull()->TEST_BGFlushesAllowed());
ASSERT_EQ(3 * options.max_background_jobs / 4,
dbfull()->TEST_BGCompactionsAllowed());
const int expected_max_compactions = 3 * expected_max_flushes;
ASSERT_EQ(expected_max_flushes, dbfull()->TEST_BGFlushesAllowed());
ASSERT_EQ(expected_max_compactions, dbfull()->TEST_BGCompactionsAllowed());
ASSERT_EQ(expected_max_flushes,
env_->GetBackgroundThreads(Env::Priority::HIGH));
ASSERT_EQ(expected_max_compactions,
env_->GetBackgroundThreads(Env::Priority::LOW));
}
}
@@ -859,10 +861,10 @@ TEST_F(DBOptionsTest, FIFOTtlBackwardCompatible) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+7 -7
View File
@@ -22,7 +22,7 @@
#include "util/random.h"
#include "util/string_util.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBPropertiesTest : public DBTestBase {
public:
@@ -391,16 +391,16 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
options.max_bytes_for_level_base = 4500 << 10;
options.target_file_size_base = 98 << 10;
options.max_write_buffer_number = 2;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.max_open_files = 11; // Make sure no proloading of table readers
// RocksDB sanitize max open files to at least 20. Modify it back.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = static_cast<int*>(arg);
*max_open_files = 11;
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
BlockBasedTableOptions table_options;
table_options.no_block_cache = true;
@@ -448,7 +448,7 @@ TEST_F(DBPropertiesTest, ReadLatencyHistogramByLevel) {
// Reopen and issue iterating. See thee latency tracked
ReopenWithColumnFamilies({"default", "pikachu"}, options);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_TRUE(dbfull()->GetProperty("rocksdb.cf-file-histogram", &prop));
ASSERT_EQ(std::string::npos, prop.find("** Level 0 read latency histogram"));
ASSERT_EQ(std::string::npos, prop.find("** Level 1 read latency histogram"));
@@ -1702,10 +1702,10 @@ TEST_F(DBPropertiesTest, BlockCacheProperties) {
}
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+17 -8
View File
@@ -5,10 +5,11 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "rocksdb/utilities/write_batch_with_index.h"
#include "test_util/testutil.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBRangeDelTest : public DBTestBase {
public:
@@ -23,8 +24,8 @@ class DBRangeDelTest : public DBTestBase {
}
};
// PlainTableFactory and NumTableFilesAtLevel() are not supported in
// ROCKSDB_LITE
// PlainTableFactory, WriteBatchWithIndex, and NumTableFilesAtLevel() are not
// supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, NonBlockBasedTableNotSupported) {
// TODO: figure out why MmapReads trips the iterator pinning assertion in
@@ -39,6 +40,13 @@ TEST_F(DBRangeDelTest, NonBlockBasedTableNotSupported) {
}
}
TEST_F(DBRangeDelTest, WriteBatchWithIndexNotSupported) {
WriteBatchWithIndex indexedBatch{};
ASSERT_TRUE(indexedBatch.DeleteRange(db_->DefaultColumnFamily(), "dr1", "dr1")
.IsNotSupported());
ASSERT_TRUE(indexedBatch.DeleteRange("dr1", "dr1").IsNotSupported());
}
TEST_F(DBRangeDelTest, FlushOutputHasOnlyRangeTombstones) {
do {
DestroyAndReopen(CurrentOptions());
@@ -917,7 +925,8 @@ TEST_F(DBRangeDelTest, MemtableBloomFilter) {
Options options = CurrentOptions();
options.memtable_prefix_bloom_size_ratio =
static_cast<double>(kMemtablePrefixFilterSize) / kMemtableSize;
options.prefix_extractor.reset(rocksdb::NewFixedPrefixTransform(kPrefixLen));
options.prefix_extractor.reset(
ROCKSDB_NAMESPACE::NewFixedPrefixTransform(kPrefixLen));
options.write_buffer_size = kMemtableSize;
Reopen(options);
@@ -1070,7 +1079,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
// endpoint (key000002#6,1) to disappear.
ASSERT_EQ(value, Get(Key(2)));
auto begin_str = Key(3);
const rocksdb::Slice begin = begin_str;
const ROCKSDB_NAMESPACE::Slice begin = begin_str;
dbfull()->TEST_CompactRange(1, &begin, nullptr);
ASSERT_EQ(1, NumTableFilesAtLevel(1));
ASSERT_EQ(2, NumTableFilesAtLevel(2));
@@ -1089,7 +1098,7 @@ TEST_F(DBRangeDelTest, RangeTombstoneEndKeyAsSstableUpperBound) {
// [key000001#5,1, key000002#72057594037927935,15]
// [key000002#6,1, key000004#72057594037927935,15]
auto begin_str = Key(0);
const rocksdb::Slice begin = begin_str;
const ROCKSDB_NAMESPACE::Slice begin = begin_str;
dbfull()->TEST_CompactRange(1, &begin, &begin);
ASSERT_EQ(0, NumTableFilesAtLevel(1));
ASSERT_EQ(3, NumTableFilesAtLevel(2));
@@ -1642,10 +1651,10 @@ TEST_F(DBRangeDelTest, OverlappedKeys) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+63 -41
View File
@@ -13,7 +13,7 @@
#include "port/stack_trace.h"
#include "rocksdb/sst_file_manager.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBSSTTest : public DBTestBase {
public:
@@ -102,6 +102,14 @@ TEST_F(DBSSTTest, SSTsWithLdbSuffixHandling) {
int const num_files = GetSstFileCount(dbname_);
ASSERT_GT(num_files, 0);
Reopen(options);
std::vector<std::string> values;
values.reserve(key_id);
for (int k = 0; k < key_id; ++k) {
values.push_back(Get(Key(k)));
}
Close();
std::vector<std::string> filenames;
GetSstFiles(env_, dbname_, &filenames);
int num_ldb_files = 0;
@@ -119,11 +127,25 @@ TEST_F(DBSSTTest, SSTsWithLdbSuffixHandling) {
Reopen(options);
for (int k = 0; k < key_id; ++k) {
ASSERT_NE("NOT_FOUND", Get(Key(k)));
ASSERT_EQ(values[k], Get(Key(k)));
}
Destroy(options);
}
// Check that we don't crash when opening DB with
// DBOptions::skip_checking_sst_file_sizes_on_db_open = true.
TEST_F(DBSSTTest, SkipCheckingSSTFileSizesOnDBOpen) {
ASSERT_OK(Put("pika", "choo"));
ASSERT_OK(Flush());
// Just open the DB with the option set to true and check that we don't crash.
Options options;
options.skip_checking_sst_file_sizes_on_db_open = true;
Reopen(options);
ASSERT_EQ("choo", Get("pika"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBSSTTest, DontDeleteMovedFile) {
// This test triggers move compaction and verifies that the file is not
@@ -261,13 +283,14 @@ TEST_F(DBSSTTest, DBWithSstFileManager) {
int files_added = 0;
int files_deleted = 0;
int files_moved = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SstFileManagerImpl::OnAddFile", [&](void* /*arg*/) { files_added++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"SstFileManagerImpl::OnDeleteFile", [&](void* /*arg*/) { files_deleted++; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SstFileManagerImpl::OnDeleteFile",
[&](void* /*arg*/) { files_deleted++; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SstFileManagerImpl::OnMoveFile", [&](void* /*arg*/) { files_moved++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Options options = CurrentOptions();
options.sst_file_manager = sst_file_manager;
@@ -315,21 +338,21 @@ TEST_F(DBSSTTest, DBWithSstFileManager) {
ASSERT_EQ(sfm->GetTrackedFiles(), files_in_db);
ASSERT_EQ(sfm->GetTotalSize(), total_files_size);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, RateLimitedDelete) {
Destroy(last_options_);
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"DBSSTTest::RateLimitedDelete:1",
"DeleteScheduler::BackgroundEmptyTrash"},
});
std::vector<uint64_t> penalties;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::BackgroundEmptyTrash:Wait",
[&](void* arg) { penalties.push_back(*(static_cast<uint64_t*>(arg))); });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"InstrumentedCondVar::TimedWaitInternal", [&](void* arg) {
// Turn timed wait into a simulated sleep
uint64_t* abs_time_us = static_cast<uint64_t*>(arg);
@@ -350,7 +373,7 @@ TEST_F(DBSSTTest, RateLimitedDelete) {
*abs_time_us = static_cast<uint64_t>(real_cur_time);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
env_->no_slowdown_ = true;
env_->time_elapse_only_sleep_ = true;
@@ -415,14 +438,14 @@ TEST_F(DBSSTTest, RateLimitedDelete) {
ASSERT_GT(time_spent_deleting, expected_penlty * 0.9);
ASSERT_LT(time_spent_deleting, expected_penlty * 1.1);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, RateLimitedWALDelete) {
Destroy(last_options_);
std::vector<uint64_t> penalties;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::BackgroundEmptyTrash:Wait",
[&](void* arg) { penalties.push_back(*(static_cast<uint64_t*>(arg))); });
@@ -443,7 +466,7 @@ TEST_F(DBSSTTest, RateLimitedWALDelete) {
sfm->delete_scheduler()->SetMaxTrashDBRatio(3.1);
ASSERT_OK(TryReopen(options));
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Create 4 files in L0
for (char v = 'a'; v <= 'd'; v++) {
@@ -467,7 +490,7 @@ TEST_F(DBSSTTest, RateLimitedWALDelete) {
sfm->WaitForEmptyTrash();
ASSERT_EQ(penalties.size(), 8);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
class DBWALTestWithParam
@@ -605,14 +628,13 @@ TEST_F(DBSSTTest, OpenDBWithExistingTrash) {
// files in the second path were not.
TEST_F(DBSSTTest, DeleteSchedulerMultipleDBPaths) {
std::atomic<int> bg_delete_file(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteTrashFile:DeleteFile",
[&](void* /*arg*/) { bg_delete_file++; });
// The deletion scheduler sometimes skips marking file as trash according to
// a heuristic. In that case the deletion will go through the below SyncPoint.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteFile",
[&](void* /*arg*/) { bg_delete_file++; });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteFile", [&](void* /*arg*/) { bg_delete_file++; });
Options options = CurrentOptions();
options.disable_auto_compactions = true;
@@ -630,7 +652,7 @@ TEST_F(DBSSTTest, DeleteSchedulerMultipleDBPaths) {
auto sfm = static_cast<SstFileManagerImpl*>(options.sst_file_manager.get());
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.disableWAL = true;
@@ -678,15 +700,15 @@ TEST_F(DBSSTTest, DeleteSchedulerMultipleDBPaths) {
sfm->WaitForEmptyTrash();
ASSERT_EQ(bg_delete_file, 10);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, DestroyDBWithRateLimitedDelete) {
int bg_delete_file = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DeleteScheduler::DeleteTrashFile:DeleteFile",
[&](void* /*arg*/) { bg_delete_file++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Status s;
Options options = CurrentOptions();
@@ -771,15 +793,15 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
DestroyAndReopen(options);
int completed_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction():CancelledCompaction", [&](void* /*arg*/) {
sfm->SetMaxAllowedSpaceUsage(0);
ASSERT_EQ(sfm->GetCompactionsReservedSize(), 0);
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun",
[&](void* /*arg*/) { completed_compactions++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
@@ -806,7 +828,7 @@ TEST_F(DBSSTTest, CancellingCompactionsWorks) {
ASSERT_EQ(sfm->GetCompactionsReservedSize(), 0);
// Make sure the stat is bumped
ASSERT_GT(dbfull()->immutable_db_options().statistics.get()->getTickerCount(COMPACTION_CANCELLED), 0);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
@@ -854,7 +876,7 @@ TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
// Now make sure CompactFiles also gets cancelled
auto l0_files = collector->GetFlushedFiles();
dbfull()->CompactFiles(rocksdb::CompactionOptions(), l0_files, 0);
dbfull()->CompactFiles(ROCKSDB_NAMESPACE::CompactionOptions(), l0_files, 0);
// Wait for manual compaction to get scheduled and finish
dbfull()->TEST_WaitForCompact(true);
@@ -868,17 +890,17 @@ TEST_F(DBSSTTest, CancellingManualCompactionsWorks) {
// returns to normal
sfm->SetMaxAllowedSpaceUsage(0);
int completed_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactFilesImpl:End", [&](void* /*arg*/) { completed_compactions++; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->CompactFiles(rocksdb::CompactionOptions(), l0_files, 0);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
dbfull()->CompactFiles(ROCKSDB_NAMESPACE::CompactionOptions(), l0_files, 0);
dbfull()->TEST_WaitForCompact(true);
ASSERT_EQ(sfm->GetCompactionsReservedSize(), 0);
ASSERT_GT(completed_compactions, 0);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
@@ -892,7 +914,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
std::atomic<int> reached_max_space_on_flush(0);
std::atomic<int> reached_max_space_on_compaction(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::FlushMemTableToOutputFile:MaxAllowedSpaceReached",
[&](void* arg) {
Status* bg_error = static_cast<Status*>(arg);
@@ -902,13 +924,13 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
*bg_error = Status::OK();
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::BackgroundCompaction():CancelledCompaction", [&](void* arg) {
bool* enough_room = static_cast<bool*>(arg);
*enough_room = true;
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::FinishCompactionOutputFile:MaxAllowedSpaceReached",
[&](void* /*arg*/) {
bg_error_set = true;
@@ -917,8 +939,8 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
for (auto limit_mb : max_space_limits_mbs) {
bg_error_set = false;
rocksdb::SyncPoint::GetInstance()->ClearTrace();
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::shared_ptr<SstFileManager> sst_file_manager(NewSstFileManager(env_));
auto sfm = static_cast<SstFileManagerImpl*>(sst_file_manager.get());
@@ -942,7 +964,7 @@ TEST_F(DBSSTTest, DBWithMaxSpaceAllowedRandomized) {
uint64_t total_sst_files_size = 0;
GetAllSSTFiles(&total_sst_files_size);
ASSERT_GE(total_sst_files_size, limit_mb * 1024 * 1024);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
ASSERT_GT(reached_max_space_on_flush, 0);
@@ -1196,10 +1218,10 @@ TEST_F(DBSSTTest, GetTotalSstFilesSizeVersionsFilesShared) {
#endif // ROCKSDB_LITE
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+7 -7
View File
@@ -10,7 +10,7 @@
#include "port/stack_trace.h"
#include "rocksdb/statistics.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBStatisticsTest : public DBTestBase {
public:
@@ -45,7 +45,7 @@ TEST_F(DBStatisticsTest, CompressionStatsTest) {
Options options = CurrentOptions();
options.compression = type;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.statistics->set_stats_level(StatsLevel::kExceptTimeForMutex);
DestroyAndReopen(options);
@@ -91,7 +91,7 @@ TEST_F(DBStatisticsTest, CompressionStatsTest) {
TEST_F(DBStatisticsTest, MutexWaitStatsDisabledByDefault) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"pikachu"}, options);
const uint64_t kMutexWaitDelay = 100;
ThreadStatusUtil::TEST_SetStateDelay(ThreadStatus::STATE_MUTEX_WAIT,
@@ -104,7 +104,7 @@ TEST_F(DBStatisticsTest, MutexWaitStatsDisabledByDefault) {
TEST_F(DBStatisticsTest, MutexWaitStats) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.statistics->set_stats_level(StatsLevel::kAll);
CreateAndReopenWithCF({"pikachu"}, options);
const uint64_t kMutexWaitDelay = 100;
@@ -118,7 +118,7 @@ TEST_F(DBStatisticsTest, MutexWaitStats) {
TEST_F(DBStatisticsTest, ResetStats) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
for (int i = 0; i < 2; ++i) {
// pick arbitrary ticker and histogram. On first iteration they're zero
@@ -140,10 +140,10 @@ TEST_F(DBStatisticsTest, ResetStats) {
}
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+3 -3
View File
@@ -19,7 +19,7 @@
#ifndef ROCKSDB_LITE
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
// A helper function that ensures the table properties returned in
// `GetPropertiesOfAllTablesTest` is correct.
@@ -325,12 +325,12 @@ TEST_F(DBTablePropertiesTest, DeletionTriggeredCompactionMarking) {
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+13 -13
View File
@@ -16,7 +16,7 @@
#include "db/forward_iterator.h"
#include "port/stack_trace.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBTestTailingIterator : public DBTestBase {
public:
@@ -98,8 +98,8 @@ TEST_F(DBTestTailingIterator, TailingIteratorSeekToNext) {
ASSERT_TRUE(itern->Valid());
ASSERT_EQ(itern->key().compare(key), 0);
}
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
for (int i = 2 * num_records; i > 0; --i) {
char buf1[32];
char buf2[32];
@@ -143,25 +143,25 @@ TEST_F(DBTestTailingIterator, TailingIteratorTrimSeekToNext) {
bool file_iters_deleted = false;
bool file_iters_renewed_null = false;
bool file_iters_renewed_copy = false;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ForwardIterator::SeekInternal:Return", [&](void* arg) {
ForwardIterator* fiter = reinterpret_cast<ForwardIterator*>(arg);
ASSERT_TRUE(!file_iters_deleted ||
fiter->TEST_CheckDeletedIters(&deleted_iters, &num_iters));
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ForwardIterator::Next:Return", [&](void* arg) {
ForwardIterator* fiter = reinterpret_cast<ForwardIterator*>(arg);
ASSERT_TRUE(!file_iters_deleted ||
fiter->TEST_CheckDeletedIters(&deleted_iters, &num_iters));
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ForwardIterator::RenewIterators:Null",
[&](void* /*arg*/) { file_iters_renewed_null = true; });
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ForwardIterator::RenewIterators:Copy",
[&](void* /*arg*/) { file_iters_renewed_copy = true; });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
const int num_records = 1000;
for (int i = 1; i < num_records; ++i) {
char buf1[32];
@@ -413,14 +413,14 @@ TEST_F(DBTestTailingIterator, TailingIteratorUpperBound) {
// This keeps track of the number of times NeedToSeekImmutable() was true.
int immutable_seeks = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"ForwardIterator::SeekInternal:Immutable",
[&](void* /*arg*/) { ++immutable_seeks; });
// Seek to 13. This should not require any immutable seeks.
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
it->Seek("13");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_FALSE(it->Valid());
ASSERT_EQ(0, immutable_seeks);
@@ -530,13 +530,13 @@ TEST_F(DBTestTailingIterator, SeekToFirstWithUpperBoundBug) {
ASSERT_EQ(iter->key().ToString(), "aa");
}
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
+171 -88
View File
@@ -68,7 +68,7 @@
#include "util/string_util.h"
#include "utilities/merge_operators.h"
namespace rocksdb {
namespace ROCKSDB_NAMESPACE {
class DBTest : public DBTestBase {
public:
@@ -232,14 +232,14 @@ TEST_F(DBTest, SkipDelay) {
// the test flaky
auto token = dbfull()->TEST_write_controler().GetDelayToken(1);
std::atomic<int> sleep_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Sleep",
[&](void* /*arg*/) { sleep_count.fetch_add(1); });
std::atomic<int> wait_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait",
[&](void* /*arg*/) { wait_count.fetch_add(1); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = sync;
@@ -290,9 +290,8 @@ TEST_F(DBTest, MixedSlowdownOptions) {
// the test flaky
auto token = dbfull()->TEST_write_controler().GetDelayToken(1);
std::atomic<int> sleep_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:BeginWriteStallDone",
[&](void* /*arg*/) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:BeginWriteStallDone", [&](void* /*arg*/) {
sleep_count.fetch_add(1);
if (threads.empty()) {
for (int i = 0; i < 2; ++i) {
@@ -303,7 +302,7 @@ TEST_F(DBTest, MixedSlowdownOptions) {
}
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = false;
@@ -345,9 +344,8 @@ TEST_F(DBTest, MixedSlowdownOptionsInQueue) {
// the test flaky
auto token = dbfull()->TEST_write_controler().GetDelayToken(1);
std::atomic<int> sleep_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Sleep",
[&](void* /*arg*/) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Sleep", [&](void* /*arg*/) {
sleep_count.fetch_add(1);
if (threads.empty()) {
for (int i = 0; i < 2; ++i) {
@@ -359,10 +357,10 @@ TEST_F(DBTest, MixedSlowdownOptionsInQueue) {
}
});
std::atomic<int> wait_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait",
[&](void* /*arg*/) { wait_count.fetch_add(1); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = false;
@@ -414,9 +412,8 @@ TEST_F(DBTest, MixedSlowdownOptionsStop) {
// the test flaky
auto token = dbfull()->TEST_write_controler().GetStopToken();
std::atomic<int> wait_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait",
[&](void* /*arg*/) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait", [&](void* /*arg*/) {
wait_count.fetch_add(1);
if (threads.empty()) {
for (int i = 0; i < 2; ++i) {
@@ -432,7 +429,7 @@ TEST_F(DBTest, MixedSlowdownOptionsStop) {
token.reset();
threads.emplace_back(wakeup_writer);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
WriteOptions wo;
wo.sync = false;
@@ -2277,6 +2274,41 @@ TEST_F(DBTest, SnapshotFiles) {
dbfull()->DisableFileDeletions();
} while (ChangeCompactOptions());
}
TEST_F(DBTest, ReadonlyDBGetLiveManifestSize) {
do {
Options options = CurrentOptions();
options.level0_file_num_compaction_trigger = 2;
DestroyAndReopen(options);
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Flush());
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact());
Close();
ASSERT_OK(ReadOnlyReopen(options));
uint64_t manifest_size = 0;
std::vector<std::string> files;
dbfull()->GetLiveFiles(files, &manifest_size);
for (const std::string& f : files) {
uint64_t number = 0;
FileType type;
if (ParseFileName(f.substr(1), &number, &type)) {
if (type == kDescriptorFile) {
uint64_t size_on_disk;
env_->GetFileSize(dbname_ + "/" + f, &size_on_disk);
ASSERT_EQ(manifest_size, size_on_disk);
break;
}
}
}
Close();
} while (ChangeCompactOptions());
}
#endif
TEST_F(DBTest, PurgeInfoLogs) {
@@ -2557,15 +2589,15 @@ TEST_F(DBTest, GroupCommitTest) {
do {
Options options = CurrentOptions();
options.env = env_;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
Reopen(options);
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"WriteThread::JoinBatchGroup:BeganWaiting",
"DBImpl::WriteImpl:BeforeLeaderEnters"},
{"WriteThread::AwaitState:BlockingWaiting",
{"WriteThread::AwaitState:BlockingWaiting",
"WriteThread::EnterAsBatchGroupLeader:End"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Start threads
GCThread thread[kGCNumThreads];
@@ -2884,13 +2916,13 @@ class ModelDB : public DB {
DBOptions GetDBOptions() const override { return options_; }
using DB::Flush;
Status Flush(const rocksdb::FlushOptions& /*options*/,
Status Flush(const ROCKSDB_NAMESPACE::FlushOptions& /*options*/,
ColumnFamilyHandle* /*column_family*/) override {
Status ret;
return ret;
}
Status Flush(
const rocksdb::FlushOptions& /*options*/,
const ROCKSDB_NAMESPACE::FlushOptions& /*options*/,
const std::vector<ColumnFamilyHandle*>& /*column_families*/) override {
return Status::OK();
}
@@ -2923,8 +2955,8 @@ class ModelDB : public DB {
Status DeleteFile(std::string /*name*/) override { return Status::OK(); }
Status GetUpdatesSince(
rocksdb::SequenceNumber,
std::unique_ptr<rocksdb::TransactionLogIterator>*,
ROCKSDB_NAMESPACE::SequenceNumber,
std::unique_ptr<ROCKSDB_NAMESPACE::TransactionLogIterator>*,
const TransactionLogIterator::ReadOptions& /*read_options*/ =
TransactionLogIterator::ReadOptions()) override {
return Status::NotSupported("Not supported in Model DB");
@@ -3174,6 +3206,57 @@ TEST_F(DBTest, BlockBasedTablePrefixIndexTest) {
ASSERT_EQ("v2", Get("k2"));
}
TEST_F(DBTest, BlockBasedTablePrefixIndexTotalOrderSeek) {
// create a DB with block prefix index
BlockBasedTableOptions table_options;
Options options = CurrentOptions();
options.max_open_files = 10;
table_options.index_type = BlockBasedTableOptions::kHashSearch;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
// RocksDB sanitize max open files to at least 20. Modify it back.
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = static_cast<int*>(arg);
*max_open_files = 11;
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
ASSERT_OK(Put("k1", "v1"));
Flush();
CompactRangeOptions cro;
cro.change_level = true;
cro.target_level = 1;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
// Force evict tables
dbfull()->TEST_table_cache()->SetCapacity(0);
// Make table cache to keep one entry.
dbfull()->TEST_table_cache()->SetCapacity(1);
ReadOptions read_options;
read_options.total_order_seek = true;
{
std::unique_ptr<Iterator> iter(db_->NewIterator(read_options));
iter->Seek("k1");
ASSERT_TRUE(iter->Valid());
ASSERT_EQ("k1", iter->key().ToString());
}
// After total order seek, prefix index should still be used.
read_options.total_order_seek = false;
{
std::unique_ptr<Iterator> iter(db_->NewIterator(read_options));
iter->Seek("k1");
ASSERT_TRUE(iter->Valid());
ASSERT_EQ("k1", iter->key().ToString());
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBTest, ChecksumTest) {
BlockBasedTableOptions table_options;
Options options = CurrentOptions();
@@ -3595,7 +3678,7 @@ TEST_F(DBTest, DISABLED_RateLimitingTest) {
options.compression = kNoCompression;
options.create_if_missing = true;
options.env = env_;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.IncreaseParallelism(4);
DestroyAndReopen(options);
@@ -3921,10 +4004,10 @@ TEST_F(DBTest, DynamicMemtableOptions) {
int count = 0;
Random rnd(301);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait",
[&](void* /*arg*/) { sleeping_task_low.WakeUp(); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
while (!sleeping_task_low.WokenUp() && count < 256) {
ASSERT_OK(Put(Key(count), RandomString(&rnd, 1024), WriteOptions()));
@@ -3982,7 +4065,7 @@ TEST_F(DBTest, DynamicMemtableOptions) {
#endif
sleeping_task_low.WaitUntilDone();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
@@ -4091,11 +4174,11 @@ TEST_F(DBTest, ThreadStatusFlush) {
options.enable_thread_tracking = true;
options = CurrentOptions(options);
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"FlushJob::FlushJob()", "DBTest::ThreadStatusFlush:1"},
{"DBTest::ThreadStatusFlush:2", "FlushJob::WriteLevel0Table"},
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
CreateAndReopenWithCF({"pikachu"}, options);
VerifyOperationCount(env_, ThreadStatus::OP_FLUSH, 0);
@@ -4120,7 +4203,7 @@ TEST_F(DBTest, ThreadStatusFlush) {
// This second sync point is to ensure the flush job will not
// be completed until we already perform VerifyOperationCount().
TEST_SYNC_POINT("DBTest::ThreadStatusFlush:2");
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_P(DBTestWithParam, ThreadStatusSingleCompaction) {
@@ -4143,15 +4226,15 @@ TEST_P(DBTestWithParam, ThreadStatusSingleCompaction) {
options.level0_file_num_compaction_trigger = kNumL0Files;
options.max_subcompactions = max_subcompactions_;
rocksdb::SyncPoint::GetInstance()->LoadDependency({
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency({
{"DBTest::ThreadStatusSingleCompaction:0", "DBImpl::BGWorkCompaction"},
{"CompactionJob::Run():Start", "DBTest::ThreadStatusSingleCompaction:1"},
{"DBTest::ThreadStatusSingleCompaction:2", "CompactionJob::Run():End"},
});
for (int tests = 0; tests < 2; ++tests) {
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->ClearTrace();
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
// The Put Phase.
@@ -4190,7 +4273,7 @@ TEST_P(DBTestWithParam, ThreadStatusSingleCompaction) {
// repeat the test with disabling thread tracking.
options.enable_thread_tracking = false;
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -4287,7 +4370,7 @@ TEST_P(DBTestWithParam, PreShutdownMultipleCompaction) {
std::vector<ThreadStatus> thread_list;
// Delay both flush and compaction
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"FlushJob::FlushJob()", "CompactionJob::Run():Start"},
{"CompactionJob::Run():Start",
"DBTest::PreShutdownMultipleCompaction:Preshutdown"},
@@ -4298,7 +4381,7 @@ TEST_P(DBTestWithParam, PreShutdownMultipleCompaction) {
{"CompactionJob::Run():End",
"DBTest::PreShutdownMultipleCompaction:VerifyPreshutdown"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Make rocksdb busy
int key = 0;
@@ -4376,7 +4459,7 @@ TEST_P(DBTestWithParam, PreShutdownCompactionMiddle) {
std::vector<ThreadStatus> thread_list;
// Delay both flush and compaction
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBTest::PreShutdownCompactionMiddle:Preshutdown",
"CompactionJob::Run():Inprogress"},
{"CompactionJob::Run():Start",
@@ -4385,7 +4468,7 @@ TEST_P(DBTestWithParam, PreShutdownCompactionMiddle) {
{"CompactionJob::Run():End",
"DBTest::PreShutdownCompactionMiddle:VerifyPreshutdown"}});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Make rocksdb busy
int key = 0;
@@ -4562,7 +4645,7 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
std::atomic<int> num_zlib(0);
std::atomic<int> num_lz4(0);
std::atomic<int> num_no(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
if (compaction->output_level() == 4) {
@@ -4570,13 +4653,13 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
num_lz4.fetch_add(1);
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FlushJob::WriteLevel0Table:output_compression", [&](void* arg) {
auto* compression = reinterpret_cast<CompressionType*>(arg);
ASSERT_TRUE(*compression == kNoCompression);
num_no.fetch_add(1);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int i = 0; i < 100; i++) {
std::string value = RandomString(&rnd, 200);
@@ -4590,8 +4673,8 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
Flush();
dbfull()->TEST_WaitForFlushMemTable();
dbfull()->TEST_WaitForCompact();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ASSERT_EQ(NumTableFilesAtLevel(1), 0);
ASSERT_EQ(NumTableFilesAtLevel(2), 0);
@@ -4604,7 +4687,7 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
// After base level turn L4->L3, L3 becomes LZ4 and L4 becomes Zlib
num_lz4.store(0);
num_no.store(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
if (compaction->output_level() == 4 && compaction->start_level() == 3) {
@@ -4615,13 +4698,13 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
num_lz4.fetch_add(1);
}
});
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FlushJob::WriteLevel0Table:output_compression", [&](void* arg) {
auto* compression = reinterpret_cast<CompressionType*>(arg);
ASSERT_TRUE(*compression == kNoCompression);
num_no.fetch_add(1);
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
for (int i = 101; i < 500; i++) {
std::string value = RandomString(&rnd, 200);
@@ -4632,8 +4715,8 @@ TEST_F(DBTest, DynamicLevelCompressionPerLevel2) {
}
}
rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ASSERT_EQ(NumTableFilesAtLevel(1), 0);
ASSERT_EQ(NumTableFilesAtLevel(2), 0);
ASSERT_GT(NumTableFilesAtLevel(3), 0);
@@ -5049,7 +5132,7 @@ TEST_F(DBTest, DynamicMiscOptions) {
options.create_if_missing = true;
options.max_sequential_skip_in_iterations = 16;
options.compression = kNoCompression;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
DestroyAndReopen(options);
auto assert_reseek_count = [this, &options](int key_start, int num_reseek) {
@@ -5141,7 +5224,7 @@ TEST_F(DBTest, L0L1L2AndUpHitCounter) {
options.max_write_buffer_number = 2;
options.max_background_compactions = 8;
options.max_background_flushes = 8;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
CreateAndReopenWithCF({"mypikachu"}, options);
int numkeys = 20000;
@@ -5244,7 +5327,7 @@ TEST_F(DBTest, CloseSpeedup) {
env_->DeleteDir(dbname_);
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
env_->SetBackgroundThreads(1, Env::LOW);
env_->SetBackgroundThreads(1, Env::HIGH);
Random rnd(301);
@@ -5299,7 +5382,7 @@ TEST_F(DBTest, MergeTestTime) {
this->env_->time_elapse_only_sleep_ = true;
this->env_->no_slowdown_ = true;
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.merge_operator.reset(new DelayedMergeOperator(this));
DestroyAndReopen(options);
@@ -5340,7 +5423,7 @@ TEST_P(DBTestWithParam, MergeCompactionTimeTest) {
SetPerfLevel(kEnableTime);
Options options = CurrentOptions();
options.compaction_filter_factory = std::make_shared<KeepFilterFactory>();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.merge_operator.reset(new DelayedMergeOperator(this));
options.compaction_style = kCompactionStyleUniversal;
options.max_subcompactions = max_subcompactions_;
@@ -5362,7 +5445,7 @@ TEST_P(DBTestWithParam, FilterCompactionTimeTest) {
std::make_shared<DelayFilterFactory>(this);
options.disable_auto_compactions = true;
options.create_if_missing = true;
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.statistics->set_stats_level(kExceptTimeForMutex);
options.max_subcompactions = max_subcompactions_;
DestroyAndReopen(options);
@@ -5626,16 +5709,16 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
// schedule automatic compactions after the manual one starts, but before it
// finishes to ensure conflict.
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::BackgroundCompaction:Start",
"DBTest::AutomaticConflictsWithManualCompaction:PrePuts"},
{"DBTest::AutomaticConflictsWithManualCompaction:PostPuts",
"DBImpl::BackgroundCompaction:NonTrivial:AfterRun"}});
std::atomic<int> callback_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MaybeScheduleFlushOrCompaction:Conflict",
[&](void* /*arg*/) { callback_count.fetch_add(1); });
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < 2; ++i) {
@@ -5665,7 +5748,7 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
for (int i = 0; i < 2; ++i) {
ASSERT_NE("NOT_FOUND", Get(Key(i)));
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
manual_compaction_thread.join();
dbfull()->TEST_WaitForCompact();
}
@@ -5689,7 +5772,7 @@ TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
ASSERT_OK(Flush());
}
rocksdb::ColumnFamilyMetaData cf_meta_data;
ROCKSDB_NAMESPACE::ColumnFamilyMetaData cf_meta_data;
db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta_data);
std::vector<std::string> input_files;
@@ -5793,19 +5876,19 @@ TEST_F(DBTest, FlushesInParallelWithCompactRange) {
}
if (iter == 1) {
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::RunManualCompaction()::1",
"DBTest::FlushesInParallelWithCompactRange:1"},
{"DBTest::FlushesInParallelWithCompactRange:2",
"DBImpl::RunManualCompaction()::2"}});
} else {
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"CompactionJob::Run():Start",
"DBTest::FlushesInParallelWithCompactRange:1"},
{"DBTest::FlushesInParallelWithCompactRange:2",
"CompactionJob::Run():End"}});
}
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<port::Thread> threads;
threads.emplace_back([&]() { Compact("a", "z"); });
@@ -5824,7 +5907,7 @@ TEST_F(DBTest, FlushesInParallelWithCompactRange) {
for (auto& t : threads) {
t.join();
}
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
}
@@ -5890,7 +5973,7 @@ TEST_F(DBTest, DelayedWriteRate) {
static_cast<int64_t>(estimated_sleep_time * 2));
env_->no_slowdown_ = false;
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
sleeping_task_low.WakeUp();
sleeping_task_low.WaitUntilDone();
}
@@ -5919,12 +6002,12 @@ TEST_F(DBTest, HardLimit) {
CreateAndReopenWithCF({"pikachu"}, options);
std::atomic<int> callback_count(0);
rocksdb::SyncPoint::GetInstance()->SetCallBack("DBImpl::DelayWrite:Wait",
[&](void* /*arg*/) {
callback_count.fetch_add(1);
sleeping_task_low.WakeUp();
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::DelayWrite:Wait", [&](void* /*arg*/) {
callback_count.fetch_add(1);
sleeping_task_low.WakeUp();
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
int key_idx = 0;
@@ -5941,7 +6024,7 @@ TEST_F(DBTest, HardLimit) {
}
ASSERT_GE(callback_count.load(), 1);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
sleeping_task_low.WaitUntilDone();
}
@@ -6013,7 +6096,7 @@ TEST_F(DBTest, SoftLimit) {
"DBImpl::BackgroundCallFlush:ContextCleanedUp");
};
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Reopen(options);
@@ -6069,7 +6152,7 @@ TEST_F(DBTest, SoftLimit) {
ASSERT_TRUE(listener->CheckCondition(WriteStallCondition::kNormal));
// Only allow one compactin going through.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"BackgroundCallCompaction:0", [&](void* /*arg*/) {
// Schedule a sleeping task.
sleeping_task_low.Reset();
@@ -6143,7 +6226,7 @@ TEST_F(DBTest, SoftLimit) {
ASSERT_TRUE(listener->CheckCondition(WriteStallCondition::kDelayed));
sleeping_task_low.WaitUntilSleeping();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
sleeping_task_low.WakeUp();
sleeping_task_low.WaitUntilDone();
}
@@ -6221,7 +6304,7 @@ TEST_F(DBTest, CreateColumnFamilyShouldFailOnIncompatibleOptions) {
#ifndef ROCKSDB_LITE
TEST_F(DBTest, RowCache) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.row_cache = NewLRUCache(8192);
DestroyAndReopen(options);
@@ -6240,7 +6323,7 @@ TEST_F(DBTest, RowCache) {
TEST_F(DBTest, PinnableSliceAndRowCache) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
options.row_cache = NewLRUCache(8192);
DestroyAndReopen(options);
@@ -6270,12 +6353,12 @@ TEST_F(DBTest, PinnableSliceAndRowCache) {
#endif // ROCKSDB_LITE
TEST_F(DBTest, DeletingOldWalAfterDrop) {
rocksdb::SyncPoint::GetInstance()->LoadDependency(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"Test:AllowFlushes", "DBImpl::BGWorkFlush"},
{"DBImpl::BGWorkFlush:done", "Test:WaitForFlush"}});
rocksdb::SyncPoint::GetInstance()->ClearTrace();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearTrace();
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
Options options = CurrentOptions();
options.max_total_wal_size = 8192;
options.compression = kNoCompression;
@@ -6285,7 +6368,7 @@ TEST_F(DBTest, DeletingOldWalAfterDrop) {
options.level0_stop_writes_trigger = (1 << 30);
options.disable_auto_compactions = true;
DestroyAndReopen(options);
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
CreateColumnFamilies({"cf1", "cf2"}, options);
ASSERT_OK(Put(0, "key1", DummyString(8192)));
@@ -6423,7 +6506,7 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
env_->GetCurrentTime(&time_2);
const uint64_t uint_time_2 = static_cast<uint64_t>(time_2);
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"PropertyBlockBuilder::AddTableProperty:Start", [&](void* arg) {
TableProperties* props = reinterpret_cast<TableProperties*>(arg);
if (set_file_creation_time_to_zero) {
@@ -6444,12 +6527,12 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
}
});
// Set file creation time in manifest all to 0.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"FileMetaData::FileMetaData", [&](void* arg) {
FileMetaData* meta = static_cast<FileMetaData*>(arg);
meta->file_creation_time = 0;
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
@@ -6499,12 +6582,12 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
Status s3 = dbfull()->GetCreationTimeOfOldestFile(&ctime);
ASSERT_EQ(s3, Status::NotSupported());
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif
} // namespace rocksdb
} // namespace ROCKSDB_NAMESPACE
#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
extern "C" {
@@ -6515,7 +6598,7 @@ void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
#endif // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
int main(int argc, char** argv) {
rocksdb::port::InstallStackTraceHandler();
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
RegisterCustomObjects(argc, argv);
return RUN_ALL_TESTS();
Loaded 100 of 965 files, more files were not shown because too many files have changed in this diff. Show more