Compare commits

...
71 Commits
Author SHA1 Message Date
Andrew Kryczka fdf403f591 update HISTORY.md for 8.0.0 2023-02-19 13:44:55 -08:00
Andrew Kryczka 4b87e21f3a Add missing override keyword in env_win.h functions (#11232)
Summary:
I couldn't figure out why this causes failures in our 8.0 release to fbcode while this issue appears to not be new in 8.0. Anyways, we can add the missing `override` keywords to these functions as the compiler insists.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11232

Reviewed By: pdillinger

Differential Revision: D43420656

Pulled By: ajkr

fbshipit-source-id: da748eeef6ba38dd113dbe4b5143d7558daf38dd
2023-02-19 08:07:07 -08:00
Alan Paxton d47126875b Fix Java API ComparatorOptions use after delete error (#11176)
Summary:
The problem
-------------
ComparatorOptions is AutoCloseable.

AbstractComparator does not hold a reference to its ComparatorOptions, but the native C++ ComparatorJniCallback holds a reference to the ComparatorOptions’ native C++ options structure. This gets deleted when the ComparatorOptions is closed, either explicitly, or as part of try-with-resources.

Later, the deleted C++ options structure gets used by the callback and the comparator options are effectively random.

The original bug report https://github.com/facebook/rocksdb/issues/8715 was caused by a GC-initiated finalization closing the still-in-use ComparatorOptions. As of 7.0, finalization of RocksDB objects no longer closes them, which worked round the reported bug, but still left ComparatorOptions with a potentially broken lifetime.

In any case, we encourage API clients to use the try-with-resources model, and so we need it to work. And if they don't use it, they leak resources.

The solution
-------------
The solution implemented here is to make a copy of the native C++ options object into the ComparatorJniCallback, rather than a reference. Then the deletion of the native object held by ComparatorOptions is *correctly* deleted when its scope is closed in try/finally.

Testing
-------
We added a regression unit test based on the original test for the reported ticket.

This checkin closes https://github.com/facebook/rocksdb/issues/8715

We expect that there are more instances of "lifecycle" bugs in the Java API. They are a major source of support time/cost, and we note that they could be addressed as a whole using the model proposed/prototyped in https://github.com/facebook/rocksdb/pull/10736

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11176

Reviewed By: cbi42

Differential Revision: D43160885

Pulled By: pdillinger

fbshipit-source-id: 60b54215a02ad9abb17363319650328c00a9ad62
2023-02-17 13:03:41 -08:00
mrambacher b6640c3117 Remove FactoryFunc from LoadXXXObject (#11203)
Summary:
The primary purpose of the FactoryFunc was to support LITE mode where the ObjectRegistry was not available.  With the removal of LITE mode, the function was no longer required.

Note that the MergeOperator had some private classes defined in header files.  To gain access to their constructors (and name methods), the class definitions were moved into header files.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11203

Reviewed By: cbi42

Differential Revision: D43160255

Pulled By: pdillinger

fbshipit-source-id: f3a465fd5d1a7049b73ecf31e4b8c3762f6dae6c
2023-02-17 12:54:07 -08:00
Andrew Kryczka 25e1365227 Merge operator failed subcode (#11231)
Summary:
From HISTORY.md: Added a subcode of `Status::Corruption`, `Status::SubCode::kMergeOperatorFailed`, for users to identify corruption failures originating in the merge operator, as opposed to RocksDB's internally identified data corruptions.

This is a followup to https://github.com/facebook/rocksdb/issues/11092, where we gave users the ability to keep running a DB despite merge operator failing. Now that the DB keeps running despite such failures, they want to be able to distinguish such failures from real corruptions.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11231

Test Plan: updated unit test

Reviewed By: akankshamahajan15

Differential Revision: D43396607

Pulled By: ajkr

fbshipit-source-id: 17fbcc779ad724dafada8abd73efd38e1c5208b9
2023-02-17 10:58:46 -08:00
Andrew Kryczka 6aef1a05d6 Use CacheDependencies() at start of ApproximateKeyAnchors() (#11230)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11230

Test Plan:
- setup command: `$ ./db_bench -benchmarks=fillrandom,compact -compression_type=none -num=1000000 -write_buffer_size=4194304 -target_file_size_base=4194304 -use_direct_io_for_flush_and_compaction=true -partition_index_and_filters=true -bloom_bits=10 -metadata_block_size=1024`
- measure small read count bucketed by size: `$ strace -fye pread64 ./db_bench.ctrl -use_existing_db=true -benchmarks=compact -compaction_readahead_size=4194304 -compression_type=none -num=1000000 -write_buffer_size=4194304 -target_file_size_base=4194304 -use_direct_io_for_flush_and_compaction=true -partition_index_and_filters=true -bloom_bits=10 -metadata_block_size=1024  -subcompactions=4 -cache_size=1048576000  2>&1 >/dev/null | awk '/= [0-9]+$/{print "[", int($NF / 1024), "KB,", int(1 + $NF / 1024), "KB)"}' | sort -n -k 2 | uniq -c | head -3`
- before:
```
   1119 [ 0 KB, 1 KB)
      1 [ 6 KB, 7 KB)
      2 [ 7 KB, 8 KB)
```
- after:
```
    242 [ 0 KB, 1 KB)
      1 [ 6 KB, 7 KB)
      2 [ 7 KB, 8 KB)
```

Reviewed By: pdillinger

Differential Revision: D43388507

Pulled By: ajkr

fbshipit-source-id: a02413c9f615b00784700646825a9870ee10f3a7
2023-02-17 09:03:37 -08:00
akankshamahajan 68e4581c67 Return NotSupported in scan if IOUring not supported and enable IOUring in db_stress for async_io testing (#11197)
Summary:
- Return NotSupported in scan if IOUring not supported if async_io is enabled
- Enable IOUring in db_stress for async_io testing
- Disable async_io in circleci crash testing as circleci doesn't support IOUring

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11197

Test Plan: CircleCI jobs

Reviewed By: anand1976

Differential Revision: D43096313

Pulled By: akankshamahajan15

fbshipit-source-id: c2c53a87636950c0243038b9f5bd0d91608e4fda
2023-02-16 18:33:06 -08:00
Peter Dillinger 64a1f7670f Customize CompressedSecondaryCache by block kind (#11204)
Summary:
Added `do_not_compress_roles` to `CompressedSecondaryCacheOptions` to disable compression on certain kinds of block. Filter blocks are now not compressed by CompressedSecondaryCache by default.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11204

Test Plan: unit test added

Reviewed By: anand1976

Differential Revision: D43147698

Pulled By: pdillinger

fbshipit-source-id: db496975ae975fa18f157f93fe131a16315ac875
2023-02-16 17:22:27 -08:00
Peter Dillinger 88056ea6cb Re-add memory_allocator.h include from cache.h (#11229)
Summary:
Enough users of NewJemallocNodumpAllocator() with cache.h to justify keeping it. (Reverting one little part of https://github.com/facebook/rocksdb/issues/11192)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11229

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D43337140

Pulled By: pdillinger

fbshipit-source-id: 886b27b96b395619a4209f51b9b7787f4fe89e57
2023-02-16 08:07:45 -08:00
Levi Tamasi ab22e79824 Support using MultiGetEntity as verification method in stress tests (#11228)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11228

Reviewed By: akankshamahajan15

Differential Revision: D43332120

Pulled By: ltamasi

fbshipit-source-id: 15f32cf335aecb7e654da24ecafc6e010dc65194
2023-02-15 17:08:25 -08:00
Levi Tamasi 94ec433833 Mention the new MultiGetEntity API in HISTORY.md (#11226)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11226

Reviewed By: akankshamahajan15

Differential Revision: D43317602

Pulled By: ltamasi

fbshipit-source-id: 4b7e063848d3cfbdb9f0c0f54d68aeab8a82595c
2023-02-15 11:50:18 -08:00
Levi Tamasi 9794acb597 Add a new MultiGetEntity API (#11222)
Summary:
The new `MultiGetEntity` API can be used to get a consistent view of
a batch of keys, with the results presented as wide-column entities.
Similarly to `GetEntity` and the iterator's `columns` API, if the entry
corresponding to the key is a wide-column entity to start with, it is
returned as-is, and if it is a plain key-value, it is wrapped into an entity
with a single default column.

Implementation-wise, the new API shares the logic of the batched `MultiGet`
API (via the `MultiGetCommon` methods). Both single-CF and multi-CF
`MultiGetEntity` APIs are provided, and blobs are also supported.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11222

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D43256950

Pulled By: ltamasi

fbshipit-source-id: 47fb2cb7e2d0470e3580f43fdb2fe9e51f0e7005
2023-02-15 09:34:17 -08:00
akankshamahajan 6d5e8604f1 Fix regression script for async_io benchmarks (#11224)
Summary:
Fix regression script for async_io benchmark using incorrect ops and threads and wrong benchmark name during reporting results.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11224

Test Plan: Ran manually

Reviewed By: anand1976

Differential Revision: D43287658

Pulled By: akankshamahajan15

fbshipit-source-id: 433e2caa0e51268e72a875549ab8f7f92a7a4216
2023-02-14 18:54:47 -08:00
sdong 1969815fa8 Remove docs/Gemfile.lock and update github-pages version (#11173)
Summary:
One system reports that a dependency in docs/Gemfile.lock is out-of-date and has a risk. I don't see a point of having Gemfile.lock checked in and dealing with dependencies all the time at all. It should be able to regenerated using `bundle install`. Update Gemfile file to a later version too.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11173

Test Plan:
Run
bundle install
bundle exec jekyll serve --host=0.0.0.0
and see website working locally.

Reviewed By: ajkr

Differential Revision: D42897698

fbshipit-source-id: aeaf065c28b8f6582f1af1b5ffbbd5fa194afe24
2023-02-14 12:17:23 -08:00
Yu Zhang c19672c187 Enable crash test to run BlobDB together with user-defined timestamp (#11199)
Summary:
I missed a stress test code sanity check when enabling this combination of tests. This PR addresses that, the "iter_start_ts" function for user defined timestamp feature is not supported when BlobDB is enabled. It's disabled for now.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11199

Test Plan:
Locally always enable BlobDB and run
tools/db_crashtest.py --stress_cmd=./db_stress --cleanup_cmd='' --enable_ts whitebox --random_kill_odd 888887

Reviewed By: ltamasi

Differential Revision: D43245657

Pulled By: jowlyzhang

fbshipit-source-id: 4cae19817bb1afd50a76f9e0e49f006fb5c0b211
2023-02-13 13:40:02 -08:00
Wentian Guo 42d6652ba2 remove dependency on options.h for port_posix.h andport_win.h (#11214)
Summary:
The files in `port/`, such as `port_posix.h`, are layering over the system libraries, so shouldn't include the DB-specific files like `options.h`. This PR remove this dependency.

# How
The reason that `port_posix.h` (or `port_win.h`) include `options.h` is to use `CpuPriority`, as there is a method `SetCpuPriority()` in `port_posix.h` that uses `CpuPriority.`
- I think `SetCpuPriority()` make sense to exist in `port_posix.h` as it provides has platform-dependent implementation
- `CpuPriority` enum is defined in `env.h`, but used in `rocksdb/include` and `port/`.

Hence, let us define `CpuPriority` enum in a common file, say `port_defs.h`, such that both directories `rocksdb/include` and `port/` can include.

When we remove this dependency, some other files have compile errors because they can't find definitions, so add header files to resolve

# Test
make all check -j

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11214

Reviewed By: pdillinger

Differential Revision: D43196910

Pulled By: guowentian

fbshipit-source-id: 70deccb72844cfb08fcc994f76c6ef6df5d55ab9
2023-02-13 02:21:38 -08:00
akankshamahajan a72f591825 Fix a minor bug in the regression script during assigning value (#11215)
Summary:
Same as title

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11215

Test Plan: Ran manually

Reviewed By: pdillinger

Differential Revision: D43194634

Pulled By: akankshamahajan15

fbshipit-source-id: 336a08a9076b222d7000e4eb2a87fc36b863b05b
2023-02-10 10:13:19 -08:00
akankshamahajan ab2157fa6f Extend existing benchmarks seekrandom and multiread to run with async_io (#11170)
Summary:
=======================================================================
Benchmark seekrandom_asyncio

=======================================================================

db_bench_cmd=$(which time) -p ./db_bench       --benchmarks=seekrandom
--db=/tmp/rocksdb/regression_test/db --wal_dir=
--use_existing_db=0       --perf_level=1
--disable_auto_compactions       --threads=1       --num=1073741824
--reads=1073741824       --writes=1073741824       --deletes=1073741824
--key_size=100       --value_size=900       --cache_size=1073741824
--statistics=0              --compression_ratio=0.5       --histogram=1
--seek_nexts=10       --stats_per_interval=1
--stats_interval_seconds=600       --max_background_flushes=4
--num_multi_db=1       --max_background_compactions=16
--num_high_pri_threads=4       --num_low_pri_threads=16
--seed=1675181789       --multiread_batched=true       --batch_size=128
--multiread_stride=12       --async_io=true
--optimize_multiget_for_io=false 2>&1
RocksDB:    version 8.0.0

=======================================================================
 Benchmark multireadrandom_asyncio

====================================================================

db_bench_cmd=$(which time) -p ./db_bench
--benchmarks=multireadrandom --db=/tmp/rocksdb/regression_test/db
--wal_dir=       --use_existing_db=0       --perf_level=1
--disable_auto_compactions       --threads=1       --num=1073741824
--reads=1073741824       --writes=1073741824       --deletes=1073741824
--key_size=100       --value_size=900       --cache_size=1073741824
--statistics=0              --compression_ratio=0.5       --histogram=1
--seek_nexts=10       --stats_per_interval=1
--stats_interval_seconds=600       --max_background_flushes=4
--num_multi_db=1       --max_background_compactions=16
--num_high_pri_threads=4       --num_low_pri_threads=16
--seed=1675181841       --multiread_batched=true       --batch_size=128
--multiread_stride=12       --async_io=true
--optimize_multiget_for_io=true 2>&1
RocksDB:    version 8.0.0
Date:       Tue Jan 31 08:17:22 2023
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11170

Reviewed By: ajkr, anand1976

Differential Revision: D42889107

Pulled By: akankshamahajan15

fbshipit-source-id: b819be2bd5f00d1db654b9e829b84f11e6bcab92
2023-02-09 18:36:53 -08:00
Peter Dillinger 3cacd4b4ec Put Cache and CacheWrapper in new public header (#11192)
Summary:
The definition of the Cache class should not be needed by the vast majority of RocksDB users, so I think it is just distracting to include it in cache.h, which is primarily needed for configuring and creating caches. This change moves the class to a new header advanced_cache.h. It is just cut-and-paste except for modifying the class API comment.

In general, operations on shared_ptr<Cache> should continue to work when only a forward declaration of Cache is available, as long as all the Cache instances provided are already shared_ptr. See https://stackoverflow.com/a/17650101/454544

Also, the most common way to customize a Cache is by wrapping an existing implementation, so it makes sense to provide CacheWrapper in the public API. This was a cut-and-paste job except removing the implementation of Name() so that derived classes must provide it.

Intended follow-up: consolidate Release() into one function to reduce customization bugs / confusion

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11192

Test Plan: `make check`

Reviewed By: anand1976

Differential Revision: D43055487

Pulled By: pdillinger

fbshipit-source-id: 7b05492df35e0f30b581b4c24c579bc275b6d110
2023-02-09 12:12:02 -08:00
Peter Dillinger b7747bbc9f Attempt fix flaky DBWriteTest.LockWALInEffect (#11209)
Summary:
Example failure:
```
[ RUN      ] DBWriteTestInstance/DBWriteTest.LockWALInEffect/1
db/db_write_test.cc:646: Failure
Put("key3", "value")
Corruption: Not active
```
Presumably from a background compaction prior to Put.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11209

Test Plan: watch CI

Reviewed By: akankshamahajan15

Differential Revision: D43147727

Pulled By: pdillinger

fbshipit-source-id: a1c34ac5ab124bfe2f23205a30777990056e9082
2023-02-09 09:21:55 -08:00
Peter Dillinger 34bb3ddc43 Improve SmallEnumSet (#11178)
Summary:
In anticipation of using this to represent sets of CacheEntryRole for including or excluding kinds of blocks in block cache tiers, add significant new features to SmallEnumSet, including at least:

* List initialization
* Applicative constexpr operations
* copy/move/equality ops
* begin/end/const_iterator for iteration
* Better comments

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11178

Test Plan: unit tests added/expanded

Reviewed By: ltamasi

Differential Revision: D42973723

Pulled By: pdillinger

fbshipit-source-id: 40783486feda931c3f7c6fcc9a300acd6a4b0a0a
2023-02-08 20:14:57 -08:00
Peter Dillinger ee5305fabb Mitigate presumed OOM in CircleCI (#11206)
Summary:
We've seen many instances of
build-linux-static_lib-alt_namespace-status_checked failing like this:
```
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make: *** [Makefile:2507: utilities/transactions/transaction_test.o]
Error 1
```

It's understandable that so many static linking jobs could exhaust memory.

The executor only has 16 vcores, so going from 32 down to 24 shouldn't hurt build time.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11206

Test Plan: will watch CI

Reviewed By: ajkr

Differential Revision: D43137246

Pulled By: pdillinger

fbshipit-source-id: 050b0f700c285dd913bcae8b4a76a44d04bb0356
2023-02-08 17:00:46 -08:00
anand76 77b61abc7b Fix bug in WAL streaming uncompression (#11198)
Summary:
Fix a bug in the calculation of the input buffer address/offset in log_reader.cc. The bug is when consecutive fragments of a compressed record are located at the same offset in the log reader buffer, the second fragment input buffer is treated as a leftover from the previous input buffer. As a result, the offset in the `ZSTD_inBuffer` is not reset.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11198

Test Plan: Add a unit test in log_test.cc that fails without the fix and passes with it.

Reviewed By: ajkr, cbi42

Differential Revision: D43102692

Pulled By: anand1976

fbshipit-source-id: aa2648f4802c33991b76a3233c5a58d4cc9e77fd
2023-02-08 12:05:49 -08:00
Levi Tamasi 876d281592 Add compaction filter support for wide-column entities (#11196)
Summary:
The patch adds compaction filter support for wide-column entities by introducing
a new `CompactionFilter` API called `FilterV3`. This API is called for regular
key-values, merge operands, and wide-column entities as well. It is passed the
existing value/operand or wide-column structure and it can update the value or
columns or keep/delete/etc. the key-value as usual. For compatibility, the default
implementation of `FilterV3` keeps all wide-column entities and falls back to calling
`FilterV2` for plain old key-values and merge operands.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11196

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D43094147

Pulled By: ltamasi

fbshipit-source-id: 75acabe9a35254f7f404ba6173ee9c2774382ebd
2023-02-07 16:17:39 -08:00
Hui Xiao 6650ca244e Remove a couple deprecated convenience.h APIs (#11120)
Summary:
**Context/Summary:**
As instructed by convenience.h comments, a few deprecated APIs are removed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11120

Test Plan:
- make check & CI
- eyeball check on test semantics.

Reviewed By: pdillinger

Differential Revision: D42937507

Pulled By: hx235

fbshipit-source-id: a9e4709387da01b1d0e9148c2e210f02e9746ee1
2023-02-07 14:11:53 -08:00
Peter Dillinger b5827c806c Revert to LIB_MODE=static for optimized builds (#11195)
Summary:
Continuous performance testing indicates there's a small performance hit with shared library (-fPIC) builds, so while retaining the motivation for https://github.com/facebook/rocksdb/issues/11168, we set the default for DEBUG_LEVEL=0 Makefile builds back to LIB_MODE=static.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11195

Test Plan: CI, with some updated checks and removal of some now obsolete LIB_MODE overrides

Reviewed By: cbi42

Differential Revision: D43090576

Pulled By: pdillinger

fbshipit-source-id: 755fe5d07005f85caf24e16f90228ffd46a6e250
2023-02-07 11:26:55 -08:00
Symious 68fa90ca43 Add kForceOptimized option to jni (#11181)
Summary:
Currently the option of "KForceOptimized" is not included in CompactRangeOptions.BottommostLevelCompaction.

This PR is to add this option.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11181

Reviewed By: ajkr

Differential Revision: D43056453

Pulled By: cbi42

fbshipit-source-id: 22fd53f980ab1a86c61dd42e948902542065128f
2023-02-07 10:14:14 -08:00
Peter Dillinger 54d72085b5 Fix regression_test.sh for LIB_MODE=shared default (#11194)
Summary:
Need to scp the .so files. Switched to tar+ssh to support symlinks, faster handling of multiple files, and compression.

Also fixing some holes in 'make clean' as I've noticed files like 'librocksdb.so.7.7.0', 'librocksdb_test_debug.so', 'librocksdb_tools_debug.so' hanging around after `make clean`

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11194

Test Plan:
Manually triggered regression test runs with change, manual `make clean`
https://fburl.com/sandcastle/gnxy5lvc
https://fburl.com/sandcastle/4pxodwh7

Reviewed By: cbi42

Differential Revision: D43069065

Pulled By: pdillinger

fbshipit-source-id: 48552b5980956784a1fdb40638d9e8ad6db51900
2023-02-06 19:44:25 -08:00
Hui Xiao 9b66331388 Simplify TEST_F(DBWALTest, FixSyncWalOnObseletedWalWithNewManifestCausingMissingWAL) (#11186)
Summary:
**Context/Summary**:
Simplify `TEST_F(DBWALTest, FixSyncWalOnObseletedWalWithNewManifestCausingMissingWAL)` based on https://github.com/facebook/rocksdb/pull/11016#pullrequestreview-1205020134 and delete unused sync points.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11186

Test Plan:
- UT failed before fix in https://github.com/facebook/rocksdb/pull/10892 and passes after
- Check UT not flaky when running with https://app.circleci.com/pipelines/github/facebook/rocksdb/21985/workflows/5f6cc355-78c1-46d8-89ee-0fd679725a8a/jobs/540878

Reviewed By: ajkr

Differential Revision: D43034723

Pulled By: hx235

fbshipit-source-id: f503774987b8f3718505f99e95080a7fad28ac66
2023-02-06 16:10:03 -08:00
Peter Dillinger 92e8874654 Fix more CircleCI jobs for LIB_MODE=shared default (#11193)
Summary:
There are a set of jobs using libbenchmark that have linker failures with new default LIB_MODE=shared. This change adds build-linux-run-microbench to the set using LIB_MODE=static to work around the linker failures. I haven't dug into how to fix them.

There is another set of jobs using folly that have linker failures with new default LIB_MODE=shared. I tried fixing these by adding --shared-libs to the folly build, but that doesn't work. It kinda looks like the folly shared libs build is simply broken with the boost dependency:

```
/usr/bin/ld: /tmp/fbcode_builder_getdeps-ZrootZprojectZthird-partyZfollyZbuildZfbcode_builder-root/installed/boost-Z1Z72zV-c0-0f3HkylpzONnr1dsHYDaR2GyTLzYdkck/lib/libboost_filesystem.a(exception.o): relocation R_X86_64_PC32 against symbol `_ZTVN5boost10filesystem16filesystem_errorE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
```

I tried updating folly to the latest commit and that didn't help. Otherwise, I didn't dig deeper into fixing that so have added build-linux-clang-13-asan-ubsan-with-folly to the set using LIB_MODE=static

Also since I saw a flaky failure (not the first time), increased the timeout on build-linux-unity-and-headers job.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11193

Test Plan: CI

Reviewed By: cbi42

Differential Revision: D43061203

Pulled By: pdillinger

fbshipit-source-id: c641671f93087f0214ea261ea895bccf657cb1a9
2023-02-06 14:35:15 -08:00
Alan Paxton 4a51900700 CI Benchmarking. Reduce runtime further as overhead appears to have risen. (#11189)
Summary:
We had miscalculated (not sure if I suddenly can’t count, or if there is something else going on), and need to leave more overhead to get the benchmarks to run reliably under 1 hour.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11189

Reviewed By: cbi42

Differential Revision: D43052045

Pulled By: ajkr

fbshipit-source-id: 3fe68432ed76a1f87d34129b0246e6b6a70a49f2
2023-02-06 11:13:39 -08:00
Peter Dillinger 27cf09172c Fix compile gettid on older Linux (#11184)
Summary:
Seen only in post-PR CI job benchmark-linux. Some context: https://stackoverflow.com/questions/30680550/c-gettid-was-not-declared-in-this-scope

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11184

Test Plan: watch CI

Reviewed By: cbi42

Differential Revision: D43013891

Pulled By: pdillinger

fbshipit-source-id: 48b3b7231080a0f803fdc36d13946d5524770302
2023-02-03 16:49:54 -08:00
Peter Dillinger cf756ed916 Use LIB_MODE=shared build by default with make (#11168)
Summary:
With https://github.com/facebook/rocksdb/issues/11150 this becomes a practical change that I think is overall good for developer efficiency.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11168

Test Plan:
More efficient build of all unit tests and tools:

```
$ git clean -fdx
$ du -sh .
522M    .
$ /usr/bin/time make -j32 LIB_MODE=static
...
14270.63user 1043.33system 11:19.85elapsed 2252%CPU (0avgtext+0avgdata 1929944maxresident)k
...
$ du -sh .
62G     .
$
```
Vs.
```
$ git clean -fdx
$ du -sh .
522M    .
$ /usr/bin/time make -j32 LIB_MODE=shared
...
9479.87user 478.26system 7:20.82elapsed 2258%CPU (0avgtext+0avgdata 1929272maxresident)k
...
$ du -sh .
5.4G    .
$
```

So 1/3 less build time and >90% less space usage.

Individual unit test edit-compile-run is not too different. Modifying an average unit test source file:
```
$ touch db/version_builder_test.cc
$ /usr/bin/time make -j32 LIB_MODE=static version_builder_test
...
34.74user 3.37system 0:38.29elapsed 99%CPU (0avgtext+0avgdata 945520maxresident)k
```
Vs.
```
$ touch db/version_builder_test.cc
$ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test
...
116.26user 43.91system 0:28.65elapsed 559%CPU (0avgtext+0avgdata 675160maxresident)k
```
A little faster with shared.

However, modifying an average DB implementation file has an extra linking step with shared lib:
```
$ touch db/db_impl/db_impl_files.cc
$ /usr/bin/time make -j32 LIB_MODE=static version_builder_test
...
33.17user 5.13system 0:39.70elapsed 96%CPU (0avgtext+0avgdata 945544maxresident)k
```
Vs.
```
$ touch db/db_impl/db_impl_files.cc
$ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test
...
40.80user 4.66system 0:45.54elapsed 99%CPU (0avgtext+0avgdata 1056340maxresident)k
```
A little slower with shared.

On the whole, should be faster and lighter weight because of the many unit test files case

Reviewed By: cbi42

Differential Revision: D42894004

Pulled By: pdillinger

fbshipit-source-id: 9e827e52ace79b86f849b6a24466e318b4b605a7
2023-02-03 15:28:52 -08:00
Peter Dillinger e17f31057b Support stack traces with gdb (and debugger invocation) (#11150)
Summary:
LIB_MODE=shared is much more efficient for building all the unit tests but comes with the downside of ugly stack traces, generally missing name demangling and source line info. Searching the internet suggests the reliable way to get stack traces with dynamic loading is with gdb.

This change automatically tries to use gdb to get a stack trace if built with LIB_MODE=shared, and only on Linux because that's where we have the capability to attach to the proper thread. (We could revise the exact conditions in the future.) If there's a failure invoking gdb, it falls back on the old method. Obscure details of making the output reasonable / pretty are in the source code comments.

Based on this, it was easy to make it so that running a test command with ROCKSDB_DEBUG=1 would invoke gdb whenever the stack trace handler was invoked, so I included that.

Intended follow-up: make LIB_MODE=shared the new default `make` build config

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11150

Test Plan:
manual, mostly by injecting an "assert(false)" into a unit test and trying different build modes etc.

Although gdb is slower to start showing stack trace output, it seems overall faster in many if not most cases, presumably because it doesn't reload the symbol table for each stack entry. At least with parallel test runs, having many tests dumping stacks with the old method can take so long it appears to hang the test run.

Reviewed By: cbi42

Differential Revision: D42894064

Pulled By: pdillinger

fbshipit-source-id: 608143309d8c69c40049c9a4abcde4f22e87b4d8
2023-02-03 13:21:03 -08:00
Peter Dillinger 0cf1008fe3 Deprecate write_global_seqno and default to false (#11179)
Summary:
This option has long been intended to be set to false by default and deprecated. It might never be practical to completely remove the feature, so that we can continue to test for backward compatibility by keeping the ability to generate DBs in the old way.

Also improved API comments.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11179

Test Plan: existing tests (with one tiny update)

Reviewed By: hx235

Differential Revision: D42973927

Pulled By: pdillinger

fbshipit-source-id: e9bc161cb933266e094aea2dff8cc03753c39dab
2023-02-03 13:00:04 -08:00
Peter Dillinger 390cc0b156 Ensure LockWAL() stall cleared for UnlockWAL() return (#11172)
Summary:
Fixes https://github.com/facebook/rocksdb/issues/11160

By counting the number of stalls placed on a write queue, we can check in UnlockWAL() whether the stall present at the start of UnlockWAL() has been cleared by the end, or wait until it's cleared.

More details in code comments and new unit test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11172

Test Plan: unit test added. Yes, it uses sleep to amplify failure on buggy behavior if present, but using a sync point to only allow new behavior would fail with the old code only because it doesn't contain the new sync point. Basically, using a sync point in UnlockWAL() could easily mask a regression by artificially limiting key behaviors. The test would only check that UnlockWAL() invokes code that *should* do the right thing, without checking that it *does* the right thing.

Reviewed By: ajkr

Differential Revision: D42894341

Pulled By: pdillinger

fbshipit-source-id: 15c9da0ca383e6aec845b29f5447d76cecbf46c3
2023-02-03 12:08:37 -08:00
anand76 63da9cfa26 Return any errors returned by ReadAsync to the MultiGet caller (#11171)
Summary:
Currently, we incorrectly return a Status::Corruption to the MultiGet caller if the file system ReadAsync cannot issue a read and returns an error for some reason, such as IOStatus::NotSupported(). In this PR, we copy the ReadAsync error to the request status so it can be returned to the user.

Tests:
Update existing unit tests and add a new one for this scenario

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11171

Reviewed By: akankshamahajan15

Differential Revision: D42950057

Pulled By: anand1976

fbshipit-source-id: 85ffcb015fa6c064c311f8a28488fec78c487869
2023-02-02 16:35:27 -08:00
Yu Zhang 701a19cc83 Enable crash test for user-defined timestamp and BlobDB combination (#11163)
Summary:
Enable the set of crash test for when user defined timestamp is enabled in combination with BlobDB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11163

Test Plan: `make check` and `db_stress`/`db_crashtest.py` with various combinations.

Reviewed By: ltamasi

Differential Revision: D42906457

Pulled By: jowlyzhang

fbshipit-source-id: 6bec6449a4213b536c787420ff30a7d17b676deb
2023-02-02 16:22:32 -08:00
changyubi fec5c8deb8 Remove NUMA setting for benchmark-linux (#11180)
Summary:
benchmark-linux is failing on main branch after https://github.com/facebook/rocksdb/issues/11074 with the following error msg:
```
/usr/bin/time -f '%e %U %S' -o /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.time numactl --interleave=all timeout 1200 ./db_bench --benchmarks=overwrite,waitforcompaction,stats --use_existing_db=1 --sync=0 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_jobs=4 --max_write_buffer_number=8 --undefok=use_blob_cache,use_shared_block_and_blob_cache,blob_cache_size,blob_cache_numshardbits,prepopulate_blob_cache,multiread_batched,cache_low_pri_pool_ratio,prepopulate_block_cache --db=/tmp/rocksdb-benchmark-datadir --wal_dir=/tmp/rocksdb-benchmark-datadir --num=20000000 --key_size=20 --value_size=400 --block_size=8192 --cache_size=10737418240 --cache_numshardbits=6 --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=none --bytes_per_sync=1048576 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --cache_low_pri_pool_ratio=0 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=1 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=1 --compaction_style=0 --num_levels=8 --min_level_to_compress=3 --level_compaction_dynamic_level_bytes=true --pin_l0_filter_and_index_blocks_in_cache=1 --duration=600 --threads=1 --merge_operator="put" --seed=1675372532 --report_file=/tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.r.csv 2>&1 | tee -a /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log
/usr/bin/time: cannot run numactl: No such file or directory
```
This PR removes the newly added NUMA setting.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11180

Test Plan: check next main branch run for benchmark-linux

Reviewed By: ajkr

Differential Revision: D42975930

Pulled By: cbi42

fbshipit-source-id: f084d39aeba9877c0752502e879c5e612b507653
2023-02-02 15:15:09 -08:00
Alan Paxton 6781009ee8 CI Benchmarking. Small configuration changes based on performance analysis. (#11074)
Summary:
First, we made a small reduction in DURATION_RW as runs were exceeding 1 hour and colliding with subsequent runs.

See Mark Callaghan’s blog post at http://smalldatum.blogspot.com/2023/01/variance-in-rocksdb-benchmarks-on-cloud.html

Configuration parameters which are not consistent with the following email from Mark (see the blog post for more context) have been updated. Where Mark has defined the parameter and we haven't, we define it explicitly. We will need to further monitor for an expected reduction in variance of test times:

To match what I did:
 ---

nsecs=1800
dbdir=/data/m/rx
resultdir=bm.lc.nt1.cm1.d0

env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=20000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=none CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb

env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=200000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=lz4 CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11074

Reviewed By: ajkr

Differential Revision: D42969668

Pulled By: cbi42

fbshipit-source-id: 1ea4e6a3901be4016108f93817eb58f74baac21a
2023-02-02 11:11:40 -08:00
Andrew Kryczka 6af16ac7c1 Update HISTORY.md for #11136 (#11177)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11177

Reviewed By: cbi42

Differential Revision: D42948946

Pulled By: ajkr

fbshipit-source-id: 783d3d9007faaa036923a0364cdd0bfbd8e78062
2023-02-02 07:50:55 -08:00
Levi Tamasi df680b24ef Clean up InvokeFilterIfNeeded a bit (#11174)
Summary:
The patch makes some code quality enhancements in `CompactionIterator::InvokeFilterIfNeeded`
including the renaming of `filter` (which is most likely a remnant of the days before the `FilterV2`
API when the compaction filter used to return a boolean) to `decision`, the removal of some
outdated comments, the elimination of an `error` flag which was only used in one failure case
out of many, as well as some small stylistic improvements. (Some the above will also come in
handy when adding compaction filter support for wide-column entities.)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11174

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D42901408

Pulled By: ltamasi

fbshipit-source-id: ab382d59a4990c5dfe1cee219d49e1d80902b666
2023-02-01 10:03:07 -08:00
Andrew Kryczka 071c33846d Allow canceling manual compaction while waiting for conflicting compaction (#11165)
Summary:
This PR adds logic to the `RunManualCompaction()` loop to check for cancellation before waiting on any conflicting compactions to finish. In case of cancellation, `RunManualCompaction()` no longer waits on conflicting compactions

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11165

Test Plan: repro test case

Reviewed By: cbi42

Differential Revision: D42864058

Pulled By: ajkr

fbshipit-source-id: ea4dd1a8f294abe212905495a8fbe8f07fca3f5a
2023-01-31 16:57:49 -08:00
Levi Tamasi 753d4d5078 Support using GetEntity as a verification method in the non-batched stress tests (#11144)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11144

Test Plan: Ran a simple blackbox crash test.

Reviewed By: akankshamahajan15

Differential Revision: D42791464

Pulled By: ltamasi

fbshipit-source-id: 8eb6e62f0bc47f709816136ff3ded0a41d04fab8
2023-01-31 10:17:48 -08:00
Levi Tamasi a82021c3d0 Fix a bug where GetEntity would expose a blob reference (#11162)
Summary:
The patch fixes a feature interaction bug between BlobDB and the `GetEntity` API:
without the patch, `GetEntity` would return the blob reference (wrapped into a
single-column entity) instead of the actual blob value.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11162

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D42854092

Pulled By: ltamasi

fbshipit-source-id: f750d0ff57def107da16f545077ddce9860ff21a
2023-01-31 09:59:25 -08:00
Peter Dillinger 94e3beec77 Cleanup, improve, stress test LockWAL() (#11143)
Summary:
The previous API comments for LockWAL didn't provide much about why you might want to use it, and didn't really meet what one would infer its contract was. Also, LockWAL was not in db_stress / crash test. In this change:

* Implement a counting semantics for LockWAL()+UnlockWAL(), so that they can safely be used concurrently across threads or recursively within a thread. This should make the API much less bug-prone and easier to use.
* Make sure no UnlockWAL() is needed after non-OK LockWAL() (to match RocksDB conventions)
* Make UnlockWAL() reliably return non-OK when there's no matching LockWAL() (for debug-ability)
* Clarify API comments on LockWAL(), UnlockWAL(), FlushWAL(), and SyncWAL(). Their exact meanings are not obvious, and I don't think it's appropriate to talk about implementation mutexes in the API comments, but about what operations might block each other.
* Add LockWAL()/UnlockWAL() to db_stress and crash test, mostly to check for assertion failures, but also checks that latest seqno doesn't change while WAL is locked. This is simpler to add when LockWAL() is allowed in multiple threads.
* Remove unnecessary use of sync points in test DBWALTest::LockWal. There was a bug during development of above changes that caused this test to fail sporadically, with and without this sync point change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11143

Test Plan: unit tests added / updated, added to stress/crash test

Reviewed By: ajkr

Differential Revision: D42848627

Pulled By: pdillinger

fbshipit-source-id: 6d976c51791941a31fd8fbf28b0f82e888d9f4b4
2023-01-30 22:52:30 -08:00
sdong 36174d89a6 DB Stress to fix a false assertion (#11164)
Summary:
Seeting this error in stress test:

db_stress: internal_repo_rocksdb/repo/db_stress_tool/db_stress_test_base.cc:2459: void rocksdb::StressTest::Open(rocksdb::SharedState *): Assertion `txn_db_ == nullptr' failed. Received signal 6 (Aborted)
......

It doesn't appear that txn_db_ is set to nullptr at all. We set ithere.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11164

Test Plan: Run db_stress transaction and non-transation with low kill rate and see restarting without assertion

Reviewed By: ajkr

Differential Revision: D42855662

fbshipit-source-id: 06816d37cce9c94a81cb54ab238fb73aa102ed46
2023-01-30 19:45:47 -08:00
Yu Zhang 24ac53d81a Use user key on sst file for blob verification for Get and MultiGet (#11105)
Summary:
Use the user key on sst file for blob verification for `Get` and `MultiGet` instead of the user key passed from caller.

Add tests for `Get` and `MultiGet` operations when user defined timestamp feature is enabled in a BlobDB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11105

Test Plan:
make V=1 db_blob_basic_test
./db_blob_basic_test --gtest_filter="DBBlobTestWithTimestamp.*"

Reviewed By: ltamasi

Differential Revision: D42716487

Pulled By: jowlyzhang

fbshipit-source-id: 5987ecbb7e56ddf46d2467a3649369390789506a
2023-01-30 10:21:21 -08:00
akankshamahajan 79e57a39a3 Move ExternalSSTTestEnv to FileSystemWrapper (#11139)
Summary:
Migrate ExternalSSTTestEnv to FileSystemWrapper

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11139

Reviewed By: anand1976

Differential Revision: D42780180

Pulled By: akankshamahajan15

fbshipit-source-id: 9a4448c9fe5186b518235fe11e1a34dcad897cdd
2023-01-27 14:51:39 -08:00
sdong 4720ba4391 Remove RocksDB LITE (#11147)
Summary:
We haven't been actively mantaining RocksDB LITE recently and the size must have been gone up significantly. We are removing the support.

Most of changes were done through following comments:

unifdef -m -UROCKSDB_LITE `git grep -l ROCKSDB_LITE | egrep '[.](cc|h)'`

by Peter Dillinger. Others changes were manually applied to build scripts, CircleCI manifests, ROCKSDB_LITE is used in an expression and file db_stress_test_base.cc.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11147

Test Plan: See CI

Reviewed By: pdillinger

Differential Revision: D42796341

fbshipit-source-id: 4920e15fc2060c2cd2221330a6d0e5e65d4b7fe2
2023-01-27 13:14:19 -08:00
Yu Zhang 6943ff6e50 Remove deprecated util functions in options_util.h (#11126)
Summary:
Remove the util functions in options_util.h that have previously been marked deprecated.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11126

Test Plan: `make check`

Reviewed By: ltamasi

Differential Revision: D42757496

Pulled By: jowlyzhang

fbshipit-source-id: 2a138a3c207d0e0e0bbb4d99548cf2cadb44bcfb
2023-01-27 11:10:53 -08:00
Andrew Kryczka 97c1024d3e Include db_stress verification method in failure message (#11133)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11133

Test Plan:
- ran it a few times on a mismatching DB+expected state; verified error messages look right:

```
Verification failed for column family 0 key 000000000000D553000000000000014C0000000000000142 (163988): value_from_db: , value_from_expected: 25E7B53421202322, msg: GetMergeOperands verification: Value not found: NotFound:
Verification failed for column family 0 key 000000000000AAE2787878 (131123): value_from_db: , value_from_expected: B2A69C18B6B7B4B5BABBB8B9BEBFBCBDA2A3A0A1A6A7A4A5, msg: Iterator verification: Value not found: NotFound:
Verification failed for column family 0 key 00000000000080C6000000000000004C78787878 (98409): value_from_db: , value_from_expected: 67AB7E1E636261606F6E6D6C6B6A6968, msg: Get verification: Value not found: NotFound:
```

Reviewed By: hx235

Differential Revision: D42757072

Pulled By: ajkr

fbshipit-source-id: b0a4a0aaa5be5d110434324853ac92aaa6972d89
2023-01-27 07:45:25 -08:00
Changyu Bi c94c8fcbd4 Remove deprecated FileSystem::Load() (#11122)
Summary:
user should use FileSystem::CreateFromString() instead.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11122

Reviewed By: ajkr

Differential Revision: D42727580

Pulled By: cbi42

fbshipit-source-id: c68b17bb82ba9dee46ba23b677d87ecf0a1e06c8
2023-01-26 20:20:58 -08:00
Karim TAAM a1e92bd956 use verify checksum option in block based table reader Open() (#11099)
Summary:
## Description
In this issue https://github.com/facebook/rocksdb/issues/11002 we found that when we use rocksdb with the `verify checksum` read_option to false the verification is done anyway

By analyzing the code along the stacktrace I saw that at the level of https://github.com/facebook/rocksdb/compare/main...matkt:feature/use-verify-checksum-in-block-based-table-reader?expand=1#diff-57ed8c49db2bdd4db7618646a177397674bbf25beacacecb104070071d30129f we are not keeping all the options and we forget the `verify_checksum`

the comment in this class suggests that it should be managed https://github.com/facebook/rocksdb/compare/main...matkt:feature/use-verify-checksum-in-block-based-table-reader?expand=1#diff-57ed8c49db2bdd4db7618646a177397674bbf25beacacecb104070071d30129fL581

<img width="1724" alt="204511641-86ab4b9b-45e5-4a2b-a13d-81fa26435d38" src="https://user-images.githubusercontent.com/26581503/213152802-c46bc1c7-a3a2-4a6f-9bb1-bf92ee93af7a.png">

this PR just adds the line to manage the `verify checksum`

## Tests

- Running unit tests
- Test without setting `verify checksum` and verifying that we are calling the checksum code
- Test by setting `verify checksum` to true and verifying that we are calling the checksum code
- Test by setting `verify checksum` to false and verifying that we are **not** calling the checksum code

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11099

Reviewed By: cbi42

Differential Revision: D42679881

Pulled By: ajkr

fbshipit-source-id: c7dd10768282fd0699f7e1bf397ceb7adbea4ab6
2023-01-26 17:38:59 -08:00
Andrew Kryczka b44cbbf709 Fix GetMergeOperands() returning MergeInProgress (#11136)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11136

Test Plan: the provided unit test used to fail due to `GetMergeOperands()` returning `Status::MergeInProgress()`; it passes now because the `GetMergeOperands()` call returns `Status::OK()`

Reviewed By: pdillinger

Differential Revision: D42759198

Pulled By: ajkr

fbshipit-source-id: 878f9f40ccc1d7e2fe7b1352814bae3a49c19939
2023-01-26 15:11:19 -08:00
dependabot[bot] dcf93b7b3e Bump commonmarker from 0.23.6 to 0.23.7 in /docs (#11128)
Summary:
Bumps [commonmarker](https://github.com/gjtorikian/commonmarker) from 0.23.6 to 0.23.7.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/gjtorikian/commonmarker/releases">commonmarker's releases</a>.</em></p>
<blockquote>
<h2>v0.23.7</h2>
<h2>What's Changed</h2>
<ul>
<li>C API stable test by <a href="https://github.com/gjtorikian"><code>@​gjtorikian</code></a> in <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/201">gjtorikian/commonmarker#201</a></li>
<li>Update to 29.0.gfm.7 by <a href="https://github.com/anticomputer"><code>@​anticomputer</code></a> in <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/224">gjtorikian/commonmarker#224</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/gjtorikian/commonmarker/compare/v0.23.6...v0.23.7">https://github.com/gjtorikian/commonmarker/compare/v0.23.6...v0.23.7</a></p>
<h2>v0.23.7.pre1</h2>
<h2>What's Changed</h2>
<ul>
<li>C API stable test by <a href="https://github.com/gjtorikian"><code>@​gjtorikian</code></a> in <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/201">gjtorikian/commonmarker#201</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/gjtorikian/commonmarker/compare/v0.23.6...v0.23.7.pre1">https://github.com/gjtorikian/commonmarker/compare/v0.23.6...v0.23.7.pre1</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/gjtorikian/commonmarker/blob/main/CHANGELOG.md">commonmarker's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2><a href="https://github.com/gjtorikian/commonmarker/tree/v1.0.0.pre6">v1.0.0.pre6</a> (2023-01-09)</h2>
<p><a href="https://github.com/gjtorikian/commonmarker/compare/v1.0.0.pre5...v1.0.0.pre6">Full Changelog</a></p>
<p><strong>Closed issues:</strong></p>
<ul>
<li>Cargo.lock prevents Ruby 3.2.0 from installing commonmarker v1.0.0.pre4 <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/211">https://github.com/facebook/rocksdb/issues/211</a></li>
</ul>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>always use rb_sys (don't use Ruby's emerging cargo tooling where available) <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/213">https://github.com/facebook/rocksdb/issues/213</a> (<a href="https://github.com/kivikakk">kivikakk</a>)</li>
</ul>
<h2><a href="https://github.com/gjtorikian/commonmarker/tree/v1.0.0.pre5">v1.0.0.pre5</a> (2023-01-08)</h2>
<p><a href="https://github.com/gjtorikian/commonmarker/compare/v1.0.0.pre4...v1.0.0.pre5">Full Changelog</a></p>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>Provide 3.2 build support <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/212">https://github.com/facebook/rocksdb/issues/212</a> (<a href="https://github.com/gjtorikian">gjtorikian</a>)</li>
</ul>
<h2><a href="https://github.com/gjtorikian/commonmarker/tree/v1.0.0.pre4">v1.0.0.pre4</a> (2022-12-28)</h2>
<p><a href="https://github.com/gjtorikian/commonmarker/compare/v1.0.0.pre3...v1.0.0.pre4">Full Changelog</a></p>
<p><strong>Closed issues:</strong></p>
<ul>
<li>Will the cmark-gfm branch continue to be maintained for awhile? <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/207">https://github.com/facebook/rocksdb/issues/207</a></li>
</ul>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>Implement native syntax highlighting <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/209">https://github.com/facebook/rocksdb/issues/209</a> (<a href="https://github.com/gjtorikian">gjtorikian</a>)</li>
<li>Bump magnus from 0.4.3 to 0.4.4 <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/208">https://github.com/facebook/rocksdb/issues/208</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li>
<li>Bump magnus from 0.4.2 to 0.4.3 <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/206">https://github.com/facebook/rocksdb/issues/206</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li>
<li>Bump comrak from 0.14.0 to 0.15.0 <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/205">https://github.com/facebook/rocksdb/issues/205</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li>
<li>Bump magnus from 0.4.1 to 0.4.2 <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/204">https://github.com/facebook/rocksdb/issues/204</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li>
</ul>
<h2><a href="https://github.com/gjtorikian/commonmarker/tree/v1.0.0.pre3">v1.0.0.pre3</a> (2022-11-30)</h2>
<p><a href="https://github.com/gjtorikian/commonmarker/compare/v1.0.0.pre.2...v1.0.0.pre3">Full Changelog</a></p>
<p><strong>Closed issues:</strong></p>
<ul>
<li>Code block incorrectly parsed in commonmarker 1.0.0.pre <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/202">https://github.com/facebook/rocksdb/issues/202</a></li>
</ul>
<p><strong>Merged pull requests:</strong></p>
<ul>
<li>Windows build <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/pull/197">https://github.com/facebook/rocksdb/issues/197</a> (<a href="https://github.com/gjtorikian">gjtorikian</a>)</li>
</ul>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/734fd86c9719643e0ef058d983daeaf2523ea78b"><code>734fd86</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/224">https://github.com/facebook/rocksdb/issues/224</a> from gjtorikian/update-to-29.0.gfm.7</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/2e724ec52a5f66116c03682b6cc76c659f8efaf5"><code>2e724ec</code></a> Turned off Rubocop.</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/9c923b0bfdb2e4087779042a483a2d6957dceb7a"><code>9c923b0</code></a> 💎 release 0.23.7</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/30419c25e8566f3e02a0fad76f5e1a82341edeb1"><code>30419c2</code></a> Added call to cmark_init_standard_node_flags()</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/9007c3798f2f6fcff80e49a7ced317887f259326"><code>9007c37</code></a> Update cmark-upstream to <a href="https://github.com/github/cmark-gfm/commit/57d5e093e">https://github.com/github/cmark-gfm/commit/57d5e093e</a>...</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/1cfec133730ecdf683b90528a6bb7253b68a8208"><code>1cfec13</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/201">https://github.com/facebook/rocksdb/issues/201</a> from gjtorikian/c-api-stable-test</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/bbf631b413cf012490ccd63d423e69dfa855369b"><code>bbf631b</code></a> lint</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/5b807a115d02dc7456467dbed871a3e91733ca7b"><code>5b807a1</code></a> ease up</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/9a24e6d2fe3a1d384e29702c1a223a4fea5bdc87"><code>9a24e6d</code></a> Test fake version</li>
<li><a href="https://github.com/gjtorikian/commonmarker/commit/d8a43bc73a6ff5c15cd12ce7dcf49902ee8031ec"><code>d8a43bc</code></a> Allow for manual dispatch</li>
<li>Additional commits viewable in <a href="https://github.com/gjtorikian/commonmarker/compare/v0.23.6...v0.23.7">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commonmarker&package-manager=bundler&previous-version=0.23.6&new-version=0.23.7)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

 ---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts).

</details>

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11128

Reviewed By: ajkr

Differential Revision: D42752086

Pulled By: cbi42

fbshipit-source-id: 6992b6f1096400a6b10b79fe36bf955fec841b71
2023-01-26 12:07:52 -08:00
Levi Tamasi a6cfdd4eda Fix the HISTORY.md entry related to the removed statistics (#11140)
Summary:
Some histograms were incorrectly categorized as tickers.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11140

Reviewed By: anand1976

Differential Revision: D42780030

Pulled By: ltamasi

fbshipit-source-id: 5aca8ec5baad8f73676aaa9d6cdbbd2a619c8a89
2023-01-26 10:38:45 -08:00
akankshamahajan 986c5b9d4e Migrate TestEnv in listener_test.cc to FileSystemWrapper (#11125)
Summary:
Migrate derived classes from EnvWrapper to FileSystemWrapper so we can eventually deprecate the storage methods in Env.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11125

Test Plan: CircleCI jobs

Reviewed By: anand1976

Differential Revision: D42732241

Pulled By: akankshamahajan15

fbshipit-source-id: c89a70a79fcfb13e158bf8919b1a87a9de133222
2023-01-25 22:42:22 -08:00
sdong e808858ae0 Remove Stats related to compressed block cache (#11135)
Summary:
Since compressed block cache is removed, those stats are not needed. They are removed in different PR in case there is a problem with it. The stats are removed in the same way in https://github.com/facebook/rocksdb/pull/11131/ . HISTORY.md was already updated by mistake, and it would be correct after merging this PR.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11135

Test Plan: Watch CI

Reviewed By: ltamasi

Differential Revision: D42757616

fbshipit-source-id: bd7cb782585c8535ce5784295225c376f3011f35
2023-01-25 15:37:50 -08:00
Levi Tamasi 6da2e20df3 Remove more obsolete statistics (#11131)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11131

Test Plan: `make check`

Reviewed By: pdillinger

Differential Revision: D42753997

Pulled By: ltamasi

fbshipit-source-id: ce8b84c1e55374257e93ed74fd255c9b759723ce
2023-01-25 15:14:13 -08:00
Heiko Becker 88edfbfb5e Fix build with gcc 13 by including <cstdint> (#11118)
Summary:
Like other versions before, gcc 13 moved some includes around and as a result <cstdint> is no longer transitively included [1]. Explicitly include it for uint{32,64}_t.

[1] https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11118

Reviewed By: cbi42

Differential Revision: D42711356

Pulled By: ajkr

fbshipit-source-id: 5ea257b85b7017f40fd8fdbce965336da95c55b2
2023-01-25 14:30:32 -08:00
Andrew Kryczka 6a5071ceb5 Support PutEntity in trace analyzer (#11127)
Summary:
Add the most basic support such that trace_analyzer commands no longer fail with
```
Cannot process the write batch in the trace
Cannot process the TraceRecord
PutEntityCF not implemented
Cannot process the trace
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11127

Reviewed By: cbi42

Differential Revision: D42732319

Pulled By: ajkr

fbshipit-source-id: 162d8a31318672a46539b1b042ec25f69b25c4ed
2023-01-25 14:27:02 -08:00
Peter Dillinger 546e213c4f Fix DelayWrite() calls for two_write_queues (#11130)
Summary:
PR https://github.com/facebook/rocksdb/issues/11020 fixed a case where it was easy to deadlock the DB with LockWAL() but introduced a bug showing up as a rare assertion failure in the stress test. Specifically, `assert(w->state == STATE_INIT)` in `WriteThread::LinkOne()` called from `BeginWriteStall()`, `DelayWrite()`, `WriteImplWALOnly()`. I haven't been about to generate a unit test that reproduces this failure but I believe the root cause is that DelayWrite() was never meant to be re-entrant, only called from the DB's write_thread_ leader. https://github.com/facebook/rocksdb/issues/11020 introduced a call to DelayWrite() from the nonmem_write_thread_ group leader.

This fix is to make DelayWrite() apply to the specific write queue that it is being called from (inject a dummy write stall entry to the head of the appropriate write queue). WriteController is re-entrant, based on polling and state changes signalled with bg_cv_, so can manage stalling two queues. The only anticipated complication (called out by Andrew in previous PR) is that we don't want timed write delays being injected in parallel for the two queues, because that dimishes the intended throttling effect. Thus, we only allow timed delays for the primary write queue.

HISTORY not updated because this is intended for the same release where the bug was introduced.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11130

Test Plan:
Although I was not able to reproduce the assertion failure, I was able to reproduce a distinct flaw with what I believe is the same root cause: a kind of deadlock if both write queues need to wake up from stopped writes. Only one will be waiting on bg_cv_ (the other waiting in `LinkOne()` for the write queue to open up), so a single SignalAll() will only unblock one of the queues, with the other re-instating the stop until another signal on bg_cv_. A simple unit test is added for this case.

Will also run crash_test_with_multiops_wc_txn for a while looking for issues.

Reviewed By: ajkr

Differential Revision: D42749330

Pulled By: pdillinger

fbshipit-source-id: 4317dd899a93d57c26fd5af7143038f82d4d4d1b
2023-01-25 14:18:27 -08:00
Peter Dillinger 9afa0f05ad Remove deprecated Env::LoadEnv() (#11121)
Summary:
Can use Env::CreateFromString() instead

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11121

Test Plan: unit tests updated

Reviewed By: cbi42

Differential Revision: D42723813

Pulled By: pdillinger

fbshipit-source-id: 5d4b5b10225dfdaf662f5f8049ee965a05d3edc9
2023-01-25 12:08:49 -08:00
Levi Tamasi 99e559533d Remove some deprecated/obsolete statistics from the API (#11123)
Summary:
These tickers/histograms have been obsolete (and not populated) for a long time.
The patch removes them from the API completely. Note that this means that the
numeric values of the remaining tickers change in the C++ code as they get shifted up.
This should be OK: the values of some existing tickers have changed many times
over the years as items have been added in the middle. (In contrast, the convention
in the Java bindings is to keep the ids, which are not guaranteed to be the same
as the ids on the C++ side, the same across releases.)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11123

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D42727793

Pulled By: ltamasi

fbshipit-source-id: e058a155a20b05b45f53e67ee380aece1b43b6c5
2023-01-24 20:56:15 -08:00
anand76 bcbab59c55 Migrate ErrorEnv from EnvWrapper to FileSystemWrapper (#11124)
Summary:
Migrate ErrorEnv from EnvWrapper to FileSystemWrapper so we can eventually deprecate the storage methods in Env.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11124

Reviewed By: akankshamahajan15

Differential Revision: D42727791

Pulled By: anand1976

fbshipit-source-id: e8362ad624dc28e55c99fc35eda12866755f62c6
2023-01-24 17:14:35 -08:00
sdong 2800aa069a Remove compressed block cache (#11117)
Summary:
Compressed block cache is replaced by compressed secondary cache. Remove the feature.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11117

Test Plan: See CI passes

Reviewed By: pdillinger

Differential Revision: D42700164

fbshipit-source-id: 6cbb24e460da29311150865f60ecb98637f9f67d
2023-01-24 17:09:19 -08:00
Peter Dillinger 4a9185340d A better contract for best_efforts_recovery (#11085)
Summary:
Capture more of the original intent at a high level, without getting bogged down in low-level details.

The old text made some weak promises about handling of LOCK files. There should be no specific concern for LOCK files, because we already rely on LockFile() to create the file if it's not present already. And the lock file is generally size 0, so don't have to worry about truncation. Added a unit test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11085

Test Plan: existing tests, and a new one.

Reviewed By: siying

Differential Revision: D42713233

Pulled By: pdillinger

fbshipit-source-id: 2fce7c974d35fac065037c9c4c7326a59c9fe340
2023-01-24 12:55:03 -08:00
Changyu Bi e0ea0dc6bd Improve documentation for allow_ingest_behind (#11119)
Summary:
update documentation to mention that only universal compaction is supported.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11119

Reviewed By: ajkr

Differential Revision: D42715986

Pulled By: cbi42

fbshipit-source-id: 91b145d3318334cb92857c5c0ffc0efed6fa4363
2023-01-24 12:12:19 -08:00
Hui Xiao 86fa2592be Fix data race on ColumnFamilyData::flush_reason by letting FlushRequest/Job owns flush_reason instead of CFD (#11111)
Summary:
**Context:**
Concurrent flushes on the same CF can set on `ColumnFamilyData::flush_reason` before each other flush finishes. An symptom is one CF has different flush_reason with others though all of them are in an atomic flush  `db_stress: db/db_impl/db_impl_compaction_flush.cc:423: rocksdb::Status rocksdb::DBImpl::AtomicFlushMemTablesToOutputFiles(const rocksdb::autovector<rocksdb::DBImpl::BGFlushArg>&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority): Assertion cfd->GetFlushReason() == cfds[0]->GetFlushReason() failed. `

**Summary:**
Suggested by ltamasi, we now refactor and let FlushRequest/Job to own flush_reason as there is no good way to define `ColumnFamilyData::flush_reason` in face of concurrent flushes on the same CF (which wasn't the case a long time ago when `ColumnFamilyData::flush_reason ` first introduced`)

**Tets:**
- new unit test
- make check
- aggressive crash test rehearsal

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11111

Reviewed By: ajkr

Differential Revision: D42644600

Pulled By: hx235

fbshipit-source-id: 8589c8184869d3415e5b780c887f877818a5ebaf
2023-01-24 09:54:04 -08:00
Hui Xiao 7e7548477c Update HISTORY.md/version.h/format compatiblity test for 7.10 release (#11114)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11114

Reviewed By: ajkr

Differential Revision: D42685234

Pulled By: hx235

fbshipit-source-id: 79908a66ab9052a2552f080049065462ebf2f94c
2023-01-23 13:26:11 -08:00
576 changed files with 4843 additions and 7711 deletions

No files matched your search

+32 -38
View File
@@ -152,15 +152,15 @@ commands:
steps:
- run:
name: "Test low-variance benchmarks"
command: ./tools/benchmark_ci.py --db_dir /tmp/rocksdb-benchmark-datadir --output_dir /tmp/benchmark-results --num_keys 10000000
command: ./tools/benchmark_ci.py --db_dir /tmp/rocksdb-benchmark-datadir --output_dir /tmp/benchmark-results --num_keys 20000000
environment:
LD_LIBRARY_PATH: /usr/local/lib
# How long to run parts of the test(s)
DURATION_RO: 400
DURATION_RW: 700
DURATION_RO: 300
DURATION_RW: 500
# Keep threads within physical capacity of server (much lower than default)
NUM_THREADS: 1
MAX_BACKGROUND_JOBS: 3
MAX_BACKGROUND_JOBS: 4
# Don't run a couple of "optional" initial tests
CI_TESTS_ONLY: "true"
# Reduce configured size of levels to ensure more levels in the leveled compaction LSM tree
@@ -170,7 +170,11 @@ commands:
# The benchmark host has 32GB memory
# The following values are tailored to work with that
# Note, tests may not exercise the targeted issues if the memory is increased on new test hosts.
COMPRESSION_TYPE: "none"
CACHE_INDEX_AND_FILTER_BLOCKS: 1
MIN_LEVEL_TO_COMPRESS: 3
CACHE_SIZE_MB: 10240
MB_WRITE_PER_SEC: 2
post-benchmarks:
steps:
@@ -269,12 +273,12 @@ jobs:
./sst_dump --help | grep -E -q 'Supported compression types: kNoCompression$' # Verify no compiled in compression
- post-steps
build-linux-shared_lib-alt_namespace-status_checked:
build-linux-static_lib-alt_namespace-status_checked:
executor: linux-docker
resource_class: 2xlarge
steps:
- pre-steps
- run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 ROCKSDB_MODIFY_NPHASH=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j32 check
- run: ASSERT_STATUS_CHECKED=1 TEST_UINT128_COMPAT=1 ROCKSDB_MODIFY_NPHASH=1 LIB_MODE=static OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j24 check
- post-steps
build-linux-release:
@@ -282,11 +286,21 @@ jobs:
resource_class: 2xlarge
steps:
- checkout # check out the code in the project directory
- run: make V=1 -j32 LIB_MODE=shared release
- run: ls librocksdb.so # ensure shared lib built
- run: ./db_stress --version # ensure with gflags
- run: make clean
- run: make V=1 -j32 release
- run: ls librocksdb.a # ensure static lib built
- run: ./db_stress --version # ensure with gflags
- run: make clean
- run: apt-get remove -y libgflags-dev
- run: make V=1 -j32 LIB_MODE=shared release
- run: ls librocksdb.so # ensure shared lib built
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- run: make clean
- run: make V=1 -j32 release
- run: ls librocksdb.a # ensure static lib built
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- post-steps
@@ -302,27 +316,6 @@ jobs:
- run: USE_RTTI=1 DEBUG_LEVEL=0 make V=1 -j16 static_lib tools db_bench
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
build-linux-lite:
executor: linux-docker
resource_class: large
steps:
- pre-steps
- run: LITE=1 make V=1 J=8 -j8 check
- post-steps
build-linux-lite-release:
executor: linux-docker
resource_class: large
steps:
- checkout # check out the code in the project directory
- run: LITE=1 make V=1 -j8 release
- run: ./db_stress --version # ensure with gflags
- run: make clean
- run: apt-get remove -y libgflags-dev
- run: LITE=1 make V=1 -j8 release
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
- post-steps
build-linux-clang-no_test_run:
executor: linux-docker
resource_class: xlarge
@@ -427,7 +420,10 @@ jobs:
steps:
- checkout # check out the code in the project directory
- run: apt-get update -y && apt-get install -y libgflags-dev
- run: make V=1 -j8 unity_test
- run:
name: "Unity build"
command: make V=1 -j8 unity_test
no_output_timeout: 20m
- run: make V=1 -j8 -k check-headers # could be moved to a different build
- post-steps
@@ -438,7 +434,7 @@ jobs:
- pre-steps
- setup-folly
- build-folly
- run: USE_FOLLY=1 CC=gcc-7 CXX=g++-7 V=1 make -j32 check
- run: USE_FOLLY=1 LIB_MODE=static CC=gcc-7 CXX=g++-7 V=1 make -j32 check # TODO: LIB_MODE only to work around unresolved linker failures
- post-steps
build-linux-gcc-7-with-folly-lite-no-test:
@@ -484,7 +480,7 @@ jobs:
resource_class: 2xlarge
steps:
- pre-steps
- run: CC=gcc-11 CXX=g++-11 V=1 make -j32 all microbench
- run: LIB_MODE=static CC=gcc-11 CXX=g++-11 V=1 make -j32 all microbench # TODO: LIB_MODE only to work around unresolved linker failures
- post-steps
build-linux-clang-13-no_test_run:
@@ -503,7 +499,7 @@ jobs:
- pre-steps
- setup-folly
- build-folly
- run: CC=clang-13 CXX=clang++-13 USE_CLANG=1 USE_FOLLY=1 COMPILE_WITH_UBSAN=1 COMPILE_WITH_ASAN=1 make -j32 check
- run: CC=clang-13 CXX=clang++-13 LIB_MODE=static USE_CLANG=1 USE_FOLLY=1 COMPILE_WITH_UBSAN=1 COMPILE_WITH_ASAN=1 make -j32 check # TODO: LIB_MODE only to work around unresolved linker failures
- post-steps
# This job is only to make sure the microbench tests are able to run, the benchmark result is not meaningful as the CI host is changing.
@@ -520,7 +516,7 @@ jobs:
resource_class: large
steps:
- pre-steps
- run: ulimit -S -n `ulimit -H -n` && make V=1 -j8 CRASH_TEST_EXT_ARGS='--duration=960 --max_key=2500000' blackbox_crash_test_with_atomic_flush
- run: ulimit -S -n `ulimit -H -n` && make V=1 -j8 CRASH_TEST_EXT_ARGS='--duration=960 --max_key=2500000 --async_io=0' blackbox_crash_test_with_atomic_flush
- post-steps
build-linux-crashtest-tiered-storage-bb:
@@ -530,7 +526,7 @@ jobs:
- pre-steps
- run:
name: "run crashtest"
command: ulimit -S -n `ulimit -H -n` && make V=1 -j32 CRASH_TEST_EXT_ARGS=--duration=10800 blackbox_crash_test_with_tiered_storage
command: ulimit -S -n `ulimit -H -n` && make V=1 -j32 CRASH_TEST_EXT_ARGS='--duration=10800 --async_io=0' blackbox_crash_test_with_tiered_storage
no_output_timeout: 100m
- post-steps
@@ -541,7 +537,7 @@ jobs:
- pre-steps
- run:
name: "run crashtest"
command: ulimit -S -n `ulimit -H -n` && make V=1 -j32 CRASH_TEST_EXT_ARGS=--duration=10800 whitebox_crash_test_with_tiered_storage
command: ulimit -S -n `ulimit -H -n` && make V=1 -j32 CRASH_TEST_EXT_ARGS='--duration=10800 --async_io=0' whitebox_crash_test_with_tiered_storage
no_output_timeout: 100m
- post-steps
@@ -821,18 +817,16 @@ workflows:
- build-linux-cmake-with-folly-coroutines
- build-linux-cmake-with-benchmark
- build-linux-encrypted_env-no_compression
- build-linux-lite
jobs-linux-run-tests-san:
jobs:
- build-linux-clang10-asan
- build-linux-clang10-ubsan
- build-linux-clang10-mini-tsan
- build-linux-shared_lib-alt_namespace-status_checked
- build-linux-static_lib-alt_namespace-status_checked
jobs-linux-no-test-run:
jobs:
- build-linux-release
- build-linux-release-rtti
- build-linux-lite-release
- build-examples
- build-fuzzers
- build-linux-clang-no_test_run
+1 -6
View File
@@ -496,12 +496,6 @@ if(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-builtin-memcmp")
endif()
option(ROCKSDB_LITE "Build RocksDBLite version" OFF)
if(ROCKSDB_LITE)
add_definitions(-DROCKSDB_LITE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-exceptions -Os")
endif()
if(CMAKE_SYSTEM_NAME MATCHES "Cygwin")
add_definitions(-fno-builtin-memcmp -DCYGWIN)
elseif(CMAKE_SYSTEM_NAME MATCHES "Darwin")
@@ -885,6 +879,7 @@ set(SOURCES
util/compression_context_cache.cc
util/concurrent_task_limiter_impl.cc
util/crc32c.cc
util/data_structure.cc
util/dynamic_bloom.cc
util/hash.cc
util/murmurhash.cc
+39 -1
View File
@@ -1,5 +1,43 @@
# Rocksdb Change Log
## Unreleased
## 8.0.0 (02/19/2023)
### Behavior changes
* `ReadOptions::verify_checksums=false` disables checksum verification for more reads of non-`CacheEntryRole::kDataBlock` blocks.
* In case of scan with async_io enabled, if posix doesn't support IOUring, Status::NotSupported error will be returned to the users. Initially that error was swallowed and reads were switched to synchronous reads.
### Bug Fixes
* Fixed a data race on `ColumnFamilyData::flush_reason` caused by concurrent flushes.
* Fixed an issue in `Get` and `MultiGet` when user-defined timestamps is enabled in combination with BlobDB.
* Fixed some atypical behaviors for `LockWAL()` such as allowing concurrent/recursive use and not expecting `UnlockWAL()` after non-OK result. See API comments.
* Fixed a feature interaction bug where for blobs `GetEntity` would expose the blob reference instead of the blob value.
* Fixed `DisableManualCompaction()` and `CompactRangeOptions::canceled` to cancel compactions even when they are waiting on conflicting compactions to finish
* Fixed a bug in which a successful `GetMergeOperands()` could transiently return `Status::MergeInProgress()`
* Return the correct error (Status::NotSupported()) to MultiGet caller when ReadOptions::async_io flag is true and IO uring is not enabled. Previously, Status::Corruption() was being returned when the actual failure was lack of async IO support.
* Fixed a bug in DB open/recovery from a compressed WAL that was caused due to incorrect handling of certain record fragments with the same offset within a WAL block.
### Feature Removal
* Remove RocksDB Lite.
* The feature block_cache_compressed is removed. Statistics related to it are removed too.
* Remove deprecated Env::LoadEnv(). Use Env::CreateFromString() instead.
* Remove deprecated FileSystem::Load(). Use FileSystem::CreateFromString() instead.
* Removed the deprecated version of these utility functions and the corresponding Java bindings: `LoadOptionsFromFile`, `LoadLatestOptions`, `CheckOptionsCompatibility`.
* Remove the FactoryFunc from the LoadObject method from the Customizable helper methods.
### Public API Changes
* Moved rarely-needed Cache class definition to new advanced_cache.h, and added a CacheWrapper class to advanced_cache.h. Minor changes to SimCache API definitions.
* Completely removed the following deprecated/obsolete statistics: the tickers `BLOCK_CACHE_INDEX_BYTES_EVICT`, `BLOCK_CACHE_FILTER_BYTES_EVICT`, `BLOOM_FILTER_MICROS`, `NO_FILE_CLOSES`, `STALL_L0_SLOWDOWN_MICROS`, `STALL_MEMTABLE_COMPACTION_MICROS`, `STALL_L0_NUM_FILES_MICROS`, `RATE_LIMIT_DELAY_MILLIS`, `NO_ITERATORS`, `NUMBER_FILTERED_DELETES`, `WRITE_TIMEDOUT`, `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`, `BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, `BLOCK_CACHE_COMPRESSION_DICT_BYTES_EVICT` as well as the histograms `STALL_L0_SLOWDOWN_COUNT`, `STALL_MEMTABLE_COMPACTION_COUNT`, `STALL_L0_NUM_FILES_COUNT`, `HARD_RATE_LIMIT_DELAY_COUNT`, `SOFT_RATE_LIMIT_DELAY_COUNT`, `BLOB_DB_GC_MICROS`, and `NUM_DATA_BLOCKS_READ_PER_LEVEL`. Note that as a result, the C++ enum values of the still supported statistics have changed. Developers are advised to not rely on the actual numeric values.
* Deprecated IngestExternalFileOptions::write_global_seqno and change default to false. This option only needs to be set to true to generate a DB compatible with RocksDB versions before 5.16.0.
* Remove deprecated APIs `GetColumnFamilyOptionsFrom{Map|String}(const ColumnFamilyOptions&, ..)`, `GetDBOptionsFrom{Map|String}(const DBOptions&, ..)`, `GetBlockBasedTableOptionsFrom{Map|String}(const BlockBasedTableOptions& table_options, ..)` and ` GetPlainTableOptionsFrom{Map|String}(const PlainTableOptions& table_options,..)`.
* Added a subcode of `Status::Corruption`, `Status::SubCode::kMergeOperatorFailed`, for users to identify corruption failures originating in the merge operator, as opposed to RocksDB's internally identified data corruptions
### Build Changes
* The `make` build now builds a shared library by default instead of a static library. Use `LIB_MODE=static` to override.
### New Features
* Compaction filters are now supported for wide-column entities by means of the `FilterV3` API. See the comment of the API for more details.
* Added `do_not_compress_roles` to `CompressedSecondaryCacheOptions` to disable compression on certain kinds of block. Filter blocks are now not compressed by CompressedSecondaryCache by default.
* Added a new `MultiGetEntity` API that enables batched wide-column point lookups. See the API comments for more details.
## 7.10.0 (01/23/2023)
### Behavior changes
* Make best-efforts recovery verify SST unique ID before Version construction (#10962)
* Introduce `epoch_number` and sort L0 files by `epoch_number` instead of `largest_seqno`. `epoch_number` represents the order of a file being flushed or ingested/imported. Compaction output file will be assigned with the minimum `epoch_number` among input files'. For L0, larger `epoch_number` indicates newer L0 file.
+1 -1
View File
@@ -178,7 +178,7 @@ to build a portable binary, add `PORTABLE=1` before your make commands, like thi
gmake rocksdbjava
* **iOS**:
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define two important pre-processing macros: `ROCKSDB_LITE` and `IOS_CROSS_COMPILE`.
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define an important pre-processing macros: `IOS_CROSS_COMPILE`.
* **Windows** (Visual Studio 2017 to up):
* Read and follow the instructions at CMakeLists.txt
+15 -49
View File
@@ -44,13 +44,6 @@ quoted_perl_command = $(subst ','\'',$(perl_command))
# Set the default DEBUG_LEVEL to 1
DEBUG_LEVEL?=1
# LIB_MODE says whether or not to use/build "shared" or "static" libraries.
# Mode "static" means to link against static libraries (.a)
# Mode "shared" means to link against shared libraries (.so, .sl, .dylib, etc)
#
# Set the default LIB_MODE to static
LIB_MODE?=static
# OBJ_DIR is where the object files reside. Default to the current directory
OBJ_DIR?=.
@@ -81,29 +74,23 @@ else ifneq ($(filter jtest rocksdbjava%, $(MAKECMDGOALS)),)
endif
endif
$(info $$DEBUG_LEVEL is ${DEBUG_LEVEL})
# LIB_MODE says whether or not to use/build "shared" or "static" libraries.
# Mode "static" means to link against static libraries (.a)
# Mode "shared" means to link against shared libraries (.so, .sl, .dylib, etc)
#
ifeq ($(DEBUG_LEVEL), 0)
# For optimized, set the default LIB_MODE to static for code size/efficiency
LIB_MODE?=static
else
# For debug, set the default LIB_MODE to shared for efficient `make check` etc.
LIB_MODE?=shared
endif
# Lite build flag.
LITE ?= 0
ifeq ($(LITE), 0)
ifneq ($(filter -DROCKSDB_LITE,$(OPT)),)
# Be backward compatible and support older format where OPT=-DROCKSDB_LITE is
# specified instead of LITE=1 on the command line.
LITE=1
endif
else ifeq ($(LITE), 1)
ifeq ($(filter -DROCKSDB_LITE,$(OPT)),)
OPT += -DROCKSDB_LITE
endif
endif
$(info $$DEBUG_LEVEL is $(DEBUG_LEVEL), $$LIB_MODE is $(LIB_MODE))
# Figure out optimize level.
ifneq ($(DEBUG_LEVEL), 2)
ifeq ($(LITE), 0)
OPTIMIZE_LEVEL ?= -O2
else
OPTIMIZE_LEVEL ?= -Os
endif
endif
# `OPTIMIZE_LEVEL` is empty when the user does not set it and `DEBUG_LEVEL=2`.
# In that case, the compiler default (`-O0` for gcc and clang) will be used.
@@ -337,13 +324,6 @@ endif
ifeq ($(PLATFORM), OS_SOLARIS)
PLATFORM_CXXFLAGS += -D _GLIBCXX_USE_C99
endif
ifneq ($(filter -DROCKSDB_LITE,$(OPT)),)
# found
CFLAGS += -fno-exceptions
CXXFLAGS += -fno-exceptions
# LUA is not supported under ROCKSDB_LITE
LUA_PATH =
endif
ifeq ($(LIB_MODE),shared)
# So that binaries are executable from build location, in addition to install location
@@ -1082,13 +1062,11 @@ check: all
rm -rf $(TEST_TMPDIR)
ifneq ($(PLATFORM), OS_AIX)
$(PYTHON) tools/check_all_python.py
ifeq ($(filter -DROCKSDB_LITE,$(OPT)),)
ifndef ASSERT_STATUS_CHECKED # not yet working with these tests
$(PYTHON) tools/ldb_test.py
sh tools/rocksdb_dump_test.sh
endif
endif
endif
ifndef SKIP_FORMAT_BUCK_CHECKS
$(MAKE) check-format
$(MAKE) check-buck-targets
@@ -1244,9 +1222,9 @@ clean: clean-ext-libraries-all clean-rocks clean-rocksjava
clean-not-downloaded: clean-ext-libraries-bin clean-rocks clean-not-downloaded-rocksjava
clean-rocks:
echo shared=$(ALL_SHARED_LIBS)
echo static=$(ALL_STATIC_LIBS)
rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(PARALLEL_TEST) $(ALL_STATIC_LIBS) $(ALL_SHARED_LIBS) $(MICROBENCHS)
# Not practical to exactly match all versions/variants in naming (e.g. debug or not)
rm -f ${LIBNAME}*.so* ${LIBNAME}*.a
rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(PARALLEL_TEST) $(MICROBENCHS)
rm -rf $(CLEAN_FILES) ios-x86 ios-arm scan_build_report
$(FIND) . -name "*.[oda]" -exec rm -f {} \;
$(FIND) . -type f \( -name "*.gcda" -o -name "*.gcno" \) -exec rm -f {} \;
@@ -2489,18 +2467,6 @@ build_size:
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib $$(stat --printf="%s" `readlink -f librocksdb.so`)
strip `readlink -f librocksdb.so`
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_stripped $$(stat --printf="%s" `readlink -f librocksdb.so`)
# === lite build, static ===
$(MAKE) clean
$(MAKE) LITE=1 static_lib
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.static_lib_lite $$(stat --printf="%s" librocksdb.a)
strip librocksdb.a
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.static_lib_lite_stripped $$(stat --printf="%s" librocksdb.a)
# === lite build, shared ===
$(MAKE) clean
$(MAKE) LITE=1 shared_lib
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_lite $$(stat --printf="%s" `readlink -f librocksdb.so`)
strip `readlink -f librocksdb.so`
$(REPORT_BUILD_STATISTIC) rocksdb.build_size.shared_lib_lite_stripped $$(stat --printf="%s" `readlink -f librocksdb.so`)
# ---------------------------------------------------------------------------
# Platform-specific compilation
-21
View File
@@ -1,21 +0,0 @@
# RocksDBLite
RocksDBLite is a project focused on mobile use cases, which don't need a lot of fancy things we've built for server workloads and they are very sensitive to binary size. For that reason, we added a compile flag ROCKSDB_LITE that comments out a lot of the nonessential code and keeps the binary lean.
Some examples of the features disabled by ROCKSDB_LITE:
* compiled-in support for LDB tool
* No backup engine
* No support for replication (which we provide in form of TransactionalIterator)
* No advanced monitoring tools
* No special-purpose memtables that are highly optimized for specific use cases
* No Transactions
When adding a new big feature to RocksDB, please add ROCKSDB_LITE compile guard if:
* Nobody from mobile really needs your feature,
* Your feature is adding a lot of weight to the binary.
Don't add ROCKSDB_LITE compile guard if:
* It would introduce a lot of code complexity. Compile guards make code harder to read. It's a trade-off.
* Your feature is not adding a lot of weight.
If unsure, ask. :)
+2
View File
@@ -247,6 +247,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
"util/concurrent_task_limiter_impl.cc",
"util/crc32c.cc",
"util/crc32c_arm64.cc",
"util/data_structure.cc",
"util/dynamic_bloom.cc",
"util/file_checksum_helper.cc",
"util/hash.cc",
@@ -589,6 +590,7 @@ cpp_library_wrapper(name="rocksdb_whole_archive_lib", srcs=[
"util/concurrent_task_limiter_impl.cc",
"util/crc32c.cc",
"util/crc32c_arm64.cc",
"util/data_structure.cc",
"util/dynamic_bloom.cc",
"util/file_checksum_helper.cc",
"util/hash.cc",
+1 -1
View File
@@ -26,7 +26,7 @@ from util import ColorString
# $python3 buckifier/buckify_rocksdb.py \
# '{"fake": {
# "extra_deps": [":test_dep", "//fakes/module:mock1"],
# "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"]
# "extra_compiler_flags": ["-DFOO_BAR", "-Os"]
# }
# }'
# (Generated TARGETS file has test_dep and mock1 as dependencies for RocksDB
+1 -1
View File
@@ -154,7 +154,7 @@ case "$TARGET_OS" in
;;
IOS)
PLATFORM=IOS
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE -DROCKSDB_LITE"
COMMON_FLAGS="$COMMON_FLAGS -DOS_MACOSX -DIOS_CROSS_COMPILE "
PLATFORM_SHARED_EXT=dylib
PLATFORM_SHARED_LDFLAGS="-dynamiclib -install_name "
CROSS_COMPILE=true
+1 -15
View File
@@ -16,7 +16,6 @@
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
static std::unordered_map<std::string, OptionTypeInfo>
lru_cache_options_type_info = {
{"capacity",
@@ -64,7 +63,6 @@ static std::unordered_map<std::string, OptionTypeInfo>
OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kMutable}},
};
#endif // ROCKSDB_LITE
Status SecondaryCache::CreateFromString(
const ConfigOptions& config_options, const std::string& value,
@@ -75,7 +73,6 @@ Status SecondaryCache::CreateFromString(
Status status;
std::shared_ptr<SecondaryCache> sec_cache;
#ifndef ROCKSDB_LITE
CompressedSecondaryCacheOptions sec_cache_opts;
status = OptionTypeInfo::ParseStruct(config_options, "",
&comp_sec_cache_options_type_info, "",
@@ -84,19 +81,13 @@ Status SecondaryCache::CreateFromString(
sec_cache = NewCompressedSecondaryCache(sec_cache_opts);
}
#else
(void)config_options;
status = Status::NotSupported(
"Cannot load compressed secondary cache in LITE mode ", args);
#endif //! ROCKSDB_LITE
if (status.ok()) {
result->swap(sec_cache);
}
return status;
} else {
return LoadSharedObject<SecondaryCache>(config_options, value, nullptr,
result);
return LoadSharedObject<SecondaryCache>(config_options, value, result);
}
}
@@ -108,7 +99,6 @@ Status Cache::CreateFromString(const ConfigOptions& config_options,
if (value.find('=') == std::string::npos) {
cache = NewLRUCache(ParseSizeT(value));
} else {
#ifndef ROCKSDB_LITE
LRUCacheOptions cache_opts;
status = OptionTypeInfo::ParseStruct(config_options, "",
&lru_cache_options_type_info, "",
@@ -116,10 +106,6 @@ Status Cache::CreateFromString(const ConfigOptions& config_options,
if (status.ok()) {
cache = NewLRUCache(cache_opts);
}
#else
(void)config_options;
status = Status::NotSupported("Cannot load cache in LITE mode ", value);
#endif //! ROCKSDB_LITE
}
if (status.ok()) {
result->swap(cache);
+1 -5
View File
@@ -16,7 +16,7 @@
#include "db/db_impl/db_impl.h"
#include "monitoring/histogram.h"
#include "port/port.h"
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "rocksdb/convenience.h"
#include "rocksdb/db.h"
#include "rocksdb/env.h"
@@ -77,11 +77,9 @@ DEFINE_bool(lean, false,
"If true, no additional computation is performed besides cache "
"operations.");
#ifndef ROCKSDB_LITE
DEFINE_string(secondary_cache_uri, "",
"Full URI for creating a custom secondary cache object");
static class std::shared_ptr<ROCKSDB_NAMESPACE::SecondaryCache> secondary_cache;
#endif // ROCKSDB_LITE
DEFINE_string(cache_type, "lru_cache", "Type of block cache.");
@@ -305,7 +303,6 @@ class CacheBench {
LRUCacheOptions opts(FLAGS_cache_size, FLAGS_num_shard_bits,
false /* strict_capacity_limit */,
0.5 /* high_pri_pool_ratio */);
#ifndef ROCKSDB_LITE
if (!FLAGS_secondary_cache_uri.empty()) {
Status s = SecondaryCache::CreateFromString(
ConfigOptions(), FLAGS_secondary_cache_uri, &secondary_cache);
@@ -318,7 +315,6 @@ class CacheBench {
}
opts.secondary_cache = secondary_cache;
}
#endif // ROCKSDB_LITE
cache_ = NewLRUCache(opts);
} else {
+1 -1
View File
@@ -7,7 +7,7 @@
#include <cassert>
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "rocksdb/rocksdb_namespace.h"
namespace ROCKSDB_NAMESPACE {
+1 -1
View File
@@ -8,7 +8,7 @@
#include <algorithm>
#include <atomic>
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "table/unique_id_impl.h"
#include "util/hash.h"
#include "util/math.h"
+1 -1
View File
@@ -8,7 +8,7 @@
#include <string>
#include "port/port.h"
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
namespace ROCKSDB_NAMESPACE {
+1 -1
View File
@@ -9,7 +9,6 @@
#include "cache/clock_cache.h"
#include <cassert>
#include <functional>
#include <numeric>
@@ -18,6 +17,7 @@
#include "monitoring/perf_context_imp.h"
#include "monitoring/statistics.h"
#include "port/lang.h"
#include "rocksdb/env.h"
#include "util/hash.h"
#include "util/math.h"
#include "util/random.h"
+11 -7
View File
@@ -22,12 +22,13 @@ CompressedSecondaryCache::CompressedSecondaryCache(
std::shared_ptr<MemoryAllocator> memory_allocator, bool use_adaptive_mutex,
CacheMetadataChargePolicy metadata_charge_policy,
CompressionType compression_type, uint32_t compress_format_version,
bool enable_custom_split_merge)
bool enable_custom_split_merge,
const CacheEntryRoleSet& do_not_compress_roles)
: cache_options_(capacity, num_shard_bits, strict_capacity_limit,
high_pri_pool_ratio, low_pri_pool_ratio, memory_allocator,
use_adaptive_mutex, metadata_charge_policy,
compression_type, compress_format_version,
enable_custom_split_merge) {
enable_custom_split_merge, do_not_compress_roles) {
cache_ =
NewLRUCache(capacity, num_shard_bits, strict_capacity_limit,
high_pri_pool_ratio, memory_allocator, use_adaptive_mutex,
@@ -71,7 +72,8 @@ std::unique_ptr<SecondaryCacheResultHandle> CompressedSecondaryCache::Lookup(
Status s;
Cache::ObjectPtr value{nullptr};
size_t charge{0};
if (cache_options_.compression_type == kNoCompression) {
if (cache_options_.compression_type == kNoCompression ||
cache_options_.do_not_compress_roles.Contains(helper->role)) {
s = helper->create_cb(Slice(ptr->get(), handle_value_charge),
create_context, allocator, &value, &charge);
} else {
@@ -143,7 +145,8 @@ Status CompressedSecondaryCache::Insert(const Slice& key,
Slice val(ptr.get(), size);
std::string compressed_val;
if (cache_options_.compression_type != kNoCompression) {
if (cache_options_.compression_type != kNoCompression &&
!cache_options_.do_not_compress_roles.Contains(helper->role)) {
PERF_COUNTER_ADD(compressed_sec_cache_uncompressed_bytes, size);
CompressionOptions compression_opts;
CompressionContext compression_context(cache_options_.compression_type);
@@ -314,12 +317,13 @@ std::shared_ptr<SecondaryCache> NewCompressedSecondaryCache(
std::shared_ptr<MemoryAllocator> memory_allocator, bool use_adaptive_mutex,
CacheMetadataChargePolicy metadata_charge_policy,
CompressionType compression_type, uint32_t compress_format_version,
bool enable_custom_split_merge) {
bool enable_custom_split_merge,
const CacheEntryRoleSet& do_not_compress_roles) {
return std::make_shared<CompressedSecondaryCache>(
capacity, num_shard_bits, strict_capacity_limit, high_pri_pool_ratio,
low_pri_pool_ratio, memory_allocator, use_adaptive_mutex,
metadata_charge_policy, compression_type, compress_format_version,
enable_custom_split_merge);
enable_custom_split_merge, do_not_compress_roles);
}
std::shared_ptr<SecondaryCache> NewCompressedSecondaryCache(
@@ -331,7 +335,7 @@ std::shared_ptr<SecondaryCache> NewCompressedSecondaryCache(
opts.high_pri_pool_ratio, opts.low_pri_pool_ratio, opts.memory_allocator,
opts.use_adaptive_mutex, opts.metadata_charge_policy,
opts.compression_type, opts.compress_format_version,
opts.enable_custom_split_merge);
opts.enable_custom_split_merge, opts.do_not_compress_roles);
}
} // namespace ROCKSDB_NAMESPACE
+3 -1
View File
@@ -78,7 +78,9 @@ class CompressedSecondaryCache : public SecondaryCache {
kDefaultCacheMetadataChargePolicy,
CompressionType compression_type = CompressionType::kLZ4Compression,
uint32_t compress_format_version = 2,
bool enable_custom_split_merge = false);
bool enable_custom_split_merge = false,
const CacheEntryRoleSet& do_not_compress_roles = {
CacheEntryRole::kFilterBlock});
~CompressedSecondaryCache() override;
const char* Name() const override { return "CompressedSecondaryCache"; }
+94 -6
View File
@@ -5,6 +5,7 @@
#include "cache/compressed_secondary_cache.h"
#include <array>
#include <iterator>
#include <memory>
#include <tuple>
@@ -75,12 +76,23 @@ class CompressedSecondaryCacheTest : public testing::Test,
return Status::OK();
}
static constexpr Cache::CacheItemHelper kHelper{
CacheEntryRole::kMisc, &DeletionCallback, &SizeCallback, &SaveToCallback,
&CreateCallback};
static constexpr auto GenerateHelpersByRole() {
std::array<Cache::CacheItemHelper, kNumCacheEntryRoles> a;
for (uint32_t i = 0; i < kNumCacheEntryRoles; ++i) {
a[i] = Cache::CacheItemHelper{static_cast<CacheEntryRole>(i),
&DeletionCallback, &SizeCallback,
&SaveToCallback, &CreateCallback};
}
return a;
}
static const std::array<Cache::CacheItemHelper, kNumCacheEntryRoles>
kHelperByRole;
static const Cache::CacheItemHelper& kHelper;
static constexpr Cache::CacheItemHelper kHelperFail{
CacheEntryRole::kMisc, &DeletionCallback, &SizeCallback,
CacheEntryRole::kDataBlock, &DeletionCallback, &SizeCallback,
&SaveToCallbackFail, &CreateCallback};
void SetFailCreate(bool fail) { fail_create_ = fail; }
@@ -787,6 +799,13 @@ class CompressedSecondaryCacheTest : public testing::Test,
bool fail_create_;
};
const std::array<Cache::CacheItemHelper, kNumCacheEntryRoles>
CompressedSecondaryCacheTest::kHelperByRole = GenerateHelpersByRole();
const Cache::CacheItemHelper& CompressedSecondaryCacheTest::kHelper =
CompressedSecondaryCacheTest::kHelperByRole[static_cast<int>(
CacheEntryRole::kDataBlock)];
class CompressedSecCacheTestWithCompressAndAllocatorParam
: public CompressedSecondaryCacheTest,
public ::testing::WithParamInterface<std::tuple<bool, bool>> {
@@ -817,7 +836,6 @@ class CompressedSecondaryCacheTestWithCompressionParam
bool sec_cache_is_compressed_;
};
#ifndef ROCKSDB_LITE
TEST_P(CompressedSecondaryCacheTestWithCompressionParam, BasicTestFromString) {
std::shared_ptr<SecondaryCache> sec_cache{nullptr};
@@ -882,7 +900,6 @@ TEST_P(CompressedSecondaryCacheTestWithCompressionParam,
BasicTestHelper(sec_cache, sec_cache_is_compressed_);
}
#endif // ROCKSDB_LITE
TEST_P(CompressedSecondaryCacheTestWithCompressionParam, FailsTest) {
FailsTest(sec_cache_is_compressed_);
@@ -908,6 +925,77 @@ TEST_P(CompressedSecondaryCacheTestWithCompressionParam,
IntegrationFullCapacityTest(sec_cache_is_compressed_);
}
TEST_P(CompressedSecondaryCacheTestWithCompressionParam, EntryRoles) {
CompressedSecondaryCacheOptions opts;
opts.capacity = 2048;
opts.num_shard_bits = 0;
if (sec_cache_is_compressed_) {
if (!LZ4_Supported()) {
ROCKSDB_GTEST_SKIP("This test requires LZ4 support.");
return;
}
} else {
opts.compression_type = CompressionType::kNoCompression;
}
// Select a random subset to include, for fast test
Random& r = *Random::GetTLSInstance();
CacheEntryRoleSet do_not_compress;
for (uint32_t i = 0; i < kNumCacheEntryRoles; ++i) {
// A few included on average, but decent chance of zero
if (r.OneIn(5)) {
do_not_compress.Add(static_cast<CacheEntryRole>(i));
}
}
opts.do_not_compress_roles = do_not_compress;
std::shared_ptr<SecondaryCache> sec_cache = NewCompressedSecondaryCache(opts);
// Fixed seed to ensure consistent compressibility (doesn't compress)
std::string junk(Random(301).RandomString(1000));
for (uint32_t i = 0; i < kNumCacheEntryRoles; ++i) {
CacheEntryRole role = static_cast<CacheEntryRole>(i);
// Uniquify `junk`
junk[0] = static_cast<char>(i);
TestItem item{junk.data(), junk.length()};
Slice ith_key = Slice(junk.data(), 5);
get_perf_context()->Reset();
ASSERT_OK(sec_cache->Insert(ith_key, &item, &kHelperByRole[i]));
ASSERT_EQ(get_perf_context()->compressed_sec_cache_insert_dummy_count, 1U);
ASSERT_OK(sec_cache->Insert(ith_key, &item, &kHelperByRole[i]));
ASSERT_EQ(get_perf_context()->compressed_sec_cache_insert_real_count, 1U);
bool is_in_sec_cache{true};
std::unique_ptr<SecondaryCacheResultHandle> handle =
sec_cache->Lookup(ith_key, &kHelperByRole[i], this, true,
/*advise_erase=*/true, is_in_sec_cache);
ASSERT_NE(handle, nullptr);
// Lookup returns the right data
std::unique_ptr<TestItem> val =
std::unique_ptr<TestItem>(static_cast<TestItem*>(handle->Value()));
ASSERT_NE(val, nullptr);
ASSERT_EQ(memcmp(val->Buf(), item.Buf(), item.Size()), 0);
bool compressed =
sec_cache_is_compressed_ && !do_not_compress.Contains(role);
if (compressed) {
ASSERT_EQ(get_perf_context()->compressed_sec_cache_uncompressed_bytes,
1000);
ASSERT_EQ(get_perf_context()->compressed_sec_cache_compressed_bytes,
1007);
} else {
ASSERT_EQ(get_perf_context()->compressed_sec_cache_uncompressed_bytes, 0);
ASSERT_EQ(get_perf_context()->compressed_sec_cache_compressed_bytes, 0);
}
}
}
INSTANTIATE_TEST_CASE_P(CompressedSecCacheTests,
CompressedSecondaryCacheTestWithCompressionParam,
testing::Bool());
-39
View File
@@ -1784,43 +1784,6 @@ TEST_F(DBSecondaryCacheTest, SecondaryCacheFailureTest) {
Destroy(options);
}
TEST_F(DBSecondaryCacheTest, TestSecondaryWithCompressedCache) {
if (!Snappy_Supported()) {
ROCKSDB_GTEST_SKIP("Compressed cache test requires snappy support");
return;
}
LRUCacheOptions opts(2000 /* capacity */, 0 /* num_shard_bits */,
false /* strict_capacity_limit */,
0.5 /* high_pri_pool_ratio */,
nullptr /* memory_allocator */, kDefaultToAdaptiveMutex,
kDontChargeCacheMetadata);
std::shared_ptr<TestSecondaryCache> secondary_cache(
new TestSecondaryCache(2048 * 1024));
opts.secondary_cache = secondary_cache;
std::shared_ptr<Cache> cache = NewLRUCache(opts);
BlockBasedTableOptions table_options;
table_options.block_cache_compressed = cache;
table_options.no_block_cache = true;
table_options.block_size = 1234;
Options options = GetDefaultOptions();
options.compression = kSnappyCompression;
options.create_if_missing = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
DestroyAndReopen(options);
Random rnd(301);
const int N = 6;
for (int i = 0; i < N; i++) {
// Partly compressible
std::string p_v = rnd.RandomString(507) + std::string(500, ' ');
ASSERT_OK(Put(Key(i), p_v));
}
ASSERT_OK(Flush());
for (int i = 0; i < 2 * N; i++) {
std::string v = Get(Key(i % N));
ASSERT_EQ(1007, v.size());
}
}
TEST_F(LRUCacheSecondaryCacheTest, BasicWaitAllTest) {
LRUCacheOptions opts(1024 /* capacity */, 2 /* num_shard_bits */,
false /* strict_capacity_limit */,
@@ -2012,7 +1975,6 @@ class LRUCacheWithStat : public LRUCache {
uint32_t lookup_count_;
};
#ifndef ROCKSDB_LITE
TEST_F(DBSecondaryCacheTest, LRUCacheDumpLoadBasic) {
LRUCacheOptions cache_opts(1024 * 1024 /* capacity */, 0 /* num_shard_bits */,
@@ -2627,7 +2589,6 @@ TEST_F(DBSecondaryCacheTest, TestSecondaryCacheOptionTwoDB) {
ASSERT_OK(DestroyDB(dbname2, options));
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+1 -1
View File
@@ -15,7 +15,7 @@
#include "port/lang.h"
#include "port/port.h"
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "util/hash.h"
#include "util/mutexlock.h"
+1 -1
View File
@@ -29,8 +29,8 @@
#include <type_traits>
#include "cache/cache_helpers.h"
#include "rocksdb/advanced_cache.h"
#include "rocksdb/advanced_options.h"
#include "rocksdb/cache.h"
namespace ROCKSDB_NAMESPACE {
+1 -1
View File
@@ -8,7 +8,7 @@
#include <memory>
#include "memory/memory_allocator.h"
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "rocksdb/rocksdb_namespace.h"
#include "rocksdb/slice.h"
#include "rocksdb/status.h"
-17
View File
@@ -23,32 +23,19 @@ class BlobFileCompletionCallback {
const std::vector<std::shared_ptr<EventListener>>& listeners,
const std::string& dbname)
: event_logger_(event_logger), listeners_(listeners), dbname_(dbname) {
#ifndef ROCKSDB_LITE
sst_file_manager_ = sst_file_manager;
mutex_ = mutex;
error_handler_ = error_handler;
#else
(void)sst_file_manager;
(void)mutex;
(void)error_handler;
#endif // ROCKSDB_LITE
}
void OnBlobFileCreationStarted(const std::string& file_name,
const std::string& column_family_name,
int job_id,
BlobFileCreationReason creation_reason) {
#ifndef ROCKSDB_LITE
// Notify the listeners.
EventHelpers::NotifyBlobFileCreationStarted(listeners_, dbname_,
column_family_name, file_name,
job_id, creation_reason);
#else
(void)file_name;
(void)column_family_name;
(void)job_id;
(void)creation_reason;
#endif
}
Status OnBlobFileCompleted(const std::string& file_name,
@@ -61,7 +48,6 @@ class BlobFileCompletionCallback {
uint64_t blob_count, uint64_t blob_bytes) {
Status s;
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(sst_file_manager_);
if (sfm) {
// Report new blob files to SstFileManagerImpl
@@ -74,7 +60,6 @@ class BlobFileCompletionCallback {
error_handler_->SetBGError(s, BackgroundErrorReason::kFlush);
}
}
#endif // !ROCKSDB_LITE
// Notify the listeners.
EventHelpers::LogAndNotifyBlobFileCreationFinished(
@@ -89,11 +74,9 @@ class BlobFileCompletionCallback {
}
private:
#ifndef ROCKSDB_LITE
SstFileManager* sst_file_manager_;
InstrumentedMutex* mutex_;
ErrorHandler* error_handler_;
#endif // ROCKSDB_LITE
EventLogger* event_logger_;
std::vector<std::shared_ptr<EventListener>> listeners_;
std::string dbname_;
-2
View File
@@ -30,7 +30,6 @@ BlobSource::BlobSource(const ImmutableOptions* immutable_options,
blob_file_cache_(blob_file_cache),
blob_cache_(immutable_options->blob_cache),
lowest_used_cache_tier_(immutable_options->lowest_used_cache_tier) {
#ifndef ROCKSDB_LITE
auto bbto =
immutable_options->table_factory->GetOptions<BlockBasedTableOptions>();
if (bbto &&
@@ -39,7 +38,6 @@ BlobSource::BlobSource(const ImmutableOptions* immutable_options,
blob_cache_ = SharedCacheInterface{std::make_shared<ChargedCache>(
immutable_options->blob_cache, bbto->block_cache)};
}
#endif // ROCKSDB_LITE
}
BlobSource::~BlobSource() = default;
-2
View File
@@ -1391,7 +1391,6 @@ class BlobSourceCacheReservationTest : public DBTestBase {
std::string db_session_id_;
};
#ifndef ROCKSDB_LITE
TEST_F(BlobSourceCacheReservationTest, SimpleCacheReservation) {
options_.cf_paths.emplace_back(
test::PerThreadDBPath(
@@ -1606,7 +1605,6 @@ TEST_F(BlobSourceCacheReservationTest, IncreaseCacheReservationOnFullCache) {
}
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+240 -8
View File
@@ -11,6 +11,7 @@
#include "db/blob/blob_index.h"
#include "db/blob/blob_log_format.h"
#include "db/db_test_util.h"
#include "db/db_with_timestamp_test_util.h"
#include "port/stack_trace.h"
#include "test_util/sync_point.h"
#include "utilities/fault_injection_env.h"
@@ -584,7 +585,6 @@ TEST_F(DBBlobBasicTest, MultiGetBlobsFromCache) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, MultiGetWithDirectIO) {
Options options = GetDefaultOptions();
@@ -773,7 +773,6 @@ TEST_F(DBBlobBasicTest, MultiGetWithDirectIO) {
ASSERT_EQ(values[2], second_blob);
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, MultiGetBlobsFromMultipleFiles) {
Options options = GetDefaultOptions();
@@ -1062,7 +1061,6 @@ TEST_F(DBBlobBasicTest, GetBlob_IndexWithInvalidFileNumber) {
.IsCorruption());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, GenerateIOTracing) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -1117,7 +1115,6 @@ TEST_F(DBBlobBasicTest, GenerateIOTracing) {
ASSERT_GT(blob_files_op_count, 2);
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, BestEffortsRecovery_MissingNewestBlobFile) {
Options options = GetDefaultOptions();
@@ -1219,7 +1216,6 @@ TEST_F(DBBlobBasicTest, MultiGetMergeBlobWithPut) {
ASSERT_EQ(values[2], "v2_0");
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, Properties) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -1382,7 +1378,6 @@ TEST_F(DBBlobBasicTest, PropertiesMultiVersion) {
BlobLogRecord::CalculateAdjustmentForRecordHeader(key_size) +
blob_size + BlobLogFooter::kSize));
}
#endif // !ROCKSDB_LITE
class DBBlobBasicIOErrorTest : public DBBlobBasicTest,
public testing::WithParamInterface<std::string> {
@@ -1632,7 +1627,6 @@ TEST_F(DBBlobBasicTest, WarmCacheWithBlobsDuringFlush) {
options.statistics->getTickerCount(BLOB_DB_CACHE_ADD));
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobBasicTest, DynamicallyWarmCacheDuringFlush) {
Options options = GetDefaultOptions();
@@ -1700,7 +1694,6 @@ TEST_F(DBBlobBasicTest, DynamicallyWarmCacheDuringFlush) {
/*end=*/nullptr));
EXPECT_EQ(0, options.statistics->getTickerCount(BLOB_DB_CACHE_ADD));
}
#endif // !ROCKSDB_LITE
TEST_F(DBBlobBasicTest, WarmCacheWithBlobsSecondary) {
CompressedSecondaryCacheOptions secondary_cache_opts;
@@ -1779,6 +1772,245 @@ TEST_F(DBBlobBasicTest, WarmCacheWithBlobsSecondary) {
1);
}
TEST_F(DBBlobBasicTest, GetEntityBlob) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
options.min_blob_size = 0;
Reopen(options);
constexpr char key[] = "key";
constexpr char blob_value[] = "blob_value";
constexpr char other_key[] = "other_key";
constexpr char other_blob_value[] = "other_blob_value";
ASSERT_OK(Put(key, blob_value));
ASSERT_OK(Put(other_key, other_blob_value));
ASSERT_OK(Flush());
WideColumns expected_columns{{kDefaultWideColumnName, blob_value}};
WideColumns other_expected_columns{
{kDefaultWideColumnName, other_blob_value}};
{
PinnableWideColumns result;
ASSERT_OK(db_->GetEntity(ReadOptions(), db_->DefaultColumnFamily(), key,
&result));
ASSERT_EQ(result.columns(), expected_columns);
}
{
PinnableWideColumns result;
ASSERT_OK(db_->GetEntity(ReadOptions(), db_->DefaultColumnFamily(),
other_key, &result));
ASSERT_EQ(result.columns(), other_expected_columns);
}
{
constexpr size_t num_keys = 2;
std::array<Slice, num_keys> keys{{key, other_key}};
std::array<PinnableWideColumns, num_keys> results;
std::array<Status, num_keys> statuses;
db_->MultiGetEntity(ReadOptions(), db_->DefaultColumnFamily(), num_keys,
&keys[0], &results[0], &statuses[0]);
ASSERT_OK(statuses[0]);
ASSERT_EQ(results[0].columns(), expected_columns);
ASSERT_OK(statuses[1]);
ASSERT_EQ(results[1].columns(), other_expected_columns);
}
}
class DBBlobWithTimestampTest : public DBBasicTestWithTimestampBase {
protected:
DBBlobWithTimestampTest()
: DBBasicTestWithTimestampBase("db_blob_with_timestamp_test") {}
};
TEST_F(DBBlobWithTimestampTest, GetBlob) {
Options options = GetDefaultOptions();
options.create_if_missing = true;
options.enable_blob_files = true;
options.min_blob_size = 0;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
constexpr char key[] = "key";
constexpr char blob_value[] = "blob_value";
ASSERT_OK(db_->Put(write_opts, key, ts, blob_value));
ASSERT_OK(Flush());
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
std::string value;
ASSERT_OK(db_->Get(read_opts, key, &value));
ASSERT_EQ(value, blob_value);
}
TEST_F(DBBlobWithTimestampTest, MultiGetBlobs) {
constexpr size_t min_blob_size = 6;
Options options = GetDefaultOptions();
options.enable_blob_files = true;
options.min_blob_size = min_blob_size;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
// Put then retrieve three key-values. The first value is below the size limit
// and is thus stored inline; the other two are stored separately as blobs.
constexpr size_t num_keys = 3;
constexpr char first_key[] = "first_key";
constexpr char first_value[] = "short";
static_assert(sizeof(first_value) - 1 < min_blob_size,
"first_value too long to be inlined");
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, first_key, ts, first_value));
constexpr char second_key[] = "second_key";
constexpr char second_value[] = "long_value";
static_assert(sizeof(second_value) - 1 >= min_blob_size,
"second_value too short to be stored as blob");
ASSERT_OK(db_->Put(write_opts, second_key, ts, second_value));
constexpr char third_key[] = "third_key";
constexpr char third_value[] = "other_long_value";
static_assert(sizeof(third_value) - 1 >= min_blob_size,
"third_value too short to be stored as blob");
ASSERT_OK(db_->Put(write_opts, third_key, ts, third_value));
ASSERT_OK(Flush());
ReadOptions read_options;
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
read_options.timestamp = &read_ts_slice;
std::array<Slice, num_keys> keys{{first_key, second_key, third_key}};
{
std::array<PinnableSlice, num_keys> values;
std::array<Status, num_keys> statuses;
db_->MultiGet(read_options, db_->DefaultColumnFamily(), num_keys, &keys[0],
&values[0], &statuses[0]);
ASSERT_OK(statuses[0]);
ASSERT_EQ(values[0], first_value);
ASSERT_OK(statuses[1]);
ASSERT_EQ(values[1], second_value);
ASSERT_OK(statuses[2]);
ASSERT_EQ(values[2], third_value);
}
}
TEST_F(DBBlobWithTimestampTest, GetMergeBlobWithPut) {
Options options = GetDefaultOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.enable_blob_files = true;
options.min_blob_size = 0;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, "Key1", ts, "v1"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v2"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v3"));
ASSERT_OK(Flush());
std::string value;
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
ASSERT_OK(db_->Get(read_opts, "Key1", &value));
ASSERT_EQ(value, "v1,v2,v3");
}
TEST_F(DBBlobWithTimestampTest, MultiGetMergeBlobWithPut) {
constexpr size_t num_keys = 3;
Options options = GetDefaultOptions();
options.merge_operator = MergeOperators::CreateStringAppendOperator();
options.enable_blob_files = true;
options.min_blob_size = 0;
options.create_if_missing = true;
const size_t kTimestampSize = Timestamp(0, 0).size();
TestComparator test_cmp(kTimestampSize);
options.comparator = &test_cmp;
DestroyAndReopen(options);
WriteOptions write_opts;
const std::string ts = Timestamp(1, 0);
ASSERT_OK(db_->Put(write_opts, "Key0", ts, "v0_0"));
ASSERT_OK(db_->Put(write_opts, "Key1", ts, "v1_0"));
ASSERT_OK(db_->Put(write_opts, "Key2", ts, "v2_0"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key0", ts, "v0_1"));
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key1", ts, "v1_1"));
ASSERT_OK(Flush());
ASSERT_OK(
db_->Merge(write_opts, db_->DefaultColumnFamily(), "Key0", ts, "v0_2"));
ASSERT_OK(Flush());
const std::string read_ts = Timestamp(2, 0);
Slice read_ts_slice(read_ts);
ReadOptions read_opts;
read_opts.timestamp = &read_ts_slice;
std::array<Slice, num_keys> keys{{"Key0", "Key1", "Key2"}};
std::array<PinnableSlice, num_keys> values;
std::array<Status, num_keys> statuses;
db_->MultiGet(read_opts, db_->DefaultColumnFamily(), num_keys, &keys[0],
&values[0], &statuses[0]);
ASSERT_OK(statuses[0]);
ASSERT_EQ(values[0], "v0_0,v0_1,v0_2");
ASSERT_OK(statuses[1]);
ASSERT_EQ(values[1], "v1_0,v1_1");
ASSERT_OK(statuses[2]);
ASSERT_EQ(values[2], "v2_0");
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
-14
View File
@@ -16,7 +16,6 @@ class DBBlobCompactionTest : public DBTestBase {
explicit DBBlobCompactionTest()
: DBTestBase("db_blob_compaction_test", /*env_do_fsync=*/false) {}
#ifndef ROCKSDB_LITE
const std::vector<InternalStats::CompactionStats>& GetCompactionStats() {
VersionSet* const versions = dbfull()->GetVersionSet();
assert(versions);
@@ -30,7 +29,6 @@ class DBBlobCompactionTest : public DBTestBase {
return internal_stats->TEST_GetCompactionStats();
}
#endif // ROCKSDB_LITE
};
namespace {
@@ -250,7 +248,6 @@ TEST_F(DBBlobCompactionTest, FilterByKeyLength) {
ASSERT_OK(db_->Get(ReadOptions(), long_key, &value));
ASSERT_EQ("value", value);
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -258,7 +255,6 @@ TEST_F(DBBlobCompactionTest, FilterByKeyLength) {
// this involves neither reading nor writing blobs
ASSERT_EQ(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -299,7 +295,6 @@ TEST_F(DBBlobCompactionTest, FilterByValueLength) {
ASSERT_EQ(long_value, value);
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -307,12 +302,10 @@ TEST_F(DBBlobCompactionTest, FilterByValueLength) {
// this involves reading but not writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
#ifndef ROCKSDB_LITE
TEST_F(DBBlobCompactionTest, BlobCompactWithStartingLevel) {
Options options = GetDefaultOptions();
@@ -388,7 +381,6 @@ TEST_F(DBBlobCompactionTest, BlobCompactWithStartingLevel) {
Close();
}
#endif
TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
Options options = GetDefaultOptions();
@@ -413,7 +405,6 @@ TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
ASSERT_EQ(new_blob_value, Get(key));
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -421,7 +412,6 @@ TEST_F(DBBlobCompactionTest, BlindWriteFilter) {
// this involves writing but not reading blobs
ASSERT_EQ(compaction_stats[1].bytes_read_blob, 0);
ASSERT_GT(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -540,7 +530,6 @@ TEST_F(DBBlobCompactionTest, CompactionFilter) {
ASSERT_EQ(kv.second + std::string(padding), Get(kv.first));
}
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -548,7 +537,6 @@ TEST_F(DBBlobCompactionTest, CompactionFilter) {
// this involves reading and writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_GT(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
@@ -606,7 +594,6 @@ TEST_F(DBBlobCompactionTest, CompactionFilterReadBlobAndKeep) {
/*end=*/nullptr));
ASSERT_EQ(blob_files, GetBlobFileNumbers());
#ifndef ROCKSDB_LITE
const auto& compaction_stats = GetCompactionStats();
ASSERT_GE(compaction_stats.size(), 2);
@@ -614,7 +601,6 @@ TEST_F(DBBlobCompactionTest, CompactionFilterReadBlobAndKeep) {
// this involves reading but not writing blobs
ASSERT_GT(compaction_stats[1].bytes_read_blob, 0);
ASSERT_EQ(compaction_stats[1].bytes_written_blob, 0);
#endif // ROCKSDB_LITE
Close();
}
-2
View File
@@ -34,7 +34,6 @@ class DBBlobCorruptionTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBBlobCorruptionTest, VerifyWholeBlobFileChecksum) {
Options options = GetDefaultOptions();
options.enable_blob_files = true;
@@ -71,7 +70,6 @@ TEST_F(DBBlobCorruptionTest, VerifyWholeBlobFileChecksum) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
-6
View File
@@ -131,9 +131,7 @@ class DBBlobIndexTest : public DBTestBase {
ASSERT_OK(Flush());
ASSERT_OK(
dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
break;
}
}
@@ -459,7 +457,6 @@ TEST_F(DBBlobIndexTest, Iterate) {
verify(15, Status::kOk, get_value(16, 0), get_value(14, 0),
create_blob_iterator, check_is_blob(false));
#ifndef ROCKSDB_LITE
// Iterator with blob support and using seek.
ASSERT_OK(dbfull()->SetOptions(
cfh(), {{"max_sequential_skip_in_iterations", "0"}}));
@@ -484,7 +481,6 @@ TEST_F(DBBlobIndexTest, Iterate) {
create_blob_iterator, check_is_blob(false));
verify(15, Status::kOk, get_value(16, 0), get_value(14, 0),
create_blob_iterator, check_is_blob(false));
#endif // !ROCKSDB_LITE
for (auto* snapshot : snapshots) {
dbfull()->ReleaseSnapshot(snapshot);
@@ -584,12 +580,10 @@ TEST_F(DBBlobIndexTest, IntegratedBlobIterate) {
Status expected_status;
verify(1, expected_status, expected_value);
#ifndef ROCKSDB_LITE
// Test DBIter::FindValueForCurrentKeyUsingSeek flow.
ASSERT_OK(dbfull()->SetOptions(cfh(),
{{"max_sequential_skip_in_iterations", "0"}}));
verify(1, expected_status, expected_value);
#endif // !ROCKSDB_LITE
}
} // namespace ROCKSDB_NAMESPACE
-2
View File
@@ -107,11 +107,9 @@ Status BuildTable(
std::vector<std::string> blob_file_paths;
std::string file_checksum = kUnknownFileChecksum;
std::string file_checksum_func_name = kUnknownFileChecksumFuncName;
#ifndef ROCKSDB_LITE
EventHelpers::NotifyTableFileCreationStarted(ioptions.listeners, dbname,
tboptions.column_family_name,
fname, job_id, tboptions.reason);
#endif // !ROCKSDB_LITE
Env* env = db_options.env;
assert(env);
FileSystem* fs = db_options.fs.get();
+8 -14
View File
@@ -7,7 +7,6 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef ROCKSDB_LITE
#include "rocksdb/c.h"
@@ -17,7 +16,7 @@
#include <vector>
#include "port/port.h"
#include "rocksdb/cache.h"
#include "rocksdb/advanced_cache.h"
#include "rocksdb/compaction_filter.h"
#include "rocksdb/comparator.h"
#include "rocksdb/convenience.h"
@@ -69,6 +68,7 @@ using ROCKSDB_NAMESPACE::CompactionOptionsFIFO;
using ROCKSDB_NAMESPACE::CompactRangeOptions;
using ROCKSDB_NAMESPACE::Comparator;
using ROCKSDB_NAMESPACE::CompressionType;
using ROCKSDB_NAMESPACE::ConfigOptions;
using ROCKSDB_NAMESPACE::CuckooTableOptions;
using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::DBOptions;
@@ -2498,8 +2498,12 @@ void rocksdb_load_latest_options(
rocksdb_options_t*** list_column_family_options, char** errptr) {
DBOptions db_opt;
std::vector<ColumnFamilyDescriptor> cf_descs;
Status s = LoadLatestOptions(std::string(db_path), env->rep, &db_opt,
&cf_descs, ignore_unknown_options, &cache->rep);
ConfigOptions config_opts;
config_opts.ignore_unknown_options = ignore_unknown_options;
config_opts.input_strings_escaped = true;
config_opts.env = env->rep;
Status s = LoadLatestOptions(config_opts, std::string(db_path), &db_opt,
&cf_descs, &cache->rep);
if (s.ok()) {
char** cf_names = (char**)malloc(cf_descs.size() * sizeof(char*));
rocksdb_options_t** cf_options = (rocksdb_options_t**)malloc(
@@ -2620,14 +2624,6 @@ void rocksdb_block_based_options_set_block_cache(
}
}
void rocksdb_block_based_options_set_block_cache_compressed(
rocksdb_block_based_table_options_t* options,
rocksdb_cache_t* block_cache_compressed) {
if (block_cache_compressed) {
options->rep.block_cache_compressed = block_cache_compressed->rep;
}
}
void rocksdb_block_based_options_set_whole_key_filtering(
rocksdb_block_based_table_options_t* options, unsigned char v) {
options->rep.whole_key_filtering = v;
@@ -6401,5 +6397,3 @@ void rocksdb_enable_manual_compaction(rocksdb_t* db) {
}
} // end extern "C"
#endif // !ROCKSDB_LITE
-11
View File
@@ -5,8 +5,6 @@
#include <stdio.h>
#ifndef ROCKSDB_LITE // Lite does not support C API
#include <assert.h>
#include <stddef.h>
#include <stdlib.h>
@@ -3468,12 +3466,3 @@ int main(int argc, char** argv) {
fprintf(stderr, "PASS\n");
return 0;
}
#else
int main(void) {
fprintf(stderr, "SKIPPED\n");
return 0;
}
#endif // !ROCKSDB_LITE
-14
View File
@@ -53,11 +53,9 @@ ColumnFamilyHandleImpl::ColumnFamilyHandleImpl(
ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl() {
if (cfd_ != nullptr) {
#ifndef ROCKSDB_LITE
for (auto& listener : cfd_->ioptions()->listeners) {
listener->OnColumnFamilyHandleDeletionStarted(this);
}
#endif // ROCKSDB_LITE
// Job id == 0 means that this is not our background process, but rather
// user thread
// Need to hold some shared pointers owned by the initial_cf_options
@@ -88,15 +86,10 @@ const std::string& ColumnFamilyHandleImpl::GetName() const {
}
Status ColumnFamilyHandleImpl::GetDescriptor(ColumnFamilyDescriptor* desc) {
#ifndef ROCKSDB_LITE
// accessing mutable cf-options requires db mutex.
InstrumentedMutexLock l(mutex_);
*desc = ColumnFamilyDescriptor(cfd()->GetName(), cfd()->GetLatestCFOptions());
return Status::OK();
#else
(void)desc;
return Status::NotSupported();
#endif // !ROCKSDB_LITE
}
const Comparator* ColumnFamilyHandleImpl::GetComparator() const {
@@ -347,7 +340,6 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
result.hard_pending_compaction_bytes_limit;
}
#ifndef ROCKSDB_LITE
// When the DB is stopped, it's possible that there are some .trash files that
// were not deleted yet, when we open the DB we will find these .trash files
// and schedule them to be deleted (or delete immediately if SstFileManager
@@ -359,7 +351,6 @@ ColumnFamilyOptions SanitizeOptions(const ImmutableDBOptions& db_options,
result.cf_paths[i].path)
.PermitUncheckedError();
}
#endif
if (result.cf_paths.empty()) {
result.cf_paths = db_options.db_paths;
@@ -557,7 +548,6 @@ ColumnFamilyData::ColumnFamilyData(
next_(nullptr),
prev_(nullptr),
log_number_(0),
flush_reason_(FlushReason::kOthers),
column_family_set_(column_family_set),
queued_for_flush_(false),
queued_for_compaction_(false),
@@ -603,7 +593,6 @@ ColumnFamilyData::ColumnFamilyData(
if (ioptions_.compaction_style == kCompactionStyleLevel) {
compaction_picker_.reset(
new LevelCompactionPicker(ioptions_, &internal_comparator_));
#ifndef ROCKSDB_LITE
} else if (ioptions_.compaction_style == kCompactionStyleUniversal) {
compaction_picker_.reset(
new UniversalCompactionPicker(ioptions_, &internal_comparator_));
@@ -617,7 +606,6 @@ ColumnFamilyData::ColumnFamilyData(
"Column family %s does not use any background compaction. "
"Compactions can only be done via CompactFiles\n",
GetName().c_str());
#endif // !ROCKSDB_LITE
} else {
ROCKS_LOG_ERROR(ioptions_.logger,
"Unable to recognize the specified compaction style %d. "
@@ -1441,7 +1429,6 @@ Status ColumnFamilyData::ValidateOptions(
return s;
}
#ifndef ROCKSDB_LITE
Status ColumnFamilyData::SetOptions(
const DBOptions& db_opts,
const std::unordered_map<std::string, std::string>& options_map) {
@@ -1460,7 +1447,6 @@ Status ColumnFamilyData::SetOptions(
}
return s;
}
#endif // ROCKSDB_LITE
// REQUIRES: DB mutex held
Env::WriteLifeTimeHint ColumnFamilyData::CalculateSSTWriteHint(int level) {
-8
View File
@@ -310,10 +310,6 @@ class ColumnFamilyData {
void SetLogNumber(uint64_t log_number) { log_number_ = log_number; }
uint64_t GetLogNumber() const { return log_number_; }
void SetFlushReason(FlushReason flush_reason) {
flush_reason_ = flush_reason;
}
FlushReason GetFlushReason() const { return flush_reason_; }
// thread-safe
const FileOptions* soptions() const;
const ImmutableOptions* ioptions() const { return &ioptions_; }
@@ -339,12 +335,10 @@ class ColumnFamilyData {
// Validate CF options against DB options
static Status ValidateOptions(const DBOptions& db_options,
const ColumnFamilyOptions& cf_options);
#ifndef ROCKSDB_LITE
// REQUIRES: DB mutex held
Status SetOptions(
const DBOptions& db_options,
const std::unordered_map<std::string, std::string>& options_map);
#endif // ROCKSDB_LITE
InternalStats* internal_stats() { return internal_stats_.get(); }
@@ -616,8 +610,6 @@ class ColumnFamilyData {
// recovered from
uint64_t log_number_;
std::atomic<FlushReason> flush_reason_;
// An object that keeps all the compaction stats
// and picks the next compaction
std::unique_ptr<CompactionPicker> compaction_picker_;
-71
View File
@@ -71,11 +71,7 @@ class ColumnFamilyTestBase : public testing::Test {
for (auto h : handles_) {
ColumnFamilyDescriptor cfdescriptor;
Status s = h->GetDescriptor(&cfdescriptor);
#ifdef ROCKSDB_LITE
EXPECT_TRUE(s.IsNotSupported());
#else
EXPECT_OK(s);
#endif // ROCKSDB_LITE
column_families.push_back(cfdescriptor);
}
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
@@ -197,12 +193,10 @@ class ColumnFamilyTestBase : public testing::Test {
&db_);
}
#ifndef ROCKSDB_LITE // ReadOnlyDB is not supported
void AssertOpenReadOnly(std::vector<std::string> cf,
std::vector<ColumnFamilyOptions> options = {}) {
ASSERT_OK(OpenReadOnly(cf, options));
}
#endif // !ROCKSDB_LITE
void Open(std::vector<std::string> cf,
std::vector<ColumnFamilyOptions> options = {}) {
@@ -224,27 +218,16 @@ class ColumnFamilyTestBase : public testing::Test {
}
bool IsDbWriteStopped() {
#ifndef ROCKSDB_LITE
uint64_t v;
EXPECT_TRUE(dbfull()->GetIntProperty("rocksdb.is-write-stopped", &v));
return (v == 1);
#else
return dbfull()->TEST_write_controler().IsStopped();
#endif // !ROCKSDB_LITE
}
uint64_t GetDbDelayedWriteRate() {
#ifndef ROCKSDB_LITE
uint64_t v;
EXPECT_TRUE(
dbfull()->GetIntProperty("rocksdb.actual-delayed-write-rate", &v));
return v;
#else
if (!dbfull()->TEST_write_controler().NeedsDelay()) {
return 0;
}
return dbfull()->TEST_write_controler().delayed_write_rate();
#endif // !ROCKSDB_LITE
}
void Destroy(const std::vector<ColumnFamilyDescriptor>& column_families =
@@ -267,7 +250,6 @@ class ColumnFamilyTestBase : public testing::Test {
db_->CreateColumnFamily(current_cf_opt, cfs[i], &handles_[cfi]));
names_[cfi] = cfs[i];
#ifndef ROCKSDB_LITE // RocksDBLite does not support GetDescriptor
// Verify the CF options of the returned CF handle.
ColumnFamilyDescriptor desc;
ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));
@@ -276,7 +258,6 @@ class ColumnFamilyTestBase : public testing::Test {
ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(
ConfigOptions(), desc.options,
SanitizeOptions(dbfull()->immutable_db_options(), current_cf_opt)));
#endif // !ROCKSDB_LITE
cfi++;
}
}
@@ -325,7 +306,6 @@ class ColumnFamilyTestBase : public testing::Test {
ASSERT_OK(db_->FlushWAL(/*sync=*/false));
}
#ifndef ROCKSDB_LITE // TEST functions in DB are not supported in lite
void WaitForFlush(int cf) {
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[cf]));
}
@@ -339,7 +319,6 @@ class ColumnFamilyTestBase : public testing::Test {
void AssertMaxTotalInMemoryState(uint64_t value) {
ASSERT_EQ(value, MaxTotalInMemoryState());
}
#endif // !ROCKSDB_LITE
Status Put(int cf, const std::string& key, const std::string& value) {
return db_->Put(WriteOptions(), handles_[cf], Slice(key), Slice(value));
@@ -377,7 +356,6 @@ class ColumnFamilyTestBase : public testing::Test {
"rocksdb.num-files-at-level" + std::to_string(level));
}
#ifndef ROCKSDB_LITE
// Return spread of files per level
std::string FilesPerLevel(int cf) {
std::string result;
@@ -394,31 +372,19 @@ class ColumnFamilyTestBase : public testing::Test {
result.resize(last_non_zero_offset);
return result;
}
#endif
void AssertFilesPerLevel(const std::string& value, int cf) {
#ifndef ROCKSDB_LITE
ASSERT_EQ(value, FilesPerLevel(cf));
#else
(void)value;
(void)cf;
#endif
}
#ifndef ROCKSDB_LITE // GetLiveFilesMetaData is not supported
int CountLiveFiles() {
std::vector<LiveFileMetaData> metadata;
db_->GetLiveFilesMetaData(&metadata);
return static_cast<int>(metadata.size());
}
#endif // !ROCKSDB_LITE
void AssertCountLiveFiles(int expected_value) {
#ifndef ROCKSDB_LITE
ASSERT_EQ(expected_value, CountLiveFiles());
#else
(void)expected_value;
#endif
}
// Do n memtable flushes, each of which produces an sstable
@@ -432,7 +398,6 @@ class ColumnFamilyTestBase : public testing::Test {
}
}
#ifndef ROCKSDB_LITE // GetSortedWalFiles is not supported
int CountLiveLogFiles() {
int micros_wait_for_log_deletion = 20000;
env_->SleepForMicroseconds(micros_wait_for_log_deletion);
@@ -461,25 +426,18 @@ class ColumnFamilyTestBase : public testing::Test {
return ret;
return 0;
}
#endif // !ROCKSDB_LITE
void AssertCountLiveLogFiles(int value) {
#ifndef ROCKSDB_LITE // GetSortedWalFiles is not supported
ASSERT_EQ(value, CountLiveLogFiles());
#else
(void)value;
#endif // !ROCKSDB_LITE
}
void AssertNumberOfImmutableMemtables(std::vector<int> num_per_cf) {
assert(num_per_cf.size() == handles_.size());
#ifndef ROCKSDB_LITE // GetProperty is not supported in lite
for (size_t i = 0; i < num_per_cf.size(); ++i) {
ASSERT_EQ(num_per_cf[i], GetProperty(static_cast<int>(i),
"rocksdb.num-immutable-mem-table"));
}
#endif // !ROCKSDB_LITE
}
void CopyFile(const std::string& source, const std::string& destination,
@@ -575,7 +533,6 @@ TEST_P(ColumnFamilyTest, DontReuseColumnFamilyID) {
}
}
#ifndef ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
Open();
@@ -598,7 +555,6 @@ TEST_P(ColumnFamilyTest, CreateCFRaceWithGetAggProperty) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // !ROCKSDB_LITE
class FlushEmptyCFTestWithParam
: public ColumnFamilyTestBase,
@@ -942,7 +898,6 @@ TEST_P(ColumnFamilyTest, IgnoreRecoveredLog) {
}
}
#ifndef ROCKSDB_LITE // TEST functions used are not supported
TEST_P(ColumnFamilyTest, FlushTest) {
Open();
CreateColumnFamiliesAndReopen({"one", "two"});
@@ -1057,7 +1012,6 @@ TEST_P(ColumnFamilyTest, LogDeletionTest) {
AssertCountLiveLogFiles(4);
Close();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CrashAfterFlush) {
std::unique_ptr<FaultInjectionTestEnv> fault_env(
@@ -1097,7 +1051,6 @@ TEST_P(ColumnFamilyTest, OpenNonexistentColumnFamily) {
ASSERT_TRUE(TryOpen({"default", "dne"}).IsInvalidArgument());
}
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
// Makes sure that obsolete log files get deleted
TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
// disable flushing stale column families
@@ -1205,14 +1158,12 @@ TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
AssertCountLiveLogFiles(7);
Close();
}
#endif // !ROCKSDB_LITE
// The test is commented out because we want to test that snapshot is
// not created for memtables not supported it, but There isn't a memtable
// that doesn't support snapshot right now. If we have one later, we can
// re-enable the test.
//
// #ifndef ROCKSDB_LITE // Cuckoo is not supported in lite
// TEST_P(ColumnFamilyTest, MemtableNotSupportSnapshot) {
// db_options_.allow_concurrent_memtable_write = false;
// Open();
@@ -1232,7 +1183,6 @@ TEST_P(ColumnFamilyTest, DifferentWriteBufferSizes) {
// {second}); auto* s3 = dbfull()->GetSnapshot(); ASSERT_TRUE(s3 == nullptr);
// Close();
// }
// #endif // !ROCKSDB_LITE
class TestComparator : public Comparator {
int Compare(const ROCKSDB_NAMESPACE::Slice& /*a*/,
@@ -1299,7 +1249,6 @@ TEST_P(ColumnFamilyTest, DifferentMergeOperators) {
Close();
}
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
TEST_P(ColumnFamilyTest, DifferentCompactionStyles) {
Open();
CreateColumnFamilies({"one", "two"});
@@ -1367,9 +1316,7 @@ TEST_P(ColumnFamilyTest, DifferentCompactionStyles) {
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE
// Sync points not supported in RocksDB Lite
TEST_P(ColumnFamilyTest, MultipleManualCompactions) {
@@ -2033,9 +1980,7 @@ TEST_P(ColumnFamilyTest, SameCFAutomaticManualCompactions) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // Tailing iterator not supported
namespace {
std::string IterStatus(Iterator* iter) {
std::string result;
@@ -2093,9 +2038,7 @@ TEST_P(ColumnFamilyTest, NewIteratorsTest) {
Destroy();
}
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // ReadOnlyDB is not supported
TEST_P(ColumnFamilyTest, ReadOnlyDBTest) {
Open();
CreateColumnFamiliesAndReopen({"one", "two", "three", "four"});
@@ -2144,9 +2087,7 @@ TEST_P(ColumnFamilyTest, ReadOnlyDBTest) {
s = OpenReadOnly({"one", "four"});
ASSERT_TRUE(!s.ok());
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported in lite
TEST_P(ColumnFamilyTest, DontRollEmptyLogs) {
Open();
CreateColumnFamiliesAndReopen({"one", "two", "three", "four"});
@@ -2168,9 +2109,7 @@ TEST_P(ColumnFamilyTest, DontRollEmptyLogs) {
ASSERT_EQ(static_cast<size_t>(total_new_writable_files), handles_.size() + 1);
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForCompaction() is not supported in lite
TEST_P(ColumnFamilyTest, FlushStaleColumnFamilies) {
Open();
CreateColumnFamilies({"one", "two"});
@@ -2217,7 +2156,6 @@ TEST_P(ColumnFamilyTest, FlushStaleColumnFamilies) {
ASSERT_EQ(0, dbfull()->TEST_total_log_size());
Close();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateMissingColumnFamilies) {
Status s = TryOpen({"one", "two"});
@@ -2457,8 +2395,6 @@ TEST_P(ColumnFamilyTest, FlushAndDropRaceCondition) {
Destroy();
}
#ifndef ROCKSDB_LITE
// skipped as persisting options is not supported in ROCKSDB_LITE
namespace {
std::atomic<int> test_stage(0);
std::atomic<bool> ordered_by_writethread(false);
@@ -2540,7 +2476,6 @@ TEST_P(ColumnFamilyTest, CreateAndDropRace) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
TEST_P(ColumnFamilyTest, WriteStallSingleColumnFamily) {
const uint64_t kBaseRate = 800000u;
@@ -2950,7 +2885,6 @@ TEST_P(ColumnFamilyTest, CreateDropAndDestroy) {
ASSERT_OK(db_->DestroyColumnFamilyHandle(cfh));
}
#ifndef ROCKSDB_LITE
TEST_P(ColumnFamilyTest, CreateDropAndDestroyWithoutFileDeletion) {
ColumnFamilyHandle* cfh;
Open();
@@ -3005,9 +2939,7 @@ TEST_P(ColumnFamilyTest, FlushCloseWALFiles) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // WaitForFlush() is not supported
TEST_P(ColumnFamilyTest, IteratorCloseWALFile1) {
SpecialEnv env(Env::Default());
db_options_.env = &env;
@@ -3114,9 +3046,7 @@ TEST_P(ColumnFamilyTest, IteratorCloseWALFile2) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
#ifndef ROCKSDB_LITE // TEST functions are not supported in lite
TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
SpecialEnv env(Env::Default());
// Allow both of flush and purge job to schedule.
@@ -3192,7 +3122,6 @@ TEST_P(ColumnFamilyTest, ForwardIteratorCloseWALFile) {
db_options_.env = env_;
Close();
}
#endif // !ROCKSDB_LITE
// Disable on windows because SyncWAL requires env->IsSyncThreadSafe()
// to return true which is not so in unbuffered mode.
-11
View File
@@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include <mutex>
#include <string>
@@ -490,13 +489,3 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as DBImpl::CompactFiles is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
+113 -43
View File
@@ -13,6 +13,7 @@
#include "db/blob/blob_index.h"
#include "db/blob/prefetch_buffer_collection.h"
#include "db/snapshot_checker.h"
#include "db/wide/wide_column_serialization.h"
#include "logging/logging.h"
#include "port/likely.h"
#include "rocksdb/listener.h"
@@ -221,39 +222,47 @@ void CompactionIterator::Next() {
bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
Slice* skip_until) {
// TODO: support compaction filter for wide-column entities
if (!compaction_filter_ ||
(ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex)) {
if (!compaction_filter_) {
return true;
}
bool error = false;
// If the user has specified a compaction filter and the sequence
// number is greater than any external snapshot, then invoke the
// filter. If the return value of the compaction filter is true,
// replace the entry with a deletion marker.
CompactionFilter::Decision filter = CompactionFilter::Decision::kUndetermined;
compaction_filter_value_.clear();
compaction_filter_skip_until_.Clear();
if (ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex &&
ikey_.type != kTypeWideColumnEntity) {
return true;
}
CompactionFilter::Decision decision =
CompactionFilter::Decision::kUndetermined;
CompactionFilter::ValueType value_type =
ikey_.type == kTypeValue ? CompactionFilter::ValueType::kValue
: CompactionFilter::ValueType::kBlobIndex;
: ikey_.type == kTypeBlobIndex
? CompactionFilter::ValueType::kBlobIndex
: CompactionFilter::ValueType::kWideColumnEntity;
// Hack: pass internal key to BlobIndexCompactionFilter since it needs
// to get sequence number.
assert(compaction_filter_);
Slice& filter_key =
(ikey_.type == kTypeValue ||
const Slice& filter_key =
(ikey_.type != kTypeBlobIndex ||
!compaction_filter_->IsStackedBlobDbInternalCompactionFilter())
? ikey_.user_key
: key_;
compaction_filter_value_.clear();
compaction_filter_skip_until_.Clear();
std::vector<std::pair<std::string, std::string>> new_columns;
{
StopWatchNano timer(clock_, report_detailed_time_);
if (kTypeBlobIndex == ikey_.type) {
filter = compaction_filter_->FilterBlobByKey(
if (ikey_.type == kTypeBlobIndex) {
decision = compaction_filter_->FilterBlobByKey(
level_, filter_key, &compaction_filter_value_,
compaction_filter_skip_until_.rep());
if (CompactionFilter::Decision::kUndetermined == filter &&
if (decision == CompactionFilter::Decision::kUndetermined &&
!compaction_filter_->IsStackedBlobDbInternalCompactionFilter()) {
if (compaction_ == nullptr) {
if (!compaction_) {
status_ =
Status::Corruption("Unexpected blob index outside of compaction");
validity_info_.Invalidate();
@@ -299,33 +308,61 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
value_type = CompactionFilter::ValueType::kValue;
}
}
if (CompactionFilter::Decision::kUndetermined == filter) {
filter = compaction_filter_->FilterV2(
level_, filter_key, value_type,
blob_value_.empty() ? value_ : blob_value_, &compaction_filter_value_,
if (decision == CompactionFilter::Decision::kUndetermined) {
const Slice* existing_val = nullptr;
const WideColumns* existing_col = nullptr;
WideColumns existing_columns;
if (ikey_.type != kTypeWideColumnEntity) {
if (!blob_value_.empty()) {
existing_val = &blob_value_;
} else {
existing_val = &value_;
}
} else {
Slice value_copy = value_;
const Status s =
WideColumnSerialization::Deserialize(value_copy, existing_columns);
if (!s.ok()) {
status_ = s;
validity_info_.Invalidate();
return false;
}
existing_col = &existing_columns;
}
decision = compaction_filter_->FilterV3(
level_, filter_key, value_type, existing_val, existing_col,
&compaction_filter_value_, &new_columns,
compaction_filter_skip_until_.rep());
}
iter_stats_.total_filter_time +=
env_ != nullptr && report_detailed_time_ ? timer.ElapsedNanos() : 0;
}
if (CompactionFilter::Decision::kUndetermined == filter) {
// Should not reach here, since FilterV2 should never return kUndetermined.
status_ =
Status::NotSupported("FilterV2() should never return kUndetermined");
if (decision == CompactionFilter::Decision::kUndetermined) {
// Should not reach here, since FilterV2/FilterV3 should never return
// kUndetermined.
status_ = Status::NotSupported(
"FilterV2/FilterV3 should never return kUndetermined");
validity_info_.Invalidate();
return false;
}
if (filter == CompactionFilter::Decision::kRemoveAndSkipUntil &&
if (decision == CompactionFilter::Decision::kRemoveAndSkipUntil &&
cmp_->Compare(*compaction_filter_skip_until_.rep(), ikey_.user_key) <=
0) {
// Can't skip to a key smaller than the current one.
// Keep the key as per FilterV2 documentation.
filter = CompactionFilter::Decision::kKeep;
// Keep the key as per FilterV2/FilterV3 documentation.
decision = CompactionFilter::Decision::kKeep;
}
if (filter == CompactionFilter::Decision::kRemove) {
if (decision == CompactionFilter::Decision::kRemove) {
// convert the current key to a delete; key_ is pointing into
// current_key_ at this point, so updating current_key_ updates key()
ikey_.type = kTypeDeletion;
@@ -333,7 +370,7 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
// no value associated with delete
value_.clear();
iter_stats_.num_record_drop_user++;
} else if (filter == CompactionFilter::Decision::kPurge) {
} else if (decision == CompactionFilter::Decision::kPurge) {
// convert the current key to a single delete; key_ is pointing into
// current_key_ at this point, so updating current_key_ updates key()
ikey_.type = kTypeSingleDeletion;
@@ -341,19 +378,19 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
// no value associated with single delete
value_.clear();
iter_stats_.num_record_drop_user++;
} else if (filter == CompactionFilter::Decision::kChangeValue) {
if (ikey_.type == kTypeBlobIndex) {
// value transfer from blob file to inlined data
} else if (decision == CompactionFilter::Decision::kChangeValue) {
if (ikey_.type != kTypeValue) {
ikey_.type = kTypeValue;
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
current_key_.UpdateInternalKey(ikey_.sequence, kTypeValue);
}
value_ = compaction_filter_value_;
} else if (filter == CompactionFilter::Decision::kRemoveAndSkipUntil) {
} else if (decision == CompactionFilter::Decision::kRemoveAndSkipUntil) {
*need_skip = true;
compaction_filter_skip_until_.ConvertFromUserKey(kMaxSequenceNumber,
kValueTypeForSeek);
*skip_until = compaction_filter_skip_until_.Encode();
} else if (filter == CompactionFilter::Decision::kChangeBlobIndex) {
} else if (decision == CompactionFilter::Decision::kChangeBlobIndex) {
// Only the StackableDB-based BlobDB impl's compaction filter should return
// kChangeBlobIndex. Decision about rewriting blob and changing blob index
// in the integrated BlobDB impl is made in subsequent call to
@@ -365,23 +402,56 @@ bool CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
validity_info_.Invalidate();
return false;
}
if (ikey_.type == kTypeValue) {
// value transfer from inlined data to blob file
if (ikey_.type != kTypeBlobIndex) {
ikey_.type = kTypeBlobIndex;
current_key_.UpdateInternalKey(ikey_.sequence, ikey_.type);
current_key_.UpdateInternalKey(ikey_.sequence, kTypeBlobIndex);
}
value_ = compaction_filter_value_;
} else if (filter == CompactionFilter::Decision::kIOError) {
} else if (decision == CompactionFilter::Decision::kIOError) {
if (!compaction_filter_->IsStackedBlobDbInternalCompactionFilter()) {
status_ = Status::NotSupported(
"CompactionFilter for integrated BlobDB should not return kIOError");
validity_info_.Invalidate();
return false;
}
status_ = Status::IOError("Failed to access blob during compaction filter");
error = true;
validity_info_.Invalidate();
return false;
} else if (decision == CompactionFilter::Decision::kChangeWideColumnEntity) {
WideColumns sorted_columns;
sorted_columns.reserve(new_columns.size());
for (const auto& column : new_columns) {
sorted_columns.emplace_back(column.first, column.second);
}
std::sort(sorted_columns.begin(), sorted_columns.end(),
[](const WideColumn& lhs, const WideColumn& rhs) {
return lhs.name().compare(rhs.name()) < 0;
});
{
const Status s = WideColumnSerialization::Serialize(
sorted_columns, compaction_filter_value_);
if (!s.ok()) {
status_ = s;
validity_info_.Invalidate();
return false;
}
}
if (ikey_.type != kTypeWideColumnEntity) {
ikey_.type = kTypeWideColumnEntity;
current_key_.UpdateInternalKey(ikey_.sequence, kTypeWideColumnEntity);
}
value_ = compaction_filter_value_;
}
return !error;
return true;
}
void CompactionIterator::NextFromInput() {
-20
View File
@@ -978,7 +978,6 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
void CompactionJob::NotifyOnSubcompactionBegin(
SubcompactionState* sub_compact) {
#ifndef ROCKSDB_LITE
Compaction* c = compact_->compaction;
if (db_options_.listeners.empty()) {
@@ -1004,14 +1003,10 @@ void CompactionJob::NotifyOnSubcompactionBegin(
}
info.status.PermitUncheckedError();
#else
(void)sub_compact;
#endif // ROCKSDB_LITE
}
void CompactionJob::NotifyOnSubcompactionCompleted(
SubcompactionState* sub_compact) {
#ifndef ROCKSDB_LITE
if (db_options_.listeners.empty()) {
return;
@@ -1032,16 +1027,12 @@ void CompactionJob::NotifyOnSubcompactionCompleted(
for (const auto& listener : db_options_.listeners) {
listener->OnSubcompactionCompleted(info);
}
#else
(void)sub_compact;
#endif // ROCKSDB_LITE
}
void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
assert(sub_compact);
assert(sub_compact->compaction);
#ifndef ROCKSDB_LITE
if (db_options_.compaction_service) {
CompactionServiceJobStatus comp_status =
ProcessKeyValueCompactionWithCompactionService(sub_compact);
@@ -1052,7 +1043,6 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
// fallback to local compaction
assert(comp_status == CompactionServiceJobStatus::kUseLocal);
}
#endif // !ROCKSDB_LITE
uint64_t prev_cpu_micros = db_options_.clock->CPUMicros();
@@ -1615,7 +1605,6 @@ Status CompactionJob::FinishCompactionOutputFile(
TableFileCreationReason::kCompaction, status_for_listener, file_checksum,
file_checksum_func_name);
#ifndef ROCKSDB_LITE
// Report new file to SstFileManagerImpl
auto sfm =
static_cast<SstFileManagerImpl*>(db_options_.sst_file_manager.get());
@@ -1634,7 +1623,6 @@ Status CompactionJob::FinishCompactionOutputFile(
db_error_handler_->SetBGError(s, BackgroundErrorReason::kCompaction);
}
}
#endif
outputs.ResetBuilder();
return s;
@@ -1759,11 +1747,9 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
std::string fname = GetTableFileName(file_number);
// Fire events.
ColumnFamilyData* cfd = sub_compact->compaction->column_family_data();
#ifndef ROCKSDB_LITE
EventHelpers::NotifyTableFileCreationStarted(
cfd->ioptions()->listeners, dbname_, cfd->GetName(), fname, job_id_,
TableFileCreationReason::kCompaction);
#endif // !ROCKSDB_LITE
// Make the output file
std::unique_ptr<FSWritableFile> writable_file;
#ifndef NDEBUG
@@ -1900,7 +1886,6 @@ void CompactionJob::CleanupCompaction() {
compact_ = nullptr;
}
#ifndef ROCKSDB_LITE
namespace {
void CopyPrefix(const Slice& src, size_t prefix_length, std::string* dst) {
assert(prefix_length > 0);
@@ -1909,7 +1894,6 @@ void CopyPrefix(const Slice& src, size_t prefix_length, std::string* dst) {
}
} // namespace
#endif // !ROCKSDB_LITE
void CompactionJob::UpdateCompactionStats() {
assert(compact_);
@@ -1956,7 +1940,6 @@ void CompactionJob::UpdateCompactionInputStatsHelper(int* num_files,
void CompactionJob::UpdateCompactionJobStats(
const InternalStats::CompactionStats& stats) const {
#ifndef ROCKSDB_LITE
compaction_job_stats_->elapsed_micros = stats.micros;
// input information
@@ -1983,9 +1966,6 @@ void CompactionJob::UpdateCompactionJobStats(
CopyPrefix(compact_->LargestUserKey(), CompactionJobStats::kMaxPrefixLength,
&compaction_job_stats_->largest_output_key_prefix);
}
#else
(void)stats;
#endif // !ROCKSDB_LITE
}
void CompactionJob::LogCompaction() {
@@ -59,7 +59,6 @@
#include "utilities/merge_operators.h"
#if !defined(IOS_CROSS_COMPILE)
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
static std::string RandomString(Random* rnd, int len, double ratio) {
@@ -959,15 +958,6 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED, not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
#else
-11
View File
@@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_job.h"
@@ -2442,13 +2441,3 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as CompactionJobStats is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
-2
View File
@@ -877,7 +877,6 @@ Compaction* CompactionPicker::CompactRange(
return compaction;
}
#ifndef ROCKSDB_LITE
namespace {
// Test whether two files have overlapping key-ranges.
bool HaveOverlappingKeyRanges(const Comparator* c, const SstFileMetaData& a,
@@ -1116,7 +1115,6 @@ Status CompactionPicker::SanitizeCompactionInputFiles(
return Status::OK();
}
#endif // !ROCKSDB_LITE
void CompactionPicker::RegisterCompaction(Compaction* c) {
if (c == nullptr) {
-6
View File
@@ -93,11 +93,9 @@ class CompactionPicker {
// into a valid one by adding more files, the function will return a
// non-ok status with specific reason.
//
#ifndef ROCKSDB_LITE
Status SanitizeCompactionInputFiles(std::unordered_set<uint64_t>* input_files,
const ColumnFamilyMetaData& cf_meta,
const int output_level) const;
#endif // ROCKSDB_LITE
// Free up the files that participated in a compaction
//
@@ -229,11 +227,9 @@ class CompactionPicker {
// A helper function to SanitizeCompactionInputFiles() that
// sanitizes "input_files" by adding necessary files.
#ifndef ROCKSDB_LITE
virtual Status SanitizeCompactionInputFilesForAllLevels(
std::unordered_set<uint64_t>* input_files,
const ColumnFamilyMetaData& cf_meta, const int output_level) const;
#endif // ROCKSDB_LITE
// Keeps track of all compactions that are running on Level0.
// Protected by DB mutex
@@ -246,7 +242,6 @@ class CompactionPicker {
const InternalKeyComparator* const icmp_;
};
#ifndef ROCKSDB_LITE
// A dummy compaction that never triggers any automatic
// compaction.
class NullCompactionPicker : public CompactionPicker {
@@ -287,7 +282,6 @@ class NullCompactionPicker : public CompactionPicker {
return false;
}
};
#endif // !ROCKSDB_LITE
// Attempts to find an intra L0 compaction conforming to the given parameters.
//
-2
View File
@@ -8,7 +8,6 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction_picker_fifo.h"
#ifndef ROCKSDB_LITE
#include <cinttypes>
#include <string>
@@ -443,4 +442,3 @@ Compaction* FIFOCompactionPicker::CompactRange(
}
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
-2
View File
@@ -8,7 +8,6 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#pragma once
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_picker.h"
@@ -60,4 +59,3 @@ class FIFOCompactionPicker : public CompactionPicker {
LogBuffer* log_buffer);
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
-5
View File
@@ -482,8 +482,6 @@ TEST_F(CompactionPickerTest, LevelTriggerDynamic4) {
ASSERT_EQ(num_levels - 1, compaction->output_level());
}
// Universal and FIFO Compactions are not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(CompactionPickerTest, NeedsCompactionUniversal) {
NewVersionStorage(1, kCompactionStyleUniversal);
UniversalCompactionPicker universal_compaction_picker(ioptions_, &icmp_);
@@ -1245,7 +1243,6 @@ TEST_F(CompactionPickerTest, FIFOToWarmWithHotBetweenWarms) {
ASSERT_EQ(2U, compaction->input(0, 0)->fd.GetNumber());
}
#endif // ROCKSDB_LITE
TEST_F(CompactionPickerTest, CompactionPriMinOverlapping1) {
NewVersionStorage(6, kCompactionStyleLevel);
@@ -2873,7 +2870,6 @@ TEST_F(CompactionPickerTest, IntraL0MaxCompactionBytesHit) {
ASSERT_EQ(0, compaction->output_level());
}
#ifndef ROCKSDB_LITE
TEST_F(CompactionPickerTest, UniversalMarkedCompactionFullOverlap) {
const uint64_t kFileSize = 100000;
@@ -3982,7 +3978,6 @@ TEST_P(PerKeyPlacementCompactionPickerTest,
INSTANTIATE_TEST_CASE_P(PerKeyPlacementCompactionPickerTest,
PerKeyPlacementCompactionPickerTest, ::testing::Bool());
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
@@ -8,7 +8,6 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/compaction/compaction_picker_universal.h"
#ifndef ROCKSDB_LITE
#include <cinttypes>
#include <limits>
@@ -1451,4 +1450,3 @@ uint64_t UniversalCompactionBuilder::GetMaxOverlappingBytes() const {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
@@ -8,7 +8,6 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#pragma once
#ifndef ROCKSDB_LITE
#include "db/compaction/compaction_picker.h"
@@ -29,4 +28,3 @@ class UniversalCompactionPicker : public CompactionPicker {
const VersionStorageInfo* vstorage) const override;
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
-2
View File
@@ -16,7 +16,6 @@
#include "options/options_helper.h"
#include "rocksdb/utilities/options_type.h"
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
class SubcompactionState;
@@ -832,4 +831,3 @@ bool CompactionServiceInput::TEST_Equals(CompactionServiceInput* other,
#endif // NDEBUG
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
-11
View File
@@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_test_util.h"
#include "port/stack_trace.h"
@@ -954,13 +953,3 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr,
"SKIPPED as CompactionService is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+1 -8
View File
@@ -15,11 +15,9 @@
namespace ROCKSDB_NAMESPACE {
static std::unordered_map<std::string, OptionTypeInfo>
sst_fixed_prefix_type_info = {
#ifndef ROCKSDB_LITE
{"length",
{0, OptionType::kSizeT, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
#endif // ROCKSDB_LITE
};
SstPartitionerFixedPrefixFactory::SstPartitionerFixedPrefixFactory(size_t len)
@@ -58,7 +56,6 @@ std::shared_ptr<SstPartitionerFactory> NewSstPartitionerFixedPrefixFactory(
return std::make_shared<SstPartitionerFixedPrefixFactory>(prefix_len);
}
#ifndef ROCKSDB_LITE
namespace {
static int RegisterSstPartitionerFactories(ObjectLibrary& library,
const std::string& /*arg*/) {
@@ -73,18 +70,14 @@ static int RegisterSstPartitionerFactories(ObjectLibrary& library,
return 1;
}
} // namespace
#endif // ROCKSDB_LITE
Status SstPartitionerFactory::CreateFromString(
const ConfigOptions& options, const std::string& value,
std::shared_ptr<SstPartitionerFactory>* result) {
#ifndef ROCKSDB_LITE
static std::once_flag once;
std::call_once(once, [&]() {
RegisterSstPartitionerFactories(*(ObjectLibrary::Default().get()), "");
});
#endif // ROCKSDB_LITE
return LoadSharedObject<SstPartitionerFactory>(options, value, nullptr,
result);
return LoadSharedObject<SstPartitionerFactory>(options, value, result);
}
} // namespace ROCKSDB_NAMESPACE
-2
View File
@@ -99,7 +99,6 @@ class SubcompactionState {
penultimate_level_outputs_.RemoveLastEmptyOutput();
}
#ifndef ROCKSDB_LITE
void BuildSubcompactionJobInfo(
SubcompactionJobInfo& subcompaction_job_info) const {
const Compaction* c = compaction;
@@ -113,7 +112,6 @@ class SubcompactionState {
subcompaction_job_info.output_level = c->output_level();
subcompaction_job_info.stats = compaction_job_stats;
}
#endif // !ROCKSDB_LITE
SubcompactionState() = delete;
SubcompactionState(const SubcompactionState&) = delete;
-8
View File
@@ -17,7 +17,6 @@
namespace ROCKSDB_NAMESPACE {
#if !defined(ROCKSDB_LITE)
class TieredCompactionTest : public DBTestBase,
public testing::WithParamInterface<bool> {
@@ -2139,18 +2138,11 @@ TEST_F(PrecludeLastLevelTest, RangeDelsCauseFileEndpointsToOverlap) {
Close();
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
-2
View File
@@ -4,7 +4,6 @@
// (found in the LICENSE.Apache file in the root directory).
//
#ifndef ROCKSDB_LITE
#include "rocksdb/convenience.h"
@@ -78,4 +77,3 @@ Status VerifySstFileChecksum(const Options& options,
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+53 -56
View File
@@ -8,7 +8,6 @@
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "rocksdb/options.h"
#ifndef ROCKSDB_LITE
#include <fcntl.h>
#include <sys/stat.h>
@@ -43,33 +42,36 @@ namespace ROCKSDB_NAMESPACE {
static constexpr int kValueSize = 1000;
namespace {
// A wrapper that allows injection of errors.
class ErrorEnv : public EnvWrapper {
class ErrorFS : public FileSystemWrapper {
public:
bool writable_file_error_;
int num_writable_file_errors_;
explicit ErrorEnv(Env* _target)
: EnvWrapper(_target),
explicit ErrorFS(const std::shared_ptr<FileSystem>& _target)
: FileSystemWrapper(_target),
writable_file_error_(false),
num_writable_file_errors_(0) {}
const char* Name() const override { return "ErrorEnv"; }
virtual Status NewWritableFile(const std::string& fname,
std::unique_ptr<WritableFile>* result,
const EnvOptions& soptions) override {
virtual IOStatus NewWritableFile(const std::string& fname,
const FileOptions& opts,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override {
result->reset();
if (writable_file_error_) {
++num_writable_file_errors_;
return Status::IOError(fname, "fake error");
return IOStatus::IOError(fname, "fake error");
}
return target()->NewWritableFile(fname, result, soptions);
return target()->NewWritableFile(fname, opts, result, dbg);
}
};
} // anonymous namespace
class CorruptionTest : public testing::Test {
public:
std::shared_ptr<Env> env_guard_;
ErrorEnv* env_;
std::shared_ptr<ErrorFS> fs_;
std::unique_ptr<Env> env_;
Env* base_env_;
std::string dbname_;
std::shared_ptr<Cache> tiny_cache_;
Options options_;
@@ -80,14 +82,15 @@ class CorruptionTest : public testing::Test {
// set it to 0), test SequenceNumberRecovery will fail, likely because of a
// bug in recovery code. Keep it 4 for now to make the test passes.
tiny_cache_ = NewLRUCache(100, 4);
Env* base_env = Env::Default();
base_env_ = Env::Default();
EXPECT_OK(
test::CreateEnvFromSystem(ConfigOptions(), &base_env, &env_guard_));
EXPECT_NE(base_env, nullptr);
env_ = new ErrorEnv(base_env);
test::CreateEnvFromSystem(ConfigOptions(), &base_env_, &env_guard_));
EXPECT_NE(base_env_, nullptr);
fs_.reset(new ErrorFS(base_env_->GetFileSystem()));
env_ = NewCompositeEnv(fs_);
options_.wal_recovery_mode = WALRecoveryMode::kTolerateCorruptedTailRecords;
options_.env = env_;
dbname_ = test::PerThreadDBPath(env_, "corruption_test");
options_.env = env_.get();
dbname_ = test::PerThreadDBPath(env_.get(), "corruption_test");
Status s = DestroyDB(dbname_, options_);
EXPECT_OK(s);
@@ -110,10 +113,9 @@ class CorruptionTest : public testing::Test {
fprintf(stdout, "db is still at %s\n", dbname_.c_str());
} else {
Options opts;
opts.env = env_->target();
opts.env = base_env_;
EXPECT_OK(DestroyDB(dbname_, opts));
}
delete env_;
}
void CloseDb() {
@@ -128,7 +130,7 @@ class CorruptionTest : public testing::Test {
if (opt.env == Options().env) {
// If env is not overridden, replace it with ErrorEnv.
// Otherwise, the test already uses a non-default Env.
opt.env = env_;
opt.env = env_.get();
}
opt.arena_block_size = 4096;
BlockBasedTableOptions table_options;
@@ -165,6 +167,10 @@ class CorruptionTest : public testing::Test {
void Build(int n, int flush_every = 0) { Build(n, 0, flush_every); }
void Check(int min_expected, int max_expected) {
Check(min_expected, max_expected, ReadOptions(false, true));
}
void Check(int min_expected, int max_expected, ReadOptions read_options) {
uint64_t next_expected = 0;
uint64_t missed = 0;
int bad_keys = 0;
@@ -176,7 +182,7 @@ class CorruptionTest : public testing::Test {
// Instead, we want the reads to be successful and this test
// will detect whether the appropriate corruptions have
// occurred.
Iterator* iter = db_->NewIterator(ReadOptions(false, true));
Iterator* iter = db_->NewIterator(read_options);
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ASSERT_OK(iter->status());
uint64_t key;
@@ -223,7 +229,7 @@ class CorruptionTest : public testing::Test {
}
ASSERT_TRUE(!fname.empty()) << filetype;
ASSERT_OK(test::CorruptFile(env_, fname, offset, bytes_to_corrupt,
ASSERT_OK(test::CorruptFile(env_.get(), fname, offset, bytes_to_corrupt,
/*verify_checksum*/ filetype == kTableFile));
}
@@ -234,7 +240,7 @@ class CorruptionTest : public testing::Test {
db_->GetLiveFilesMetaData(&metadata);
for (const auto& m : metadata) {
if (m.level == level) {
ASSERT_OK(test::CorruptFile(env_, dbname_ + "/" + m.name, offset,
ASSERT_OK(test::CorruptFile(env_.get(), dbname_ + "/" + m.name, offset,
bytes_to_corrupt));
return;
}
@@ -308,7 +314,7 @@ class CorruptionTest : public testing::Test {
if (bytes_to_truncate == 0) {
new_size = 0;
}
ASSERT_OK(test::TruncateFile(env_, path, new_size));
ASSERT_OK(test::TruncateFile(env_.get(), path, new_size));
}
};
@@ -402,14 +408,14 @@ TEST_F(CorruptionTest, PostPITRCorruptionWALsRetained) {
}
TEST_F(CorruptionTest, RecoverWriteError) {
env_->writable_file_error_ = true;
fs_->writable_file_error_ = true;
Status s = TryReopen();
ASSERT_TRUE(!s.ok());
}
TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
// Do enough writing to force minor compaction
env_->writable_file_error_ = true;
fs_->writable_file_error_ = true;
const int num =
static_cast<int>(3 + (Options().write_buffer_size / kValueSize));
std::string value_storage;
@@ -425,8 +431,8 @@ TEST_F(CorruptionTest, NewFileErrorDuringWrite) {
ASSERT_TRUE(!failed || !s.ok());
}
ASSERT_TRUE(!s.ok());
ASSERT_GE(env_->num_writable_file_errors_, 1);
env_->writable_file_error_ = false;
ASSERT_GE(fs_->num_writable_file_errors_, 1);
fs_->writable_file_error_ = false;
Reopen();
}
@@ -444,7 +450,7 @@ TEST_F(CorruptionTest, TableFile) {
TEST_F(CorruptionTest, VerifyChecksumReadahead) {
Options options;
SpecialEnv senv(env_->target());
SpecialEnv senv(base_env_);
options.env = &senv;
// Disable block cache as we are going to check checksum for
// the same file twice and measure number of reads.
@@ -512,7 +518,7 @@ TEST_F(CorruptionTest, TableFileIndexData) {
dbi = static_cast_with_check<DBImpl>(db_);
// one full file may be readable, since only one was corrupted
// the other file should be fully non-readable, since index was corrupted
Check(0, 5000);
Check(0, 5000, ReadOptions(true, true));
ASSERT_NOK(dbi->VerifyChecksum());
// In paranoid mode, the db cannot be opened due to the corrupted file.
@@ -587,8 +593,8 @@ TEST_F(CorruptionTest, TableFileWrongSize) {
// Make the file smaller with truncation.
// First leaving a partial footer, and then completely removing footer.
for (size_t bytes_lost : {8, 100}) {
ASSERT_OK(
test::TruncateFile(env_, filename, metadata[0].size - bytes_lost));
ASSERT_OK(test::TruncateFile(env_.get(), filename,
metadata[0].size - bytes_lost));
// Reported well with paranoid checks
options_.paranoid_checks = true;
@@ -653,7 +659,7 @@ TEST_F(CorruptionTest, CorruptedDescriptor) {
TEST_F(CorruptionTest, CompactionInputError) {
Options options;
options.env = env_;
options.env = env_.get();
Reopen(&options);
Build(10);
DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
@@ -674,7 +680,7 @@ TEST_F(CorruptionTest, CompactionInputError) {
TEST_F(CorruptionTest, CompactionInputErrorParanoid) {
Options options;
options.env = env_;
options.env = env_.get();
options.paranoid_checks = true;
options.write_buffer_size = 131072;
options.max_write_buffer_number = 2;
@@ -761,7 +767,7 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
ImmutableOptions(options_), kRangeDelBlockName, &range_del_handle));
ASSERT_OK(TryReopen());
ASSERT_OK(test::CorruptFile(env_, filename,
ASSERT_OK(test::CorruptFile(env_.get(), filename,
static_cast<int>(range_del_handle.offset()), 1));
ASSERT_TRUE(TryReopen().IsCorruption());
}
@@ -769,7 +775,7 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
TEST_F(CorruptionTest, FileSystemStateCorrupted) {
for (int iter = 0; iter < 2; ++iter) {
Options options;
options.env = env_;
options.env = env_.get();
options.paranoid_checks = true;
options.create_if_missing = true;
Reopen(&options);
@@ -808,7 +814,7 @@ static const auto& corruption_modes = {
TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
Options options;
options.env = env_;
options.env = env_.get();
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -836,7 +842,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnFlush) {
TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
Options options;
options.env = env_;
options.env = env_.get();
options.paranoid_file_checks = true;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
@@ -869,7 +875,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
Options options;
options.env = env_;
options.env = env_.get();
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -905,7 +911,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
Options options;
options.env = env_;
options.env = env_.get();
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -944,7 +950,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
Options options;
options.env = env_;
options.env = env_.get();
options.check_flush_compaction_key_order = false;
options.paranoid_file_checks = true;
options.create_if_missing = true;
@@ -980,7 +986,7 @@ TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
Options options;
options.env = env_;
options.env = env_.get();
options.create_if_missing = true;
options.allow_data_in_errors = true;
auto mode = mock::MockTableFactory::kCorruptKey;
@@ -1009,7 +1015,7 @@ TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
Options options;
options.env = env_;
options.env = env_.get();
options.paranoid_file_checks = false;
options.create_if_missing = true;
options.check_flush_compaction_key_order = false;
@@ -1036,7 +1042,7 @@ TEST_F(CorruptionTest, CompactionKeyOrderCheck) {
TEST_F(CorruptionTest, FlushKeyOrderCheck) {
Options options;
options.env = env_;
options.env = env_.get();
options.paranoid_file_checks = false;
options.create_if_missing = true;
ASSERT_OK(db_->SetOptions({{"check_flush_compaction_key_order", "true"}}));
@@ -1089,7 +1095,7 @@ TEST_F(CorruptionTest, DisableKeyOrderCheck) {
TEST_F(CorruptionTest, VerifyWholeTableChecksum) {
CloseDb();
Options options;
options.env = env_;
options.env = env_.get();
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.file_checksum_gen_factory =
@@ -1178,7 +1184,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, CrashDuringRecovery) {
track_and_verify_wals_in_manifest_;
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.avoid_flush_during_recovery = false;
options.env = env_;
options.env = env_.get();
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.max_write_buffer_number = 8;
@@ -1350,7 +1356,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, TxnDbCrashDuringRecovery) {
options.track_and_verify_wals_in_manifest =
track_and_verify_wals_in_manifest_;
options.avoid_flush_during_recovery = false;
options.env = env_;
options.env = env_.get();
ASSERT_OK(DestroyDB(dbname_, options));
options.create_if_missing = true;
options.max_write_buffer_number = 3;
@@ -1545,7 +1551,7 @@ TEST_P(CrashDuringRecoveryWithCorruptionTest, CrashDuringRecoveryWithFlush) {
Options options;
options.wal_recovery_mode = WALRecoveryMode::kPointInTimeRecovery;
options.avoid_flush_during_recovery = false;
options.env = env_;
options.env = env_.get();
options.create_if_missing = true;
ASSERT_OK(DestroyDB(dbname_, options));
@@ -1661,12 +1667,3 @@ int main(int argc, char** argv) {
return RUN_ALL_TESTS();
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED as RepairDB() is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // !ROCKSDB_LITE
-10
View File
@@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_impl/db_impl.h"
#include "db/db_test_util.h"
@@ -350,12 +349,3 @@ int main(int argc, char** argv) {
}
}
#else
#include <stdio.h>
int main(int /*argc*/, char** /*argv*/) {
fprintf(stderr, "SKIPPED as Cuckoo table is not supported in ROCKSDB_LITE\n");
return 0;
}
#endif // ROCKSDB_LITE
+121 -50
View File
@@ -20,9 +20,7 @@
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/block_builder.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "util/file_checksum_helper.h"
#include "util/random.h"
#include "utilities/counted_fs.h"
@@ -32,6 +30,9 @@
namespace ROCKSDB_NAMESPACE {
static bool enable_io_uring = true;
extern "C" bool RocksDbIOUringEnable() { return enable_io_uring; }
class DBBasicTest : public DBTestBase {
public:
DBBasicTest() : DBTestBase("db_basic_test", /*env_do_fsync=*/false) {}
@@ -95,7 +96,6 @@ TEST_F(DBBasicTest, UniqueSession) {
EXPECT_MATCHES_REGEX(sid2, expected);
EXPECT_MATCHES_REGEX(sid3, expected);
#ifndef ROCKSDB_LITE
Close();
ASSERT_OK(ReadOnlyReopen(options));
ASSERT_OK(db_->GetDbSessionId(sid1));
@@ -110,7 +110,6 @@ TEST_F(DBBasicTest, UniqueSession) {
ASSERT_NE(sid1, sid2);
ASSERT_EQ(sid2, sid3);
#endif // ROCKSDB_LITE
CreateAndReopenWithCF({"goku"}, options);
ASSERT_OK(db_->GetDbSessionId(sid1));
@@ -127,7 +126,6 @@ TEST_F(DBBasicTest, UniqueSession) {
ASSERT_NE(sid1, sid4);
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, ReadOnlyDB) {
ASSERT_OK(Put("foo", "v1"));
ASSERT_OK(Put("bar", "v2"));
@@ -364,7 +362,6 @@ TEST_F(DBBasicTest, LevelLimitReopen) {
options.max_bytes_for_level_multiplier_additional.resize(10, 1);
ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#endif // ROCKSDB_LITE
TEST_F(DBBasicTest, PutDeleteGet) {
do {
@@ -426,7 +423,6 @@ TEST_F(DBBasicTest, GetFromVersions) {
} while (ChangeOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, GetSnapshot) {
anon::OptionsOverride options_override;
options_override.skip_policy = kSkipNoSnapshot;
@@ -447,7 +443,6 @@ TEST_F(DBBasicTest, GetSnapshot) {
}
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
TEST_F(DBBasicTest, CheckLock) {
do {
@@ -679,7 +674,27 @@ TEST_F(DBBasicTest, IdentityAcrossRestarts) {
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, LockFileRecovery) {
Options options = CurrentOptions();
// Regardless of best_efforts_recovery
for (bool ber : {false, true}) {
options.best_efforts_recovery = ber;
DestroyAndReopen(options);
std::string id1, id2;
ASSERT_OK(db_->GetDbIdentity(id1));
Close();
// Should be OK to re-open DB after lock file deleted
std::string lockfilename = LockFileName(dbname_);
ASSERT_OK(env_->DeleteFile(lockfilename));
Reopen(options);
// Should be same DB as before
ASSERT_OK(db_->GetDbIdentity(id2));
ASSERT_EQ(id1, id2);
}
}
TEST_F(DBBasicTest, Snapshot) {
env_->SetMockSleep();
anon::OptionsOverride options_override;
@@ -751,7 +766,6 @@ TEST_F(DBBasicTest, Snapshot) {
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
class DBBasicMultiConfigs : public DBBasicTest,
public ::testing::WithParamInterface<int> {
@@ -2162,6 +2176,7 @@ class DBMultiGetAsyncIOTest : public DBBasicTest,
options_.disable_auto_compactions = true;
options_.statistics = statistics_;
options_.table_factory.reset(NewBlockBasedTableFactory(bbto));
options_.env = Env::Default();
Reopen(options_);
int num_keys = 0;
@@ -2228,6 +2243,20 @@ class DBMultiGetAsyncIOTest : public DBBasicTest,
const std::shared_ptr<Statistics>& statistics() { return statistics_; }
protected:
void PrepareDBForTest() {
#ifdef ROCKSDB_IOURING_PRESENT
Reopen(options_);
#else // ROCKSDB_IOURING_PRESENT
// Warm up the block cache so we don't need to use the IO uring
Iterator* iter = dbfull()->NewIterator(ReadOptions());
for (iter->SeekToFirst(); iter->Valid() && iter->status().ok();
iter->Next())
;
EXPECT_OK(iter->status());
delete iter;
#endif // ROCKSDB_IOURING_PRESENT
}
void ReopenDB() { Reopen(options_); }
private:
@@ -2242,6 +2271,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
std::vector<PinnableSlice> values(key_strs.size());
std::vector<Status> statuses(key_strs.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2260,6 +2291,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
// With async IO, lookups will happen in parallel for each key
#ifdef ROCKSDB_IOURING_PRESENT
if (GetParam()) {
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
@@ -2269,6 +2301,11 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL0) {
// L0 file
ASSERT_EQ(multiget_io_batch_size.count, 3);
}
#else // ROCKSDB_IOURING_PRESENT
if (GetParam()) {
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
@@ -2286,6 +2323,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2299,6 +2338,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
ASSERT_EQ(values[1], "val_l1_" + std::to_string(54));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2306,9 +2346,11 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1) {
// A batch of 3 async IOs is expected, one for each overlapping file in L1
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
#endif // ROCKSDB_IOURING_PRESENT
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#ifdef ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
@@ -2324,9 +2366,9 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
values.resize(keys.size());
statuses.resize(keys.size());
int count = 0;
SyncPoint::GetInstance()->SetCallBack(
"TableCache::GetTableReader:BeforeOpenFile", [&](void* status) {
static int count = 0;
count++;
// Fail the last table reader open, which is the 6th SST file
// since 3 overlapping L0 files + 3 L1 files containing the keys
@@ -2349,7 +2391,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
});
SyncPoint::GetInstance()->EnableProcessing();
ReopenDB();
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
@@ -2371,6 +2413,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1Error) {
ASSERT_EQ(multiget_io_batch_size.max, 2);
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 2);
}
#endif // ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
std::vector<std::string> key_strs;
@@ -2388,6 +2431,8 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2401,6 +2446,7 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
ASSERT_EQ(values[1], "val_l1_" + std::to_string(54));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2411,6 +2457,7 @@ TEST_P(DBMultiGetAsyncIOTest, LastKeyInFile) {
// will lookup 2 files in parallel and issue 2 async reads
ASSERT_EQ(multiget_io_batch_size.count, 2);
ASSERT_EQ(multiget_io_batch_size.max, 2);
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
@@ -2429,6 +2476,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2442,6 +2491,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
ASSERT_EQ(values[1], "val_l2_" + std::to_string(56));
ASSERT_EQ(values[2], "val_l1_" + std::to_string(102));
#ifdef ROCKSDB_IOURING_PRESENT
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
@@ -2451,6 +2501,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2) {
// Otherwise, the L2 lookup will happen after L1.
ASSERT_EQ(multiget_io_batch_size.count, GetParam() ? 1 : 2);
ASSERT_EQ(multiget_io_batch_size.max, GetParam() ? 3 : 2);
#endif // ROCKSDB_IOURING_PRESENT
}
TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
@@ -2467,6 +2518,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2482,6 +2535,7 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeOverlapL0L1) {
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 2);
}
#ifdef ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeDelInL1) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
@@ -2496,6 +2550,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL2WithRangeDelInL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2525,6 +2581,8 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2WithRangeDelInL1) {
values.resize(keys.size());
statuses.resize(keys.size());
PrepareDBForTest();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
@@ -2539,6 +2597,45 @@ TEST_P(DBMultiGetAsyncIOTest, GetFromL1AndL2WithRangeDelInL1) {
// Bloom filters in L0/L1 will avoid the coroutine calls in those levels
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
#endif // ROCKSDB_IOURING_PRESENT
TEST_P(DBMultiGetAsyncIOTest, GetNoIOUring) {
std::vector<std::string> key_strs;
std::vector<Slice> keys;
std::vector<PinnableSlice> values;
std::vector<Status> statuses;
key_strs.push_back(Key(33));
key_strs.push_back(Key(54));
key_strs.push_back(Key(102));
keys.push_back(key_strs[0]);
keys.push_back(key_strs[1]);
keys.push_back(key_strs[2]);
values.resize(keys.size());
statuses.resize(keys.size());
enable_io_uring = false;
ReopenDB();
ReadOptions ro;
ro.async_io = true;
ro.optimize_multiget_for_io = GetParam();
dbfull()->MultiGet(ro, dbfull()->DefaultColumnFamily(), keys.size(),
keys.data(), values.data(), statuses.data());
ASSERT_EQ(values.size(), 3);
ASSERT_EQ(statuses[0], Status::NotSupported());
ASSERT_EQ(statuses[1], Status::NotSupported());
ASSERT_EQ(statuses[2], Status::NotSupported());
HistogramData multiget_io_batch_size;
statistics()->histogramData(MULTIGET_IO_BATCH_SIZE, &multiget_io_batch_size);
// A batch of 3 async IOs is expected, one for each overlapping file in L1
ASSERT_EQ(multiget_io_batch_size.count, 1);
ASSERT_EQ(multiget_io_batch_size.max, 3);
ASSERT_EQ(statistics()->getTickerCount(MULTIGET_COROUTINE_COUNT), 3);
}
INSTANTIATE_TEST_CASE_P(DBMultiGetAsyncIOTest, DBMultiGetAsyncIOTest,
testing::Bool());
@@ -2708,7 +2805,6 @@ TEST_P(MultiGetPrefixExtractorTest, Batched) {
INSTANTIATE_TEST_CASE_P(MultiGetPrefix, MultiGetPrefixExtractorTest,
::testing::Bool());
#ifndef ROCKSDB_LITE
class DBMultiGetRowCacheTest : public DBBasicTest,
public ::testing::WithParamInterface<bool> {};
@@ -2863,7 +2959,6 @@ TEST_F(DBBasicTest, ValueTypeString) {
ASSERT_TRUE(key_version.GetTypeName() != "Invalid");
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBBasicTest, MultiGetIOBufferOverrun) {
Options options = CurrentOptions();
@@ -2964,7 +3059,6 @@ TEST_F(DBBasicTest, BestEffortsRecoveryWithVersionBuildingFailure) {
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
namespace {
class TableFileListener : public EventListener {
public:
@@ -3239,7 +3333,6 @@ TEST_F(DBBasicTest, DisableTrackWal) {
ASSERT_TRUE(dbfull()->GetVersionSet()->GetWalSet().GetWals().empty());
Close();
}
#endif // !ROCKSDB_LITE
TEST_F(DBBasicTest, ManifestChecksumMismatch) {
Options options = CurrentOptions();
@@ -3283,7 +3376,6 @@ TEST_F(DBBasicTest, ConcurrentlyCloseDB) {
}
}
#ifndef ROCKSDB_LITE
class DBBasicTestTrackWal : public DBTestBase,
public testing::WithParamInterface<bool> {
public:
@@ -3339,21 +3431,16 @@ TEST_P(DBBasicTestTrackWal, DoNotTrackObsoleteWal) {
INSTANTIATE_TEST_CASE_P(DBBasicTestTrackWal, DBBasicTestTrackWal,
testing::Bool());
#endif // ROCKSDB_LITE
class DBBasicTestMultiGet : public DBTestBase {
public:
DBBasicTestMultiGet(std::string test_dir, int num_cfs, bool compressed_cache,
DBBasicTestMultiGet(std::string test_dir, int num_cfs,
bool uncompressed_cache, bool _compression_enabled,
bool _fill_cache, uint32_t compression_parallel_threads)
: DBTestBase(test_dir, /*env_do_fsync=*/false) {
compression_enabled_ = _compression_enabled;
fill_cache_ = _fill_cache;
if (compressed_cache) {
std::shared_ptr<Cache> cache = NewLRUCache(1048576);
compressed_cache_ = std::make_shared<MyBlockCache>(cache);
}
if (uncompressed_cache) {
std::shared_ptr<Cache> cache = NewLRUCache(1048576);
uncompressed_cache_ = std::make_shared<MyBlockCache>(cache);
@@ -3365,7 +3452,6 @@ class DBBasicTestMultiGet : public DBTestBase {
Random rnd(301);
BlockBasedTableOptions table_options;
#ifndef ROCKSDB_LITE
if (compression_enabled_) {
std::vector<CompressionType> compression_types;
compression_types = GetSupportedCompressions();
@@ -3384,12 +3470,6 @@ class DBBasicTestMultiGet : public DBTestBase {
compression_enabled_ = false;
}
}
#else
// GetSupportedCompressions() is not available in LITE build
if (!Snappy_Supported()) {
compression_enabled_ = false;
}
#endif // ROCKSDB_LITE
table_options.block_cache = uncompressed_cache_;
if (table_options.block_cache == nullptr) {
@@ -3397,7 +3477,6 @@ class DBBasicTestMultiGet : public DBTestBase {
} else {
table_options.pin_l0_filter_and_index_blocks_in_cache = true;
}
table_options.block_cache_compressed = compressed_cache_;
table_options.flush_block_policy_factory.reset(
new MyFlushBlockPolicyFactory());
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@@ -3580,16 +3659,14 @@ class DBBasicTestMultiGet : public DBTestBase {
std::vector<std::string> cf_names_;
};
class DBBasicTestWithParallelIO
: public DBBasicTestMultiGet,
public testing::WithParamInterface<
std::tuple<bool, bool, bool, bool, uint32_t>> {
class DBBasicTestWithParallelIO : public DBBasicTestMultiGet,
public testing::WithParamInterface<
std::tuple<bool, bool, bool, uint32_t>> {
public:
DBBasicTestWithParallelIO()
: DBBasicTestMultiGet("/db_basic_test_with_parallel_io", 1,
std::get<0>(GetParam()), std::get<1>(GetParam()),
std::get<2>(GetParam()), std::get<3>(GetParam()),
std::get<4>(GetParam())) {}
std::get<2>(GetParam()), std::get<3>(GetParam())) {}
};
TEST_P(DBBasicTestWithParallelIO, MultiGet) {
@@ -3715,7 +3792,6 @@ TEST_P(DBBasicTestWithParallelIO, MultiGet) {
}
}
#ifndef ROCKSDB_LITE
TEST_P(DBBasicTestWithParallelIO, MultiGetDirectIO) {
class FakeDirectIOEnv : public EnvWrapper {
class FakeDirectIOSequentialFile;
@@ -3832,7 +3908,6 @@ TEST_P(DBBasicTestWithParallelIO, MultiGetDirectIO) {
}
Close();
}
#endif // ROCKSDB_LITE
TEST_P(DBBasicTestWithParallelIO, MultiGetWithChecksumMismatch) {
std::vector<std::string> key_data(10);
@@ -3920,13 +3995,12 @@ TEST_P(DBBasicTestWithParallelIO, MultiGetWithMissingFile) {
INSTANTIATE_TEST_CASE_P(ParallelIO, DBBasicTestWithParallelIO,
// Params are as follows -
// Param 0 - Compressed cache enabled
// Param 1 - Uncompressed cache enabled
// Param 2 - Data compression enabled
// Param 3 - ReadOptions::fill_cache
// Param 4 - CompressionOptions::parallel_threads
// Param 0 - Uncompressed cache enabled
// Param 1 - Data compression enabled
// Param 2 - ReadOptions::fill_cache
// Param 3 - CompressionOptions::parallel_threads
::testing::Combine(::testing::Bool(), ::testing::Bool(),
::testing::Bool(), ::testing::Bool(),
::testing::Bool(),
::testing::Values(1, 4)));
// Forward declaration
@@ -4137,9 +4211,8 @@ class DBBasicTestMultiGetDeadline : public DBBasicTestMultiGet,
DBBasicTestMultiGetDeadline()
: DBBasicTestMultiGet(
"db_basic_test_multiget_deadline" /*Test dir*/,
10 /*# of column families*/, false /*compressed cache enabled*/,
true /*uncompressed cache enabled*/, true /*compression enabled*/,
true /*ReadOptions.fill_cache*/,
10 /*# of column families*/, true /*uncompressed cache enabled*/,
true /*compression enabled*/, true /*ReadOptions.fill_cache*/,
1 /*# of parallel compression threads*/) {}
inline void CheckStatus(std::vector<Status>& statuses, size_t num_ok) {
@@ -4375,7 +4448,6 @@ TEST_F(DBBasicTest, FailOpenIfLoggerCreationFail) {
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
TEST_F(DBBasicTest, VerifyFileChecksums) {
Options options = GetDefaultOptions();
options.create_if_missing = true;
@@ -4441,7 +4513,6 @@ TEST_F(DBBasicTest, DISABLED_ManualWalSync) {
ASSERT_TRUE(TryReopen(options).IsCorruption());
}
#endif // !ROCKSDB_LITE
// A test class for intercepting random reads and injecting artificial
// delays. Used for testing the deadline/timeout feature
+15 -358
View File
@@ -44,10 +44,6 @@ class DBBlockCacheTest : public DBTestBase {
size_t compression_dict_miss_count_ = 0;
size_t compression_dict_hit_count_ = 0;
size_t compression_dict_insert_count_ = 0;
size_t compressed_miss_count_ = 0;
size_t compressed_hit_count_ = 0;
size_t compressed_insert_count_ = 0;
size_t compressed_failure_count_ = 0;
public:
const size_t kNumBlocks = 10;
@@ -85,14 +81,6 @@ class DBBlockCacheTest : public DBTestBase {
hit_count_ = TestGetTickerCount(options, BLOCK_CACHE_HIT);
insert_count_ = TestGetTickerCount(options, BLOCK_CACHE_ADD);
failure_count_ = TestGetTickerCount(options, BLOCK_CACHE_ADD_FAILURES);
compressed_miss_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS);
compressed_hit_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT);
compressed_insert_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD);
compressed_failure_count_ =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD_FAILURES);
}
void RecordCacheCountersForCompressionDict(const Options& options) {
@@ -144,30 +132,6 @@ class DBBlockCacheTest : public DBTestBase {
compression_dict_insert_count_ = new_compression_dict_insert_count;
}
void CheckCompressedCacheCounters(const Options& options,
size_t expected_misses,
size_t expected_hits,
size_t expected_inserts,
size_t expected_failures) {
size_t new_miss_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS);
size_t new_hit_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT);
size_t new_insert_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD);
size_t new_failure_count =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD_FAILURES);
ASSERT_EQ(compressed_miss_count_ + expected_misses, new_miss_count);
ASSERT_EQ(compressed_hit_count_ + expected_hits, new_hit_count);
ASSERT_EQ(compressed_insert_count_ + expected_inserts, new_insert_count);
ASSERT_EQ(compressed_failure_count_ + expected_failures, new_failure_count);
compressed_miss_count_ = new_miss_count;
compressed_hit_count_ = new_hit_count;
compressed_insert_count_ = new_insert_count;
compressed_failure_count_ = new_failure_count;
}
#ifndef ROCKSDB_LITE
const std::array<size_t, kNumCacheEntryRoles> GetCacheEntryRoleCountsBg() {
// Verify in cache entry role stats
std::array<size_t, kNumCacheEntryRoles> cache_entry_role_counts;
@@ -181,7 +145,6 @@ class DBBlockCacheTest : public DBTestBase {
}
return cache_entry_role_counts;
}
#endif // ROCKSDB_LITE
};
TEST_F(DBBlockCacheTest, IteratorBlockCacheUsage) {
@@ -274,84 +237,6 @@ TEST_F(DBBlockCacheTest, TestWithoutCompressedBlockCache) {
}
#ifdef SNAPPY
TEST_F(DBBlockCacheTest, TestWithCompressedBlockCache) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
table_options.no_block_cache = true;
table_options.block_cache_compressed = nullptr;
table_options.block_size = 1;
table_options.filter_policy.reset(NewBloomFilterPolicy(20));
table_options.cache_index_and_filter_blocks = false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.compression = CompressionType::kSnappyCompression;
DestroyAndReopen(options);
std::string value(kValueSize, 'a');
for (size_t i = 0; i < kNumBlocks; i++) {
ASSERT_OK(Put(std::to_string(i), value));
ASSERT_OK(Flush());
}
ReadOptions read_options;
std::shared_ptr<Cache> compressed_cache = NewLRUCache(1 << 25, 0, false);
LRUCacheOptions co;
co.capacity = 0;
co.num_shard_bits = 0;
co.strict_capacity_limit = false;
// Needed not to count entry stats collector
co.metadata_charge_policy = kDontChargeCacheMetadata;
std::shared_ptr<Cache> cache = NewLRUCache(co);
table_options.block_cache = cache;
table_options.no_block_cache = false;
table_options.block_cache_compressed = compressed_cache;
table_options.max_auto_readahead_size = 0;
table_options.cache_index_and_filter_blocks = false;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
Reopen(options);
RecordCacheCounters(options);
// Load blocks into cache.
for (size_t i = 0; i < kNumBlocks - 1; i++) {
ASSERT_EQ(value, Get(std::to_string(i)));
CheckCacheCounters(options, 1, 0, 1, 0);
CheckCompressedCacheCounters(options, 1, 0, 1, 0);
}
size_t usage = cache->GetUsage();
ASSERT_EQ(0, usage);
ASSERT_EQ(usage, cache->GetPinnedUsage());
size_t compressed_usage = compressed_cache->GetUsage();
ASSERT_LT(0, compressed_usage);
// Compressed block cache cannot be pinned.
ASSERT_EQ(0, compressed_cache->GetPinnedUsage());
// Set strict capacity limit flag. Now block will only load into compressed
// block cache.
cache->SetCapacity(usage);
cache->SetStrictCapacityLimit(true);
ASSERT_EQ(usage, cache->GetPinnedUsage());
// Load last key block.
ASSERT_EQ(
"Operation aborted: Memory limit reached: Insert failed due to LRU cache "
"being full.",
Get(std::to_string(kNumBlocks - 1)));
// Failure will also record the miss counter.
CheckCacheCounters(options, 1, 0, 0, 1);
CheckCompressedCacheCounters(options, 1, 0, 1, 0);
// Clear strict capacity limit flag. This time we shall hit compressed block
// cache and load into block cache.
cache->SetStrictCapacityLimit(false);
// Load last key block.
ASSERT_EQ(value, Get(std::to_string(kNumBlocks - 1)));
CheckCacheCounters(options, 1, 0, 1, 0);
CheckCompressedCacheCounters(options, 0, 1, 0, 0);
}
namespace {
class PersistentCacheFromCache : public PersistentCache {
@@ -402,9 +287,11 @@ class PersistentCacheFromCache : public PersistentCache {
};
class ReadOnlyCacheWrapper : public CacheWrapper {
public:
using CacheWrapper::CacheWrapper;
using Cache::Insert;
const char* Name() const override { return "ReadOnlyCacheWrapper"; }
Status Insert(const Slice& /*key*/, Cache::ObjectPtr /*value*/,
const CacheItemHelper* /*helper*/, size_t /*charge*/,
Handle** /*handle*/, Priority /*priority*/) override {
@@ -413,83 +300,8 @@ class ReadOnlyCacheWrapper : public CacheWrapper {
};
} // anonymous namespace
TEST_F(DBBlockCacheTest, TestWithSameCompressed) {
auto table_options = GetTableOptions();
auto options = GetOptions(table_options);
InitTable(options);
std::shared_ptr<Cache> rw_cache{NewLRUCache(1000000)};
std::shared_ptr<PersistentCacheFromCache> rw_pcache{
new PersistentCacheFromCache(rw_cache, /*read_only*/ false)};
// Exercise some obscure behavior with read-only wrappers
std::shared_ptr<Cache> ro_cache{new ReadOnlyCacheWrapper(rw_cache)};
std::shared_ptr<PersistentCacheFromCache> ro_pcache{
new PersistentCacheFromCache(rw_cache, /*read_only*/ true)};
// Simple same pointer
table_options.block_cache = rw_cache;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache same as block_cache_compressed not "
"currently supported, and would be bad for performance anyway");
// Other cases
table_options.block_cache = ro_cache;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache and block_cache_compressed share "
"the same key space, which is not supported");
table_options.block_cache = rw_cache;
table_options.block_cache_compressed = ro_cache;
table_options.persistent_cache.reset();
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache_compressed and block_cache share "
"the same key space, which is not supported");
table_options.block_cache = ro_cache;
table_options.block_cache_compressed.reset();
table_options.persistent_cache = rw_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache and persistent_cache share the same "
"key space, which is not supported");
table_options.block_cache = rw_cache;
table_options.block_cache_compressed.reset();
table_options.persistent_cache = ro_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: persistent_cache and block_cache share the same "
"key space, which is not supported");
table_options.block_cache.reset();
table_options.no_block_cache = true;
table_options.block_cache_compressed = ro_cache;
table_options.persistent_cache = rw_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: block_cache_compressed and persistent_cache "
"share the same key space, which is not supported");
table_options.block_cache.reset();
table_options.no_block_cache = true;
table_options.block_cache_compressed = rw_cache;
table_options.persistent_cache = ro_pcache;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ASSERT_EQ(TryReopen(options).ToString(),
"Invalid argument: persistent_cache and block_cache_compressed "
"share the same key space, which is not supported");
}
#endif // SNAPPY
#ifndef ROCKSDB_LITE
// Make sure that when options.block_cache is set, after a new table is
// created its index/filter blocks are added to block cache.
@@ -609,11 +421,6 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
ASSERT_EQ(cache->GetUsage(), index_bytes_insert + filter_bytes_insert);
// set the cache capacity to the current usage
cache->SetCapacity(index_bytes_insert + filter_bytes_insert);
// The index and filter eviction statistics were broken by the refactoring
// that moved the readers out of the block cache. Disabling these until we can
// bring the stats back.
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT), 0);
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT), 0);
// Note that the second key needs to be no longer than the first one.
// Otherwise the second index block may not fit in cache.
ASSERT_OK(Put(1, "key", "val"));
@@ -624,13 +431,6 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
index_bytes_insert);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_INSERT),
filter_bytes_insert);
// The index and filter eviction statistics were broken by the refactoring
// that moved the readers out of the block cache. Disabling these until we can
// bring the stats back.
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT),
// index_bytes_insert);
// ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT),
// filter_bytes_insert);
}
#if (defined OS_LINUX || defined OS_WIN)
@@ -913,7 +713,8 @@ class LookupLiarCache : public CacheWrapper {
explicit LookupLiarCache(std::shared_ptr<Cache> target)
: CacheWrapper(std::move(target)) {}
using Cache::Lookup;
const char* Name() const override { return "LookupLiarCache"; }
Handle* Lookup(const Slice& key, const CacheItemHelper* helper = nullptr,
CreateContext* create_context = nullptr,
Priority priority = Priority::LOW, bool wait = true,
@@ -1086,124 +887,6 @@ TEST_F(DBBlockCacheTest, ParanoidFileChecks) {
TestGetTickerCount(options, BLOCK_CACHE_ADD));
}
TEST_F(DBBlockCacheTest, CompressedCache) {
if (!Snappy_Supported()) {
return;
}
int num_iter = 80;
// Run this test three iterations.
// Iteration 1: only a uncompressed block cache
// Iteration 2: only a compressed block cache
// Iteration 3: both block cache and compressed cache
// Iteration 4: both block cache and compressed cache, but DB is not
// compressed
for (int iter = 0; iter < 4; iter++) {
Options options = CurrentOptions();
options.write_buffer_size = 64 * 1024; // small write buffer
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
BlockBasedTableOptions table_options;
switch (iter) {
case 0:
// only uncompressed block cache
table_options.block_cache = NewLRUCache(8 * 1024);
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 1:
// no block cache, only compressed cache
table_options.no_block_cache = true;
table_options.block_cache = nullptr;
table_options.block_cache_compressed = NewLRUCache(8 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 2:
// both compressed and uncompressed block cache
table_options.block_cache = NewLRUCache(1024);
table_options.block_cache_compressed = NewLRUCache(8 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
break;
case 3:
// both block cache and compressed cache, but DB is not compressed
// also, make block cache sizes bigger, to trigger block cache hits
table_options.block_cache = NewLRUCache(1024 * 1024);
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.compression = kNoCompression;
break;
default:
FAIL();
}
CreateAndReopenWithCF({"pikachu"}, options);
// default column family doesn't have block cache
Options no_block_cache_opts;
no_block_cache_opts.statistics = options.statistics;
no_block_cache_opts = CurrentOptions(no_block_cache_opts);
BlockBasedTableOptions table_options_no_bc;
table_options_no_bc.no_block_cache = true;
no_block_cache_opts.table_factory.reset(
NewBlockBasedTableFactory(table_options_no_bc));
ReopenWithColumnFamilies(
{"default", "pikachu"},
std::vector<Options>({no_block_cache_opts, options}));
Random rnd(301);
// Write 8MB (80 values, each 100K)
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 0);
std::vector<std::string> values;
std::string str;
for (int i = 0; i < num_iter; i++) {
if (i % 4 == 0) { // high compression ratio
str = rnd.RandomString(1000);
}
values.push_back(str);
ASSERT_OK(Put(1, Key(i), values[i]));
}
// flush all data from memtable so that reads are from block cache
ASSERT_OK(Flush(1));
for (int i = 0; i < num_iter; i++) {
ASSERT_EQ(Get(1, Key(i)), values[i]);
}
// check that we triggered the appropriate code paths in the cache
switch (iter) {
case 0:
// only uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 1:
// no block cache, only compressed cache
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 2:
// both compressed and uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
break;
case 3:
// both compressed and uncompressed block cache
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_MISS), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_HIT), 0);
ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_MISS), 0);
// compressed doesn't have any hits since blocks are not compressed on
// storage
ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_HIT), 0);
break;
default:
FAIL();
}
options.create_if_missing = true;
DestroyAndReopen(options);
}
}
TEST_F(DBBlockCacheTest, CacheCompressionDict) {
const int kNumFiles = 4;
const int kNumEntriesPerFile = 128;
@@ -1652,7 +1335,6 @@ TEST_F(DBBlockCacheTest, HyperClockCacheReportProblems) {
EXPECT_EQ(logger->PopCounts(), (std::array<int, 3>{{0, 1, 0}}));
}
#endif // ROCKSDB_LITE
class DBBlockCacheKeyTest
: public DBTestBase,
@@ -1708,31 +1390,16 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
uint64_t expected_stat = 0;
std::function<void()> verify_stats;
if (use_compressed_cache_) {
if (!Snappy_Supported()) {
ROCKSDB_GTEST_SKIP("Compressed cache test requires snappy support");
return;
}
options.compression = CompressionType::kSnappyCompression;
table_options.no_block_cache = true;
table_options.block_cache_compressed = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
// One for ordinary SST file and one for external SST file
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_ADD));
};
} else {
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_DATA_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_INDEX_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_FILTER_ADD));
};
}
table_options.cache_index_and_filter_blocks = true;
table_options.block_cache = NewLRUCache(1 << 25, 0, false);
verify_stats = [&options, &expected_stat] {
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_DATA_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_INDEX_ADD));
ASSERT_EQ(expected_stat,
options.statistics->getTickerCount(BLOCK_CACHE_FILTER_ADD));
};
table_options.filter_policy.reset(NewBloomFilterPolicy(10, false));
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@@ -1774,7 +1441,6 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
++key_count;
}
#ifndef ROCKSDB_LITE
// Save an export of those ordinary SST files for later
std::string export_files_dir = dbname_ + "/exported";
ExportImportFilesMetaData* metadata_ptr_ = nullptr;
@@ -1803,13 +1469,6 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
ASSERT_OK(db_->IngestExternalFile(handles_[1], {f}, ingest_opts));
}
if (exclude_file_numbers_) {
// FIXME(peterd): figure out where these extra ADDs are coming from
options.statistics->recordTick(BLOCK_CACHE_COMPRESSED_ADD,
uint64_t{0} - uint64_t{2});
}
#endif
perform_gets();
verify_stats();
@@ -1822,7 +1481,6 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
// Make sure we can cache hit even on a full copy of the DB. Using
// StableCacheKeyTestFS, Checkpoint will resort to full copy not hard link.
// (Checkpoint not available in LITE mode to test this.)
#ifndef ROCKSDB_LITE
auto db_copy_name = dbname_ + "-copy";
ASSERT_OK(Checkpoint::Create(db_, &checkpoint));
ASSERT_OK(checkpoint->CreateCheckpoint(db_copy_name));
@@ -1861,7 +1519,6 @@ TEST_P(DBBlockCacheKeyTest, StableCacheKeys) {
perform_gets();
verify_stats();
#endif // !ROCKSDB_LITE
Close();
Destroy(options);
-24
View File
@@ -245,14 +245,12 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloomCustomPrefixExtractor) {
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
// No bloom on extractor changed
#ifndef ROCKSDB_LITE
ASSERT_OK(db_->SetOptions({{"prefix_extractor", "capped:10"}}));
ASSERT_EQ("NOT_FOUND", Get("foobarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 3);
ASSERT_EQ(
3,
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
#endif // ROCKSDB_LITE
// No bloom on extractor changed, after re-open
options.prefix_extractor.reset(NewCappedPrefixTransform(10));
@@ -317,14 +315,12 @@ TEST_F(DBBloomFilterTest, GetFilterByPrefixBloom) {
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
// No bloom on extractor changed
#ifndef ROCKSDB_LITE
ASSERT_OK(db_->SetOptions({{"prefix_extractor", "capped:10"}}));
ASSERT_EQ("NOT_FOUND", Get("foobarbar"));
ASSERT_EQ(TestGetTickerCount(options, BLOOM_FILTER_USEFUL), 3);
ASSERT_EQ(
3,
(*(get_perf_context()->level_to_perf_context))[0].bloom_filter_useful);
#endif // ROCKSDB_LITE
get_perf_context()->Reset();
}
@@ -564,7 +560,6 @@ TEST_P(DBBloomFilterTestWithParam, BloomFilter) {
ASSERT_LE(reads, 3 * N / 100);
}
#ifndef ROCKSDB_LITE
// Sanity check some table properties
std::map<std::string, std::string> props;
ASSERT_TRUE(db_->GetMapProperty(
@@ -583,7 +578,6 @@ TEST_P(DBBloomFilterTestWithParam, BloomFilter) {
uint64_t num_filter_entries = ParseUint64(props["num_filter_entries"]);
EXPECT_EQ(num_filter_entries, nkeys);
#endif // ROCKSDB_LITE
env_->delay_sstable_sync_.store(false, std::memory_order_release);
Close();
@@ -649,10 +643,8 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
PutFn();
GetFn();
};
#ifndef ROCKSDB_LITE
std::map<std::string, std::string> props;
const auto& kAggTableProps = DB::Properties::kAggregatedTableProperties;
#endif // ROCKSDB_LITE
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -675,11 +667,9 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Test 2: use custom API to skip filters -> no filter constructed
// or read.
@@ -693,11 +683,9 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Control test: using an actual filter with 100% FP rate -> the filter
// is constructed and checked on read.
@@ -713,11 +701,9 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(
TestGetAndResetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE),
maxKey);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_NE(props["filter_size"], "0");
#endif // ROCKSDB_LITE
// Test 3 (options test): Able to read existing filters with longstanding
// generated options file entry `filter_policy=rocksdb.BuiltinBloomFilter`
@@ -743,11 +729,9 @@ TEST_P(DBBloomFilterTestWithParam, SkipFilterOnEssentiallyZeroBpk) {
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_POSITIVE), 0);
EXPECT_EQ(TestGetTickerCount(options, BLOOM_FILTER_FULL_TRUE_POSITIVE), 0);
#ifndef ROCKSDB_LITE
props.clear();
ASSERT_TRUE(db_->GetMapProperty(kAggTableProps, &props));
EXPECT_EQ(props["filter_size"], "0");
#endif // ROCKSDB_LITE
}
#if !defined(ROCKSDB_VALGRIND_RUN) || defined(ROCKSDB_FULL_VALGRIND_RUN)
@@ -1503,7 +1487,6 @@ TEST_P(DBFilterConstructionCorruptionTestWithParam, DetectCorruption) {
}
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
TEST_P(DBFilterConstructionCorruptionTestWithParam,
DynamicallyTurnOnAndOffDetectConstructCorruption) {
Options options = CurrentOptions();
@@ -1587,7 +1570,6 @@ TEST_P(DBFilterConstructionCorruptionTestWithParam,
db_->GetOptions().table_factory->GetOptions<BlockBasedTableOptions>();
EXPECT_FALSE(updated_table_options->detect_filter_construct_corruption);
}
#endif // ROCKSDB_LITE
namespace {
// NOTE: This class is referenced by HISTORY.md as a model for a wrapper
@@ -1756,7 +1738,6 @@ TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
EXPECT_LE(useful_count, maxKey * 0.91);
}
} else {
#ifndef ROCKSDB_LITE
// Also try external SST file
{
std::string file_path = dbname_ + "/external.sst";
@@ -1768,7 +1749,6 @@ TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
// Note: kCompactionStyleLevel is default, ignored if num_levels == -1
EXPECT_EQ(policy->DumpTestReport(),
"cf=abe,s=kCompactionStyleLevel,n=-1,l=-1,b=0,r=kMisc\n");
#endif
}
// Destroy
@@ -2036,9 +2016,7 @@ TEST_P(DBBloomFilterTestVaryPrefixAndFormatVer, PartitionedMultiGet) {
ASSERT_OK(Put(UKey(i), UKey(i)));
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ(TotalTableFiles(), 1);
#endif
constexpr uint32_t Q = 29;
// MultiGet In
@@ -2190,7 +2168,6 @@ INSTANTIATE_TEST_CASE_P(DBBloomFilterTestVaryPrefixAndFormatVer,
std::make_tuple(true, 3), std::make_tuple(true, 4),
std::make_tuple(true, 5)));
#ifndef ROCKSDB_LITE
namespace {
static const std::string kPlainTable = "test_PlainTableBloom";
} // anonymous namespace
@@ -3486,7 +3463,6 @@ TEST_F(DBBloomFilterTest, WeirdPrefixExtractorWithFilter3) {
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-6
View File
@@ -285,7 +285,6 @@ class ChangeFilterFactory : public CompactionFilterFactory {
const char* Name() const override { return "ChangeFilterFactory"; }
};
#ifndef ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilter) {
Options options = CurrentOptions();
options.max_open_files = -1;
@@ -469,7 +468,6 @@ TEST_F(DBTestCompactionFilter, CompactionFilterDeletesAll) {
delete itr;
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterFlush) {
// Tests a `CompactionFilterFactory` that filters when table file is created
@@ -655,7 +653,6 @@ TEST_F(DBTestCompactionFilter, CompactionFilterWithMergeOperator) {
ASSERT_EQ(newvalue, four);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterContextManual) {
KeepFilterFactory* filter = new KeepFilterFactory(true, true);
@@ -715,7 +712,6 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextManual) {
ASSERT_EQ(count, 0);
}
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, CompactionFilterContextCfId) {
KeepFilterFactory* filter = new KeepFilterFactory(false, true);
@@ -746,7 +742,6 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextCfId) {
ASSERT_TRUE(filter->compaction_filter_created());
}
#ifndef ROCKSDB_LITE
// Compaction filters aplies to all records, regardless snapshots.
TEST_F(DBTestCompactionFilter, CompactionFilterIgnoreSnapshot) {
std::string five = std::to_string(5);
@@ -807,7 +802,6 @@ TEST_F(DBTestCompactionFilter, CompactionFilterIgnoreSnapshot) {
// removed.
db_->ReleaseSnapshot(snapshot);
}
#endif // ROCKSDB_LITE
TEST_F(DBTestCompactionFilter, SkipUntil) {
Options options = CurrentOptions();
+53 -8
View File
@@ -30,7 +30,6 @@
namespace ROCKSDB_NAMESPACE {
// SYNC_POINT is not supported in released Windows mode.
#if !defined(ROCKSDB_LITE)
class CompactionStatsCollector : public EventListener {
public:
@@ -3585,6 +3584,59 @@ TEST_P(DBCompactionTestWithParam, FullCompactionInBottomPriThreadPool) {
Env::Default()->SetBackgroundThreads(0, Env::Priority::BOTTOM);
}
TEST_F(DBCompactionTest, CancelCompactionWaitingOnConflict) {
// This test verifies cancellation of a compaction waiting to be scheduled due
// to conflict with a running compaction.
//
// A `CompactRange()` in universal compacts all files, waiting for files to
// become available if they are locked for another compaction. This test
// triggers an automatic compaction that blocks a `CompactRange()`, and
// verifies that `DisableManualCompaction()` can successfully cancel the
// `CompactRange()` without waiting for the automatic compaction to finish.
const int kNumSortedRuns = 4;
Options options = CurrentOptions();
options.compaction_style = kCompactionStyleUniversal;
options.level0_file_num_compaction_trigger = kNumSortedRuns;
options.memtable_factory.reset(
test::NewSpecialSkipListFactory(KNumKeysByGenerateNewFile - 1));
Reopen(options);
test::SleepingBackgroundTask auto_compaction_sleeping_task;
// Block automatic compaction when it runs in the callback
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::Run():Start",
[&](void* /*arg*/) { auto_compaction_sleeping_task.DoSleep(); });
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
// Fill overlapping files in L0 to trigger an automatic compaction
Random rnd(301);
for (int i = 0; i < kNumSortedRuns; ++i) {
int key_idx = 0;
GenerateNewFile(&rnd, &key_idx, true /* nowait */);
}
auto_compaction_sleeping_task.WaitUntilSleeping();
// Make sure the manual compaction has seen the conflict before being canceled
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
{{"ColumnFamilyData::CompactRange:Return",
"DBCompactionTest::CancelCompactionWaitingOnConflict:"
"PreDisableManualCompaction"}});
auto manual_compaction_thread = port::Thread([this]() {
ASSERT_TRUE(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr)
.IsIncomplete());
});
// Cancel it. Thread should be joinable, i.e., manual compaction was unblocked
// despite finding a conflict with an automatic compaction that is still
// running
TEST_SYNC_POINT(
"DBCompactionTest::CancelCompactionWaitingOnConflict:"
"PreDisableManualCompaction");
db_->DisableManualCompaction();
manual_compaction_thread.join();
}
TEST_F(DBCompactionTest, OptimizedDeletionObsoleting) {
// Deletions can be dropped when compacted to non-last level if they fall
// outside the lower-level files' key-ranges.
@@ -9056,18 +9108,11 @@ TEST_F(DBCompactionTest, BottommostFileCompactionAllowIngestBehind) {
// ASSERT_OK(dbfull()->TEST_WaitForCompact(true /* wait_unscheduled */));
}
#endif // !defined(ROCKSDB_LITE)
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
-8
View File
@@ -10,7 +10,6 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "port/port.h"
@@ -492,16 +491,9 @@ TEST_F(DBTestDynamicLevel, DISABLED_MigrateToDynamicLevelMaxBytesBase) {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
-4
View File
@@ -6,9 +6,7 @@
#include "db/db_test_util.h"
#include "port/stack_trace.h"
#include "rocksdb/perf_context.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include <iostream>
#include <string>
@@ -27,7 +25,6 @@ class DBEncryptionTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBEncryptionTest, CheckEncrypted) {
ASSERT_OK(Put("foo567", "v1.fetdq"));
@@ -119,7 +116,6 @@ TEST_F(DBEncryptionTest, ReadEmptyFile) {
ASSERT_TRUE(data.empty());
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-2
View File
@@ -4,7 +4,6 @@
// (found in the LICENSE.Apache file in the root directory).
//
#ifndef ROCKSDB_LITE
#include <algorithm>
#include <cstdint>
@@ -439,4 +438,3 @@ Status DBImpl::GetLiveFilesStorageInfo(
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+62 -34
View File
@@ -73,9 +73,7 @@ TEST_F(DBFlushTest, FlushWhileWritingManifest) {
ASSERT_OK(dbfull()->Flush(no_wait));
// If the issue is hit we will wait here forever.
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
ASSERT_EQ(2, TotalTableFiles());
#endif // ROCKSDB_LITE
}
// Disable this test temporarily on Travis as it fails intermittently.
@@ -105,9 +103,7 @@ TEST_F(DBFlushTest, SyncFail) {
// Now the background job will do the flush; wait for it.
// Returns the IO error happend during flush.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
ASSERT_EQ("", FilesPerLevel()); // flush failed.
#endif // ROCKSDB_LITE
Destroy(options);
}
@@ -664,7 +660,6 @@ TEST_F(DBFlushTest, StatisticsGarbageRangeDeletes) {
Close();
}
#ifndef ROCKSDB_LITE
// This simple Listener can only handle one flush at a time.
class TestFlushListener : public EventListener {
public:
@@ -744,7 +739,62 @@ class TestFlushListener : public EventListener {
Env* env_;
DBFlushTest* test_;
};
#endif // !ROCKSDB_LITE
// RocksDB lite does not support GetLiveFiles()
TEST_F(DBFlushTest, FixFlushReasonRaceFromConcurrentFlushes) {
Options options = CurrentOptions();
options.atomic_flush = true;
options.disable_auto_compactions = true;
CreateAndReopenWithCF({"cf1"}, options);
for (int idx = 0; idx < 1; ++idx) {
ASSERT_OK(Put(0, Key(idx), std::string(1, 'v')));
ASSERT_OK(Put(1, Key(idx), std::string(1, 'v')));
}
// To coerce a manual flush happenning in the middle of GetLiveFiles's flush,
// we need to pause background flush thread and enable it later.
std::shared_ptr<test::SleepingBackgroundTask> sleeping_task =
std::make_shared<test::SleepingBackgroundTask>();
env_->SetBackgroundThreads(1, Env::HIGH);
env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask,
sleeping_task.get(), Env::Priority::HIGH);
sleeping_task->WaitUntilSleeping();
// Coerce a manual flush happenning in the middle of GetLiveFiles's flush
bool get_live_files_paused_at_sync_point = false;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"DBImpl::AtomicFlushMemTables:AfterScheduleFlush", [&](void* /* arg */) {
if (get_live_files_paused_at_sync_point) {
// To prevent non-GetLiveFiles() flush from pausing at this sync point
return;
}
get_live_files_paused_at_sync_point = true;
FlushOptions fo;
fo.wait = false;
fo.allow_write_stall = true;
ASSERT_OK(dbfull()->Flush(fo));
// Resume background flush thread so GetLiveFiles() can finish
sleeping_task->WakeUp();
sleeping_task->WaitUntilDone();
});
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
std::vector<std::string> files;
uint64_t manifest_file_size;
// Before the fix, a race condition on default cf's flush reason due to
// concurrent GetLiveFiles's flush and manual flush will fail
// an internal assertion.
// After the fix, such race condition is fixed and there is no assertion
// failure.
ASSERT_OK(db_->GetLiveFiles(files, &manifest_file_size, /*flush*/ true));
ASSERT_TRUE(get_live_files_paused_at_sync_point);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBFlushTest, MemPurgeBasic) {
Options options = CurrentOptions();
@@ -777,24 +827,16 @@ TEST_F(DBFlushTest, MemPurgeBasic) {
// Enforce size of a single MemTable to 64MB (64MB = 67108864 bytes).
options.write_buffer_size = 1 << 20;
#ifndef ROCKSDB_LITE
// Initially deactivate the MemPurge prototype.
options.experimental_mempurge_threshold = 0.0;
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#else
// Activate directly the MemPurge prototype.
// (RocksDB lite does not support dynamic options)
options.experimental_mempurge_threshold = 1.0;
#endif // !ROCKSDB_LITE
ASSERT_OK(TryReopen(options));
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
// Dynamically activate the MemPurge prototype without restarting the DB.
ColumnFamilyHandle* cfh = db_->DefaultColumnFamily();
ASSERT_OK(db_->SetOptions(cfh, {{"experimental_mempurge_threshold", "1.0"}}));
#endif
std::atomic<uint32_t> mempurge_count{0};
std::atomic<uint32_t> sst_count{0};
@@ -927,7 +969,6 @@ TEST_F(DBFlushTest, MemPurgeBasic) {
}
// RocksDB lite does not support dynamic options
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, MemPurgeBasicToggle) {
Options options = CurrentOptions();
@@ -1040,12 +1081,10 @@ TEST_F(DBFlushTest, MemPurgeBasicToggle) {
Close();
}
// Closes the "#ifndef ROCKSDB_LITE"
// End of MemPurgeBasicToggle, which is not
// supported with RocksDB LITE because it
// relies on dynamically changing the option
// flag experimental_mempurge_threshold.
#endif
// At the moment, MemPurge feature is deactivated
// when atomic_flush is enabled. This is because the level
@@ -1163,10 +1202,8 @@ TEST_F(DBFlushTest, MemPurgeDeleteAndDeleteRange) {
options.compression = kNoCompression;
options.inplace_update_support = false;
options.allow_concurrent_memtable_write = true;
#ifndef ROCKSDB_LITE
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#endif // !ROCKSDB_LITE
// Enforce size of a single MemTable to 64MB (64MB = 67108864 bytes).
options.write_buffer_size = 1 << 20;
// Activate the MemPurge prototype.
@@ -1364,10 +1401,8 @@ TEST_F(DBFlushTest, MemPurgeAndCompactionFilter) {
options.compression = kNoCompression;
options.inplace_update_support = false;
options.allow_concurrent_memtable_write = true;
#ifndef ROCKSDB_LITE
TestFlushListener* listener = new TestFlushListener(options.env, this);
options.listeners.emplace_back(listener);
#endif // !ROCKSDB_LITE
// Create a ConditionalUpdate compaction filter
// that will update all the values of the KV pairs
// where the keys are "lower" than KEY4.
@@ -1820,12 +1855,10 @@ TEST_F(DBFlushTest, ManualFlushFailsInReadOnlyMode) {
ASSERT_OK(db_->ContinueBackgroundWork());
// We ingested the error to env, so the returned status is not OK.
ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
#ifndef ROCKSDB_LITE
uint64_t num_bg_errors;
ASSERT_TRUE(
db_->GetIntProperty(DB::Properties::kBackgroundErrors, &num_bg_errors));
ASSERT_GT(num_bg_errors, 0);
#endif // ROCKSDB_LITE
// In the bug scenario, triggering another flush would cause the second flush
// to hang forever. After the fix we expect it to return an error.
@@ -1867,7 +1900,6 @@ TEST_F(DBFlushTest, CFDropRaceWithWaitForFlushMemTables) {
SyncPoint::GetInstance()->DisableProcessing();
}
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
class TestListener : public EventListener {
public:
@@ -1958,7 +1990,6 @@ TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // !ROCKSDB_LITE
TEST_F(DBFlushTest, FlushWithBlob) {
constexpr uint64_t min_blob_size = 10;
@@ -2020,7 +2051,6 @@ TEST_F(DBFlushTest, FlushWithBlob) {
ASSERT_EQ(blob_file->GetTotalBlobCount(), 1);
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
@@ -2036,7 +2066,6 @@ TEST_F(DBFlushTest, FlushWithBlob) {
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED],
compaction_stats[0].bytes_written +
compaction_stats[0].bytes_written_blob);
#endif // ROCKSDB_LITE
}
TEST_F(DBFlushTest, FlushWithChecksumHandoff1) {
@@ -2350,7 +2379,6 @@ TEST_P(DBFlushTestBlobError, FlushError) {
ASSERT_NE(type, kBlobFile);
}
#ifndef ROCKSDB_LITE
const InternalStats* const internal_stats = cfd->internal_stats();
assert(internal_stats);
@@ -2374,10 +2402,8 @@ TEST_P(DBFlushTestBlobError, FlushError) {
ASSERT_EQ(cf_stats_value[InternalStats::BYTES_FLUSHED],
compaction_stats[0].bytes_written +
compaction_stats[0].bytes_written_blob);
#endif // ROCKSDB_LITE
}
#ifndef ROCKSDB_LITE
TEST_F(DBFlushTest, TombstoneVisibleInSnapshot) {
class SimpleTestFlushListener : public EventListener {
public:
@@ -2440,7 +2466,9 @@ TEST_P(DBAtomicFlushTest, ManualFlushUnder2PC) {
options.atomic_flush = GetParam();
// 64MB so that memtable flush won't be trigger by the small writes.
options.write_buffer_size = (static_cast<size_t>(64) << 20);
auto flush_listener = std::make_shared<FlushCounterListener>();
flush_listener->expected_flush_reason = FlushReason::kManualFlush;
options.listeners.push_back(flush_listener);
// Destroy the DB to recreate as a TransactionDB.
Close();
Destroy(options, true);
@@ -2507,7 +2535,6 @@ TEST_P(DBAtomicFlushTest, ManualFlushUnder2PC) {
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
ASSERT_EQ(cfh->cfd()->GetFlushReason(), FlushReason::kManualFlush);
}
// The recovered min log number with prepared data should be non-zero.
@@ -2520,13 +2547,15 @@ TEST_P(DBAtomicFlushTest, ManualFlushUnder2PC) {
ASSERT_TRUE(db_impl->allow_2pc());
ASSERT_NE(db_impl->MinLogNumberToKeep(), 0);
}
#endif // ROCKSDB_LITE
TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.atomic_flush = GetParam();
options.write_buffer_size = (static_cast<size_t>(64) << 20);
auto flush_listener = std::make_shared<FlushCounterListener>();
flush_listener->expected_flush_reason = FlushReason::kManualFlush;
options.listeners.push_back(flush_listener);
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
size_t num_cfs = handles_.size();
@@ -2551,7 +2580,6 @@ TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
for (size_t i = 0; i != num_cfs; ++i) {
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
ASSERT_EQ(cfh->cfd()->GetFlushReason(), FlushReason::kManualFlush);
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
}
-2
View File
@@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "db/db_impl/compacted_db_impl.h"
#include "db/db_impl/db_impl.h"
@@ -254,4 +253,3 @@ Status CompactedDBImpl::Open(const Options& options, const std::string& dbname,
}
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
-4
View File
@@ -4,7 +4,6 @@
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -133,12 +132,10 @@ class CompactedDBImpl : public DBImpl {
// Share with DBImplReadOnly?
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
private:
friend class DB;
@@ -151,4 +148,3 @@ class CompactedDBImpl : public DBImpl {
LevelFilesBrief files_;
};
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
+151 -103
View File
@@ -213,17 +213,13 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
has_unpersisted_data_(false),
unable_to_release_oldest_log_(false),
num_running_ingest_file_(0),
#ifndef ROCKSDB_LITE
wal_manager_(immutable_db_options_, file_options_, io_tracer_,
seq_per_batch),
#endif // ROCKSDB_LITE
bg_work_paused_(0),
bg_compaction_paused_(0),
refitting_level_(false),
opened_successfully_(false),
#ifndef ROCKSDB_LITE
periodic_task_scheduler_(),
#endif // ROCKSDB_LITE
two_write_queues_(options.two_write_queues),
manual_wal_flush_(options.manual_wal_flush),
// last_sequencee_ is always maintained by the main queue that also writes
@@ -247,7 +243,8 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
atomic_flush_install_cv_(&mutex_),
blob_callback_(immutable_db_options_.sst_file_manager.get(), &mutex_,
&error_handler_, &event_logger_,
immutable_db_options_.listeners, dbname_) {
immutable_db_options_.listeners, dbname_),
lock_wal_count_(0) {
// !batch_per_trx_ implies seq_per_batch_ because it is only unset for
// WriteUnprepared, which should use seq_per_batch_.
assert(batch_per_txn_ || seq_per_batch_);
@@ -265,7 +262,6 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
SetDbSessionId();
assert(!db_session_id_.empty());
#ifndef ROCKSDB_LITE
periodic_task_functions_.emplace(PeriodicTaskType::kDumpStats,
[this]() { this->DumpStats(); });
periodic_task_functions_.emplace(PeriodicTaskType::kPersistStats,
@@ -275,7 +271,6 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
periodic_task_functions_.emplace(
PeriodicTaskType::kRecordSeqnoTime,
[this]() { this->RecordSeqnoToTimeMapping(); });
#endif // ROCKSDB_LITE
versions_.reset(new VersionSet(dbname_, &immutable_db_options_, file_options_,
table_cache_.get(), write_buffer_manager_,
@@ -496,7 +491,6 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Shutdown: canceling all background work");
#ifndef ROCKSDB_LITE
for (uint8_t task_type = 0;
task_type < static_cast<uint8_t>(PeriodicTaskType::kMax); task_type++) {
Status s = periodic_task_scheduler_.Unregister(
@@ -507,7 +501,6 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
task_type, s.ToString().c_str());
}
}
#endif // !ROCKSDB_LITE
InstrumentedMutexLock l(&mutex_);
if (!shutting_down_.load(std::memory_order_acquire) &&
@@ -604,7 +597,7 @@ Status DBImpl::CloseHelper() {
while (!flush_queue_.empty()) {
const FlushRequest& flush_req = PopFirstFromFlushQueue();
for (const auto& iter : flush_req) {
for (const auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
iter.first->UnrefAndTryDelete();
}
}
@@ -704,7 +697,6 @@ Status DBImpl::CloseHelper() {
ROCKS_LOG_INFO(immutable_db_options_.info_log, "Shutdown complete");
LogFlush(immutable_db_options_.info_log);
#ifndef ROCKSDB_LITE
// If the sst_file_manager was allocated by us during DB::Open(), ccall
// Close() on it before closing the info_log. Otherwise, background thread
// in SstFileManagerImpl might try to log something
@@ -713,7 +705,6 @@ Status DBImpl::CloseHelper() {
immutable_db_options_.sst_file_manager.get());
sfm->Close();
}
#endif // ROCKSDB_LITE
if (immutable_db_options_.info_log && own_info_log_) {
Status s = immutable_db_options_.info_log->Close();
@@ -791,7 +782,6 @@ void DBImpl::PrintStatistics() {
}
Status DBImpl::StartPeriodicTaskScheduler() {
#ifndef ROCKSDB_LITE
#ifndef NDEBUG
// It only used by test to disable scheduler
@@ -834,13 +824,9 @@ Status DBImpl::StartPeriodicTaskScheduler() {
periodic_task_functions_.at(PeriodicTaskType::kFlushInfoLog));
return s;
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
Status DBImpl::RegisterRecordSeqnoTimeWorker() {
#ifndef ROCKSDB_LITE
uint64_t min_time_duration = std::numeric_limits<uint64_t>::max();
uint64_t max_time_duration = std::numeric_limits<uint64_t>::min();
{
@@ -883,9 +869,6 @@ Status DBImpl::RegisterRecordSeqnoTimeWorker() {
}
return s;
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
// esitmate the total size of stats_history_
@@ -906,7 +889,6 @@ size_t DBImpl::EstimateInMemoryStatsHistorySize() const {
void DBImpl::PersistStats() {
TEST_SYNC_POINT("DBImpl::PersistStats:Entry");
#ifndef ROCKSDB_LITE
if (shutdown_initiated_) {
return;
}
@@ -1011,7 +993,6 @@ void DBImpl::PersistStats() {
stats_history_size, stats_history_.size());
}
TEST_SYNC_POINT("DBImpl::PersistStats:End");
#endif // !ROCKSDB_LITE
}
bool DBImpl::FindStatsByTime(uint64_t start_time, uint64_t end_time,
@@ -1053,7 +1034,6 @@ Status DBImpl::GetStatsHistory(
void DBImpl::DumpStats() {
TEST_SYNC_POINT("DBImpl::DumpStats:1");
#ifndef ROCKSDB_LITE
std::string stats;
if (shutdown_initiated_) {
return;
@@ -1118,7 +1098,6 @@ void DBImpl::DumpStats() {
ROCKS_LOG_INFO(immutable_db_options_.info_log, "%s", stats.c_str());
}
}
#endif // !ROCKSDB_LITE
PrintStatistics();
}
@@ -1173,11 +1152,6 @@ FSDirectory* DBImpl::GetDataDir(ColumnFamilyData* cfd, size_t path_id) const {
Status DBImpl::SetOptions(
ColumnFamilyHandle* column_family,
const std::unordered_map<std::string, std::string>& options_map) {
#ifdef ROCKSDB_LITE
(void)column_family;
(void)options_map;
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
auto* cfd =
static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
if (options_map.empty()) {
@@ -1235,15 +1209,10 @@ Status DBImpl::SetOptions(
}
LogFlush(immutable_db_options_.info_log);
return s;
#endif // ROCKSDB_LITE
}
Status DBImpl::SetDBOptions(
const std::unordered_map<std::string, std::string>& options_map) {
#ifdef ROCKSDB_LITE
(void)options_map;
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
if (options_map.empty()) {
ROCKS_LOG_WARN(immutable_db_options_.info_log,
"SetDBOptions(), empty input.");
@@ -1408,7 +1377,6 @@ Status DBImpl::SetDBOptions(
}
LogFlush(immutable_db_options_.info_log);
return s;
#endif // ROCKSDB_LITE
}
// return the same level if it cannot be moved
@@ -1462,15 +1430,10 @@ Status DBImpl::FlushWAL(bool sync) {
return SyncWAL();
}
bool DBImpl::WALBufferIsEmpty(bool lock) {
if (lock) {
log_write_mutex_.Lock();
}
bool DBImpl::WALBufferIsEmpty() {
InstrumentedMutexLock l(&log_write_mutex_);
log::Writer* cur_log_writer = logs_.back().writer;
auto res = cur_log_writer->BufferIsEmpty();
if (lock) {
log_write_mutex_.Unlock();
}
return res;
}
@@ -1572,29 +1535,83 @@ Status DBImpl::ApplyWALToManifest(VersionEdit* synced_wals) {
Status DBImpl::LockWAL() {
{
InstrumentedMutexLock lock(&mutex_);
WriteThread::Writer w;
write_thread_.EnterUnbatched(&w, &mutex_);
WriteThread::Writer nonmem_w;
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
if (lock_wal_count_ > 0) {
assert(lock_wal_write_token_);
++lock_wal_count_;
} else {
// NOTE: this will "unnecessarily" wait for other non-LockWAL() write
// stalls to clear before LockWAL returns, however fixing that would
// not be simple because if we notice the primary queue is already
// stalled, that stall might clear while we release DB mutex in
// EnterUnbatched() for the nonmem queue. And if we work around that in
// the naive way, we could deadlock by locking the two queues in different
// orders.
lock_wal_write_token_ = write_controller_.GetStopToken();
WriteThread::Writer w;
write_thread_.EnterUnbatched(&w, &mutex_);
WriteThread::Writer nonmem_w;
if (two_write_queues_) {
nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
}
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
// NOTE: releasing mutex in EnterUnbatched might mean we are actually
// now lock_wal_count > 0
if (lock_wal_count_ == 0) {
assert(!lock_wal_write_token_);
lock_wal_write_token_ = write_controller_.GetStopToken();
}
++lock_wal_count_;
if (two_write_queues_) {
nonmem_write_thread_.ExitUnbatched(&nonmem_w);
}
write_thread_.ExitUnbatched(&w);
}
write_thread_.ExitUnbatched(&w);
}
return FlushWAL(/*sync=*/false);
// NOTE: avoid I/O holding DB mutex
Status s = FlushWAL(/*sync=*/false);
if (!s.ok()) {
// Non-OK return should not be in locked state
UnlockWAL().PermitUncheckedError();
}
return s;
}
Status DBImpl::UnlockWAL() {
bool signal = false;
uint64_t maybe_stall_begun_count = 0;
uint64_t nonmem_maybe_stall_begun_count = 0;
{
InstrumentedMutexLock lock(&mutex_);
lock_wal_write_token_.reset();
if (lock_wal_count_ == 0) {
return Status::Aborted("No LockWAL() in effect");
}
--lock_wal_count_;
if (lock_wal_count_ == 0) {
lock_wal_write_token_.reset();
signal = true;
// For the last UnlockWAL, we don't want to return from UnlockWAL()
// until the thread(s) that called BeginWriteStall() have had a chance to
// call EndWriteStall(), so that no_slowdown writes after UnlockWAL() are
// guaranteed to succeed if there's no other source of stall.
maybe_stall_begun_count = write_thread_.GetBegunCountOfOutstandingStall();
if (two_write_queues_) {
nonmem_maybe_stall_begun_count =
nonmem_write_thread_.GetBegunCountOfOutstandingStall();
}
}
}
if (signal) {
// SignalAll outside of mutex for efficiency
bg_cv_.SignalAll();
}
// Ensure stalls have cleared
if (maybe_stall_begun_count) {
write_thread_.WaitForStallEndedCount(maybe_stall_begun_count);
}
if (nonmem_maybe_stall_begun_count) {
nonmem_write_thread_.WaitForStallEndedCount(nonmem_maybe_stall_begun_count);
}
bg_cv_.SignalAll();
return Status::OK();
}
@@ -2588,14 +2605,25 @@ void DBImpl::MultiGet(const ReadOptions& read_options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, Status* statuses,
const bool sorted_input) {
return MultiGet(read_options, num_keys, column_families, keys, values,
/*timestamps=*/nullptr, statuses, sorted_input);
MultiGet(read_options, num_keys, column_families, keys, values,
/* timestamps */ nullptr, statuses, sorted_input);
}
void DBImpl::MultiGet(const ReadOptions& read_options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, std::string* timestamps,
Status* statuses, const bool sorted_input) {
MultiGetCommon(read_options, num_keys, column_families, keys, values,
/* columns */ nullptr, timestamps, statuses, sorted_input);
}
void DBImpl::MultiGetCommon(const ReadOptions& read_options,
const size_t num_keys,
ColumnFamilyHandle** column_families,
const Slice* keys, PinnableSlice* values,
PinnableWideColumns* columns,
std::string* timestamps, Status* statuses,
const bool sorted_input) {
if (num_keys == 0) {
return;
}
@@ -2641,8 +2669,20 @@ void DBImpl::MultiGet(const ReadOptions& read_options, const size_t num_keys,
autovector<KeyContext*, MultiGetContext::MAX_BATCH_SIZE> sorted_keys;
sorted_keys.resize(num_keys);
for (size_t i = 0; i < num_keys; ++i) {
values[i].Reset();
key_context.emplace_back(column_families[i], keys[i], &values[i],
PinnableSlice* val = nullptr;
PinnableWideColumns* col = nullptr;
if (values) {
val = &values[i];
val->Reset();
} else {
assert(columns);
col = &columns[i];
col->Reset();
}
key_context.emplace_back(column_families[i], keys[i], val, col,
timestamps ? &timestamps[i] : nullptr,
&statuses[i]);
}
@@ -2766,8 +2806,8 @@ void DBImpl::MultiGet(const ReadOptions& read_options,
ColumnFamilyHandle* column_family, const size_t num_keys,
const Slice* keys, PinnableSlice* values,
Status* statuses, const bool sorted_input) {
return MultiGet(read_options, column_family, num_keys, keys, values,
/*timestamp=*/nullptr, statuses, sorted_input);
MultiGet(read_options, column_family, num_keys, keys, values,
/* timestamps */ nullptr, statuses, sorted_input);
}
void DBImpl::MultiGet(const ReadOptions& read_options,
@@ -2775,6 +2815,16 @@ void DBImpl::MultiGet(const ReadOptions& read_options,
const Slice* keys, PinnableSlice* values,
std::string* timestamps, Status* statuses,
const bool sorted_input) {
MultiGetCommon(read_options, column_family, num_keys, keys, values,
/* columns */ nullptr, timestamps, statuses, sorted_input);
}
void DBImpl::MultiGetCommon(const ReadOptions& read_options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, PinnableWideColumns* columns,
std::string* timestamps, Status* statuses,
bool sorted_input) {
if (tracer_) {
// TODO: This mutex should be removed later, to improve performance when
// tracing is enabled.
@@ -2788,8 +2838,20 @@ void DBImpl::MultiGet(const ReadOptions& read_options,
autovector<KeyContext*, MultiGetContext::MAX_BATCH_SIZE> sorted_keys;
sorted_keys.resize(num_keys);
for (size_t i = 0; i < num_keys; ++i) {
values[i].Reset();
key_context.emplace_back(column_family, keys[i], &values[i],
PinnableSlice* val = nullptr;
PinnableWideColumns* col = nullptr;
if (values) {
val = &values[i];
val->Reset();
} else {
assert(columns);
col = &columns[i];
col->Reset();
}
key_context.emplace_back(column_family, keys[i], val, col,
timestamps ? &timestamps[i] : nullptr,
&statuses[i]);
}
@@ -2951,8 +3013,17 @@ Status DBImpl::MultiGetImpl(
uint64_t bytes_read = 0;
for (size_t i = start_key; i < start_key + num_keys - keys_left; ++i) {
KeyContext* key = (*sorted_keys)[i];
assert(key);
assert(key->s);
if (key->s->ok()) {
bytes_read += key->value->size();
if (key->value) {
bytes_read += key->value->size();
} else {
assert(key->columns);
bytes_read += key->columns->serialized_size();
}
num_found++;
}
}
@@ -2976,6 +3047,22 @@ Status DBImpl::MultiGetImpl(
return s;
}
void DBImpl::MultiGetEntity(const ReadOptions& options, size_t num_keys,
ColumnFamilyHandle** column_families,
const Slice* keys, PinnableWideColumns* results,
Status* statuses, bool sorted_input) {
MultiGetCommon(options, num_keys, column_families, keys, /* values */ nullptr,
results, /* timestamps */ nullptr, statuses, sorted_input);
}
void DBImpl::MultiGetEntity(const ReadOptions& options,
ColumnFamilyHandle* column_family, size_t num_keys,
const Slice* keys, PinnableWideColumns* results,
Status* statuses, bool sorted_input) {
MultiGetCommon(options, column_family, num_keys, keys, /* values */ nullptr,
results, /* timestamps */ nullptr, statuses, sorted_input);
}
Status DBImpl::CreateColumnFamily(const ColumnFamilyOptions& cf_options,
const std::string& column_family,
ColumnFamilyHandle** handle) {
@@ -3295,11 +3382,6 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
assert(cfd != nullptr);
ReadCallback* read_callback = nullptr; // No read callback provided.
if (read_options.tailing) {
#ifdef ROCKSDB_LITE
// not supported in lite version
result = nullptr;
#else
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
auto iter = new ForwardIterator(this, read_options, cfd, sv,
/* allow_unprepared_value */ true);
@@ -3308,7 +3390,6 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
cfd->user_comparator(), iter, sv->current, kMaxSequenceNumber,
sv->mutable_cf_options.max_sequential_skip_in_iterations, read_callback,
this, cfd);
#endif
} else {
// Note: no need to consider the special case of
// last_seq_same_as_publish_seq_==false since NewIterator is overridden in
@@ -3439,10 +3520,6 @@ Status DBImpl::NewIterators(
iterators->clear();
iterators->reserve(column_families.size());
if (read_options.tailing) {
#ifdef ROCKSDB_LITE
return Status::InvalidArgument(
"Tailing iterator not supported in RocksDB lite");
#else
for (auto cfh : column_families) {
auto cfd = static_cast_with_check<ColumnFamilyHandleImpl>(cfh)->cfd();
SuperVersion* sv = cfd->GetReferencedSuperVersion(this);
@@ -3454,7 +3531,6 @@ Status DBImpl::NewIterators(
sv->mutable_cf_options.max_sequential_skip_in_iterations,
read_callback, this, cfd));
}
#endif
} else {
// Note: no need to consider the special case of
// last_seq_same_as_publish_seq_==false since NewIterators is overridden in
@@ -3476,11 +3552,9 @@ Status DBImpl::NewIterators(
const Snapshot* DBImpl::GetSnapshot() { return GetSnapshotImpl(false); }
#ifndef ROCKSDB_LITE
const Snapshot* DBImpl::GetSnapshotForWriteConflictBoundary() {
return GetSnapshotImpl(true);
}
#endif // ROCKSDB_LITE
std::pair<Status, std::shared_ptr<const Snapshot>>
DBImpl::CreateTimestampedSnapshot(SequenceNumber snapshot_seq, uint64_t ts) {
@@ -3728,7 +3802,6 @@ void DBImpl::ReleaseSnapshot(const Snapshot* s) {
delete casted_s;
}
#ifndef ROCKSDB_LITE
Status DBImpl::GetPropertiesOfAllTables(ColumnFamilyHandle* column_family,
TablePropertiesCollection* props) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
@@ -3772,7 +3845,6 @@ Status DBImpl::GetPropertiesOfTablesInRange(ColumnFamilyHandle* column_family,
return s;
}
#endif // ROCKSDB_LITE
const std::string& DBImpl::GetName() const { return dbname_; }
@@ -3791,7 +3863,6 @@ SystemClock* DBImpl::GetSystemClock() const {
return immutable_db_options_.clock;
}
#ifndef ROCKSDB_LITE
Status DBImpl::StartIOTrace(const TraceOptions& trace_options,
std::unique_ptr<TraceWriter>&& trace_writer) {
@@ -3805,7 +3876,6 @@ Status DBImpl::EndIOTrace() {
return Status::OK();
}
#endif // ROCKSDB_LITE
Options DBImpl::GetOptions(ColumnFamilyHandle* column_family) const {
InstrumentedMutexLock l(&mutex_);
@@ -3934,7 +4004,6 @@ bool DBImpl::GetPropertyHandleOptionsStatistics(std::string* value) {
return true;
}
#ifndef ROCKSDB_LITE
Status DBImpl::ResetStats() {
InstrumentedMutexLock l(&mutex_);
for (auto* cfd : *versions_->GetColumnFamilySet()) {
@@ -3944,7 +4013,6 @@ Status DBImpl::ResetStats() {
}
return Status::OK();
}
#endif // ROCKSDB_LITE
bool DBImpl::GetAggregatedIntProperty(const Slice& property,
uint64_t* aggregated_value) {
@@ -4153,7 +4221,6 @@ void DBImpl::ReleaseFileNumberFromPendingOutputs(
}
}
#ifndef ROCKSDB_LITE
Status DBImpl::GetUpdatesSince(
SequenceNumber seq, std::unique_ptr<TransactionLogIterator>* iter,
const TransactionLogIterator::ReadOptions& read_options) {
@@ -4402,7 +4469,6 @@ void DBImpl::GetAllColumnFamilyMetaData(
}
}
#endif // ROCKSDB_LITE
Status DBImpl::CheckConsistency() {
mutex_.AssertHeld();
@@ -4753,7 +4819,6 @@ Status DestroyDB(const std::string& dbname, const Options& options,
Status DBImpl::WriteOptionsFile(bool need_mutex_lock,
bool need_enter_write_thread) {
#ifndef ROCKSDB_LITE
WriteThread::Writer w;
if (need_mutex_lock) {
mutex_.Lock();
@@ -4810,14 +4875,9 @@ Status DBImpl::WriteOptionsFile(bool need_mutex_lock,
s.ToString().c_str());
}
}
#else
(void)need_mutex_lock;
(void)need_enter_write_thread;
#endif // !ROCKSDB_LITE
return Status::OK();
}
#ifndef ROCKSDB_LITE
namespace {
void DeleteOptionsFilesHelper(const std::map<uint64_t, std::string>& filenames,
const size_t num_files_to_keep,
@@ -4835,10 +4895,8 @@ void DeleteOptionsFilesHelper(const std::map<uint64_t, std::string>& filenames,
}
}
} // namespace
#endif // !ROCKSDB_LITE
Status DBImpl::DeleteObsoleteOptionsFiles() {
#ifndef ROCKSDB_LITE
std::vector<std::string> filenames;
// use ordered map to store keep the filenames sorted from the newest
// to the oldest.
@@ -4866,13 +4924,9 @@ Status DBImpl::DeleteObsoleteOptionsFiles() {
DeleteOptionsFilesHelper(options_filenames, kNumOptionsFilesKept,
immutable_db_options_.info_log, GetEnv());
return Status::OK();
#else
return Status::OK();
#endif // !ROCKSDB_LITE
}
Status DBImpl::RenameTempFileToOptionsFile(const std::string& file_name) {
#ifndef ROCKSDB_LITE
Status s;
uint64_t options_file_number = versions_->NewFileNumber();
@@ -4916,10 +4970,6 @@ Status DBImpl::RenameTempFileToOptionsFile(const std::string& file_name) {
DeleteObsoleteOptionsFiles().PermitUncheckedError();
}
return s;
#else
(void)file_name;
return Status::OK();
#endif // !ROCKSDB_LITE
}
#ifdef ROCKSDB_USING_THREAD_STATUS
@@ -4967,7 +5017,6 @@ void DumpRocksDBBuildVersion(Logger* log) {
}
}
#ifndef ROCKSDB_LITE
SequenceNumber DBImpl::GetEarliestMemTableSequenceNumber(SuperVersion* sv,
bool include_history) {
// Find the earliest sequence number that we know we can rely on reading
@@ -5924,6 +5973,5 @@ void DBImpl::RecordSeqnoToTimeMapping() {
seqno, unix_time);
}
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
+77 -61
View File
@@ -16,6 +16,7 @@
#include <map>
#include <set>
#include <string>
#include <unordered_map>
#include <utility>
#include <vector>
@@ -53,13 +54,9 @@
#include "rocksdb/env.h"
#include "rocksdb/memtablerep.h"
#include "rocksdb/status.h"
#ifndef ROCKSDB_LITE
#include "rocksdb/trace_reader_writer.h"
#endif // ROCKSDB_LITE
#include "rocksdb/transaction_log.h"
#ifndef ROCKSDB_LITE
#include "rocksdb/utilities/replayer.h"
#endif // ROCKSDB_LITE
#include "rocksdb/write_buffer_manager.h"
#include "table/merging_iterator.h"
#include "table/scoped_arena_iterator.h"
@@ -280,33 +277,40 @@ class DBImpl : public DB {
// The values and statuses parameters are arrays with number of elements
// equal to keys.size(). This allows the storage for those to be alloacted
// by the caller on the stack for small batches
virtual void MultiGet(const ReadOptions& options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
const bool sorted_input = false) override;
virtual void MultiGet(const ReadOptions& options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, std::string* timestamps,
Status* statuses,
const bool sorted_input = false) override;
void MultiGet(const ReadOptions& options, ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys, PinnableSlice* values,
Status* statuses, const bool sorted_input = false) override;
void MultiGet(const ReadOptions& options, ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys, PinnableSlice* values,
std::string* timestamps, Status* statuses,
const bool sorted_input = false) override;
virtual void MultiGet(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, Status* statuses,
const bool sorted_input = false) override;
virtual void MultiGet(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, std::string* timestamps,
Status* statuses,
const bool sorted_input = false) override;
void MultiGet(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, Status* statuses,
const bool sorted_input = false) override;
void MultiGet(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, std::string* timestamps,
Status* statuses, const bool sorted_input = false) override;
virtual void MultiGetWithCallback(
void MultiGetWithCallback(
const ReadOptions& options, ColumnFamilyHandle* column_family,
ReadCallback* callback,
autovector<KeyContext*, MultiGetContext::MAX_BATCH_SIZE>* sorted_keys);
using DB::MultiGetEntity;
void MultiGetEntity(const ReadOptions& options,
ColumnFamilyHandle* column_family, size_t num_keys,
const Slice* keys, PinnableWideColumns* results,
Status* statuses, bool sorted_input) override;
void MultiGetEntity(const ReadOptions& options, size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableWideColumns* results, Status* statuses,
bool sorted_input) override;
virtual Status CreateColumnFamily(const ColumnFamilyOptions& cf_options,
const std::string& column_family,
ColumnFamilyHandle** handle) override;
@@ -429,7 +433,7 @@ class DBImpl : public DB {
const FlushOptions& options,
const std::vector<ColumnFamilyHandle*>& column_families) override;
virtual Status FlushWAL(bool sync) override;
bool WALBufferIsEmpty(bool lock = true);
bool WALBufferIsEmpty();
virtual Status SyncWAL() override;
virtual Status LockWAL() override;
virtual Status UnlockWAL() override;
@@ -468,7 +472,6 @@ class DBImpl : public DB {
uint64_t start_time, uint64_t end_time,
std::unique_ptr<StatsHistoryIterator>* stats_iterator) override;
#ifndef ROCKSDB_LITE
using DB::ResetStats;
virtual Status ResetStats() override;
// All the returned filenames start with "/"
@@ -596,7 +599,6 @@ class DBImpl : public DB {
ColumnFamilyHandle* column_family, const Range* range, std::size_t n,
TablePropertiesCollection* props) override;
#endif // ROCKSDB_LITE
// ---- End of implementations of the DB interface ----
SystemClock* GetSystemClock() const;
@@ -654,7 +656,6 @@ class DBImpl : public DB {
// depends also on data written to the WAL but not to the memtable.
SequenceNumber TEST_GetLastVisibleSequence() const;
#ifndef ROCKSDB_LITE
// Similar to Write() but will call the callback once on the single write
// thread to determine whether it is safe to perform the write.
virtual Status WriteWithCallback(const WriteOptions& write_options,
@@ -719,7 +720,6 @@ class DBImpl : public DB {
Status TraceIteratorSeekForPrev(const uint32_t& cf_id, const Slice& key,
const Slice& lower_bound,
const Slice upper_bound);
#endif // ROCKSDB_LITE
// Similar to GetSnapshot(), but also lets the db know that this snapshot
// will be used for transaction write-conflict checking. The DB can then
@@ -1124,6 +1124,8 @@ class DBImpl : public DB {
void TEST_UnlockMutex();
void TEST_SignalAllBgCv();
// REQUIRES: mutex locked
void* TEST_BeginWrite();
@@ -1175,9 +1177,7 @@ class DBImpl : public DB {
return files_grabbed_for_purge_;
}
#ifndef ROCKSDB_LITE
const PeriodicTaskScheduler& TEST_GetPeriodicTaskScheduler() const;
#endif // !ROCKSDB_LITE
#endif // NDEBUG
@@ -1383,7 +1383,7 @@ class DBImpl : public DB {
void NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
const MutableCFOptions& mutable_cf_options,
int job_id);
int job_id, FlushReason flush_reason);
void NotifyOnFlushCompleted(
ColumnFamilyData* cfd, const MutableCFOptions& mutable_cf_options,
@@ -1400,12 +1400,10 @@ class DBImpl : public DB {
void NotifyOnMemTableSealed(ColumnFamilyData* cfd,
const MemTableInfo& mem_table_info);
#ifndef ROCKSDB_LITE
void NotifyOnExternalFileIngested(
ColumnFamilyData* cfd, const ExternalSstFileIngestionJob& ingestion_job);
virtual Status FlushForGetLiveFiles();
#endif // !ROCKSDB_LITE
void NewThreadStatusCfInfo(ColumnFamilyData* cfd) const;
@@ -1553,9 +1551,7 @@ class DBImpl : public DB {
friend class WriteUnpreparedTxnDB;
friend class WriteUnpreparedTxn;
#ifndef ROCKSDB_LITE
friend class ForwardIterator;
#endif
friend struct SuperVersion;
friend class CompactedDBImpl;
friend class DBTest_ConcurrentFlushWAL_Test;
@@ -1675,12 +1671,17 @@ class DBImpl : public DB {
// Argument required by background flush thread.
struct BGFlushArg {
BGFlushArg()
: cfd_(nullptr), max_memtable_id_(0), superversion_context_(nullptr) {}
: cfd_(nullptr),
max_memtable_id_(0),
superversion_context_(nullptr),
flush_reason_(FlushReason::kOthers) {}
BGFlushArg(ColumnFamilyData* cfd, uint64_t max_memtable_id,
SuperVersionContext* superversion_context)
SuperVersionContext* superversion_context,
FlushReason flush_reason)
: cfd_(cfd),
max_memtable_id_(max_memtable_id),
superversion_context_(superversion_context) {}
superversion_context_(superversion_context),
flush_reason_(flush_reason) {}
// Column family to flush.
ColumnFamilyData* cfd_;
@@ -1691,6 +1692,7 @@ class DBImpl : public DB {
// installs a new superversion for the column family. This operation
// requires a SuperVersionContext object (currently embedded in JobContext).
SuperVersionContext* superversion_context_;
FlushReason flush_reason_;
};
// Argument passed to flush thread.
@@ -1819,7 +1821,7 @@ class DBImpl : public DB {
// installs a new super version for the column family.
Status FlushMemTableToOutputFile(
ColumnFamilyData* cfd, const MutableCFOptions& mutable_cf_options,
bool* madeProgress, JobContext* job_context,
bool* madeProgress, JobContext* job_context, FlushReason flush_reason,
SuperVersionContext* superversion_context,
std::vector<SequenceNumber>& snapshot_seqs,
SequenceNumber earliest_write_conflict_snapshot,
@@ -1865,7 +1867,8 @@ class DBImpl : public DB {
// num_bytes: for slowdown case, delay time is calculated based on
// `num_bytes` going through.
Status DelayWrite(uint64_t num_bytes, const WriteOptions& write_options);
Status DelayWrite(uint64_t num_bytes, WriteThread& write_thread,
const WriteOptions& write_options);
// Begin stalling of writes when memory usage increases beyond a certain
// threshold.
@@ -2015,7 +2018,6 @@ class DBImpl : public DB {
// Used by WriteImpl to update bg_error_ in case of memtable insert error.
void MemTableInsertStatusCheck(const Status& memtable_insert_status);
#ifndef ROCKSDB_LITE
Status CompactFilesImpl(const CompactionOptions& compact_options,
ColumnFamilyData* cfd, Version* version,
const std::vector<std::string>& input_file_names,
@@ -2023,24 +2025,27 @@ class DBImpl : public DB {
const int output_level, int output_path_id,
JobContext* job_context, LogBuffer* log_buffer,
CompactionJobInfo* compaction_job_info);
#endif // ROCKSDB_LITE
ColumnFamilyData* GetColumnFamilyDataByName(const std::string& cf_name);
void MaybeScheduleFlushOrCompaction();
// A flush request specifies the column families to flush as well as the
// largest memtable id to persist for each column family. Once all the
// memtables whose IDs are smaller than or equal to this per-column-family
// specified value, this flush request is considered to have completed its
// work of flushing this column family. After completing the work for all
// column families in this request, this flush is considered complete.
using FlushRequest = std::vector<std::pair<ColumnFamilyData*, uint64_t>>;
struct FlushRequest {
FlushReason flush_reason;
// A map from column family to flush to largest memtable id to persist for
// each column family. Once all the memtables whose IDs are smaller than or
// equal to this per-column-family specified value, this flush request is
// considered to have completed its work of flushing this column family.
// After completing the work for all column families in this request, this
// flush is considered complete.
std::unordered_map<ColumnFamilyData*, uint64_t>
cfd_to_max_mem_id_to_persist;
};
void GenerateFlushRequest(const autovector<ColumnFamilyData*>& cfds,
FlushRequest* req);
FlushReason flush_reason, FlushRequest* req);
void SchedulePendingFlush(const FlushRequest& req, FlushReason flush_reason);
void SchedulePendingFlush(const FlushRequest& req);
void SchedulePendingCompaction(ColumnFamilyData* cfd);
void SchedulePendingPurge(std::string fname, std::string dir_to_sync,
@@ -2158,7 +2163,6 @@ class DBImpl : public DB {
bool ShouldntRunManualCompaction(ManualCompactionState* m);
bool HaveManualCompaction(ColumnFamilyData* cfd);
bool MCOverlap(ManualCompactionState* m, ManualCompactionState* m1);
#ifndef ROCKSDB_LITE
void BuildCompactionJobInfo(const ColumnFamilyData* cfd, Compaction* c,
const Status& st,
const CompactionJobStats& compaction_job_stats,
@@ -2171,7 +2175,6 @@ class DBImpl : public DB {
ColumnFamilyData* cfd, uint64_t num,
std::unique_ptr<std::list<uint64_t>::iterator>& pending_output_elem,
uint64_t* next_file_number);
#endif //! ROCKSDB_LITE
bool ShouldPurge(uint64_t file_number) const;
void MarkAsGrabbedForPurge(uint64_t file_number);
@@ -2195,6 +2198,18 @@ class DBImpl : public DB {
const size_t num_keys, bool sorted,
autovector<KeyContext*, MultiGetContext::MAX_BATCH_SIZE>* key_ptrs);
void MultiGetCommon(const ReadOptions& options,
ColumnFamilyHandle* column_family, const size_t num_keys,
const Slice* keys, PinnableSlice* values,
PinnableWideColumns* columns, std::string* timestamps,
Status* statuses, bool sorted_input);
void MultiGetCommon(const ReadOptions& options, const size_t num_keys,
ColumnFamilyHandle** column_families, const Slice* keys,
PinnableSlice* values, PinnableWideColumns* columns,
std::string* timestamps, Status* statuses,
bool sorted_input);
// A structure to hold the information required to process MultiGet of keys
// belonging to one column family. For a multi column family MultiGet, there
// will be a container of these objects.
@@ -2581,9 +2596,7 @@ class DBImpl : public DB {
// REQUIRES: mutex held
int num_running_ingest_file_;
#ifndef ROCKSDB_LITE
WalManager wal_manager_;
#endif // ROCKSDB_LITE
// A value of > 0 temporarily disables scheduling of background work
int bg_work_paused_;
@@ -2611,14 +2624,12 @@ class DBImpl : public DB {
// Only to be set during initialization
std::unique_ptr<PreReleaseCallback> recoverable_state_pre_release_callback_;
#ifndef ROCKSDB_LITE
// Scheduler to run DumpStats(), PersistStats(), and FlushInfoLog().
// Currently, internally it has a global timer instance for running the tasks.
PeriodicTaskScheduler periodic_task_scheduler_;
// It contains the implementations for each periodic task.
std::map<PeriodicTaskType, const PeriodicTaskFunc> periodic_task_functions_;
#endif
// When set, we use a separate queue for writes that don't write to memtable.
// In 2PC these are the writes at Prepare phase.
@@ -2673,9 +2684,14 @@ class DBImpl : public DB {
// thread safe, both read and write need db mutex hold.
SeqnoToTimeMapping seqno_time_mapping_;
// stop write token that is acquired when LockWal() is called. Destructed
// when UnlockWal() is called.
// Stop write token that is acquired when first LockWAL() is called.
// Destroyed when last UnlockWAL() is called. Controlled by DB mutex.
// See lock_wal_count_
std::unique_ptr<WriteControllerToken> lock_wal_write_token_;
// The number of LockWAL called without matching UnlockWAL call.
// See also lock_wal_write_token_
uint32_t lock_wal_count_;
};
class GetWithTimestampReadCallback : public ReadCallback {
+77 -114
View File
@@ -30,7 +30,6 @@ bool DBImpl::EnoughRoomForCompaction(
bool* sfm_reserved_compact_space, LogBuffer* log_buffer) {
// Check if we have enough room to do the compaction
bool enough_room = true;
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm) {
@@ -45,11 +44,6 @@ bool DBImpl::EnoughRoomForCompaction(
*sfm_reserved_compact_space = true;
}
}
#else
(void)cfd;
(void)inputs;
(void)sfm_reserved_compact_space;
#endif // ROCKSDB_LITE
if (!enough_room) {
// Just in case tests want to change the value of enough_room
TEST_SYNC_POINT_CALLBACK(
@@ -155,7 +149,7 @@ IOStatus DBImpl::SyncClosedLogs(JobContext* job_context,
Status DBImpl::FlushMemTableToOutputFile(
ColumnFamilyData* cfd, const MutableCFOptions& mutable_cf_options,
bool* made_progress, JobContext* job_context,
bool* made_progress, JobContext* job_context, FlushReason flush_reason,
SuperVersionContext* superversion_context,
std::vector<SequenceNumber>& snapshot_seqs,
SequenceNumber earliest_write_conflict_snapshot,
@@ -215,7 +209,8 @@ Status DBImpl::FlushMemTableToOutputFile(
dbname_, cfd, immutable_db_options_, mutable_cf_options, max_memtable_id,
file_options_for_compaction_, versions_.get(), &mutex_, &shutting_down_,
snapshot_seqs, earliest_write_conflict_snapshot, snapshot_checker,
job_context, log_buffer, directories_.GetDbDir(), GetDataDir(cfd, 0U),
job_context, flush_reason, log_buffer, directories_.GetDbDir(),
GetDataDir(cfd, 0U),
GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_,
&event_logger_, mutable_cf_options.report_bg_io_stats,
true /* sync_output_directory */, true /* write_manifest */, thread_pri,
@@ -258,10 +253,9 @@ Status DBImpl::FlushMemTableToOutputFile(
TEST_SYNC_POINT_CALLBACK(
"DBImpl::FlushMemTableToOutputFile:AfterPickMemtables", &flush_job);
#ifndef ROCKSDB_LITE
// may temporarily unlock and lock the mutex.
NotifyOnFlushBegin(cfd, &file_meta, mutable_cf_options, job_context->job_id);
#endif // ROCKSDB_LITE
NotifyOnFlushBegin(cfd, &file_meta, mutable_cf_options, job_context->job_id,
flush_reason);
bool switched_to_mempurge = false;
// Within flush_job.Run, rocksdb may call event listener to notify
@@ -342,7 +336,6 @@ Status DBImpl::FlushMemTableToOutputFile(
// If flush ran smoothly and no mempurge happened
// install new SST file path.
if (s.ok() && (!switched_to_mempurge)) {
#ifndef ROCKSDB_LITE
// may temporarily unlock and lock the mutex.
NotifyOnFlushCompleted(cfd, mutable_cf_options,
flush_job.GetCommittedFlushJobsInfo());
@@ -365,7 +358,6 @@ Status DBImpl::FlushMemTableToOutputFile(
error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
}
}
#endif // ROCKSDB_LITE
}
TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:Finish");
return s;
@@ -390,8 +382,9 @@ Status DBImpl::FlushMemTablesToOutputFiles(
MutableCFOptions mutable_cf_options_copy = *cfd->GetLatestMutableCFOptions();
SuperVersionContext* superversion_context =
bg_flush_arg.superversion_context_;
FlushReason flush_reason = bg_flush_arg.flush_reason_;
Status s = FlushMemTableToOutputFile(
cfd, mutable_cf_options_copy, made_progress, job_context,
cfd, mutable_cf_options_copy, made_progress, job_context, flush_reason,
superversion_context, snapshot_seqs, earliest_write_conflict_snapshot,
snapshot_checker, log_buffer, thread_pri);
return s;
@@ -420,7 +413,9 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
for (const auto cfd : cfds) {
assert(cfd->imm()->NumNotFlushed() != 0);
assert(cfd->imm()->IsFlushPending());
assert(cfd->GetFlushReason() == cfds[0]->GetFlushReason());
}
for (const auto bg_flush_arg : bg_flush_args) {
assert(bg_flush_arg.flush_reason_ == bg_flush_args[0].flush_reason_);
}
#endif /* !NDEBUG */
@@ -459,13 +454,15 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
all_mutable_cf_options.emplace_back(*cfd->GetLatestMutableCFOptions());
const MutableCFOptions& mutable_cf_options = all_mutable_cf_options.back();
uint64_t max_memtable_id = bg_flush_args[i].max_memtable_id_;
FlushReason flush_reason = bg_flush_args[i].flush_reason_;
jobs.emplace_back(new FlushJob(
dbname_, cfd, immutable_db_options_, mutable_cf_options,
max_memtable_id, file_options_for_compaction_, versions_.get(), &mutex_,
&shutting_down_, snapshot_seqs, earliest_write_conflict_snapshot,
snapshot_checker, job_context, log_buffer, directories_.GetDbDir(),
data_dir, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
stats_, &event_logger_, mutable_cf_options.report_bg_io_stats,
snapshot_checker, job_context, flush_reason, log_buffer,
directories_.GetDbDir(), data_dir,
GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_,
&event_logger_, mutable_cf_options.report_bg_io_stats,
false /* sync_output_directory */, false /* write_manifest */,
thread_pri, io_tracer_, seqno_time_mapping_, db_id_, db_session_id_,
cfd->GetFullHistoryTsLow(), &blob_callback_));
@@ -479,14 +476,13 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
IOStatus log_io_s = IOStatus::OK();
assert(num_cfs == static_cast<int>(jobs.size()));
#ifndef ROCKSDB_LITE
for (int i = 0; i != num_cfs; ++i) {
const MutableCFOptions& mutable_cf_options = all_mutable_cf_options.at(i);
// may temporarily unlock and lock the mutex.
FlushReason flush_reason = bg_flush_args[i].flush_reason_;
NotifyOnFlushBegin(cfds[i], &file_meta[i], mutable_cf_options,
job_context->job_id);
job_context->job_id, flush_reason);
}
#endif /* !ROCKSDB_LITE */
if (logfile_number_ > 0) {
// TODO (yanqin) investigate whether we should sync the closed logs for
@@ -642,8 +638,9 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
bool resuming_from_bg_err =
error_handler_.IsDBStopped() ||
(cfds[0]->GetFlushReason() == FlushReason::kErrorRecovery ||
cfds[0]->GetFlushReason() == FlushReason::kErrorRecoveryRetryFlush);
(bg_flush_args[0].flush_reason_ == FlushReason::kErrorRecovery ||
bg_flush_args[0].flush_reason_ ==
FlushReason::kErrorRecoveryRetryFlush);
while ((!resuming_from_bg_err || error_handler_.GetRecoveryError().ok())) {
std::pair<Status, bool> res = wait_to_install_func();
@@ -660,8 +657,9 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
resuming_from_bg_err =
error_handler_.IsDBStopped() ||
(cfds[0]->GetFlushReason() == FlushReason::kErrorRecovery ||
cfds[0]->GetFlushReason() == FlushReason::kErrorRecoveryRetryFlush);
(bg_flush_args[0].flush_reason_ == FlushReason::kErrorRecovery ||
bg_flush_args[0].flush_reason_ ==
FlushReason::kErrorRecoveryRetryFlush);
}
if (!resuming_from_bg_err) {
@@ -693,10 +691,8 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
mems_list.emplace_back(&mems);
mutable_cf_options_list.emplace_back(&all_mutable_cf_options[i]);
tmp_file_meta.emplace_back(&file_meta[i]);
#ifndef ROCKSDB_LITE
committed_flush_jobs_info.emplace_back(
jobs[i]->GetCommittedFlushJobsInfo());
#endif //! ROCKSDB_LITE
}
}
@@ -748,7 +744,6 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
if (made_progress) {
*made_progress = true;
}
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
assert(all_mutable_cf_options.size() == static_cast<size_t>(num_cfs));
@@ -779,7 +774,6 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
}
}
}
#endif // ROCKSDB_LITE
}
// Need to undo atomic flush if something went wrong, i.e. s is not OK and
@@ -816,8 +810,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
void DBImpl::NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
const MutableCFOptions& mutable_cf_options,
int job_id) {
#ifndef ROCKSDB_LITE
int job_id, FlushReason flush_reason) {
if (immutable_db_options_.listeners.size() == 0U) {
return;
}
@@ -849,7 +842,7 @@ void DBImpl::NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
info.triggered_writes_stop = triggered_writes_stop;
info.smallest_seqno = file_meta->fd.smallest_seqno;
info.largest_seqno = file_meta->fd.largest_seqno;
info.flush_reason = cfd->GetFlushReason();
info.flush_reason = flush_reason;
for (auto listener : immutable_db_options_.listeners) {
listener->OnFlushBegin(this, info);
}
@@ -857,18 +850,11 @@ void DBImpl::NotifyOnFlushBegin(ColumnFamilyData* cfd, FileMetaData* file_meta,
mutex_.Lock();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)file_meta;
(void)mutable_cf_options;
(void)job_id;
#endif // ROCKSDB_LITE
}
void DBImpl::NotifyOnFlushCompleted(
ColumnFamilyData* cfd, const MutableCFOptions& mutable_cf_options,
std::list<std::unique_ptr<FlushJobInfo>>* flush_jobs_info) {
#ifndef ROCKSDB_LITE
assert(flush_jobs_info != nullptr);
if (immutable_db_options_.listeners.size() == 0U) {
return;
@@ -900,11 +886,6 @@ void DBImpl::NotifyOnFlushCompleted(
mutex_.Lock();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)mutable_cf_options;
(void)flush_jobs_info;
#endif // ROCKSDB_LITE
}
Status DBImpl::CompactRange(const CompactRangeOptions& options,
@@ -1287,17 +1268,6 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
const int output_level, const int output_path_id,
std::vector<std::string>* const output_file_names,
CompactionJobInfo* compaction_job_info) {
#ifdef ROCKSDB_LITE
(void)compact_options;
(void)column_family;
(void)input_file_names;
(void)output_level;
(void)output_path_id;
(void)output_file_names;
(void)compaction_job_info;
// not supported in lite version
return Status::NotSupported("Not supported in ROCKSDB LITE");
#else
if (column_family == nullptr) {
return Status::InvalidArgument("ColumnFamilyHandle must be non-null.");
}
@@ -1356,10 +1326,8 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
}
return s;
#endif // ROCKSDB_LITE
}
#ifndef ROCKSDB_LITE
Status DBImpl::CompactFilesImpl(
const CompactionOptions& compact_options, ColumnFamilyData* cfd,
Version* version, const std::vector<std::string>& input_file_names,
@@ -1501,14 +1469,12 @@ Status DBImpl::CompactFilesImpl(
// SetBGError
compaction_job.io_status().PermitUncheckedError();
c->ReleaseCompactionFiles(s);
#ifndef ROCKSDB_LITE
// Need to make sure SstFileManager does its bookkeeping
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm && sfm_reserved_compact_space) {
sfm->OnCompactionCompletion(c.get());
}
#endif // ROCKSDB_LITE
ReleaseFileNumberFromPendingOutputs(pending_outputs_inserted_elem);
@@ -1565,7 +1531,6 @@ Status DBImpl::CompactFilesImpl(
return status;
}
#endif // ROCKSDB_LITE
Status DBImpl::PauseBackgroundWork() {
InstrumentedMutexLock guard_lock(&mutex_);
@@ -1599,7 +1564,6 @@ void DBImpl::NotifyOnCompactionBegin(ColumnFamilyData* cfd, Compaction* c,
const Status& st,
const CompactionJobStats& job_stats,
int job_id) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.listeners.empty()) {
return;
}
@@ -1628,19 +1592,11 @@ void DBImpl::NotifyOnCompactionBegin(ColumnFamilyData* cfd, Compaction* c,
}
mutex_.Lock();
current->Unref();
#else
(void)cfd;
(void)c;
(void)st;
(void)job_stats;
(void)job_id;
#endif // ROCKSDB_LITE
}
void DBImpl::NotifyOnCompactionCompleted(
ColumnFamilyData* cfd, Compaction* c, const Status& st,
const CompactionJobStats& compaction_job_stats, const int job_id) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.listeners.size() == 0U) {
return;
}
@@ -1670,13 +1626,6 @@ void DBImpl::NotifyOnCompactionCompleted(
current->Unref();
// no need to signal bg_cv_ as it will be signaled at the end of the
// flush process.
#else
(void)cfd;
(void)c;
(void)st;
(void)compaction_job_stats;
(void)job_id;
#endif // ROCKSDB_LITE
}
// REQUIREMENT: block all background work by calling PauseBackgroundWork()
@@ -2023,8 +1972,19 @@ Status DBImpl::RunManualCompaction(
manual.begin, manual.end, &manual.manual_end, &manual_conflict,
max_file_num_to_ignore, trim_ts)) == nullptr &&
manual_conflict))) {
// Running either this or some other manual compaction
bg_cv_.Wait();
if (!scheduled) {
// There is a conflicting compaction
if (manual_compaction_paused_ > 0 || manual.canceled == true) {
// Stop waiting since it was canceled. Pretend the error came from
// compaction so the below cleanup/error handling code can process it.
manual.done = true;
manual.status =
Status::Incomplete(Status::SubCode::kManualCompactionPaused);
}
}
if (!manual.done) {
bg_cv_.Wait();
}
if (manual_compaction_paused_ > 0 && scheduled && !unscheduled) {
assert(thread_pool_priority != Env::Priority::TOTAL);
// unschedule all manual compactions
@@ -2102,16 +2062,17 @@ Status DBImpl::RunManualCompaction(
}
void DBImpl::GenerateFlushRequest(const autovector<ColumnFamilyData*>& cfds,
FlushRequest* req) {
FlushReason flush_reason, FlushRequest* req) {
assert(req != nullptr);
req->reserve(cfds.size());
req->flush_reason = flush_reason;
req->cfd_to_max_mem_id_to_persist.reserve(cfds.size());
for (const auto cfd : cfds) {
if (nullptr == cfd) {
// cfd may be null, see DBImpl::ScheduleFlushes
continue;
}
uint64_t max_memtable_id = cfd->imm()->GetLatestMemTableID();
req->emplace_back(cfd, max_memtable_id);
req->cfd_to_max_mem_id_to_persist.emplace(cfd, max_memtable_id);
}
}
@@ -2169,7 +2130,7 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
if (s.ok()) {
if (cfd->imm()->NumNotFlushed() != 0 || !cfd->mem()->IsEmpty() ||
!cached_recoverable_state_empty_.load()) {
FlushRequest req{{cfd, flush_memtable_id}};
FlushRequest req{flush_reason, {{cfd, flush_memtable_id}}};
flush_reqs.emplace_back(std::move(req));
memtable_ids_to_wait.emplace_back(cfd->imm()->GetLatestMemTableID());
}
@@ -2197,7 +2158,7 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
"to avoid holding old logs",
cfd->GetName().c_str());
s = SwitchMemtable(cfd_stats, &context);
FlushRequest req{{cfd_stats, flush_memtable_id}};
FlushRequest req{flush_reason, {{cfd_stats, flush_memtable_id}}};
flush_reqs.emplace_back(std::move(req));
memtable_ids_to_wait.emplace_back(
cfd_stats->imm()->GetLatestMemTableID());
@@ -2208,8 +2169,9 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
if (s.ok() && !flush_reqs.empty()) {
for (const auto& req : flush_reqs) {
assert(req.size() == 1);
ColumnFamilyData* loop_cfd = req[0].first;
assert(req.cfd_to_max_mem_id_to_persist.size() == 1);
ColumnFamilyData* loop_cfd =
req.cfd_to_max_mem_id_to_persist.begin()->first;
loop_cfd->imm()->FlushRequested();
}
// If the caller wants to wait for this flush to complete, it indicates
@@ -2218,13 +2180,14 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
// Therefore, we increase the cfd's ref count.
if (flush_options.wait) {
for (const auto& req : flush_reqs) {
assert(req.size() == 1);
ColumnFamilyData* loop_cfd = req[0].first;
assert(req.cfd_to_max_mem_id_to_persist.size() == 1);
ColumnFamilyData* loop_cfd =
req.cfd_to_max_mem_id_to_persist.begin()->first;
loop_cfd->Ref();
}
}
for (const auto& req : flush_reqs) {
SchedulePendingFlush(req, flush_reason);
SchedulePendingFlush(req);
}
MaybeScheduleFlushOrCompaction();
}
@@ -2243,8 +2206,8 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
autovector<const uint64_t*> flush_memtable_ids;
assert(flush_reqs.size() == memtable_ids_to_wait.size());
for (size_t i = 0; i < flush_reqs.size(); ++i) {
assert(flush_reqs[i].size() == 1);
cfds.push_back(flush_reqs[i][0].first);
assert(flush_reqs[i].cfd_to_max_mem_id_to_persist.size() == 1);
cfds.push_back(flush_reqs[i].cfd_to_max_mem_id_to_persist.begin()->first);
flush_memtable_ids.push_back(&(memtable_ids_to_wait[i]));
}
s = WaitForFlushMemTables(
@@ -2341,8 +2304,8 @@ Status DBImpl::AtomicFlushMemTables(
cfd->Ref();
}
}
GenerateFlushRequest(cfds, &flush_req);
SchedulePendingFlush(flush_req, flush_reason);
GenerateFlushRequest(cfds, flush_reason, &flush_req);
SchedulePendingFlush(flush_req);
MaybeScheduleFlushOrCompaction();
}
@@ -2357,7 +2320,7 @@ Status DBImpl::AtomicFlushMemTables(
TEST_SYNC_POINT("DBImpl::AtomicFlushMemTables:BeforeWaitForBgFlush");
if (s.ok() && flush_options.wait) {
autovector<const uint64_t*> flush_memtable_ids;
for (auto& iter : flush_req) {
for (auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
flush_memtable_ids.push_back(&(iter.second));
}
s = WaitForFlushMemTables(
@@ -2704,9 +2667,9 @@ DBImpl::FlushRequest DBImpl::PopFirstFromFlushQueue() {
FlushRequest flush_req = flush_queue_.front();
flush_queue_.pop_front();
if (!immutable_db_options_.atomic_flush) {
assert(flush_req.size() == 1);
assert(flush_req.cfd_to_max_mem_id_to_persist.size() == 1);
}
for (const auto& elem : flush_req) {
for (const auto& elem : flush_req.cfd_to_max_mem_id_to_persist) {
if (!immutable_db_options_.atomic_flush) {
ColumnFamilyData* cfd = elem.first;
assert(cfd);
@@ -2714,7 +2677,6 @@ DBImpl::FlushRequest DBImpl::PopFirstFromFlushQueue() {
cfd->set_queued_for_flush(false);
}
}
// TODO: need to unset flush reason?
return flush_req;
}
@@ -2744,31 +2706,29 @@ ColumnFamilyData* DBImpl::PickCompactionFromQueue(
return cfd;
}
void DBImpl::SchedulePendingFlush(const FlushRequest& flush_req,
FlushReason flush_reason) {
void DBImpl::SchedulePendingFlush(const FlushRequest& flush_req) {
mutex_.AssertHeld();
if (flush_req.empty()) {
if (flush_req.cfd_to_max_mem_id_to_persist.empty()) {
return;
}
if (!immutable_db_options_.atomic_flush) {
// For the non-atomic flush case, we never schedule multiple column
// families in the same flush request.
assert(flush_req.size() == 1);
ColumnFamilyData* cfd = flush_req[0].first;
assert(flush_req.cfd_to_max_mem_id_to_persist.size() == 1);
ColumnFamilyData* cfd =
flush_req.cfd_to_max_mem_id_to_persist.begin()->first;
assert(cfd);
if (!cfd->queued_for_flush() && cfd->imm()->IsFlushPending()) {
cfd->Ref();
cfd->set_queued_for_flush(true);
cfd->SetFlushReason(flush_reason);
++unscheduled_flushes_;
flush_queue_.push_back(flush_req);
}
} else {
for (auto& iter : flush_req) {
for (auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
ColumnFamilyData* cfd = iter.first;
cfd->Ref();
cfd->SetFlushReason(flush_reason);
}
++unscheduled_flushes_;
flush_queue_.push_back(flush_req);
@@ -2900,10 +2860,12 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
while (!flush_queue_.empty()) {
// This cfd is already referenced
const FlushRequest& flush_req = PopFirstFromFlushQueue();
FlushReason flush_reason = flush_req.flush_reason;
superversion_contexts.clear();
superversion_contexts.reserve(flush_req.size());
superversion_contexts.reserve(
flush_req.cfd_to_max_mem_id_to_persist.size());
for (const auto& iter : flush_req) {
for (const auto& iter : flush_req.cfd_to_max_mem_id_to_persist) {
ColumnFamilyData* cfd = iter.first;
if (cfd->GetMempurgeUsed()) {
// If imm() contains silent memtables (e.g.: because
@@ -2919,7 +2881,7 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
}
superversion_contexts.emplace_back(SuperVersionContext(true));
bg_flush_args.emplace_back(cfd, iter.second,
&(superversion_contexts.back()));
&(superversion_contexts.back()), flush_reason);
}
if (!bg_flush_args.empty()) {
break;
@@ -2943,9 +2905,14 @@ Status DBImpl::BackgroundFlush(bool* made_progress, JobContext* job_context,
status = FlushMemTablesToOutputFiles(bg_flush_args, made_progress,
job_context, log_buffer, thread_pri);
TEST_SYNC_POINT("DBImpl::BackgroundFlush:BeforeFlush");
// All the CFDs in the FlushReq must have the same flush reason, so just
// grab the first one
*reason = bg_flush_args[0].cfd_->GetFlushReason();
// All the CFD/bg_flush_arg in the FlushReq must have the same flush reason, so
// just grab the first one
#ifndef NDEBUG
for (const auto bg_flush_arg : bg_flush_args) {
assert(bg_flush_arg.flush_reason_ == bg_flush_args[0].flush_reason_);
}
#endif /* !NDEBUG */
*reason = bg_flush_args[0].flush_reason_;
for (auto& arg : bg_flush_args) {
ColumnFamilyData* cfd = arg.cfd_;
if (cfd->UnrefAndTryDelete()) {
@@ -3572,14 +3539,12 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
c->ReleaseCompactionFiles(status);
*made_progress = true;
#ifndef ROCKSDB_LITE
// Need to make sure SstFileManager does its bookkeeping
auto sfm = static_cast<SstFileManagerImpl*>(
immutable_db_options_.sst_file_manager.get());
if (sfm && sfm_reserved_compact_space) {
sfm->OnCompactionCompletion(c.get());
}
#endif // ROCKSDB_LITE
NotifyOnCompactionCompleted(c->column_family_data(), c.get(), status,
compaction_job_stats, job_context->job_id);
@@ -3756,7 +3721,6 @@ bool DBImpl::MCOverlap(ManualCompactionState* m, ManualCompactionState* m1) {
return false;
}
#ifndef ROCKSDB_LITE
void DBImpl::BuildCompactionJobInfo(
const ColumnFamilyData* cfd, Compaction* c, const Status& st,
const CompactionJobStats& compaction_job_stats, const int job_id,
@@ -3825,7 +3789,6 @@ void DBImpl::BuildCompactionJobInfo(
std::move(blob_file_garbage_info));
}
}
#endif
// SuperVersionContext gets created and destructed outside of the lock --
// we use this conveniently to:
+2 -2
View File
@@ -198,6 +198,8 @@ void DBImpl::TEST_LockMutex() { mutex_.Lock(); }
void DBImpl::TEST_UnlockMutex() { mutex_.Unlock(); }
void DBImpl::TEST_SignalAllBgCv() { bg_cv_.SignalAll(); }
void* DBImpl::TEST_BeginWrite() {
auto w = new WriteThread::Writer();
write_thread_.EnterUnbatched(w, &mutex_);
@@ -289,7 +291,6 @@ size_t DBImpl::TEST_GetWalPreallocateBlockSize(
return GetWalPreallocateBlockSize(write_buffer_size);
}
#ifndef ROCKSDB_LITE
void DBImpl::TEST_WaitForPeriodicTaskRun(std::function<void()> callback) const {
periodic_task_scheduler_.TEST_WaitForRun(callback);
}
@@ -303,7 +304,6 @@ SeqnoToTimeMapping DBImpl::TEST_GetSeqnoToTimeMapping() const {
return seqno_time_mapping_;
}
#endif // !ROCKSDB_LITE
size_t DBImpl::TEST_EstimateInMemoryStatsHistorySize() const {
return EstimateInMemoryStatsHistorySize();
-2
View File
@@ -20,7 +20,6 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
Status DBImpl::SuggestCompactRange(ColumnFamilyHandle* column_family,
const Slice* begin, const Slice* end) {
auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
@@ -154,6 +153,5 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
return status;
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-6
View File
@@ -361,8 +361,6 @@ void DBImpl::DeleteObsoleteFileImpl(int job_id, const std::string& fname,
}
TEST_SYNC_POINT_CALLBACK("DBImpl::DeleteObsoleteFileImpl:AfterDeletion",
&file_deletion_status);
TEST_SYNC_POINT_CALLBACK("DBImpl::DeleteObsoleteFileImpl:AfterDeletion2",
const_cast<std::string*>(&fname));
if (file_deletion_status.ok()) {
ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[JOB %d] Delete %s type=%d #%" PRIu64 " -- %s\n", job_id,
@@ -599,13 +597,11 @@ void DBImpl::PurgeObsoleteFiles(JobContext& state, bool schedule_only) {
to_delete;
}
#ifndef ROCKSDB_LITE
if (type == kWalFile && (immutable_db_options_.WAL_ttl_seconds > 0 ||
immutable_db_options_.WAL_size_limit_MB > 0)) {
wal_manager_.ArchiveWALFile(fname, number);
continue;
}
#endif // !ROCKSDB_LITE
// If I do not own these files, e.g. secondary instance with max_open_files
// = -1, then no need to delete or schedule delete these files since they
@@ -669,9 +665,7 @@ void DBImpl::PurgeObsoleteFiles(JobContext& state, bool schedule_only) {
}
}
}
#ifndef ROCKSDB_LITE
wal_manager_.PurgeObsoleteWALFiles();
#endif // ROCKSDB_LITE
LogFlush(immutable_db_options_.info_log);
InstrumentedMutexLock l(&mutex_);
--pending_purge_obsolete_files_;
-16
View File
@@ -153,7 +153,6 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
result.avoid_flush_during_recovery = false;
}
#ifndef ROCKSDB_LITE
ImmutableDBOptions immutable_db_options(result);
if (!immutable_db_options.IsWalDirSameAsDBPath()) {
// Either the WAL dir and db_paths[0]/db_name are not the same, or we
@@ -195,7 +194,6 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
NewSstFileManager(result.env, result.info_log));
result.sst_file_manager = sst_file_manager;
}
#endif // !ROCKSDB_LITE
// Supported wal compression types
if (!StreamingCompressionTypeSupported(result.wal_compression)) {
@@ -845,7 +843,6 @@ Status DBImpl::LogAndApplyForRecovery(const RecoveryContext& recovery_ctx) {
}
void DBImpl::InvokeWalFilterIfNeededOnColumnFamilyToWalNumberMap() {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.wal_filter == nullptr) {
return;
}
@@ -863,7 +860,6 @@ void DBImpl::InvokeWalFilterIfNeededOnColumnFamilyToWalNumberMap() {
}
wal_filter.ColumnFamilyLogNumberMap(cf_lognumber_map, cf_name_id_map);
#endif // !ROCKSDB_LITE
}
bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
@@ -872,7 +868,6 @@ bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
Status& status,
bool& stop_replay,
WriteBatch& batch) {
#ifndef ROCKSDB_LITE
if (immutable_db_options_.wal_filter == nullptr) {
return true;
}
@@ -958,15 +953,6 @@ bool DBImpl::InvokeWalFilterIfNeededOnWalRecord(uint64_t wal_number,
batch = new_batch;
}
return true;
#else // !ROCKSDB_LITE
(void)wal_number;
(void)wal_fname;
(void)reporter;
(void)status;
(void)stop_replay;
(void)batch;
return true;
#endif // ROCKSDB_LITE
}
// REQUIRES: wal_numbers are sorted in ascending order
@@ -1981,7 +1967,6 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
}
impl->mutex_.Unlock();
#ifndef ROCKSDB_LITE
auto sfm = static_cast<SstFileManagerImpl*>(
impl->immutable_db_options_.sst_file_manager.get());
if (s.ok() && sfm) {
@@ -2065,7 +2050,6 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
impl->immutable_db_options_.db_paths[0].path);
}
#endif // !ROCKSDB_LITE
if (s.ok()) {
ROCKS_LOG_HEADER(impl->immutable_db_options_.info_log, "DB pointer %p",
-17
View File
@@ -16,7 +16,6 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options,
const std::string& dbname)
@@ -321,21 +320,5 @@ Status DBImplReadOnly::OpenForReadOnlyWithoutCheck(
return s;
}
#else // !ROCKSDB_LITE
Status DB::OpenForReadOnly(const Options& /*options*/,
const std::string& /*dbname*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
Status DB::OpenForReadOnly(
const DBOptions& /*db_options*/, const std::string& /*dbname*/,
const std::vector<ColumnFamilyDescriptor>& /*column_families*/,
std::vector<ColumnFamilyHandle*>* /*handles*/, DB** /*dbptr*/,
bool /*error_if_wal_file_exists*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-4
View File
@@ -5,7 +5,6 @@
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -146,12 +145,10 @@ class DBImplReadOnly : public DBImpl {
// FIXME: some missing overrides for more "write" functions
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
private:
// A "helper" function for DB::OpenForReadOnly without column families
@@ -167,4 +164,3 @@ class DBImplReadOnly : public DBImpl {
};
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
-18
View File
@@ -17,7 +17,6 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
DBImplSecondary::DBImplSecondary(const DBOptions& db_options,
const std::string& dbname,
std::string secondary_path)
@@ -946,22 +945,5 @@ Status DB::OpenAndCompact(
output, override_options);
}
#else // !ROCKSDB_LITE
Status DB::OpenAsSecondary(const Options& /*options*/,
const std::string& /*name*/,
const std::string& /*secondary_path*/,
DB** /*dbptr*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
Status DB::OpenAsSecondary(
const DBOptions& /*db_options*/, const std::string& /*dbname*/,
const std::string& /*secondary_path*/,
const std::vector<ColumnFamilyDescriptor>& /*column_families*/,
std::vector<ColumnFamilyHandle*>* /*handles*/, DB** /*dbptr*/) {
return Status::NotSupported("Not supported in ROCKSDB_LITE.");
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-4
View File
@@ -5,7 +5,6 @@
#pragma once
#ifndef ROCKSDB_LITE
#include <string>
#include <vector>
@@ -269,12 +268,10 @@ class DBImplSecondary : public DBImpl {
#endif // NDEBUG
protected:
#ifndef ROCKSDB_LITE
Status FlushForGetLiveFiles() override {
// No-op for read-only DB
return Status::OK();
}
#endif // !ROCKSDB_LITE
// ColumnFamilyCollector is a write batch handler which does nothing
// except recording unique column family IDs
@@ -407,4 +404,3 @@ class DBImplSecondary : public DBImpl {
} // namespace ROCKSDB_NAMESPACE
#endif // !ROCKSDB_LITE
+40 -33
View File
@@ -149,7 +149,6 @@ Status DBImpl::Write(const WriteOptions& write_options, WriteBatch* my_batch) {
return s;
}
#ifndef ROCKSDB_LITE
Status DBImpl::WriteWithCallback(const WriteOptions& write_options,
WriteBatch* my_batch,
WriteCallback* callback) {
@@ -163,7 +162,6 @@ Status DBImpl::WriteWithCallback(const WriteOptions& write_options,
}
return s;
}
#endif // ROCKSDB_LITE
// The main write queue. This is the only write queue that updates LastSequence.
// When using one write queue, the same sequence also indicates the last
@@ -926,7 +924,8 @@ Status DBImpl::WriteImplWALOnly(
}
} else {
InstrumentedMutexLock lock(&mutex_);
Status status = DelayWrite(/*num_bytes=*/0ull, write_options);
Status status =
DelayWrite(/*num_bytes=*/0ull, *write_thread, write_options);
if (!status.ok()) {
WriteThread::WriteGroup write_group;
write_thread->EnterAsBatchGroupLeader(&w, &write_group);
@@ -1201,7 +1200,7 @@ Status DBImpl::PreprocessWrite(const WriteOptions& write_options,
// might happen for smaller writes but larger writes can go through.
// Can optimize it if it is an issue.
InstrumentedMutexLock l(&mutex_);
status = DelayWrite(last_batch_group_size_, write_options);
status = DelayWrite(last_batch_group_size_, write_thread_, write_options);
PERF_TIMER_START(write_pre_and_post_process_time);
}
@@ -1653,14 +1652,14 @@ Status DBImpl::SwitchWAL(WriteContext* write_context) {
cfd->imm()->FlushRequested();
if (!immutable_db_options_.atomic_flush) {
FlushRequest flush_req;
GenerateFlushRequest({cfd}, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWalFull);
GenerateFlushRequest({cfd}, FlushReason::kWalFull, &flush_req);
SchedulePendingFlush(flush_req);
}
}
if (immutable_db_options_.atomic_flush) {
FlushRequest flush_req;
GenerateFlushRequest(cfds, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWalFull);
GenerateFlushRequest(cfds, FlushReason::kWalFull, &flush_req);
SchedulePendingFlush(flush_req);
}
MaybeScheduleFlushOrCompaction();
}
@@ -1744,14 +1743,15 @@ Status DBImpl::HandleWriteBufferManagerFlush(WriteContext* write_context) {
cfd->imm()->FlushRequested();
if (!immutable_db_options_.atomic_flush) {
FlushRequest flush_req;
GenerateFlushRequest({cfd}, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWriteBufferManager);
GenerateFlushRequest({cfd}, FlushReason::kWriteBufferManager,
&flush_req);
SchedulePendingFlush(flush_req);
}
}
if (immutable_db_options_.atomic_flush) {
FlushRequest flush_req;
GenerateFlushRequest(cfds, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWriteBufferManager);
GenerateFlushRequest(cfds, FlushReason::kWriteBufferManager, &flush_req);
SchedulePendingFlush(flush_req);
}
MaybeScheduleFlushOrCompaction();
}
@@ -1768,8 +1768,8 @@ uint64_t DBImpl::GetMaxTotalWalSize() const {
}
// REQUIRES: mutex_ is held
// REQUIRES: this thread is currently at the front of the writer queue
Status DBImpl::DelayWrite(uint64_t num_bytes,
// REQUIRES: this thread is currently at the leader for write_thread
Status DBImpl::DelayWrite(uint64_t num_bytes, WriteThread& write_thread,
const WriteOptions& write_options) {
mutex_.AssertHeld();
uint64_t time_delayed = 0;
@@ -1777,8 +1777,16 @@ Status DBImpl::DelayWrite(uint64_t num_bytes,
{
StopWatch sw(immutable_db_options_.clock, stats_, WRITE_STALL,
&time_delayed);
uint64_t delay =
write_controller_.GetDelay(immutable_db_options_.clock, num_bytes);
// To avoid parallel timed delays (bad throttling), only support them
// on the primary write queue.
uint64_t delay;
if (&write_thread == &write_thread_) {
delay =
write_controller_.GetDelay(immutable_db_options_.clock, num_bytes);
} else {
assert(num_bytes == 0);
delay = 0;
}
TEST_SYNC_POINT("DBImpl::DelayWrite:Start");
if (delay > 0) {
if (write_options.no_slowdown) {
@@ -1786,9 +1794,9 @@ Status DBImpl::DelayWrite(uint64_t num_bytes,
}
TEST_SYNC_POINT("DBImpl::DelayWrite:Sleep");
// Notify write_thread_ about the stall so it can setup a barrier and
// Notify write_thread about the stall so it can setup a barrier and
// fail any pending writers with no_slowdown
write_thread_.BeginWriteStall();
write_thread.BeginWriteStall();
mutex_.Unlock();
TEST_SYNC_POINT("DBImpl::DelayWrite:BeginWriteStallDone");
// We will delay the write until we have slept for `delay` microseconds
@@ -1808,7 +1816,7 @@ Status DBImpl::DelayWrite(uint64_t num_bytes,
immutable_db_options_.clock->SleepForMicroseconds(kDelayInterval);
}
mutex_.Lock();
write_thread_.EndWriteStall();
write_thread.EndWriteStall();
}
// Don't wait if there's a background error, even if its a soft error. We
@@ -1822,12 +1830,17 @@ Status DBImpl::DelayWrite(uint64_t num_bytes,
}
delayed = true;
// Notify write_thread_ about the stall so it can setup a barrier and
// Notify write_thread about the stall so it can setup a barrier and
// fail any pending writers with no_slowdown
write_thread_.BeginWriteStall();
TEST_SYNC_POINT("DBImpl::DelayWrite:Wait");
write_thread.BeginWriteStall();
if (&write_thread == &write_thread_) {
TEST_SYNC_POINT("DBImpl::DelayWrite:Wait");
} else {
TEST_SYNC_POINT("DBImpl::DelayWrite:NonmemWait");
}
bg_cv_.Wait();
write_thread_.EndWriteStall();
TEST_SYNC_POINT_CALLBACK("DBImpl::DelayWrite:AfterWait", &mutex_);
write_thread.EndWriteStall();
}
}
assert(!delayed || !write_options.no_slowdown);
@@ -2008,13 +2021,13 @@ Status DBImpl::ScheduleFlushes(WriteContext* context) {
if (immutable_db_options_.atomic_flush) {
AssignAtomicFlushSeq(cfds);
FlushRequest flush_req;
GenerateFlushRequest(cfds, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWriteBufferFull);
GenerateFlushRequest(cfds, FlushReason::kWriteBufferFull, &flush_req);
SchedulePendingFlush(flush_req);
} else {
for (auto* cfd : cfds) {
FlushRequest flush_req;
GenerateFlushRequest({cfd}, &flush_req);
SchedulePendingFlush(flush_req, FlushReason::kWriteBufferFull);
GenerateFlushRequest({cfd}, FlushReason::kWriteBufferFull, &flush_req);
SchedulePendingFlush(flush_req);
}
}
MaybeScheduleFlushOrCompaction();
@@ -2022,7 +2035,6 @@ Status DBImpl::ScheduleFlushes(WriteContext* context) {
return status;
}
#ifndef ROCKSDB_LITE
void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
const MemTableInfo& mem_table_info) {
if (immutable_db_options_.listeners.size() == 0U) {
@@ -2038,7 +2050,6 @@ void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
}
mutex_.Lock();
}
#endif // ROCKSDB_LITE
// REQUIRES: mutex_ is held
// REQUIRES: this thread is currently at the front of the writer queue
@@ -2077,14 +2088,12 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
const MutableCFOptions mutable_cf_options = *cfd->GetLatestMutableCFOptions();
// Set memtable_info for memtable sealed callback
#ifndef ROCKSDB_LITE
MemTableInfo memtable_info;
memtable_info.cf_name = cfd->GetName();
memtable_info.first_seqno = cfd->mem()->GetFirstSequenceNumber();
memtable_info.earliest_seqno = cfd->mem()->GetEarliestSequenceNumber();
memtable_info.num_entries = cfd->mem()->num_entries();
memtable_info.num_deletes = cfd->mem()->num_deletes();
#endif // ROCKSDB_LITE
// Log this later after lock release. It may be outdated, e.g., if background
// flush happens before logging, but that should be ok.
int num_imm_unflushed = cfd->imm()->NumNotFlushed();
@@ -2243,11 +2252,9 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
InstallSuperVersionAndScheduleWork(cfd, &context->superversion_context,
mutable_cf_options);
#ifndef ROCKSDB_LITE
// Notify client that memtable is sealed, now that we have successfully
// installed a new memtable
NotifyOnMemTableSealed(cfd, memtable_info);
#endif // ROCKSDB_LITE
// It is possible that we got here without checking the value of i_os, but
// that is okay. If we did, it most likely means that s was already an error.
// In any case, ignore any unchecked error for i_os here.
-4
View File
@@ -19,7 +19,6 @@ class DBIOFailureTest : public DBTestBase {
DBIOFailureTest() : DBTestBase("db_io_failure_test", /*env_do_fsync=*/true) {}
};
#ifndef ROCKSDB_LITE
// Check that number of files does not grow when writes are dropped
TEST_F(DBIOFailureTest, DropWrites) {
do {
@@ -123,7 +122,6 @@ TEST_F(DBIOFailureTest, NoSpaceCompactRange) {
env_->no_space_.store(false, std::memory_order_release);
} while (ChangeCompactOptions());
}
#endif // ROCKSDB_LITE
TEST_F(DBIOFailureTest, NonWritableFileSystem) {
do {
@@ -147,7 +145,6 @@ TEST_F(DBIOFailureTest, NonWritableFileSystem) {
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBIOFailureTest, ManifestWriteError) {
// Test for the following problem:
// (a) Compaction produces file F
@@ -582,7 +579,6 @@ TEST_F(DBIOFailureTest, CompactionSstSyncError) {
ASSERT_EQ("bar3", Get(1, "foo"));
}
#endif // !(defined NDEBUG) || !defined(OS_WIN)
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
-4
View File
@@ -1439,7 +1439,6 @@ void DBIter::Seek(const Slice& target) {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
StopWatch sw(clock_, statistics_, DB_SEEK);
#ifndef ROCKSDB_LITE
if (db_impl_ != nullptr && cfd_ != nullptr) {
// TODO: What do we do if this returns an error?
Slice lower_bound, upper_bound;
@@ -1456,7 +1455,6 @@ void DBIter::Seek(const Slice& target) {
db_impl_->TraceIteratorSeek(cfd_->GetID(), target, lower_bound, upper_bound)
.PermitUncheckedError();
}
#endif // ROCKSDB_LITE
status_ = Status::OK();
ReleaseTempPinnedData();
@@ -1514,7 +1512,6 @@ void DBIter::SeekForPrev(const Slice& target) {
PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
StopWatch sw(clock_, statistics_, DB_SEEK);
#ifndef ROCKSDB_LITE
if (db_impl_ != nullptr && cfd_ != nullptr) {
// TODO: What do we do if this returns an error?
Slice lower_bound, upper_bound;
@@ -1533,7 +1530,6 @@ void DBIter::SeekForPrev(const Slice& target) {
upper_bound)
.PermitUncheckedError();
}
#endif // ROCKSDB_LITE
status_ = Status::OK();
ReleaseTempPinnedData();
-6
View File
@@ -388,13 +388,7 @@ class DBIter final : public Iterator {
MergeContext merge_context_;
LocalStatistics local_stats_;
PinnedIteratorsManager pinned_iters_mgr_;
#ifdef ROCKSDB_LITE
ROCKSDB_FIELD_UNUSED
#endif
DBImpl* db_impl_;
#ifdef ROCKSDB_LITE
ROCKSDB_FIELD_UNUSED
#endif
ColumnFamilyData* cfd_;
const Slice* const timestamp_ub_;
const Slice* const timestamp_lb_;
-12
View File
@@ -891,7 +891,6 @@ TEST_P(DBIteratorTest, IteratorDeleteAfterCfDrop) {
}
// SetOptions not defined in ROCKSDB LITE
#ifndef ROCKSDB_LITE
TEST_P(DBIteratorTest, DBIteratorBoundTest) {
Options options = CurrentOptions();
options.env = env_;
@@ -1119,7 +1118,6 @@ TEST_P(DBIteratorTest, DBIteratorBoundMultiSeek) {
TestGetTickerCount(options, BLOCK_CACHE_MISS));
}
}
#endif
TEST_P(DBIteratorTest, DBIteratorBoundOptimizationTest) {
for (auto format_version : {2, 3, 4}) {
@@ -1564,7 +1562,6 @@ INSTANTIATE_TEST_CASE_P(DBIteratorTestForPinnedDataInstance,
DBIteratorTestForPinnedData,
testing::Values(true, false));
#ifndef ROCKSDB_LITE
TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
Options options = CurrentOptions();
BlockBasedTableOptions table_options;
@@ -1634,7 +1631,6 @@ TEST_P(DBIteratorTest, PinnedDataIteratorMultipleFiles) {
delete iter;
}
#endif
TEST_P(DBIteratorTest, PinnedDataIteratorMergeOperator) {
Options options = CurrentOptions();
@@ -2216,9 +2212,7 @@ TEST_P(DBIteratorTest, ReadAhead) {
ASSERT_OK(Put(Key(i), value));
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ("1,1,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
env_->random_read_bytes_counter_ = 0;
options.statistics->setTickerCount(NO_FILE_OPENS, 0);
@@ -2229,7 +2223,6 @@ TEST_P(DBIteratorTest, ReadAhead) {
size_t bytes_read = env_->random_read_bytes_counter_;
delete iter;
int64_t num_file_closes = TestGetTickerCount(options, NO_FILE_CLOSES);
env_->random_read_bytes_counter_ = 0;
options.statistics->setTickerCount(NO_FILE_OPENS, 0);
read_options.readahead_size = 1024 * 10;
@@ -2238,10 +2231,7 @@ TEST_P(DBIteratorTest, ReadAhead) {
int64_t num_file_opens_readahead = TestGetTickerCount(options, NO_FILE_OPENS);
size_t bytes_read_readahead = env_->random_read_bytes_counter_;
delete iter;
int64_t num_file_closes_readahead =
TestGetTickerCount(options, NO_FILE_CLOSES);
ASSERT_EQ(num_file_opens, num_file_opens_readahead);
ASSERT_EQ(num_file_closes, num_file_closes_readahead);
ASSERT_GT(bytes_read_readahead, bytes_read);
ASSERT_GT(bytes_read_readahead, read_options.readahead_size * 3);
@@ -2285,12 +2275,10 @@ TEST_P(DBIteratorTest, DBIteratorSkipRecentDuplicatesTest) {
ASSERT_OK(Put("b", std::to_string(i + 1).c_str()));
}
#ifndef ROCKSDB_LITE
// Check that memtable wasn't flushed.
std::string val;
ASSERT_TRUE(db_->GetProperty("rocksdb.num-files-at-level0", &val));
EXPECT_EQ("0", val);
#endif
// Seek iterator to a smaller key.
get_perf_context()->Reset();
-8
View File
@@ -10,7 +10,6 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "env/mock_env.h"
@@ -290,16 +289,9 @@ TEST_F(DBTestXactLogIterator, TransactionLogIteratorBlobs) {
}
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
-16
View File
@@ -72,12 +72,8 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenClose) {
printf("Open\n");
ASSERT_OK(DB::Open(options, dbname_, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db));
#endif
}
ASSERT_EQ(2, cache_->Size());
ASSERT_TRUE(cache_->Contains(data_path_0_));
@@ -104,12 +100,8 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenDelete) {
printf("Open\n");
ASSERT_OK(DB::Open(options, dbname_, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db));
#endif
}
ASSERT_EQ(1, cache_->Size());
ASSERT_TRUE(cache_->Contains(dbname_));
@@ -261,16 +253,12 @@ TEST_F(DBLogicalBlockSizeCacheTest, OpenWithColumnFamilies) {
{"default", ColumnFamilyOptions()}},
&cfs, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(options, dbname_,
{{"cf1", cf_options},
{"cf2", cf_options},
{"default", ColumnFamilyOptions()}},
&cfs, &db));
#endif
}
// Logical block sizes of dbname_ and cf_path_0_ are cached during Open.
@@ -360,14 +348,10 @@ TEST_F(DBLogicalBlockSizeCacheTest, DestroyColumnFamilyHandle) {
options, dbname_,
{{"cf", cf_options}, {"default", ColumnFamilyOptions()}}, &cfs, &db));
} else {
#ifdef ROCKSDB_LITE
break;
#else
printf("OpenForReadOnly\n");
ASSERT_OK(DB::OpenForReadOnly(
options, dbname_,
{{"cf", cf_options}, {"default", ColumnFamilyOptions()}}, &cfs, &db));
#endif
}
// cf_path_0_ and dbname_ are cached.
ASSERT_EQ(2, cache_->Size());
+42 -2
View File
@@ -8,9 +8,7 @@
#include "rocksdb/perf_context.h"
#include "rocksdb/utilities/debug.h"
#include "table/block_based/block_builder.h"
#if !defined(ROCKSDB_LITE)
#include "test_util/sync_point.h"
#endif
#include "rocksdb/merge_operator.h"
#include "utilities/fault_injection_env.h"
#include "utilities/merge_operators.h"
@@ -439,6 +437,48 @@ TEST_F(DBMergeOperandTest, GetMergeOperandsLargeResultOptimization) {
}
}
TEST_F(DBMergeOperandTest, GetMergeOperandsBaseDeletionInImmMem) {
// In this test, "k1" has a MERGE in a mutable memtable on top of a base
// DELETE in an immutable memtable.
Options opts = CurrentOptions();
opts.max_write_buffer_number = 10;
opts.min_write_buffer_number_to_merge = 10;
opts.merge_operator = MergeOperators::CreateDeprecatedPutOperator();
Reopen(opts);
ASSERT_OK(Put("k1", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("k0", "val"));
ASSERT_OK(Delete("k1"));
ASSERT_OK(Put("k2", "val"));
ASSERT_OK(dbfull()->TEST_SwitchMemtable());
ASSERT_OK(Merge("k1", "val"));
{
std::vector<PinnableSlice> values(2);
GetMergeOperandsOptions merge_operands_info;
merge_operands_info.expected_max_number_of_operands =
static_cast<int>(values.size());
std::string key = "k1", from_db;
int number_of_operands = 0;
ASSERT_OK(db_->GetMergeOperands(ReadOptions(), db_->DefaultColumnFamily(),
key, values.data(), &merge_operands_info,
&number_of_operands));
ASSERT_EQ(1, number_of_operands);
from_db = values[0].ToString();
ASSERT_EQ("val", from_db);
}
{
std::string val;
ASSERT_OK(db_->Get(ReadOptions(), "k1", &val));
ASSERT_EQ("val", val);
}
}
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
+5 -5
View File
@@ -202,7 +202,6 @@ TEST_F(DBMergeOperatorTest, MergeErrorOnIteration) {
VerifyDBInternal({{"k1", "v1"}, {"k2", "corrupted"}, {"k2", "v2"}});
}
#ifndef ROCKSDB_LITE
TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
// This is like a mini-stress test dedicated to `OpFailureScope::kMustMerge`.
@@ -232,7 +231,9 @@ TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
{
std::string value;
ASSERT_OK(db_->Get(ReadOptions(), "k0", &value));
ASSERT_TRUE(db_->Get(ReadOptions(), "k1", &value).IsCorruption());
Status s = db_->Get(ReadOptions(), "k1", &value);
ASSERT_TRUE(s.IsCorruption());
ASSERT_EQ(Status::SubCode::kMergeOperatorFailed, s.subcode());
ASSERT_OK(db_->Get(ReadOptions(), "k2", &value));
}
@@ -244,6 +245,8 @@ TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
ASSERT_EQ("k0", iter->key());
iter->Next();
ASSERT_TRUE(iter->status().IsCorruption());
ASSERT_EQ(Status::SubCode::kMergeOperatorFailed,
iter->status().subcode());
iter->SeekToLast();
ASSERT_TRUE(iter->Valid());
@@ -355,7 +358,6 @@ TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
}
}
#endif // ROCKSDB_LITE
class MergeOperatorPinningTest : public DBMergeOperatorTest,
public testing::WithParamInterface<bool> {
@@ -368,7 +370,6 @@ class MergeOperatorPinningTest : public DBMergeOperatorTest,
INSTANTIATE_TEST_CASE_P(MergeOperatorPinningTest, MergeOperatorPinningTest,
::testing::Bool());
#ifndef ROCKSDB_LITE
TEST_P(MergeOperatorPinningTest, OperandsMultiBlocks) {
Options options = CurrentOptions();
BlockBasedTableOptions table_options;
@@ -639,7 +640,6 @@ TEST_F(DBMergeOperatorTest, TailingIteratorMemtableUnrefedBySomeoneElse) {
EXPECT_TRUE(pushed_first_operand);
EXPECT_TRUE(stepped_to_next_operand);
}
#endif // ROCKSDB_LITE
TEST_F(DBMergeOperatorTest, SnapshotCheckerAndReadCallback) {
Options options = CurrentOptions();
-4
View File
@@ -29,7 +29,6 @@ class DBOptionsTest : public DBTestBase {
public:
DBOptionsTest() : DBTestBase("db_options_test", /*env_do_fsync=*/true) {}
#ifndef ROCKSDB_LITE
std::unordered_map<std::string, std::string> GetMutableDBOptionsMap(
const DBOptions& options) {
std::string options_str;
@@ -76,7 +75,6 @@ class DBOptionsTest : public DBTestBase {
auto sanitized_options = SanitizeOptions(dbname_, db_options);
return GetMutableDBOptionsMap(sanitized_options);
}
#endif // ROCKSDB_LITE
};
TEST_F(DBOptionsTest, ImmutableTrackAndVerifyWalsInManifest) {
@@ -112,7 +110,6 @@ TEST_F(DBOptionsTest, ImmutableVerifySstUniqueIdInManifest) {
}
// RocksDB lite don't support dynamic options.
#ifndef ROCKSDB_LITE
TEST_F(DBOptionsTest, AvoidUpdatingOptions) {
Options options;
@@ -1148,7 +1145,6 @@ TEST_F(DBOptionsTest, ChangeCompression) {
SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
TEST_F(DBOptionsTest, BottommostCompressionOptsWithFallbackType) {
// Verify the bottommost compression options still take effect even when the
-6
View File
@@ -55,7 +55,6 @@ class DBPropertiesTest : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, Empty) {
do {
Options options;
@@ -1112,7 +1111,6 @@ TEST_F(DBPropertiesTest, EstimateCompressionRatio) {
ASSERT_GT(CompressionRatioAtLevel(1), 10.0);
}
#endif // ROCKSDB_LITE
class CountingUserTblPropCollector : public TablePropertiesCollector {
public:
@@ -1263,7 +1261,6 @@ class BlockCountingTablePropertiesCollectorFactory
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
Options options = CurrentOptions();
options.level0_file_num_compaction_trigger = (1 << 30);
@@ -1303,7 +1300,6 @@ TEST_F(DBPropertiesTest, GetUserDefinedTableProperties) {
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
ASSERT_GT(collector_factory->num_created_, 0U);
}
#endif // ROCKSDB_LITE
TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
Options options = CurrentOptions();
@@ -1365,7 +1361,6 @@ TEST_F(DBPropertiesTest, UserDefinedTablePropertiesContext) {
ASSERT_GT(collector_factory->num_created_, 0U);
}
#ifndef ROCKSDB_LITE
TEST_F(DBPropertiesTest, TablePropertiesNeedCompactTest) {
Random rnd(301);
@@ -2195,7 +2190,6 @@ TEST_F(DBPropertiesTest, TableMetaIndexKeys) {
} while (ChangeOptions());
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-10
View File
@@ -27,9 +27,6 @@ class DBRangeDelTest : public DBTestBase {
}
};
// PlainTableFactory, WriteBatchWithIndex, and NumTableFilesAtLevel() are not
// supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, NonBlockBasedTableNotSupported) {
// TODO: figure out why MmapReads trips the iterator pinning assertion in
// RangeDelAggregator. Ideally it would be supported; otherwise it should at
@@ -298,7 +295,6 @@ TEST_F(DBRangeDelTest, CompactRangeDelsSameStartKey) {
ASSERT_TRUE(db_->Get(ReadOptions(), "b1", &value).IsNotFound());
}
}
#endif // ROCKSDB_LITE
TEST_F(DBRangeDelTest, FlushRemovesCoveredKeys) {
const int kNum = 300, kRangeBegin = 50, kRangeEnd = 250;
@@ -335,8 +331,6 @@ TEST_F(DBRangeDelTest, FlushRemovesCoveredKeys) {
db_->ReleaseSnapshot(snapshot);
}
// NumTableFilesAtLevel() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, CompactionRemovesCoveredKeys) {
const int kNumPerFile = 100, kNumFiles = 4;
Options opts = CurrentOptions();
@@ -517,7 +511,6 @@ TEST_F(DBRangeDelTest, ValidUniversalSubcompactionBoundaries) {
std::numeric_limits<uint64_t>::max() /* max_file_num_to_ignore */,
"" /*trim_ts*/));
}
#endif // ROCKSDB_LITE
TEST_F(DBRangeDelTest, CompactionRemovesCoveredMergeOperands) {
const int kNumPerFile = 3, kNumFiles = 3;
@@ -589,8 +582,6 @@ TEST_F(DBRangeDelTest, PutDeleteRangeMergeFlush) {
ASSERT_EQ(expected, actual);
}
// NumTableFilesAtLevel() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBRangeDelTest, ObsoleteTombstoneCleanup) {
// During compaction to bottommost level, verify range tombstones older than
// the oldest snapshot are removed, while others are preserved.
@@ -3024,7 +3015,6 @@ TEST_F(DBRangeDelTest, DoubleCountRangeTombstoneCompensatedSize) {
db_->ReleaseSnapshot(snapshot);
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-15
View File
@@ -96,19 +96,10 @@ std::string GetTestNameSuffix(
return oss.str();
}
#ifndef ROCKSDB_LITE
INSTANTIATE_TEST_CASE_P(DBRateLimiterOnReadTest, DBRateLimiterOnReadTest,
::testing::Combine(::testing::Bool(), ::testing::Bool(),
::testing::Bool()),
GetTestNameSuffix);
#else // ROCKSDB_LITE
// Cannot use direct I/O in lite mode.
INSTANTIATE_TEST_CASE_P(DBRateLimiterOnReadTest, DBRateLimiterOnReadTest,
::testing::Combine(::testing::Values(false),
::testing::Bool(),
::testing::Bool()),
GetTestNameSuffix);
#endif // ROCKSDB_LITE
TEST_P(DBRateLimiterOnReadTest, Get) {
if (use_direct_io_ && !IsDirectIOSupported()) {
@@ -234,7 +225,6 @@ TEST_P(DBRateLimiterOnReadTest, Iterator) {
ASSERT_EQ(expected, options_.rate_limiter->GetTotalRequests(Env::IO_USER));
}
#if !defined(ROCKSDB_LITE)
TEST_P(DBRateLimiterOnReadTest, VerifyChecksum) {
if (use_direct_io_ && !IsDirectIOSupported()) {
@@ -264,7 +254,6 @@ TEST_P(DBRateLimiterOnReadTest, VerifyFileChecksums) {
ASSERT_EQ(expected, options_.rate_limiter->GetTotalRequests(Env::IO_USER));
}
#endif // !defined(ROCKSDB_LITE)
class DBRateLimiterOnWriteTest : public DBTestBase {
public:
@@ -319,10 +308,8 @@ TEST_F(DBRateLimiterOnWriteTest, Compact) {
// Pre-comaction:
// level-0 : `kNumFiles` SST files overlapping on [kStartKey, kEndKey]
#ifndef ROCKSDB_LITE
std::string files_per_level_pre_compaction = std::to_string(kNumFiles);
ASSERT_EQ(files_per_level_pre_compaction, FilesPerLevel(0 /* cf */));
#endif // !ROCKSDB_LITE
std::int64_t prev_total_request =
options_.rate_limiter->GetTotalRequests(Env::IO_TOTAL);
@@ -337,10 +324,8 @@ TEST_F(DBRateLimiterOnWriteTest, Compact) {
// Post-comaction:
// level-0 : 0 SST file
// level-1 : 1 SST file
#ifndef ROCKSDB_LITE
std::string files_per_level_post_compaction = "0,1";
ASSERT_EQ(files_per_level_post_compaction, FilesPerLevel(0 /* cf */));
#endif // !ROCKSDB_LITE
std::int64_t exepcted_compaction_request = 1;
EXPECT_EQ(actual_compaction_request, exepcted_compaction_request);
-4
View File
@@ -17,7 +17,6 @@ class DBReadOnlyTestWithTimestamp : public DBBasicTestWithTimestampBase {
: DBBasicTestWithTimestampBase("db_readonly_test_with_timestamp") {}
protected:
#ifndef ROCKSDB_LITE
void CheckDBOpenedAsCompactedDBWithOneLevel0File() {
VersionSet* const versions = dbfull()->GetVersionSet();
ASSERT_NE(versions, nullptr);
@@ -63,10 +62,8 @@ class DBReadOnlyTestWithTimestamp : public DBBasicTestWithTimestampBase {
ASSERT_TRUE(
storage_info->LevelFilesBrief(highest_non_empty_level).num_files > 0);
}
#endif // !ROCKSDB_LITE
};
#ifndef ROCKSDB_LITE
TEST_F(DBReadOnlyTestWithTimestamp, IteratorAndGetReadTimestampSizeMismatch) {
const int kNumKeysPerFile = 128;
const uint64_t kMaxKey = 1024;
@@ -949,7 +946,6 @@ TEST_F(DBReadOnlyTestWithTimestamp,
Close();
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
int main(int argc, char** argv) {
-2
View File
@@ -18,7 +18,6 @@
namespace ROCKSDB_NAMESPACE {
#ifndef ROCKSDB_LITE
class DBSecondaryTestBase : public DBBasicTestWithTimestampBase {
public:
explicit DBSecondaryTestBase(const std::string& dbname)
@@ -1682,7 +1681,6 @@ TEST_F(DBSecondaryTestWithTimestamp, Iterators) {
Close();
}
#endif //! ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-4
View File
@@ -24,7 +24,6 @@ class DBSSTTest : public DBTestBase {
DBSSTTest() : DBTestBase("db_sst_test", /*env_do_fsync=*/true) {}
};
#ifndef ROCKSDB_LITE
// A class which remembers the name of each flushed file.
class FlushedFileCollector : public EventListener {
public:
@@ -53,7 +52,6 @@ class FlushedFileCollector : public EventListener {
std::vector<std::string> flushed_files_;
std::mutex mutex_;
};
#endif // ROCKSDB_LITE
TEST_F(DBSSTTest, DontDeletePendingOutputs) {
Options options;
@@ -151,7 +149,6 @@ TEST_F(DBSSTTest, SkipCheckingSSTFileSizesOnDBOpen) {
ASSERT_EQ("choo", Get("pika"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBSSTTest, DontDeleteMovedFile) {
// This test triggers move compaction and verifies that the file is not
// deleted when it's part of move compaction
@@ -1856,7 +1853,6 @@ TEST_F(DBSSTTest, DBWithSFMForBlobFilesAtomicFlush) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-2
View File
@@ -157,7 +157,6 @@ TEST_F(DBStatisticsTest, ExcludeTickers) {
ASSERT_GT(options.statistics->getTickerCount(BYTES_READ), 0);
}
#ifndef ROCKSDB_LITE
TEST_F(DBStatisticsTest, VerifyChecksumReadStat) {
Options options = CurrentOptions();
@@ -204,7 +203,6 @@ TEST_F(DBStatisticsTest, VerifyChecksumReadStat) {
}
}
#endif // !ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-2
View File
@@ -24,7 +24,6 @@
#include "test_util/testutil.h"
#include "util/random.h"
#ifndef ROCKSDB_LITE
namespace ROCKSDB_NAMESPACE {
@@ -616,7 +615,6 @@ INSTANTIATE_TEST_CASE_P(DBTablePropertiesTest, DBTablePropertiesTest,
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE
int main(int argc, char** argv) {
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
-9
View File
@@ -10,7 +10,6 @@
// Introduction of SyncPoint effectively disabled building and running this test
// in Release build.
// which is a pity, it is a good test
#if !defined(ROCKSDB_LITE)
#include "db/db_test_util.h"
#include "db/forward_iterator.h"
@@ -241,7 +240,6 @@ TEST_P(DBTestTailingIterator, TailingIteratorTrimSeekToNext) {
iterh = nullptr;
BlockBasedTableOptions table_options;
table_options.no_block_cache = true;
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
ReopenWithColumnFamilies({"default", "pikachu"}, options);
read_options.read_tier = kBlockCacheTier;
@@ -589,16 +587,9 @@ TEST_P(DBTestTailingIterator, SeekToFirstWithUpperBoundBug) {
} // namespace ROCKSDB_NAMESPACE
#endif // !defined(ROCKSDB_LITE)
int main(int argc, char** argv) {
#if !defined(ROCKSDB_LITE)
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
#else
(void)argc;
(void)argv;
return 0;
#endif
}
+2 -64
View File
@@ -129,8 +129,6 @@ TEST_F(DBTest, MockEnvTest) {
ASSERT_TRUE(!iterator->Valid());
delete iterator;
// TEST_FlushMemTable() is not supported in ROCKSDB_LITE
#ifndef ROCKSDB_LITE
DBImpl* dbi = static_cast_with_check<DBImpl>(db);
ASSERT_OK(dbi->TEST_FlushMemTable());
@@ -139,14 +137,10 @@ TEST_F(DBTest, MockEnvTest) {
ASSERT_OK(db->Get(ReadOptions(), keys[i], &res));
ASSERT_TRUE(res == vals[i]);
}
#endif // ROCKSDB_LITE
delete db;
}
// NewMemEnv returns nullptr in ROCKSDB_LITE since class InMemoryEnv isn't
// defined.
#ifndef ROCKSDB_LITE
TEST_F(DBTest, MemEnvTest) {
std::unique_ptr<Env> env{NewMemEnv(Env::Default())};
Options options;
@@ -199,7 +193,6 @@ TEST_F(DBTest, MemEnvTest) {
}
delete db;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, WriteEmptyBatch) {
Options options = CurrentOptions();
@@ -458,7 +451,6 @@ TEST_F(DBTest, MixedSlowdownOptionsStop) {
wo.no_slowdown = true;
ASSERT_OK(dbfull()->Put(wo, "foo3", "bar"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, LevelLimitReopen) {
Options options = CurrentOptions();
@@ -481,9 +473,7 @@ TEST_F(DBTest, LevelLimitReopen) {
options.max_bytes_for_level_multiplier_additional.resize(10, 1);
ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, LevelReopenWithFIFO) {
const int kLevelCount = 4;
const int kKeyCount = 5;
@@ -610,7 +600,6 @@ TEST_F(DBTest, LevelReopenWithFIFO) {
ASSERT_EQ(expected_files_per_level_after_fifo[i], FilesPerLevel(kCF));
}
}
#endif // !ROCKSDB_LITE
TEST_F(DBTest, PutSingleDeleteGet) {
do {
@@ -857,9 +846,7 @@ TEST_F(DBTest, DISABLED_VeryLargeValue) {
ASSERT_OK(Put(key2, raw));
dbfull()->TEST_WaitForFlushMemTable();
#ifndef ROCKSDB_LITE
ASSERT_EQ(1, NumTableFilesAtLevel(0));
#endif // !ROCKSDB_LITE
std::string value;
Status s = db_->Get(ReadOptions(), key1, &value);
@@ -936,7 +923,6 @@ TEST_F(DBTest, WrongLevel0Config) {
ASSERT_OK(DB::Open(options, dbname_, &db_));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, GetOrderedByLevels) {
do {
CreateAndReopenWithCF({"pikachu"}, CurrentOptions());
@@ -1007,7 +993,6 @@ TEST_F(DBTest, GetEncountersEmptyLevel) {
ASSERT_EQ(NumTableFilesAtLevel(0, 1), 1); // XXX
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FlushMultipleMemtable) {
do {
@@ -1027,7 +1012,6 @@ TEST_F(DBTest, FlushMultipleMemtable) {
ASSERT_OK(Flush(1));
} while (ChangeCompactOptions());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, FlushSchedule) {
Options options = CurrentOptions();
options.disable_auto_compactions = true;
@@ -1072,7 +1056,6 @@ TEST_F(DBTest, FlushSchedule) {
ASSERT_LE(pikachu_tables, static_cast<uint64_t>(10));
ASSERT_GT(pikachu_tables, static_cast<uint64_t>(0));
}
#endif // ROCKSDB_LITE
namespace {
class KeepFilter : public CompactionFilter {
@@ -1137,14 +1120,12 @@ class DelayFilterFactory : public CompactionFilterFactory {
};
} // anonymous namespace
#ifndef ROCKSDB_LITE
static std::string CompressibleString(Random* rnd, int len) {
std::string r;
test::CompressibleString(rnd, 0.8, len, &r);
return r;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FailMoreDbPaths) {
Options options = CurrentOptions();
@@ -1266,7 +1247,6 @@ void CheckLiveFilesMeta(
}
}
#ifndef ROCKSDB_LITE
void AddBlobFile(const ColumnFamilyHandle* cfh, uint64_t blob_file_number,
uint64_t total_blob_count, uint64_t total_blob_bytes,
const std::string& checksum_method,
@@ -1558,9 +1538,7 @@ TEST_F(DBTest, DISABLED_RepeatedWritesToSameKey) {
}
} while (ChangeCompactOptions());
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
static bool Between(uint64_t val, uint64_t low, uint64_t high) {
bool result = (val >= low) && (val <= high);
if (!result) {
@@ -1957,9 +1935,7 @@ TEST_F(DBTest, ApproximateSizes_MixOfSmallAndLarge) {
// ApproximateOffsetOf() is not yet implemented in plain table format.
} while (ChangeOptions(kSkipPlainTable));
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, Snapshot) {
env_->SetMockSleep();
anon::OptionsOverride options_override;
@@ -2081,7 +2057,6 @@ TEST_F(DBTest, HiddenValuesAreRemoved) {
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction |
kSkipPlainTable));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, UnremovableSingleDelete) {
// If we compact:
@@ -2132,7 +2107,6 @@ TEST_F(DBTest, UnremovableSingleDelete) {
kSkipMergePut));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DeletionMarkers1) {
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
@@ -2252,7 +2226,6 @@ TEST_F(DBTest, OverlapInLevel0) {
ASSERT_EQ("NOT_FOUND", Get(1, "600"));
} while (ChangeOptions(kSkipUniversalCompaction | kSkipFIFOCompaction));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, ComparatorCheck) {
class NewComparator : public Comparator {
@@ -2440,7 +2413,6 @@ TEST_F(DBTest, DestroyDBMetaDatabase) {
ASSERT_TRUE(!(DB::Open(options, metametadbname, &db)).ok());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, SnapshotFiles) {
do {
Options options = CurrentOptions();
@@ -2667,7 +2639,6 @@ TEST_F(DBTest, GetLiveBlobFiles) {
ASSERT_EQ(cfmd.blob_file_count, 1U);
ASSERT_EQ(cfmd.blob_file_size, bmd.blob_file_size);
}
#endif
TEST_F(DBTest, PurgeInfoLogs) {
Options options = CurrentOptions();
@@ -2715,7 +2686,6 @@ TEST_F(DBTest, PurgeInfoLogs) {
}
}
#ifndef ROCKSDB_LITE
// Multi-threaded test:
namespace {
@@ -2902,7 +2872,6 @@ INSTANTIATE_TEST_CASE_P(
::testing::Combine(
::testing::ValuesIn(MultiThreadedDBTest::GenerateOptionConfigs()),
::testing::Bool()));
#endif // ROCKSDB_LITE
// Group commit test:
#if !defined(OS_WIN)
@@ -3109,7 +3078,6 @@ class ModelDB : public DB {
return s;
}
#ifndef ROCKSDB_LITE
using DB::IngestExternalFile;
Status IngestExternalFile(
ColumnFamilyHandle* /*column_family*/,
@@ -3151,7 +3119,6 @@ class ModelDB : public DB {
std::size_t /*n*/, TablePropertiesCollection* /*props*/) override {
return Status();
}
#endif // ROCKSDB_LITE
using DB::KeyMayExist;
bool KeyMayExist(const ReadOptions& /*options*/,
@@ -3332,7 +3299,6 @@ class ModelDB : public DB {
Status DisableFileDeletions() override { return Status::OK(); }
Status EnableFileDeletions(bool /*force*/) override { return Status::OK(); }
#ifndef ROCKSDB_LITE
Status GetLiveFiles(std::vector<std::string>&, uint64_t* /*size*/,
bool /*flush_memtable*/ = true) override {
@@ -3376,7 +3342,6 @@ class ModelDB : public DB {
void GetColumnFamilyMetaData(ColumnFamilyHandle* /*column_family*/,
ColumnFamilyMetaData* /*metadata*/) override {}
#endif // ROCKSDB_LITE
Status GetDbIdentity(std::string& /*identity*/) const override {
return Status::OK();
@@ -3625,12 +3590,10 @@ TEST_F(DBTest, BlockBasedTablePrefixIndexTest) {
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#ifndef ROCKSDB_LITE
// Back to original
ASSERT_OK(dbfull()->SetOptions({{"prefix_extractor", "fixed:1"}}));
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#endif // !ROCKSDB_LITE
// Same if there's a problem initally loading prefix transform
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
@@ -3642,12 +3605,10 @@ TEST_F(DBTest, BlockBasedTablePrefixIndexTest) {
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#ifndef ROCKSDB_LITE
// Change again
ASSERT_OK(dbfull()->SetOptions({{"prefix_extractor", "fixed:2"}}));
ASSERT_EQ("v1", Get("k1"));
ASSERT_EQ("v2", Get("k2"));
#endif // !ROCKSDB_LITE
SyncPoint::GetInstance()->DisableProcessing();
// Reopen with no prefix extractor, make sure everything still works.
@@ -3769,7 +3730,6 @@ TEST_F(DBTest, ChecksumTest) {
ASSERT_EQ("h", Get("g"));
}
#ifndef ROCKSDB_LITE
TEST_P(DBTestWithParam, FIFOCompactionTest) {
for (int iter = 0; iter < 2; ++iter) {
// first iteration -- auto compaction
@@ -4141,9 +4101,7 @@ TEST_F(DBTest, FIFOCompactionWithTTLTest) {
options.compaction_options_fifo.max_table_files_size);
}
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
/*
* This test is not reliable enough as it heavily depends on disk behavior.
* Disable as it is flaky.
@@ -4324,7 +4282,6 @@ TEST_F(DBTest, ConcurrentMemtableNotSupported) {
ASSERT_NOK(db_->CreateColumnFamily(cf_options, "name", &handle));
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, SanitizeNumThreads) {
for (int attempt = 0; attempt < 2; attempt++) {
@@ -4478,7 +4435,6 @@ TEST_F(DBTest, ManualFlushWalAndWriteRace) {
ASSERT_EQ("value2", Get("foo2"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DynamicMemtableOptions) {
const uint64_t k64KB = 1 << 16;
const uint64_t k128KB = 1 << 17;
@@ -4632,7 +4588,6 @@ TEST_F(DBTest, DynamicMemtableOptions) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
#ifdef ROCKSDB_USING_THREAD_STATUS
namespace {
@@ -5082,7 +5037,6 @@ TEST_P(DBTestWithParam, PreShutdownCompactionMiddle) {
#endif // ROCKSDB_USING_THREAD_STATUS
#ifndef ROCKSDB_LITE
TEST_F(DBTest, FlushOnDestroy) {
WriteOptions wo;
wo.disableWAL = true;
@@ -5635,7 +5589,6 @@ TEST_F(DBTest, DynamicUniversalCompactionOptions) {
dbfull()->GetOptions().compaction_options_universal.allow_trivial_move,
false);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FileCreationRandomFailure) {
Options options;
@@ -5698,7 +5651,6 @@ TEST_F(DBTest, FileCreationRandomFailure) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, DynamicMiscOptions) {
// Test max_sequential_skip_in_iterations
@@ -5793,7 +5745,6 @@ TEST_F(DBTest, DynamicMiscOptions) {
&mutable_cf_options));
ASSERT_FALSE(mutable_cf_options.check_flush_compaction_key_order);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, L0L1L2AndUpHitCounter) {
const int kNumLevels = 3;
@@ -6012,7 +5963,6 @@ TEST_F(DBTest, MergeTestTime) {
#endif // ROCKSDB_USING_THREAD_STATUS
}
#ifndef ROCKSDB_LITE
TEST_P(DBTestWithParam, MergeCompactionTimeTest) {
SetPerfLevel(kEnableTime);
Options options = CurrentOptions();
@@ -6075,7 +6025,6 @@ TEST_P(DBTestWithParam, FilterCompactionTimeTest) {
TestGetTickerCount(options, FILTER_OPERATION_TOTAL_TIME));
delete itr;
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, TestLogCleanup) {
Options options = CurrentOptions();
@@ -6092,7 +6041,6 @@ TEST_F(DBTest, TestLogCleanup) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, EmptyCompactedDB) {
Options options = CurrentOptions();
options.max_open_files = -1;
@@ -6102,9 +6050,7 @@ TEST_F(DBTest, EmptyCompactedDB) {
ASSERT_TRUE(s.IsNotSupported());
Close();
}
#endif // ROCKSDB_LITE
#ifndef ROCKSDB_LITE
TEST_F(DBTest, SuggestCompactRangeTest) {
class CompactionFilterFactoryGetContext : public CompactionFilterFactory {
public:
@@ -6333,7 +6279,6 @@ TEST_F(DBTest, CompactRangeWithEmptyBottomLevel) {
ASSERT_EQ(NumTableFilesAtLevel(0), 0);
ASSERT_EQ(NumTableFilesAtLevel(1), kNumL0Files);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
const int kNumL0Files = 50;
@@ -6391,7 +6336,6 @@ TEST_F(DBTest, AutomaticConflictsWithManualCompaction) {
ASSERT_OK(dbfull()->TEST_WaitForCompact());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
Options options = CurrentOptions();
options.max_background_compactions = 1;
@@ -6451,7 +6395,6 @@ TEST_F(DBTest, CompactFilesShouldTriggerAutoCompaction) {
ASSERT_LE(cf_meta_data.levels[0].files.size(),
options.level0_file_num_compaction_trigger);
}
#endif // ROCKSDB_LITE
// Github issue #595
// Large write batch with column families
@@ -6662,7 +6605,7 @@ TEST_F(DBTest, HardLimit) {
sleeping_task_low.WaitUntilDone();
}
#if !defined(ROCKSDB_LITE) && !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
#if !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
class WriteStallListener : public EventListener {
public:
WriteStallListener() : condition_(WriteStallCondition::kNormal) {}
@@ -6904,8 +6847,7 @@ TEST_F(DBTest, LastWriteBufferDelay) {
sleeping_task.WakeUp();
sleeping_task.WaitUntilDone();
}
#endif // !defined(ROCKSDB_LITE) &&
// !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
#endif // !defined(ROCKSDB_DISABLE_STALL_NOTIFICATION)
TEST_F(DBTest, FailWhenCompressionNotSupportedTest) {
CompressionType compressions[] = {kZlibCompression, kBZip2Compression,
@@ -6941,7 +6883,6 @@ TEST_F(DBTest, CreateColumnFamilyShouldFailOnIncompatibleOptions) {
delete handle;
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, RowCache) {
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -7073,7 +7014,6 @@ TEST_F(DBTest, ReusePinnableSlice) {
1);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, DeletingOldWalAfterDrop) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
@@ -7199,7 +7139,6 @@ TEST_F(DBTest, LargeBlockSizeTest) {
ASSERT_NOK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest, CreationTimeOfOldestFile) {
const int kNumKeysPerFile = 32;
@@ -7388,7 +7327,6 @@ TEST_F(DBTest, ShuttingDownNotBlockStalledWrites) {
thd.join();
}
#endif
} // namespace ROCKSDB_NAMESPACE
-138
View File
@@ -48,7 +48,6 @@ class DBTest2 : public DBTestBase {
}
};
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, OpenForReadOnly) {
DB* db_ptr = nullptr;
std::string dbname = test::PerThreadDBPath("db_readonly");
@@ -106,66 +105,6 @@ TEST_F(DBTest2, OpenForReadOnlyWithColumnFamilies) {
ASSERT_NOK(env_->FileExists(dbname));
}
class TestReadOnlyWithCompressedCache
: public DBTestBase,
public testing::WithParamInterface<std::tuple<int, bool>> {
public:
TestReadOnlyWithCompressedCache()
: DBTestBase("test_readonly_with_compressed_cache",
/*env_do_fsync=*/true) {
max_open_files_ = std::get<0>(GetParam());
use_mmap_ = std::get<1>(GetParam());
}
int max_open_files_;
bool use_mmap_;
};
TEST_P(TestReadOnlyWithCompressedCache, ReadOnlyWithCompressedCache) {
if (use_mmap_ && !IsMemoryMappedAccessSupported()) {
ROCKSDB_GTEST_SKIP("Test requires MMAP support");
return;
}
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Put("foo2", "barbarbarbarbarbarbarbar"));
ASSERT_OK(Flush());
DB* db_ptr = nullptr;
Options options = CurrentOptions();
options.allow_mmap_reads = use_mmap_;
options.max_open_files = max_open_files_;
options.compression = kSnappyCompression;
BlockBasedTableOptions table_options;
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
table_options.no_block_cache = true;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
options.statistics = CreateDBStatistics();
ASSERT_OK(DB::OpenForReadOnly(options, dbname_, &db_ptr));
std::string v;
ASSERT_OK(db_ptr->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("bar", v);
ASSERT_EQ(0, options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
ASSERT_OK(db_ptr->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("bar", v);
if (Snappy_Supported()) {
if (use_mmap_) {
ASSERT_EQ(0,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
} else {
ASSERT_EQ(1,
options.statistics->getTickerCount(BLOCK_CACHE_COMPRESSED_HIT));
}
}
delete db_ptr;
}
INSTANTIATE_TEST_CASE_P(TestReadOnlyWithCompressedCache,
TestReadOnlyWithCompressedCache,
::testing::Combine(::testing::Values(-1, 100),
::testing::Bool()));
class PartitionedIndexTestListener : public EventListener {
public:
void OnFlushCompleted(DB* /*db*/, const FlushJobInfo& info) override {
@@ -206,7 +145,6 @@ TEST_F(DBTest2, PartitionedIndexUserToInternalKey) {
}
}
#endif // ROCKSDB_LITE
class PrefixFullBloomWithReverseComparator
: public DBTestBase,
@@ -350,7 +288,6 @@ TEST_F(DBTest2, MaxSuccessiveMergesChangeWithDBRecovery) {
Reopen(options);
}
#ifndef ROCKSDB_LITE
class DBTestSharedWriteBufferAcrossCFs
: public DBTestBase,
public testing::WithParamInterface<std::tuple<bool, bool>> {
@@ -1604,9 +1541,7 @@ TEST_P(PresetCompressionDictTest, CompactNonBottommost) {
}
ASSERT_OK(Flush());
}
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
uint64_t prev_compression_dict_bytes_inserted =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT);
@@ -1614,9 +1549,7 @@ TEST_P(PresetCompressionDictTest, CompactNonBottommost) {
// file is not bottommost due to the existing L2 file covering the same key-
// range.
ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// We can use `BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT` to detect whether a
// compression dictionary exists since dictionaries would be preloaded when
// the compaction finishes.
@@ -1680,17 +1613,13 @@ TEST_P(PresetCompressionDictTest, CompactBottommost) {
}
ASSERT_OK(Flush());
}
#ifndef ROCKSDB_LITE
ASSERT_EQ("2", FilesPerLevel(0));
#endif // ROCKSDB_LITE
uint64_t prev_compression_dict_bytes_inserted =
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT);
CompactRangeOptions cro;
ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
ASSERT_GT(
TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT),
prev_compression_dict_bytes_inserted);
@@ -2057,7 +1986,6 @@ TEST_F(DBTest2, CompactionStall) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, FirstSnapshotTest) {
Options options;
@@ -2074,7 +2002,6 @@ TEST_F(DBTest2, FirstSnapshotTest) {
db_->ReleaseSnapshot(s1);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, DuplicateSnapshot) {
Options options;
options = CurrentOptions(options);
@@ -2106,7 +2033,6 @@ TEST_F(DBTest2, DuplicateSnapshot) {
db_->ReleaseSnapshot(s);
}
}
#endif // ROCKSDB_LITE
class PinL0IndexAndFilterBlocksTest
: public DBTestBase,
@@ -2351,7 +2277,6 @@ INSTANTIATE_TEST_CASE_P(PinL0IndexAndFilterBlocksTest,
std::make_tuple(false, false),
std::make_tuple(false, true)));
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, MaxCompactionBytesTest) {
Options options = CurrentOptions();
options.memtable_factory.reset(test::NewSpecialSkipListFactory(
@@ -2625,7 +2550,6 @@ TEST_F(DBTest2, PersistentCache) {
new MockPersistentCache(type, 10 * 1024));
table_options.no_block_cache = true;
table_options.block_cache = bsize ? NewLRUCache(bsize) : nullptr;
table_options.block_cache_compressed = nullptr;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
DestroyAndReopen(options);
@@ -2718,7 +2642,6 @@ TEST_F(DBTest2, SyncPointMarker) {
ASSERT_EQ(sync_point_called.load(), 1);
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
#endif
size_t GetEncodedEntrySize(size_t key_size, size_t value_size) {
std::string buffer;
@@ -2925,7 +2848,6 @@ TEST_F(DBTest2, ReadAmpBitmapLiveInCacheAfterDBClose) {
}
#endif // !OS_SOLARIS
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, AutomaticCompactionOverlapManualCompaction) {
Options options = CurrentOptions();
options.num_levels = 3;
@@ -3206,9 +3128,7 @@ TEST_F(DBTest2, PausingManualCompaction3) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::Run():PausingManualCompaction:1",
@@ -3222,18 +3142,14 @@ TEST_F(DBTest2, PausingManualCompaction3) {
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
// As manual compaction disabled, not even reach sync point
ASSERT_EQ(run_manual_compactions, 0);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"CompactionJob::Run():PausingManualCompaction:1");
dbfull()->EnableManualCompaction();
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3262,9 +3178,7 @@ TEST_F(DBTest2, PausingManualCompaction4) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
"CompactionJob::Run():PausingManualCompaction:2", [&](void* arg) {
@@ -3291,17 +3205,13 @@ TEST_F(DBTest2, PausingManualCompaction4) {
.IsManualCompactionPaused());
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
ASSERT_EQ(run_manual_compactions, 1);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
"CompactionJob::Run():PausingManualCompaction:2");
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3334,9 +3244,7 @@ TEST_F(DBTest2, CancelManualCompaction1) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
int run_manual_compactions = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
@@ -3360,9 +3268,7 @@ TEST_F(DBTest2, CancelManualCompaction1) {
// E.g. we should call the compaction function exactly one time.
ASSERT_EQ(compactions_run, 0);
ASSERT_EQ(run_manual_compactions, 0);
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
compactions_run = 0;
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearCallBack(
@@ -3393,9 +3299,7 @@ TEST_F(DBTest2, CancelManualCompaction1) {
compact_options.canceled->store(false, std::memory_order_relaxed);
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3429,9 +3333,7 @@ TEST_F(DBTest2, CancelManualCompaction2) {
DestroyAndReopen(options);
generate_files();
#ifndef ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
@@ -3479,9 +3381,7 @@ TEST_F(DBTest2, CancelManualCompaction2) {
compact_options.canceled->store(false, std::memory_order_relaxed);
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
}
@@ -3709,7 +3609,6 @@ TEST_F(DBTest2, OptimizeForSmallDB) {
value.Reset();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, IterRaceFlush1) {
ASSERT_OK(Put("foo", "v1"));
@@ -3992,7 +3891,6 @@ TEST_F(DBTest2, LowPriWrite) {
ASSERT_EQ(1, rate_limit_count.load());
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, RateLimitedCompactionReads) {
// compaction input has 512KB data
const int kNumKeysPerFile = 128;
@@ -4073,7 +3971,6 @@ TEST_F(DBTest2, RateLimitedCompactionReads) {
}
}
}
#endif // ROCKSDB_LITE
// Make sure DB can be reopen with reduced number of levels, given no file
// is on levels higher than the new num_levels.
@@ -4086,21 +3983,15 @@ TEST_F(DBTest2, ReduceLevel) {
ASSERT_OK(Put("foo", "bar"));
ASSERT_OK(Flush());
MoveFilesToLevel(6);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
CompactRangeOptions compact_options;
compact_options.change_level = true;
compact_options.target_level = 1;
ASSERT_OK(dbfull()->CompactRange(compact_options, nullptr, nullptr));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
options.num_levels = 3;
Reopen(options);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel());
#endif // !ROCKSDB_LITE
}
// Test that ReadCallback is actually used in both memtbale and sst tables
@@ -4135,18 +4026,14 @@ TEST_F(DBTest2, ReadCallbackTest) {
}
ASSERT_OK(Flush());
MoveFilesToLevel(6);
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
for (; i < 30; i++) {
ASSERT_OK(Put(key, value + std::to_string(i)));
auto snapshot = dbfull()->GetSnapshot();
snapshots.push_back(snapshot);
}
ASSERT_OK(Flush());
#ifndef ROCKSDB_LITE
ASSERT_EQ("1,0,0,0,0,0,2", FilesPerLevel());
#endif // !ROCKSDB_LITE
// And also add some values to the memtable
for (; i < 40; i++) {
ASSERT_OK(Put(key, value + std::to_string(i)));
@@ -4188,7 +4075,6 @@ TEST_F(DBTest2, ReadCallbackTest) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, LiveFilesOmitObsoleteFiles) {
// Regression test for race condition where an obsolete file is returned to
@@ -5257,7 +5143,6 @@ TEST_F(DBTest2, TraceWithFilter) {
ASSERT_EQ(count, 6);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, PinnableSliceAndMmapReads) {
Options options = CurrentOptions();
@@ -5288,7 +5173,6 @@ TEST_F(DBTest2, PinnableSliceAndMmapReads) {
// compaction. It crashes if it does.
ASSERT_EQ(pinned_value.ToString(), "bar");
#ifndef ROCKSDB_LITE
pinned_value.Reset();
// Unsafe to pin mmap files when they could be kicked out of table cache
Close();
@@ -5306,7 +5190,6 @@ TEST_F(DBTest2, PinnableSliceAndMmapReads) {
ASSERT_EQ(Get("foo", &pinned_value), Status::OK());
ASSERT_TRUE(pinned_value.IsPinned());
ASSERT_EQ(pinned_value.ToString(), "bar");
#endif
}
TEST_F(DBTest2, DISABLED_IteratorPinnedMemory) {
@@ -5539,7 +5422,6 @@ TEST_F(DBTest2, TestGetColumnFamilyHandleUnlocked) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, TestCompactFiles) {
// Setup sync point dependency to reproduce the race condition of
// DBImpl::GetColumnFamilyHandleUnlocked
@@ -5607,7 +5489,6 @@ TEST_F(DBTest2, TestCompactFiles) {
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, MultiDBParallelOpenTest) {
const int kNumDbs = 2;
@@ -5793,7 +5674,6 @@ TEST_F(DBTest2, PrefixBloomFilteredOut) {
delete iter;
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, RowCacheSnapshot) {
Options options = CurrentOptions();
options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -5837,7 +5717,6 @@ TEST_F(DBTest2, RowCacheSnapshot) {
db_->ReleaseSnapshot(s2);
db_->ReleaseSnapshot(s3);
}
#endif // ROCKSDB_LITE
// When DB is reopened with multiple column families, the manifest file
// is written after the first CF is flushed, and it is written again
@@ -6027,9 +5906,7 @@ TEST_F(DBTest2, SameSmallestInSameLevel) {
ASSERT_OK(db_->Merge(WriteOptions(), "key", "8"));
ASSERT_OK(Flush());
ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,4,1", FilesPerLevel());
#endif // ROCKSDB_LITE
ASSERT_EQ("2,3,4,5,6,7,8", Get("key"));
}
@@ -6162,11 +6039,7 @@ TEST_F(DBTest2, ChangePrefixExtractor) {
// Sometimes filter is checked based on upper bound. Assert counters
// for that case. Otherwise, only check data correctness.
#ifndef ROCKSDB_LITE
bool expect_filter_check = !use_partitioned_filter;
#else
bool expect_filter_check = false;
#endif
table_options.partition_filters = use_partitioned_filter;
if (use_partitioned_filter) {
table_options.index_type =
@@ -6364,7 +6237,6 @@ TEST_F(DBTest2, BlockBasedTablePrefixGetIndexNotFound) {
ASSERT_EQ("ok", Get("b1"));
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, AutoPrefixMode1) {
do {
// create a DB with block prefix index
@@ -7242,7 +7114,6 @@ TEST_F(DBTest2, FileTemperatureManifestFixup) {
std::vector<ColumnFamilyDescriptor> column_families;
for (size_t i = 0; i < handles_.size(); ++i) {
ColumnFamilyDescriptor cfdescriptor;
// GetDescriptor is not implemented for ROCKSDB_LITE
handles_[i]->GetDescriptor(&cfdescriptor).PermitUncheckedError();
column_families.push_back(cfdescriptor);
}
@@ -7282,7 +7153,6 @@ TEST_F(DBTest2, FileTemperatureManifestFixup) {
Close();
}
#endif // ROCKSDB_LITE
// WAL recovery mode is WALRecoveryMode::kPointInTimeRecovery.
TEST_F(DBTest2, PointInTimeRecoveryWithIOErrorWhileReadingWal) {
@@ -7338,7 +7208,6 @@ TEST_F(DBTest2, PointInTimeRecoveryWithSyncFailureInCFCreation) {
ReopenWithColumnFamilies({"default", "test1", "test2"}, options);
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, SortL0FilesByEpochNumber) {
Options options = CurrentOptions();
options.num_levels = 1;
@@ -7548,7 +7417,6 @@ TEST_F(DBTest2, RecoverEpochNumber) {
}
}
#endif // ROCKSDB_LITE
TEST_F(DBTest2, RenameDirectory) {
Options options = CurrentOptions();
@@ -7626,9 +7494,7 @@ TEST_F(DBTest2, SstUniqueIdVerifyBackwardCompatible) {
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// Reopen (with verification)
ASSERT_TRUE(options.verify_sst_unique_id_in_manifest);
@@ -7683,9 +7549,7 @@ TEST_F(DBTest2, SstUniqueIdVerify) {
}
ASSERT_OK(dbfull()->TEST_WaitForCompact());
#ifndef ROCKSDB_LITE
ASSERT_EQ("0,1", FilesPerLevel(0));
#endif // ROCKSDB_LITE
// Reopen with verification should fail
options.verify_sst_unique_id_in_manifest = true;
@@ -7808,7 +7672,6 @@ TEST_F(DBTest2, BestEffortsRecoveryWithSstUniqueIdVerification) {
}
}
#ifndef ROCKSDB_LITE
TEST_F(DBTest2, GetLatestSeqAndTsForKey) {
Destroy(last_options_);
@@ -7865,7 +7728,6 @@ TEST_F(DBTest2, GetLatestSeqAndTsForKey) {
// Verify that no read to SST files.
ASSERT_EQ(0, options.statistics->getTickerCount(GET_HIT_L0));
}
#endif // ROCKSDB_LITE
} // namespace ROCKSDB_NAMESPACE
-33
View File
@@ -70,7 +70,6 @@ DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
if (getenv("MEM_ENV")) {
mem_env_ = MockEnv::Create(base_env, base_env->GetSystemClock());
}
#ifndef ROCKSDB_LITE
if (getenv("ENCRYPTED_ENV")) {
std::shared_ptr<EncryptionProvider> provider;
std::string provider_id = getenv("ENCRYPTED_ENV");
@@ -82,7 +81,6 @@ DBTestBase::DBTestBase(const std::string path, bool env_do_fsync)
&provider));
encrypted_env_ = NewEncryptedEnv(mem_env_ ? mem_env_ : base_env, provider);
}
#endif // !ROCKSDB_LITE
env_ = new SpecialEnv(encrypted_env_ ? encrypted_env_
: (mem_env_ ? mem_env_ : base_env));
env_->SetBackgroundThreads(1, Env::LOW);
@@ -124,22 +122,6 @@ DBTestBase::~DBTestBase() {
}
bool DBTestBase::ShouldSkipOptions(int option_config, int skip_mask) {
#ifdef ROCKSDB_LITE
// These options are not supported in ROCKSDB_LITE
if (option_config == kHashSkipList ||
option_config == kPlainTableFirstBytePrefix ||
option_config == kPlainTableCappedPrefix ||
option_config == kPlainTableCappedPrefixNonMmap ||
option_config == kPlainTableAllBytesPrefix ||
option_config == kVectorRep || option_config == kHashLinkList ||
option_config == kUniversalCompaction ||
option_config == kUniversalCompactionMultiLevel ||
option_config == kUniversalSubcompactions ||
option_config == kFIFOCompaction ||
option_config == kConcurrentSkipList) {
return true;
}
#endif
if ((skip_mask & kSkipUniversalCompaction) &&
(option_config == kUniversalCompaction ||
@@ -377,7 +359,6 @@ Options DBTestBase::GetOptions(
bool can_allow_mmap = IsMemoryMappedAccessSupported();
switch (option_config) {
#ifndef ROCKSDB_LITE
case kHashSkipList:
options.prefix_extractor.reset(NewFixedPrefixTransform(1));
options.memtable_factory.reset(NewHashSkipListRepFactory(16));
@@ -431,7 +412,6 @@ Options DBTestBase::GetOptions(
SetupSyncPointsToMockDirectIO();
break;
}
#endif // ROCKSDB_LITE
case kMergePut:
options.merge_operator = MergeOperators::CreatePutOperator();
break;
@@ -480,10 +460,6 @@ Options DBTestBase::GetOptions(
options.compaction_style = kCompactionStyleUniversal;
options.num_levels = 8;
break;
case kCompressedBlockCache:
options.allow_mmap_writes = can_allow_mmap;
table_options.block_cache_compressed = NewLRUCache(8 * 1024 * 1024);
break;
case kInfiniteMaxOpenFiles:
options.max_open_files = -1;
break;
@@ -706,7 +682,6 @@ void DBTestBase::Destroy(const Options& options, bool delete_cf_paths) {
if (delete_cf_paths) {
for (size_t i = 0; i < handles_.size(); ++i) {
ColumnFamilyDescriptor cfdescriptor;
// GetDescriptor is not implemented for ROCKSDB_LITE
handles_[i]->GetDescriptor(&cfdescriptor).PermitUncheckedError();
column_families.push_back(cfdescriptor);
}
@@ -1047,7 +1022,6 @@ std::string DBTestBase::AllEntriesFor(const Slice& user_key, int cf) {
return result;
}
#ifndef ROCKSDB_LITE
int DBTestBase::NumSortedRuns(int cf) {
ColumnFamilyMetaData cf_meta;
if (cf == 0) {
@@ -1166,7 +1140,6 @@ std::string DBTestBase::FilesPerLevel(int cf) {
return result;
}
#endif // !ROCKSDB_LITE
std::vector<uint64_t> DBTestBase::GetBlobFileNumbers() {
VersionSet* const versions = dbfull()->GetVersionSet();
@@ -1284,7 +1257,6 @@ void DBTestBase::MoveFilesToLevel(int level, int cf) {
}
}
#ifndef ROCKSDB_LITE
void DBTestBase::DumpFileCounts(const char* label) {
fprintf(stderr, "---\n%s:\n", label);
fprintf(stderr, "maxoverlap: %" PRIu64 "\n",
@@ -1296,7 +1268,6 @@ void DBTestBase::DumpFileCounts(const char* label) {
}
}
}
#endif // !ROCKSDB_LITE
std::string DBTestBase::DumpSSTableList() {
std::string property;
@@ -1622,7 +1593,6 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
}
if (tailing_iter) {
#ifndef ROCKSDB_LITE
// Tailing iterator
int iter_cnt = 0;
ReadOptions ro;
@@ -1651,7 +1621,6 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
}
delete iter;
#endif // ROCKSDB_LITE
}
if (total_reads_res) {
@@ -1679,7 +1648,6 @@ void DBTestBase::VerifyDBInternal(
iter->~InternalIterator();
}
#ifndef ROCKSDB_LITE
uint64_t DBTestBase::GetNumberOfSstFilesForColumnFamily(
DB* db, std::string column_family_name) {
@@ -1700,7 +1668,6 @@ uint64_t DBTestBase::GetSstSizeHelper(Temperature temperature) {
&prop));
return static_cast<uint64_t>(std::atoi(prop.c_str()));
}
#endif // ROCKSDB_LITE
void VerifySstUniqueIds(const TablePropertiesCollection& props) {
ASSERT_FALSE(props.empty()); // suspicious test if empty
+11 -94
View File
@@ -695,7 +695,6 @@ class SpecialEnv : public EnvWrapper {
bool no_slowdown_;
};
#ifndef ROCKSDB_LITE
class FileTemperatureTestFS : public FileSystemWrapper {
public:
explicit FileTemperatureTestFS(const std::shared_ptr<FileSystem>& fs)
@@ -870,7 +869,6 @@ class FlushCounterListener : public EventListener {
ASSERT_EQ(expected_flush_reason.load(), flush_job_info.flush_reason);
}
};
#endif
// A test merge operator mimics put but also fails if one of merge operands is
// "corrupted", "corrupted_try_merge", or "corrupted_must_merge".
@@ -911,82 +909,6 @@ class TestPutOperator : public MergeOperator {
virtual const char* Name() const override { return "TestPutOperator"; }
};
// A wrapper around Cache that can easily be extended with instrumentation,
// etc.
class CacheWrapper : public Cache {
public:
explicit CacheWrapper(std::shared_ptr<Cache> target)
: target_(std::move(target)) {}
const char* Name() const override { return target_->Name(); }
Status Insert(const Slice& key, ObjectPtr value,
const CacheItemHelper* helper, size_t charge,
Handle** handle = nullptr,
Priority priority = Priority::LOW) override {
return target_->Insert(key, value, helper, charge, handle, priority);
}
Handle* Lookup(const Slice& key, const CacheItemHelper* helper,
CreateContext* create_context,
Priority priority = Priority::LOW, bool wait = true,
Statistics* stats = nullptr) override {
return target_->Lookup(key, helper, create_context, priority, wait, stats);
}
bool Ref(Handle* handle) override { return target_->Ref(handle); }
using Cache::Release;
bool Release(Handle* handle, bool erase_if_last_ref = false) override {
return target_->Release(handle, erase_if_last_ref);
}
ObjectPtr Value(Handle* handle) override { return target_->Value(handle); }
void Erase(const Slice& key) override { target_->Erase(key); }
uint64_t NewId() override { return target_->NewId(); }
void SetCapacity(size_t capacity) override { target_->SetCapacity(capacity); }
void SetStrictCapacityLimit(bool strict_capacity_limit) override {
target_->SetStrictCapacityLimit(strict_capacity_limit);
}
bool HasStrictCapacityLimit() const override {
return target_->HasStrictCapacityLimit();
}
size_t GetCapacity() const override { return target_->GetCapacity(); }
size_t GetUsage() const override { return target_->GetUsage(); }
size_t GetUsage(Handle* handle) const override {
return target_->GetUsage(handle);
}
size_t GetPinnedUsage() const override { return target_->GetPinnedUsage(); }
size_t GetCharge(Handle* handle) const override {
return target_->GetCharge(handle);
}
const CacheItemHelper* GetCacheItemHelper(Handle* handle) const override {
return target_->GetCacheItemHelper(handle);
}
void ApplyToAllEntries(
const std::function<void(const Slice& key, ObjectPtr value, size_t charge,
const CacheItemHelper* helper)>& callback,
const ApplyToAllEntriesOptions& opts) override {
target_->ApplyToAllEntries(callback, opts);
}
void EraseUnRefEntries() override { target_->EraseUnRefEntries(); }
protected:
std::shared_ptr<Cache> target_;
};
/*
* A cache wrapper that tracks certain CacheEntryRole's cache charge, its
* peaks and increments
@@ -1004,6 +926,8 @@ class TargetCacheChargeTrackingCache : public CacheWrapper {
public:
explicit TargetCacheChargeTrackingCache(std::shared_ptr<Cache> target);
const char* Name() const override { return "TargetCacheChargeTrackingCache"; }
Status Insert(const Slice& key, ObjectPtr value,
const CacheItemHelper* helper, size_t charge,
Handle** handle = nullptr,
@@ -1056,16 +980,15 @@ class DBTestBase : public testing::Test {
kHashSkipList = 18,
kUniversalCompaction = 19,
kUniversalCompactionMultiLevel = 20,
kCompressedBlockCache = 21,
kInfiniteMaxOpenFiles = 22,
kCRC32cChecksum = 23,
kFIFOCompaction = 24,
kOptimizeFiltersForHits = 25,
kRowCache = 26,
kRecycleLogFiles = 27,
kConcurrentSkipList = 28,
kPipelinedWrite = 29,
kConcurrentWALWrites = 30,
kInfiniteMaxOpenFiles = 21,
kCRC32cChecksum = 22,
kFIFOCompaction = 23,
kOptimizeFiltersForHits = 24,
kRowCache = 25,
kRecycleLogFiles = 26,
kConcurrentSkipList = 27,
kPipelinedWrite = 28,
kConcurrentWALWrites = 29,
kDirectIO,
kLevelSubcompactions,
kBlockBasedTableWithIndexRestartInterval,
@@ -1261,7 +1184,6 @@ class DBTestBase : public testing::Test {
const std::vector<std::string>& cfs,
const Options& options);
#ifndef ROCKSDB_LITE
int NumSortedRuns(int cf = 0);
uint64_t TotalSize(int cf = 0);
@@ -1277,7 +1199,6 @@ class DBTestBase : public testing::Test {
double CompressionRatioAtLevel(int level, int cf = 0);
int TotalTableFiles(int cf = 0, int levels = -1);
#endif // ROCKSDB_LITE
std::vector<uint64_t> GetBlobFileNumbers();
@@ -1313,9 +1234,7 @@ class DBTestBase : public testing::Test {
void MoveFilesToLevel(int level, int cf = 0);
#ifndef ROCKSDB_LITE
void DumpFileCounts(const char* label);
#endif // ROCKSDB_LITE
std::string DumpSSTableList();
@@ -1384,12 +1303,10 @@ class DBTestBase : public testing::Test {
void VerifyDBInternal(
std::vector<std::pair<std::string, std::string>> true_data);
#ifndef ROCKSDB_LITE
uint64_t GetNumberOfSstFilesForColumnFamily(DB* db,
std::string column_family_name);
uint64_t GetSstSizeHelper(Temperature temperature);
#endif // ROCKSDB_LITE
uint64_t TestGetTickerCount(const Options& options, Tickers ticker_type) {
return options.statistics->getTickerCount(ticker_type);
Loaded 100 of 576 files, more files were not shown because too many files have changed in this diff. Show more