add missing release notes to HISTORY.md

Summary:
Added unit tests that have paranoid_check = true that perform range deletions.  At the moment, the deleted ranges do not appear to be checked as part of the paranoid checks.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7521

Reviewed By: zhichao-cao

Differential Revision: D24262175

Pulled By: ajkr

fbshipit-source-id: 1035e968f7ab8ccaa7af086b835a4e72c7e56743

.circleci/config.yml

@@ -10,9 +10,26 @@ aliases:
     only_for_branches: master
 
 commands:
+  install-pyenv-on-macos:
+    steps:
+      - run:
+          name: Install pyenv on macos
+          command: |
+            HOMEBREW_NO_AUTO_UPDATE=1 brew install pyenv
+
+  increase-max-open-files-on-macos:
+    steps:
+      - run:
+          name: Increase max open files
+          command: |
+            sudo sysctl -w kern.maxfiles=1048576
+            sudo sysctl -w kern.maxfilesperproc=1048576
+            sudo launchctl limit maxfiles 1048576
+
   pre-steps:
     steps:
       - checkout
+      - run: pyenv install --skip-existing 3.5.9
       - run: pyenv global 3.5.9
       - run:
           name: Setup Environment Variables
@@ -46,6 +63,13 @@ commands:
           name: Install gflags
           command: |
             sudo apt-get update -y && sudo apt-get install -y libgflags-dev
+  
+  install-gflags-on-macos:
+    steps:
+      - run:
+          name: Install gflags on macos
+          command: |
+            HOMEBREW_NO_AUTO_UPDATE=1 brew install gflags
 
   install-gtest-parallel:
     steps:
@@ -63,6 +87,17 @@ executors:
       shell: bash.exe
 
 jobs:
+  build-macos:
+    macos:
+      xcode: 11.3.0
+    steps:
+      - increase-max-open-files-on-macos
+      - install-pyenv-on-macos
+      - pre-steps
+      - install-gflags-on-macos
+      - run: ulimit -S -n 1048576 && OPT=-DCIRCLECI make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
+      - post-steps
+
   build-linux:
     machine:
       image: ubuntu-1604:202007-01
@@ -419,3 +454,6 @@ workflows:
   build-linux-gcc-4-8:
     jobs:
       - build-linux-gcc-4-8
+  build-macos:
+    jobs:
+      - build-macos

HISTORY.md

@@ -1,26 +1,13 @@
 # Rocksdb Change Log
 ## Unreleased
-### Public API Change
-* Deprecate `BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`. These options still take effect until users migrate to the replacement APIs in `BlockBasedTableOptions::metadata_cache_options`. Migration guidance can be found in the API comments on the deprecated options.
-
-### Behavior Changes
-* The dictionary compression settings specified in `ColumnFamilyOptions::compression_opts` now additionally affect files generated by flush and compaction to non-bottommost level. Previously those settings at most affected files generated by compaction to bottommost level, depending on whether `ColumnFamilyOptions::bottommost_compression_opts` overrode them. Users who relied on dictionary compression settings in `ColumnFamilyOptions::compression_opts` affecting only the bottommost level can keep the behavior by moving their dictionary settings to `ColumnFamilyOptions::bottommost_compression_opts` and setting its `enabled` flag.
-
-## 6.14.3 (10/30/2020)
 ### Bug Fixes
-* Reverted a behavior change silently introduced in 6.14.2, in which the effects of the `ignore_unknown_options` flag (used in option parsing/loading functions) changed.
-* Reverted a behavior change silently introduced in 6.14, in which options parsing/loading functions began returning `NotFound` instead of `InvalidArgument` for option names not available in the present version.
-
-## 6.14.2 (10/21/2020)
-### Bug Fixes
-* Fixed a bug which causes hang in closing DB when refit level is set in opt build. It was because ContinueBackgroundWork() was called in assert statement which is a no op. It was introduced in 6.14.
-
-## 6.14.1 (10/13/2020)
-### Bug Fixes
-* Since 6.12, memtable lookup should report unrecognized value_type as corruption (#7121).
-* Since 6.14, fix false positive flush/compaction `Status::Corruption` failure when `paranoid_file_checks == true` and range tombstones were written to the compaction output files.
 * Fixed a bug in the following combination of features: indexes with user keys (`format_version >= 3`), indexes are partitioned (`index_type == kTwoLevelIndexSearch`), and some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`). The bug could cause keys to be truncated when read from the index leading to wrong read results or other unexpected behavior.
 * Fixed a bug when indexes are partitioned (`index_type == kTwoLevelIndexSearch`), some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`), and partitions reads could be mixed between block cache and directly from the file (e.g., with `enable_index_compression == 1` and `mmap_read == 1`, partitions that were stored uncompressed due to poor compression ratio would be read directly from the file via mmap, while partitions that were stored compressed would be read from block cache). The bug could cause index partitions to be mistakenly considered empty during reads leading to wrong read results.
+* Since 6.12, memtable lookup should report unrecognized value_type as corruption (#7121).
+* Since 6.14, fix false positive flush/compaction `Status::Corruption` failure when `paranoid_file_checks == true` and range tombstones were written to the compaction output files.
+
+### Public API Change
+* Deprecate `BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`. These options still take effect until users migrate to the replacement APIs in `BlockBasedTableOptions::metadata_cache_options`. Migration guidance can be found in the API comments on the deprecated options.
 
 ## 6.14 (10/09/2020)
 ### Bug fixes

Makefile

@@ -588,6 +588,7 @@ ifdef ASSERT_STATUS_CHECKED
 		crc32c_test \
 		dbformat_test \
 		db_basic_test \
+		db_flush_test \
 		db_with_timestamp_basic_test \
 		db_with_timestamp_compaction_test \
 		db_options_test \

db/builder.cc

@@ -124,6 +124,11 @@ Status BuildTable(
   if (iter->Valid() || !range_del_agg->IsEmpty()) {
     TableBuilder* builder;
     std::unique_ptr<WritableFileWriter> file_writer;
+    // Currently we only enable dictionary compression during compaction to the
+    // bottommost level.
+    CompressionOptions compression_opts_for_flush(compression_opts);
+    compression_opts_for_flush.max_dict_bytes = 0;
+    compression_opts_for_flush.zstd_max_train_bytes = 0;
     {
       std::unique_ptr<FSWritableFile> file;
 #ifndef NDEBUG
@@ -155,7 +160,7 @@ Status BuildTable(
           ioptions, mutable_cf_options, internal_comparator,
           int_tbl_prop_collector_factories, column_family_id,
           column_family_name, file_writer.get(), compression,
-          sample_for_compression, compression_opts, level,
+          sample_for_compression, compression_opts_for_flush, level,
           false /* skip_filters */, creation_time, oldest_key_time,
           0 /*target_file_size*/, file_creation_time, db_id, db_session_id);
     }

db/compaction/compaction.cc

@@ -248,6 +248,12 @@ Compaction::Compaction(VersionStorageInfo* vstorage,
   if (max_subcompactions_ == 0) {
     max_subcompactions_ = _mutable_db_options.max_subcompactions;
   }
+  if (!bottommost_level_) {
+    // Currently we only enable dictionary compression during compaction to the
+    // bottommost level.
+    output_compression_opts_.max_dict_bytes = 0;
+    output_compression_opts_.zstd_max_train_bytes = 0;
+  }
 
 #ifndef NDEBUG
   for (size_t i = 1; i < inputs_.size(); ++i) {

db/corruption_test.cc

@@ -117,7 +117,7 @@ class CorruptionTest : public testing::Test {
       //if ((i % 100) == 0) fprintf(stderr, "@ %d of %d\n", i, n);
       Slice key = Key(i + start, &key_space);
       batch.Clear();
-      ASSERT_OK(batch.Put(key, Value(i, &value_space)));
+      ASSERT_OK(batch.Put(key, Value(i + start, &value_space)));
       ASSERT_OK(db_->Write(WriteOptions(), &batch));
     }
   }
@@ -628,6 +628,105 @@ TEST_F(CorruptionTest, ParanoidFileChecksOnCompact) {
   }
 }
 
+TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
+  Options options;
+  options.check_flush_compaction_key_order = false;
+  options.paranoid_file_checks = true;
+  options.create_if_missing = true;
+  for (bool do_flush : {true, false}) {
+    delete db_;
+    db_ = nullptr;
+    ASSERT_OK(DestroyDB(dbname_, options));
+    ASSERT_OK(DB::Open(options, dbname_, &db_));
+    std::string start, end;
+    assert(db_ != nullptr);
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(3, &start), Key(7, &end)));
+    auto snap = db_->GetSnapshot();
+    ASSERT_NE(snap, nullptr);
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(8, &start), Key(9, &end)));
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(2, &start), Key(5, &end)));
+    Build(10);
+    if (do_flush) {
+      ASSERT_OK(db_->Flush(FlushOptions()));
+    } else {
+      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
+      ASSERT_OK(dbi->TEST_FlushMemTable());
+      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
+    }
+    db_->ReleaseSnapshot(snap);
+  }
+}
+
+TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
+  Options options;
+  options.check_flush_compaction_key_order = false;
+  options.paranoid_file_checks = true;
+  options.create_if_missing = true;
+  for (bool do_flush : {true, false}) {
+    delete db_;
+    db_ = nullptr;
+    ASSERT_OK(DestroyDB(dbname_, options));
+    ASSERT_OK(DB::Open(options, dbname_, &db_));
+    assert(db_ != nullptr);
+    Build(10, 0, 0);
+    std::string start, end;
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(5, &start), Key(15, &end)));
+    auto snap = db_->GetSnapshot();
+    ASSERT_NE(snap, nullptr);
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(8, &start), Key(9, &end)));
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(12, &start), Key(17, &end)));
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(2, &start), Key(4, &end)));
+    Build(10, 10, 0);
+    if (do_flush) {
+      ASSERT_OK(db_->Flush(FlushOptions()));
+    } else {
+      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
+      ASSERT_OK(dbi->TEST_FlushMemTable());
+      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
+    }
+    db_->ReleaseSnapshot(snap);
+  }
+}
+
+TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
+  Options options;
+  options.check_flush_compaction_key_order = false;
+  options.paranoid_file_checks = true;
+  options.create_if_missing = true;
+  for (bool do_flush : {true, false}) {
+    delete db_;
+    db_ = nullptr;
+    ASSERT_OK(DestroyDB(dbname_, options));
+    ASSERT_OK(DB::Open(options, dbname_, &db_));
+    assert(db_ != nullptr);
+    std::string start, end;
+    Build(10);
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(3, &start), Key(7, &end)));
+    auto snap = db_->GetSnapshot();
+    ASSERT_NE(snap, nullptr);
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(6, &start), Key(8, &end)));
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(2, &start), Key(5, &end)));
+    if (do_flush) {
+      ASSERT_OK(db_->Flush(FlushOptions()));
+    } else {
+      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
+      ASSERT_OK(dbi->TEST_FlushMemTable());
+      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
+    }
+    db_->ReleaseSnapshot(snap);
+  }
+}
+
 TEST_F(CorruptionTest, LogCorruptionErrorsInCompactionIterator) {
   Options options;
   options.create_if_missing = true;
@@ -725,102 +824,6 @@ TEST_F(CorruptionTest, DisableKeyOrderCheck) {
   ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
 }
 
-TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeFirst) {
-  Options options;
-  options.paranoid_file_checks = true;
-  options.create_if_missing = true;
-  for (bool do_flush : {true, false}) {
-    delete db_;
-    db_ = nullptr;
-    ASSERT_OK(DestroyDB(dbname_, options));
-    ASSERT_OK(DB::Open(options, dbname_, &db_));
-    std::string start, end;
-    assert(db_ != nullptr);
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(3, &start), Key(7, &end)));
-    auto snap = db_->GetSnapshot();
-    ASSERT_NE(snap, nullptr);
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(8, &start), Key(9, &end)));
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(2, &start), Key(5, &end)));
-    Build(10);
-    if (do_flush) {
-      ASSERT_OK(db_->Flush(FlushOptions()));
-    } else {
-      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
-      ASSERT_OK(dbi->TEST_FlushMemTable());
-      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
-    }
-    db_->ReleaseSnapshot(snap);
-  }
-}
-
-TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRange) {
-  Options options;
-  options.paranoid_file_checks = true;
-  options.create_if_missing = true;
-  for (bool do_flush : {true, false}) {
-    delete db_;
-    db_ = nullptr;
-    ASSERT_OK(DestroyDB(dbname_, options));
-    ASSERT_OK(DB::Open(options, dbname_, &db_));
-    assert(db_ != nullptr);
-    Build(10, 0, 0);
-    std::string start, end;
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(5, &start), Key(15, &end)));
-    auto snap = db_->GetSnapshot();
-    ASSERT_NE(snap, nullptr);
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(8, &start), Key(9, &end)));
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(12, &start), Key(17, &end)));
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(2, &start), Key(4, &end)));
-    Build(10, 10, 0);
-    if (do_flush) {
-      ASSERT_OK(db_->Flush(FlushOptions()));
-    } else {
-      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
-      ASSERT_OK(dbi->TEST_FlushMemTable());
-      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
-    }
-    db_->ReleaseSnapshot(snap);
-  }
-}
-
-TEST_F(CorruptionTest, ParanoidFileChecksWithDeleteRangeLast) {
-  Options options;
-  options.paranoid_file_checks = true;
-  options.create_if_missing = true;
-  for (bool do_flush : {true, false}) {
-    delete db_;
-    db_ = nullptr;
-    ASSERT_OK(DestroyDB(dbname_, options));
-    ASSERT_OK(DB::Open(options, dbname_, &db_));
-    assert(db_ != nullptr);
-    std::string start, end;
-    Build(10);
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(3, &start), Key(7, &end)));
-    auto snap = db_->GetSnapshot();
-    ASSERT_NE(snap, nullptr);
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(6, &start), Key(8, &end)));
-    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
-                               Key(2, &start), Key(5, &end)));
-    if (do_flush) {
-      ASSERT_OK(db_->Flush(FlushOptions()));
-    } else {
-      DBImpl* dbi = static_cast_with_check<DBImpl>(db_);
-      ASSERT_OK(dbi->TEST_FlushMemTable());
-      ASSERT_OK(dbi->TEST_CompactRange(0, nullptr, nullptr, nullptr, true));
-    }
-    db_->ReleaseSnapshot(snap);
-  }
-}
-
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/db_block_cache_test.cc

@@ -837,9 +837,8 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
   Random rnd(301);
   for (auto compression_type : compression_types) {
     Options options = CurrentOptions();
-    options.bottommost_compression = compression_type;
-    options.bottommost_compression_opts.max_dict_bytes = 4096;
-    options.bottommost_compression_opts.enabled = true;
+    options.compression = compression_type;
+    options.compression_opts.max_dict_bytes = 4096;
     options.create_if_missing = true;
     options.num_levels = 2;
     options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
@@ -992,9 +991,6 @@ TEST_P(DBBlockCachePinningTest, TwoLevelDB) {
       ++expected_index_misses;
     }
   }
-  if (unpartitioned_pinning_ == PinningTier::kNone) {
-    ++expected_compression_dict_misses;
-  }
   ASSERT_EQ(expected_filter_misses,
             TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS));
   ASSERT_EQ(expected_index_misses,

db/db_flush_test.cc

@@ -63,7 +63,7 @@ TEST_F(DBFlushTest, FlushWhileWritingManifest) {
   ASSERT_OK(Put("bar", "v"));
   ASSERT_OK(dbfull()->Flush(no_wait));
   // If the issue is hit we will wait here forever.
-  dbfull()->TEST_WaitForFlushMemTable();
+  ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
 #ifndef ROCKSDB_LITE
   ASSERT_EQ(2, TotalTableFiles());
 #endif  // ROCKSDB_LITE
@@ -88,7 +88,7 @@ TEST_F(DBFlushTest, SyncFail) {
   SyncPoint::GetInstance()->EnableProcessing();
 
   CreateAndReopenWithCF({"pikachu"}, options);
-  Put("key", "value");
+  ASSERT_OK(Put("key", "value"));
   auto* cfd =
       static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
           ->cfd();
@@ -107,7 +107,8 @@ TEST_F(DBFlushTest, SyncFail) {
   TEST_SYNC_POINT("DBFlushTest::SyncFail:2");
   fault_injection_env->SetFilesystemActive(true);
   // Now the background job will do the flush; wait for it.
-  dbfull()->TEST_WaitForFlushMemTable();
+  // Returns the IO error happend during flush.
+  ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
 #ifndef ROCKSDB_LITE
   ASSERT_EQ("", FilesPerLevel());  // flush failed.
 #endif                             // ROCKSDB_LITE
@@ -126,7 +127,7 @@ TEST_F(DBFlushTest, SyncSkip) {
   SyncPoint::GetInstance()->EnableProcessing();
 
   Reopen(options);
-  Put("key", "value");
+  ASSERT_OK(Put("key", "value"));
 
   FlushOptions flush_options;
   flush_options.wait = false;
@@ -136,7 +137,7 @@ TEST_F(DBFlushTest, SyncSkip) {
   TEST_SYNC_POINT("DBFlushTest::SyncSkip:2");
 
   // Now the background job will do the flush; wait for it.
-  dbfull()->TEST_WaitForFlushMemTable();
+  ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
 
   Destroy(options);
 }
@@ -171,9 +172,9 @@ TEST_F(DBFlushTest, FlushInLowPriThreadPool) {
   ASSERT_OK(Put("key", "val"));
   for (int i = 0; i < 4; ++i) {
     ASSERT_OK(Put("key", "val"));
-    dbfull()->TEST_WaitForFlushMemTable();
+    ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
   }
-  dbfull()->TEST_WaitForCompact();
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
   ASSERT_EQ(4, num_flushes);
   ASSERT_EQ(1, num_compactions);
 }
@@ -306,7 +307,8 @@ TEST_F(DBFlushTest, ManualFlushFailsInReadOnlyMode) {
   // mode.
   fault_injection_env->SetFilesystemActive(false);
   ASSERT_OK(db_->ContinueBackgroundWork());
-  dbfull()->TEST_WaitForFlushMemTable();
+  // We ingested the error to env, so the returned status is not OK.
+  ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable());
 #ifndef ROCKSDB_LITE
   uint64_t num_bg_errors;
   ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBackgroundErrors,
@@ -713,7 +715,8 @@ TEST_P(DBAtomicFlushTest, AtomicFlushRollbackSomeJobs) {
   fault_injection_env->SetFilesystemActive(false);
   TEST_SYNC_POINT("DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:2");
   for (auto* cfh : handles_) {
-    dbfull()->TEST_WaitForFlushMemTable(cfh);
+    // Returns the IO error happend during flush.
+    ASSERT_NOK(dbfull()->TEST_WaitForFlushMemTable(cfh));
   }
   for (size_t i = 0; i != num_cfs; ++i) {
     auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);

db/db_impl/db_impl.cc

@@ -459,7 +459,9 @@ void DBImpl::CancelAllBackgroundWork(bool wait) {
       autovector<ColumnFamilyData*> cfds;
       SelectColumnFamiliesForAtomicFlush(&cfds);
       mutex_.Unlock();
-      AtomicFlushMemTables(cfds, FlushOptions(), FlushReason::kShutDown);
+      Status s =
+          AtomicFlushMemTables(cfds, FlushOptions(), FlushReason::kShutDown);
+      s.PermitUncheckedError();  //**TODO: What to do on error?
       mutex_.Lock();
     } else {
       for (auto cfd : *versions_->GetColumnFamilySet()) {
@@ -974,7 +976,6 @@ Status DBImpl::SetOptions(
   MutableCFOptions new_options;
   Status s;
   Status persist_options_status;
-  persist_options_status.PermitUncheckedError();  // Allow uninitialized access
   SuperVersionContext sv_context(/* create_superversion */ true);
   {
     auto db_options = GetDBOptions();
@@ -2931,7 +2932,7 @@ const Snapshot* DBImpl::GetSnapshotForWriteConflictBoundary() {
 SnapshotImpl* DBImpl::GetSnapshotImpl(bool is_write_conflict_boundary,
                                       bool lock) {
   int64_t unix_time = 0;
-  env_->GetCurrentTime(&unix_time);  // Ignore error
+  env_->GetCurrentTime(&unix_time).PermitUncheckedError();  // Ignore error
   SnapshotImpl* s = new SnapshotImpl;
 
   if (lock) {
@@ -4718,8 +4719,7 @@ Status DBImpl::CreateColumnFamilyWithImport(
                       temp_s.ToString().c_str());
     }
     // Always returns Status::OK()
-    temp_s = DestroyColumnFamilyHandle(*handle);
-    assert(temp_s.ok());
+    assert(DestroyColumnFamilyHandle(*handle).ok());
     *handle = nullptr;
   }
   return status;

db/db_impl/db_impl_compaction_flush.cc

@@ -950,8 +950,7 @@ Status DBImpl::CompactRange(const CompactRangeOptions& options,
       s = ReFitLevel(cfd, final_output_level, options.target_level);
       TEST_SYNC_POINT("DBImpl::CompactRange:PostRefitLevel");
       // ContinueBackgroundWork always return Status::OK().
-      Status temp_s = ContinueBackgroundWork();
-      assert(temp_s.ok());
+      assert(ContinueBackgroundWork().ok());
     }
     EnableManualCompaction();
   }

db/db_impl/db_impl_write.cc

@@ -1784,9 +1784,11 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
     // the current log, so treat it as a fatal error and set bg_error
     // Should handle return error?
     if (!io_s.ok()) {
+      // Should handle return error?
       error_handler_.SetBGError(io_s, BackgroundErrorReason::kMemTable)
           .PermitUncheckedError();
     } else {
+      // Should handle return error?
       error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable)
           .PermitUncheckedError();
     }

db/db_options_test.cc

@@ -79,6 +79,21 @@ class DBOptionsTest : public DBTestBase {
 #endif  // ROCKSDB_LITE
 };
 
+TEST_F(DBOptionsTest, ImmutableTrackAndVerifyWalsInManifest) {
+  Options options;
+  options.track_and_verify_wals_in_manifest = true;
+
+  ImmutableDBOptions db_options(options);
+  ASSERT_TRUE(db_options.track_and_verify_wals_in_manifest);
+
+  Reopen(options);
+  ASSERT_TRUE(dbfull()->GetDBOptions().track_and_verify_wals_in_manifest);
+
+  Status s =
+      dbfull()->SetDBOptions({{"track_and_verify_wals_in_manifest", "false"}});
+  ASSERT_FALSE(s.ok());
+}
+
 // RocksDB lite don't support dynamic options.
 #ifndef ROCKSDB_LITE
 

db/db_test2.cc

@@ -12,7 +12,6 @@
 
 #include "db/db_test_util.h"
 #include "db/read_callback.h"
-#include "options/options_helper.h"
 #include "port/port.h"
 #include "port/stack_trace.h"
 #include "rocksdb/persistent_cache.h"
@@ -1390,178 +1389,6 @@ TEST_F(DBTest2, PresetCompressionDictLocality) {
   }
 }
 
-class PresetCompressionDictTest
-    : public DBTestBase,
-      public testing::WithParamInterface<std::tuple<CompressionType, bool>> {
- public:
-  PresetCompressionDictTest()
-      : DBTestBase("/db_test2", false /* env_do_fsync */),
-        compression_type_(std::get<0>(GetParam())),
-        bottommost_(std::get<1>(GetParam())) {}
-
- protected:
-  const CompressionType compression_type_;
-  const bool bottommost_;
-};
-
-INSTANTIATE_TEST_CASE_P(
-    DBTest2, PresetCompressionDictTest,
-    ::testing::Combine(::testing::ValuesIn(GetSupportedDictCompressions()),
-                       ::testing::Bool()));
-
-TEST_P(PresetCompressionDictTest, Flush) {
-  // Verifies that dictionary is generated and written during flush only when
-  // `ColumnFamilyOptions::compression` enables dictionary.
-  const size_t kValueLen = 256;
-  const size_t kKeysPerFile = 1 << 10;
-  const size_t kDictLen = 4 << 10;
-
-  Options options = CurrentOptions();
-  if (bottommost_) {
-    options.bottommost_compression = compression_type_;
-    options.bottommost_compression_opts.enabled = true;
-    options.bottommost_compression_opts.max_dict_bytes = kDictLen;
-  } else {
-    options.compression = compression_type_;
-    options.compression_opts.max_dict_bytes = kDictLen;
-  }
-  options.memtable_factory.reset(new SpecialSkipListFactory(kKeysPerFile));
-  options.statistics = CreateDBStatistics();
-  BlockBasedTableOptions bbto;
-  bbto.cache_index_and_filter_blocks = true;
-  options.table_factory.reset(NewBlockBasedTableFactory(bbto));
-  Reopen(options);
-
-  uint64_t prev_compression_dict_misses =
-      TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
-  Random rnd(301);
-  for (size_t i = 0; i <= kKeysPerFile; ++i) {
-    ASSERT_OK(Put(Key(static_cast<int>(i)), rnd.RandomString(kValueLen)));
-  }
-  ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
-
-  // If there's a compression dictionary, it should have been loaded when the
-  // flush finished, incurring a cache miss.
-  uint64_t expected_compression_dict_misses;
-  if (bottommost_) {
-    expected_compression_dict_misses = prev_compression_dict_misses;
-  } else {
-    expected_compression_dict_misses = prev_compression_dict_misses + 1;
-  }
-  ASSERT_EQ(expected_compression_dict_misses,
-            TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
-}
-
-TEST_P(PresetCompressionDictTest, CompactNonBottommost) {
-  // Verifies that dictionary is generated and written during compaction to
-  // non-bottommost level only when `ColumnFamilyOptions::compression` enables
-  // dictionary.
-  const size_t kValueLen = 256;
-  const size_t kKeysPerFile = 1 << 10;
-  const size_t kDictLen = 4 << 10;
-
-  Options options = CurrentOptions();
-  if (bottommost_) {
-    options.bottommost_compression = compression_type_;
-    options.bottommost_compression_opts.enabled = true;
-    options.bottommost_compression_opts.max_dict_bytes = kDictLen;
-  } else {
-    options.compression = compression_type_;
-    options.compression_opts.max_dict_bytes = kDictLen;
-  }
-  options.disable_auto_compactions = true;
-  options.statistics = CreateDBStatistics();
-  BlockBasedTableOptions bbto;
-  bbto.cache_index_and_filter_blocks = true;
-  options.table_factory.reset(NewBlockBasedTableFactory(bbto));
-  Reopen(options);
-
-  Random rnd(301);
-  for (size_t j = 0; j <= kKeysPerFile; ++j) {
-    ASSERT_OK(Put(Key(static_cast<int>(j)), rnd.RandomString(kValueLen)));
-  }
-  ASSERT_OK(Flush());
-  MoveFilesToLevel(2);
-
-  for (int i = 0; i < 2; ++i) {
-    for (size_t j = 0; j <= kKeysPerFile; ++j) {
-      ASSERT_OK(Put(Key(static_cast<int>(j)), rnd.RandomString(kValueLen)));
-    }
-    ASSERT_OK(Flush());
-  }
-#ifndef ROCKSDB_LITE
-  ASSERT_EQ("2,0,1", FilesPerLevel(0));
-#endif  // ROCKSDB_LITE
-
-  uint64_t prev_compression_dict_misses =
-      TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
-  // This L0->L1 compaction merges the two L0 files into L1. The produced L1
-  // file is not bottommost due to the existing L2 file covering the same key-
-  // range.
-  ASSERT_OK(dbfull()->TEST_CompactRange(0, nullptr, nullptr));
-#ifndef ROCKSDB_LITE
-  ASSERT_EQ("0,1,1", FilesPerLevel(0));
-#endif  // ROCKSDB_LITE
-  // If there's a compression dictionary, it should have been loaded when the
-  // compaction finished, incurring a cache miss.
-  uint64_t expected_compression_dict_misses;
-  if (bottommost_) {
-    expected_compression_dict_misses = prev_compression_dict_misses;
-  } else {
-    expected_compression_dict_misses = prev_compression_dict_misses + 1;
-  }
-  ASSERT_EQ(expected_compression_dict_misses,
-            TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
-}
-
-TEST_P(PresetCompressionDictTest, CompactBottommost) {
-  // Verifies that dictionary is generated and written during compaction to
-  // non-bottommost level only when either `ColumnFamilyOptions::compression` or
-  // `ColumnFamilyOptions::bottommost_compression` enables dictionary.
-  const size_t kValueLen = 256;
-  const size_t kKeysPerFile = 1 << 10;
-  const size_t kDictLen = 4 << 10;
-
-  Options options = CurrentOptions();
-  if (bottommost_) {
-    options.bottommost_compression = compression_type_;
-    options.bottommost_compression_opts.enabled = true;
-    options.bottommost_compression_opts.max_dict_bytes = kDictLen;
-  } else {
-    options.compression = compression_type_;
-    options.compression_opts.max_dict_bytes = kDictLen;
-  }
-  options.disable_auto_compactions = true;
-  options.statistics = CreateDBStatistics();
-  BlockBasedTableOptions bbto;
-  bbto.cache_index_and_filter_blocks = true;
-  options.table_factory.reset(NewBlockBasedTableFactory(bbto));
-  Reopen(options);
-
-  Random rnd(301);
-  for (int i = 0; i < 2; ++i) {
-    for (size_t j = 0; j <= kKeysPerFile; ++j) {
-      ASSERT_OK(Put(Key(static_cast<int>(j)), rnd.RandomString(kValueLen)));
-    }
-    ASSERT_OK(Flush());
-  }
-#ifndef ROCKSDB_LITE
-  ASSERT_EQ("2", FilesPerLevel(0));
-#endif  // ROCKSDB_LITE
-
-  uint64_t prev_compression_dict_misses =
-      TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
-  CompactRangeOptions cro;
-  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
-#ifndef ROCKSDB_LITE
-  ASSERT_EQ("0,1", FilesPerLevel(0));
-#endif  // ROCKSDB_LITE
-  // If there's a compression dictionary, it should have been loaded when the
-  // compaction finished, incurring a cache miss.
-  ASSERT_EQ(prev_compression_dict_misses + 1,
-            TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
-}
-
 class CompactionCompressionListener : public EventListener {
  public:
   explicit CompactionCompressionListener(Options* db_options)
@@ -4549,8 +4376,8 @@ class DummyOldStats : public Statistics {
   }
   bool HistEnabledForType(uint32_t /*type*/) const override { return false; }
   std::string ToString() const override { return ""; }
-  int num_rt = 0;
-  int num_mt = 0;
+  std::atomic<int> num_rt{0};
+  std::atomic<int> num_mt{0};
 };
 }  // namespace
 

db/forward_iterator.cc

@@ -662,9 +662,10 @@ void ForwardIterator::RebuildIterators(bool refresh_sv) {
             read_options_, sv_->current->version_set()->LastSequence()));
     range_del_agg.AddTombstones(std::move(range_del_iter));
     // Always return Status::OK().
-    Status temp_s = sv_->imm->AddRangeTombstoneIterators(read_options_, &arena_,
-                                                         &range_del_agg);
-    assert(temp_s.ok());
+    assert(
+        sv_->imm
+            ->AddRangeTombstoneIterators(read_options_, &arena_, &range_del_agg)
+            .ok());
   }
   has_iter_trimmed_for_upper_bound_ = false;
 
@@ -727,9 +728,10 @@ void ForwardIterator::RenewIterators() {
             read_options_, sv_->current->version_set()->LastSequence()));
     range_del_agg.AddTombstones(std::move(range_del_iter));
     // Always return Status::OK().
-    Status temp_s = svnew->imm->AddRangeTombstoneIterators(
-        read_options_, &arena_, &range_del_agg);
-    assert(temp_s.ok());
+    assert(
+        svnew->imm
+            ->AddRangeTombstoneIterators(read_options_, &arena_, &range_del_agg)
+            .ok());
   }
 
   const auto* vstorage = sv_->current->storage_info();

include/rocksdb/options.h

@@ -390,6 +390,20 @@ struct DBOptions {
   // Default: true
   bool paranoid_checks = true;
 
+  // If true, track WALs in MANIFEST and verify them on recovery.
+  //
+  // If a WAL is tracked in MANIFEST but is missing from disk on recovery,
+  // or the size of the tracked WAL is larger than the WAL's on-disk size,
+  // an error is reported and recovery is aborted.
+  //
+  // If a WAL is not tracked in MANIFEST, then no verification will happen
+  // during recovery.
+  //
+  // Default: false
+  // FIXME(cheng): This option is part of a work in progress and does not yet
+  // work
+  bool track_and_verify_wals_in_manifest = false;
+
   // Use the specified object to interact with the environment,
   // e.g. to read/write files, schedule background work, etc. In the near
   // future, support for doing storage operations such as read/write files

include/rocksdb/status.h

@@ -162,15 +162,9 @@ class Status {
   static Status NotFound(const Slice& msg, const Slice& msg2 = Slice()) {
     return Status(kNotFound, msg, msg2);
   }
-
   // Fast path for not found without malloc;
   static Status NotFound(SubCode msg = kNone) { return Status(kNotFound, msg); }
 
-  static Status NotFound(SubCode sc, const Slice& msg,
-                         const Slice& msg2 = Slice()) {
-    return Status(kNotFound, sc, msg, msg2);
-  }
-
   static Status Corruption(const Slice& msg, const Slice& msg2 = Slice()) {
     return Status(kCorruption, msg, msg2);
   }
@@ -469,8 +463,7 @@ class Status {
 #ifdef ROCKSDB_ASSERT_STATUS_CHECKED
     checked_ = true;
 #endif  // ROCKSDB_ASSERT_STATUS_CHECKED
-    return (code() == kIOError || code() == kNotFound) &&
-           (subcode() == kPathNotFound);
+    return (code() == kIOError) && (subcode() == kPathNotFound);
   }
 
   // Returns true iff the status indicates manual compaction paused. This

include/rocksdb/version.h

@@ -6,7 +6,7 @@
 
 #define ROCKSDB_MAJOR 6
 #define ROCKSDB_MINOR 14
-#define ROCKSDB_PATCH 3
+#define ROCKSDB_PATCH 0
 
 // Do not use these. We made the mistake of declaring macros starting with
 // double underscore. Now we have to live with our choice. We'll deprecate these

memtable/write_buffer_manager.cc

@@ -91,10 +91,8 @@ void WriteBufferManager::ReserveMemWithCache(size_t mem) {
     // Expand size by at least 256KB.
     // Add a dummy record to the cache
     Cache::Handle* handle = nullptr;
-    Status s =
-        cache_rep_->cache_->Insert(cache_rep_->GetNextCacheKey(), nullptr,
-                                   kSizeDummyEntry, nullptr, &handle);
-    s.PermitUncheckedError();  // TODO: What to do on error?
+    cache_rep_->cache_->Insert(cache_rep_->GetNextCacheKey(), nullptr,
+                               kSizeDummyEntry, nullptr, &handle);
     // We keep the handle even if insertion fails and a null handle is
     // returned, so that when memory shrinks, we don't release extra
     // entries from cache.

options/db_options.cc

@@ -198,6 +198,11 @@ static std::unordered_map<std::string, OptionTypeInfo>
          {offsetof(struct ImmutableDBOptions, paranoid_checks),
           OptionType::kBoolean, OptionVerificationType::kNormal,
           OptionTypeFlags::kNone}},
+        {"track_and_verify_wals_in_manifest",
+         {offsetof(struct ImmutableDBOptions,
+                   track_and_verify_wals_in_manifest),
+          OptionType::kBoolean, OptionVerificationType::kNormal,
+          OptionTypeFlags::kNone}},
         {"skip_log_error_on_recovery",
          {0, OptionType::kBoolean, OptionVerificationType::kDeprecated,
           OptionTypeFlags::kNone}},
@@ -497,6 +502,8 @@ ImmutableDBOptions::ImmutableDBOptions(const DBOptions& options)
       create_missing_column_families(options.create_missing_column_families),
       error_if_exists(options.error_if_exists),
       paranoid_checks(options.paranoid_checks),
+      track_and_verify_wals_in_manifest(
+          options.track_and_verify_wals_in_manifest),
       env(options.env),
       fs(options.env->GetFileSystem()),
       rate_limiter(options.rate_limiter),
@@ -579,6 +586,10 @@ void ImmutableDBOptions::Dump(Logger* log) const {
                    create_if_missing);
   ROCKS_LOG_HEADER(log, "                        Options.paranoid_checks: %d",
                    paranoid_checks);
+  ROCKS_LOG_HEADER(log,
+                   "                              "
+                   "Options.track_and_verify_wals_in_manifest: %d",
+                   track_and_verify_wals_in_manifest);
   ROCKS_LOG_HEADER(log, "                                    Options.env: %p",
                    env);
   ROCKS_LOG_HEADER(log, "                                     Options.fs: %s",

options/db_options.h

@@ -23,6 +23,7 @@ struct ImmutableDBOptions {
   bool create_missing_column_families;
   bool error_if_exists;
   bool paranoid_checks;
+  bool track_and_verify_wals_in_manifest;
   Env* env;
   std::shared_ptr<FileSystem> fs;
   std::shared_ptr<RateLimiter> rate_limiter;

options/options_helper.cc

@@ -51,6 +51,8 @@ DBOptions BuildDBOptions(const ImmutableDBOptions& immutable_db_options,
       immutable_db_options.create_missing_column_families;
   options.error_if_exists = immutable_db_options.error_if_exists;
   options.paranoid_checks = immutable_db_options.paranoid_checks;
+  options.track_and_verify_wals_in_manifest =
+      immutable_db_options.track_and_verify_wals_in_manifest;
   options.env = immutable_db_options.env;
   options.rate_limiter = immutable_db_options.rate_limiter;
   options.sst_file_manager = immutable_db_options.sst_file_manager;
@@ -296,17 +298,6 @@ std::vector<CompressionType> GetSupportedCompressions() {
   return supported_compressions;
 }
 
-std::vector<CompressionType> GetSupportedDictCompressions() {
-  std::vector<CompressionType> dict_compression_types;
-  for (const auto& comp_to_name : OptionsHelper::compression_type_string_map) {
-    CompressionType t = comp_to_name.second;
-    if (t != kDisableCompressionOption && DictCompressionTypeSupported(t)) {
-      dict_compression_types.push_back(t);
-    }
-  }
-  return dict_compression_types;
-}
-
 #ifndef ROCKSDB_LITE
 bool ParseSliceTransformHelper(
     const std::string& kFixedPrefixName, const std::string& kCappedPrefixName,
@@ -735,15 +726,9 @@ Status GetColumnFamilyOptionsFromMap(
   *new_options = base_options;
 
   const auto config = CFOptionsAsConfigurable(base_options);
-  Status s = ConfigureFromMap<ColumnFamilyOptions>(
-      config_options, opts_map, OptionsHelper::kCFOptionsName, config.get(),
-      new_options);
-  // Translate any errors (NotFound, NotSupported, to InvalidArgument
-  if (s.ok() || s.IsInvalidArgument()) {
-    return s;
-  } else {
-    return Status::InvalidArgument(s.getState());
-  }
+  return ConfigureFromMap<ColumnFamilyOptions>(config_options, opts_map,
+                                               OptionsHelper::kCFOptionsName,
+                                               config.get(), new_options);
 }
 
 Status GetColumnFamilyOptionsFromString(
@@ -790,15 +775,9 @@ Status GetDBOptionsFromMap(
   assert(new_options);
   *new_options = base_options;
   auto config = DBOptionsAsConfigurable(base_options);
-  Status s = ConfigureFromMap<DBOptions>(config_options, opts_map,
-                                         OptionsHelper::kDBOptionsName,
-                                         config.get(), new_options);
-  // Translate any errors (NotFound, NotSupported, to InvalidArgument
-  if (s.ok() || s.IsInvalidArgument()) {
-    return s;
-  } else {
-    return Status::InvalidArgument(s.getState());
-  }
+  return ConfigureFromMap<DBOptions>(config_options, opts_map,
+                                     OptionsHelper::kDBOptionsName,
+                                     config.get(), new_options);
 }
 
 Status GetDBOptionsFromString(const DBOptions& base_options,
@@ -864,12 +843,7 @@ Status GetOptionsFromString(const ConfigOptions& config_options,
       *new_options = Options(*new_db_options, base_options);
     }
   }
-  // Translate any errors (NotFound, NotSupported, to InvalidArgument
-  if (s.ok() || s.IsInvalidArgument()) {
-    return s;
-  } else {
-    return Status::InvalidArgument(s.getState());
-  }
+  return s;
 }
 
 std::unordered_map<std::string, EncodingType>

options/options_helper.h

@@ -25,8 +25,6 @@ struct Options;
 
 std::vector<CompressionType> GetSupportedCompressions();
 
-std::vector<CompressionType> GetSupportedDictCompressions();
-
 // Checks that the combination of DBOptions and ColumnFamilyOptions are valid
 Status ValidateOptions(const DBOptions& db_opts,
                        const ColumnFamilyOptions& cf_opts);

options/options_parser.cc

@@ -460,13 +460,7 @@ Status RocksDBOptionsParser::EndSection(
             opt_section_titles[kOptionSectionTableOptions].size()),
         &(cf_opt->table_factory));
     if (s.ok()) {
-      s = cf_opt->table_factory->ConfigureFromMap(config_options, opt_map);
-      // Translate any errors (NotFound, NotSupported, to InvalidArgument
-      if (s.ok() || s.IsInvalidArgument()) {
-        return s;
-      } else {
-        return Status::InvalidArgument(s.getState());
-      }
+      return cf_opt->table_factory->ConfigureFromMap(config_options, opt_map);
     } else {
       // Return OK for not supported table factories as TableFactory
       // Deserialization is optional.

options/options_settable_test.cc

@@ -281,6 +281,7 @@ TEST_F(OptionsSettableTest, DBOptionsAllFieldsSettable) {
                              "skip_log_error_on_recovery=true;"
                              "writable_file_max_buffer_size=1048576;"
                              "paranoid_checks=true;"
+                             "track_and_verify_wals_in_manifest=true;"
                              "is_fd_close_on_exec=false;"
                              "bytes_per_sync=4295013613;"
                              "strict_bytes_per_sync=true;"

options/options_test.cc

@@ -109,6 +109,7 @@ TEST_F(OptionsTest, GetOptionsFromMapTest) {
       {"create_missing_column_families", "true"},
       {"error_if_exists", "false"},
       {"paranoid_checks", "true"},
+      {"track_and_verify_wals_in_manifest", "true"},
       {"max_open_files", "32"},
       {"max_total_wal_size", "33"},
       {"use_fsync", "true"},
@@ -263,6 +264,7 @@ TEST_F(OptionsTest, GetOptionsFromMapTest) {
   ASSERT_EQ(new_db_opt.create_missing_column_families, true);
   ASSERT_EQ(new_db_opt.error_if_exists, false);
   ASSERT_EQ(new_db_opt.paranoid_checks, true);
+  ASSERT_EQ(new_db_opt.track_and_verify_wals_in_manifest, true);
   ASSERT_EQ(new_db_opt.max_open_files, 32);
   ASSERT_EQ(new_db_opt.max_total_wal_size, static_cast<uint64_t>(33));
   ASSERT_EQ(new_db_opt.use_fsync, true);
@@ -302,11 +304,8 @@ TEST_F(OptionsTest, GetOptionsFromMapTest) {
   ASSERT_EQ(new_db_opt.strict_bytes_per_sync, true);
 
   db_options_map["max_open_files"] = "hello";
-  Status s =
-      GetDBOptionsFromMap(exact, base_db_opt, db_options_map, &new_db_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
+  ASSERT_NOK(
+      GetDBOptionsFromMap(exact, base_db_opt, db_options_map, &new_db_opt));
   ASSERT_OK(
       RocksDBOptionsParser::VerifyDBOptions(exact, base_db_opt, new_db_opt));
   ASSERT_OK(
@@ -314,9 +313,8 @@ TEST_F(OptionsTest, GetOptionsFromMapTest) {
 
   // unknow options should fail parsing without ignore_unknown_options = true
   db_options_map["unknown_db_option"] = "1";
-  s = GetDBOptionsFromMap(exact, base_db_opt, db_options_map, &new_db_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+  ASSERT_NOK(
+      GetDBOptionsFromMap(exact, base_db_opt, db_options_map, &new_db_opt));
   ASSERT_OK(
       RocksDBOptionsParser::VerifyDBOptions(exact, base_db_opt, new_db_opt));
 
@@ -401,29 +399,22 @@ TEST_F(OptionsTest, GetColumnFamilyOptionsFromStringTest) {
   ASSERT_EQ(kMoName, std::string(new_cf_opt.merge_operator->Name()));
 
   // Wrong key/value pair
-  Status s = GetColumnFamilyOptionsFromString(
+  ASSERT_NOK(GetColumnFamilyOptionsFromString(
       config_options, base_cf_opt,
-      "write_buffer_size=13;max_write_buffer_number;", &new_cf_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
+      "write_buffer_size=13;max_write_buffer_number;", &new_cf_opt));
   ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_options, base_cf_opt,
                                                   new_cf_opt));
 
-  // Error Parsing value
-  s = GetColumnFamilyOptionsFromString(
+  // Error Paring value
+  ASSERT_NOK(GetColumnFamilyOptionsFromString(
       config_options, base_cf_opt,
-      "write_buffer_size=13;max_write_buffer_number=;", &new_cf_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      "write_buffer_size=13;max_write_buffer_number=;", &new_cf_opt));
   ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_options, base_cf_opt,
                                                   new_cf_opt));
 
   // Missing option name
-  s = GetColumnFamilyOptionsFromString(
-      config_options, base_cf_opt, "write_buffer_size=13; =100;", &new_cf_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+  ASSERT_NOK(GetColumnFamilyOptionsFromString(
+      config_options, base_cf_opt, "write_buffer_size=13; =100;", &new_cf_opt));
   ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_options, base_cf_opt,
                                                   new_cf_opt));
 
@@ -785,6 +776,7 @@ TEST_F(OptionsTest, OldInterfaceTest) {
       {"create_missing_column_families", "true"},
       {"error_if_exists", "false"},
       {"paranoid_checks", "true"},
+      {"track_and_verify_wals_in_manifest", "true"},
       {"max_open_files", "32"},
   };
   ASSERT_OK(GetDBOptionsFromMap(base_db_opt, db_options_map, &new_db_opt));
@@ -792,12 +784,10 @@ TEST_F(OptionsTest, OldInterfaceTest) {
   ASSERT_EQ(new_db_opt.create_missing_column_families, true);
   ASSERT_EQ(new_db_opt.error_if_exists, false);
   ASSERT_EQ(new_db_opt.paranoid_checks, true);
+  ASSERT_EQ(new_db_opt.track_and_verify_wals_in_manifest, true);
   ASSERT_EQ(new_db_opt.max_open_files, 32);
   db_options_map["unknown_option"] = "1";
-  Status s = GetDBOptionsFromMap(base_db_opt, db_options_map, &new_db_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
+  ASSERT_NOK(GetDBOptionsFromMap(base_db_opt, db_options_map, &new_db_opt));
   ASSERT_OK(
       RocksDBOptionsParser::VerifyDBOptions(exact, base_db_opt, new_db_opt));
   ASSERT_OK(GetDBOptionsFromMap(base_db_opt, db_options_map, &new_db_opt, true,
@@ -809,13 +799,11 @@ TEST_F(OptionsTest, OldInterfaceTest) {
   ASSERT_EQ(new_db_opt.create_if_missing, false);
   ASSERT_EQ(new_db_opt.error_if_exists, false);
   ASSERT_EQ(new_db_opt.max_open_files, 42);
-  s = GetDBOptionsFromString(
+  ASSERT_NOK(GetDBOptionsFromString(
       base_db_opt,
       "create_if_missing=false;error_if_exists=false;max_open_files=42;"
       "unknown_option=1;",
-      &new_db_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      &new_db_opt));
   ASSERT_OK(
       RocksDBOptionsParser::VerifyDBOptions(exact, base_db_opt, new_db_opt));
 }
@@ -860,23 +848,19 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
   EXPECT_EQ(bfp.GetWholeBitsPerKey(), 5);
 
   // unknown option
-  Status s = GetBlockBasedTableOptionsFromString(
+  ASSERT_NOK(GetBlockBasedTableOptionsFromString(
       config_options, table_opt,
       "cache_index_and_filter_blocks=1;index_type=kBinarySearch;"
       "bad_option=1",
-      &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      &new_opt));
   ASSERT_EQ(static_cast<bool>(table_opt.cache_index_and_filter_blocks),
             new_opt.cache_index_and_filter_blocks);
   ASSERT_EQ(table_opt.index_type, new_opt.index_type);
 
   // unrecognized index type
-  s = GetBlockBasedTableOptionsFromString(
+  ASSERT_NOK(GetBlockBasedTableOptionsFromString(
       config_options, table_opt,
-      "cache_index_and_filter_blocks=1;index_type=kBinarySearchXX", &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      "cache_index_and_filter_blocks=1;index_type=kBinarySearchXX", &new_opt));
   ASSERT_EQ(table_opt.cache_index_and_filter_blocks,
             new_opt.cache_index_and_filter_blocks);
   ASSERT_EQ(table_opt.index_type, new_opt.index_type);
@@ -890,23 +874,21 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
   ASSERT_EQ(table_opt.index_type, new_opt.index_type);
 
   // unrecognized filter policy name
-  s = GetBlockBasedTableOptionsFromString(config_options, table_opt,
+  ASSERT_NOK(
+      GetBlockBasedTableOptionsFromString(config_options, table_opt,
                                           "cache_index_and_filter_blocks=1;"
                                           "filter_policy=bloomfilterxx:4:true",
-                                          &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+                                          &new_opt));
   ASSERT_EQ(table_opt.cache_index_and_filter_blocks,
             new_opt.cache_index_and_filter_blocks);
   ASSERT_EQ(table_opt.filter_policy, new_opt.filter_policy);
 
   // unrecognized filter policy config
-  s = GetBlockBasedTableOptionsFromString(config_options, table_opt,
+  ASSERT_NOK(
+      GetBlockBasedTableOptionsFromString(config_options, table_opt,
                                           "cache_index_and_filter_blocks=1;"
                                           "filter_policy=bloomfilter:4",
-                                          &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+                                          &new_opt));
   ASSERT_EQ(table_opt.cache_index_and_filter_blocks,
             new_opt.cache_index_and_filter_blocks);
   ASSERT_EQ(table_opt.filter_policy, new_opt.filter_policy);
@@ -1039,22 +1021,18 @@ TEST_F(OptionsTest, GetPlainTableOptionsFromString) {
   ASSERT_TRUE(new_opt.store_index_in_file);
 
   // unknown option
-  Status s = GetPlainTableOptionsFromString(
+  ASSERT_NOK(GetPlainTableOptionsFromString(
       config_options, table_opt,
       "user_key_len=66;bloom_bits_per_key=20;hash_table_ratio=0.5;"
       "bad_option=1",
-      &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      &new_opt));
 
   // unrecognized EncodingType
-  s = GetPlainTableOptionsFromString(
+  ASSERT_NOK(GetPlainTableOptionsFromString(
       config_options, table_opt,
       "user_key_len=66;bloom_bits_per_key=20;hash_table_ratio=0.5;"
       "encoding_type=kPrefixXX",
-      &new_opt);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+      &new_opt));
 }
 #endif  // !ROCKSDB_LITE
 
@@ -1173,29 +1151,23 @@ TEST_F(OptionsTest, GetOptionsFromStringTest) {
   base_options.dump_malloc_stats = false;
   base_options.write_buffer_size = 1024;
   Options bad_options = new_options;
-  Status s = GetOptionsFromString(config_options, base_options,
+  ASSERT_NOK(GetOptionsFromString(config_options, base_options,
                                   "create_if_missing=XX;dump_malloc_stats=true",
-                                  &bad_options);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
+                                  &bad_options));
   ASSERT_EQ(bad_options.dump_malloc_stats, false);
 
   bad_options = new_options;
-  s = GetOptionsFromString(config_options, base_options,
-                           "write_buffer_size=XX;dump_malloc_stats=true",
-                           &bad_options);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
+  ASSERT_NOK(GetOptionsFromString(config_options, base_options,
+                                  "write_buffer_size=XX;dump_malloc_stats=true",
+                                  &bad_options));
   ASSERT_EQ(bad_options.dump_malloc_stats, false);
 
   // Test a bad value for a TableFactory Option returns a failure
   bad_options = new_options;
-  s = GetOptionsFromString(config_options, base_options,
-                           "write_buffer_size=16;dump_malloc_stats=true"
-                           "block_based_table_factory={block_size=XX;};",
-                           &bad_options);
-  ASSERT_TRUE(s.IsInvalidArgument());
+  ASSERT_NOK(GetOptionsFromString(config_options, base_options,
+                                  "write_buffer_size=16;dump_malloc_stats=true"
+                                  "block_based_table_factory={block_size=XX;};",
+                                  &bad_options));
   ASSERT_EQ(bad_options.dump_malloc_stats, false);
   ASSERT_EQ(bad_options.write_buffer_size, 1024);
 
@@ -1652,6 +1624,7 @@ TEST_F(OptionsOldApiTest, GetOptionsFromMapTest) {
       {"create_missing_column_families", "true"},
       {"error_if_exists", "false"},
       {"paranoid_checks", "true"},
+      {"track_and_verify_wals_in_manifest", "true"},
       {"max_open_files", "32"},
       {"max_total_wal_size", "33"},
       {"use_fsync", "true"},
@@ -1800,6 +1773,7 @@ TEST_F(OptionsOldApiTest, GetOptionsFromMapTest) {
   ASSERT_EQ(new_db_opt.create_missing_column_families, true);
   ASSERT_EQ(new_db_opt.error_if_exists, false);
   ASSERT_EQ(new_db_opt.paranoid_checks, true);
+  ASSERT_EQ(new_db_opt.track_and_verify_wals_in_manifest, true);
   ASSERT_EQ(new_db_opt.max_open_files, 32);
   ASSERT_EQ(new_db_opt.max_total_wal_size, static_cast<uint64_t>(33));
   ASSERT_EQ(new_db_opt.use_fsync, true);

table/block_based/block_based_table_factory.cc

@@ -762,14 +762,8 @@ Status GetBlockBasedTableOptionsFromString(
   if (!s.ok()) {
     return s;
   }
-  s = GetBlockBasedTableOptionsFromMap(config_options, table_options, opts_map,
-                                       new_table_options);
-  // Translate any errors (NotFound, NotSupported, to InvalidArgument
-  if (s.ok() || s.IsInvalidArgument()) {
-    return s;
-  } else {
-    return Status::InvalidArgument(s.getState());
-  }
+  return GetBlockBasedTableOptionsFromMap(config_options, table_options,
+                                          opts_map, new_table_options);
 }
 
 Status GetBlockBasedTableOptionsFromMap(

table/plain/plain_table_factory.cc

@@ -142,14 +142,8 @@ Status GetPlainTableOptionsFromString(const ConfigOptions& config_options,
     return s;
   }
 
-  s = GetPlainTableOptionsFromMap(config_options, table_options, opts_map,
-                                  new_table_options);
-  // Translate any errors (NotFound, NotSupported, to InvalidArgument
-  if (s.ok() || s.IsInvalidArgument()) {
-    return s;
-  } else {
-    return Status::InvalidArgument(s.getState());
-  }
+  return GetPlainTableOptionsFromMap(config_options, table_options, opts_map,
+                                     new_table_options);
 }
 
 Status GetMemTableRepFactoryFromString(

test_util/testutil.cc

@@ -309,6 +309,7 @@ void RandomInitDBOptions(DBOptions* db_opt, Random* rnd) {
   db_opt->error_if_exists = rnd->Uniform(2);
   db_opt->is_fd_close_on_exec = rnd->Uniform(2);
   db_opt->paranoid_checks = rnd->Uniform(2);
+  db_opt->track_and_verify_wals_in_manifest = rnd->Uniform(2);
   db_opt->skip_log_error_on_recovery = rnd->Uniform(2);
   db_opt->skip_stats_update_on_db_open = rnd->Uniform(2);
   db_opt->skip_checking_sst_file_sizes_on_db_open = rnd->Uniform(2);

tools/db_crashtest.py

@@ -25,7 +25,6 @@ import argparse
 #   for txn:
 #       default_params < {blackbox,whitebox}_default_params < txn_params < args
 
-expected_values_file = tempfile.NamedTemporaryFile()
 
 default_params = {
     "acquire_snapshot_one_in": 10000,
@@ -59,7 +58,7 @@ default_params = {
     "destroy_db_initially": 0,
     "enable_pipelined_write": lambda: random.randint(0, 1),
     "enable_compaction_filter": lambda: random.choice([0, 0, 0, 1]),
-    "expected_values_path": expected_values_file.name,
+    "expected_values_path": lambda: setup_expected_values_file(),
     "flush_one_in": 1000000,
     "file_checksum_impl": lambda: random.choice(["none", "crc32c", "xxh64", "big"]),
     "get_live_files_one_in": 1000000,
@@ -159,6 +158,24 @@ def get_dbname(test_name):
         os.mkdir(dbname)
     return dbname
 
+expected_values_file = None
+def setup_expected_values_file():
+    global expected_values_file
+    if expected_values_file is not None:
+        return expected_values_file
+    expected_file_name = "rocksdb_crashtest_" + "expected"
+    test_tmpdir = os.environ.get(_TEST_DIR_ENV_VAR)
+    if test_tmpdir is None or test_tmpdir == "":
+        expected_values_file = tempfile.NamedTemporaryFile(
+            prefix=expected_file_name, delete=False).name
+    else:
+        # if tmpdir is specified, store the expected_values_file in the same dir
+        expected_values_file = test_tmpdir + "/" + expected_file_name
+        if os.path.exists(expected_values_file):
+            os.remove(expected_values_file)
+        open(expected_values_file, 'a').close()
+    return expected_values_file
+
 
 def is_direct_io_supported(dbname):
     with tempfile.NamedTemporaryFile(dir=dbname) as f:
@@ -618,6 +635,10 @@ def main():
         blackbox_crash_main(args, unknown_args)
     if args.test_type == 'whitebox':
         whitebox_crash_main(args, unknown_args)
+    # Only delete the `expected_values_file` if test passes
+    if os.path.exists(expected_values_file):
+        os.remove(expected_values_file)
+
 
 if __name__ == '__main__':
     main()

util/compression.h

@@ -540,43 +540,6 @@ inline bool CompressionTypeSupported(CompressionType compression_type) {
   }
 }
 
-inline bool DictCompressionTypeSupported(CompressionType compression_type) {
-  switch (compression_type) {
-    case kNoCompression:
-      return false;
-    case kSnappyCompression:
-      return false;
-    case kZlibCompression:
-      return Zlib_Supported();
-    case kBZip2Compression:
-      return false;
-    case kLZ4Compression:
-    case kLZ4HCCompression:
-#if LZ4_VERSION_NUMBER >= 10400  // r124+
-      return LZ4_Supported();
-#else
-      return false;
-#endif
-    case kXpressCompression:
-      return false;
-    case kZSTDNotFinalCompression:
-#if ZSTD_VERSION_NUMBER >= 500  // v0.5.0+
-      return ZSTDNotFinal_Supported();
-#else
-      return false;
-#endif
-    case kZSTD:
-#if ZSTD_VERSION_NUMBER >= 500  // v0.5.0+
-      return ZSTD_Supported();
-#else
-      return false;
-#endif
-    default:
-      assert(false);
-      return false;
-  }
-}
-
 inline std::string CompressionTypeToString(CompressionType compression_type) {
   switch (compression_type) {
     case kNoCompression:

util/rate_limiter_test.cc

@@ -59,7 +59,7 @@ TEST_F(RateLimiterTest, Modes) {
   }
 }
 
-#if !(defined(TRAVIS) && defined(OS_MACOSX))
+#if !((defined(TRAVIS) || defined(CIRCLECI)) && defined(OS_MACOSX))
 TEST_F(RateLimiterTest, Rate) {
   auto* env = Env::Default();
   struct Arg {

utilities/options/options_util.cc

@@ -65,11 +65,7 @@ Status GetLatestOptionsFileName(const std::string& dbpath,
   uint64_t latest_time_stamp = 0;
   std::vector<std::string> file_names;
   s = env->GetChildren(dbpath, &file_names);
-  if (s.IsNotFound()) {
-    return Status::NotFound(Status::kPathNotFound,
-                            "No options files found in the DB directory.",
-                            dbpath);
-  } else if (!s.ok()) {
+  if (!s.ok()) {
     return s;
   }
   for (auto& file_name : file_names) {
@@ -83,9 +79,7 @@ Status GetLatestOptionsFileName(const std::string& dbpath,
     }
   }
   if (latest_file_name.size() == 0) {
-    return Status::NotFound(Status::kPathNotFound,
-                            "No options files found in the DB directory.",
-                            dbpath);
+    return Status::NotFound("No options files found in the DB directory.");
   }
   *options_file_name = latest_file_name;
   return Status::OK();

utilities/options/options_util_test.cc

@@ -11,8 +11,6 @@
 #include <cinttypes>
 #include <unordered_map>
 
-#include "env/mock_env.h"
-#include "file/filename.h"
 #include "options/options_parser.h"
 #include "rocksdb/convenience.h"
 #include "rocksdb/db.h"
@@ -33,12 +31,14 @@ namespace ROCKSDB_NAMESPACE {
 class OptionsUtilTest : public testing::Test {
  public:
   OptionsUtilTest() : rnd_(0xFB) {
-    env_.reset(NewMemEnv(Env::Default()));
+    env_.reset(new test::StringEnv(Env::Default()));
+    fs_.reset(new LegacyFileSystemWrapper(env_.get()));
     dbname_ = test::PerThreadDBPath("options_util_test");
   }
 
  protected:
-  std::unique_ptr<Env> env_;
+  std::unique_ptr<test::StringEnv> env_;
+  std::unique_ptr<LegacyFileSystemWrapper> fs_;
   std::string dbname_;
   Random rnd_;
 };
@@ -58,8 +58,8 @@ TEST_F(OptionsUtilTest, SaveAndLoad) {
   }
 
   const std::string kFileName = "OPTIONS-123456";
-  ASSERT_OK(PersistRocksDBOptions(db_opt, cf_names, cf_opts, kFileName,
-                                  env_->GetFileSystem().get()));
+  ASSERT_OK(
+      PersistRocksDBOptions(db_opt, cf_names, cf_opts, kFileName, fs_.get()));
 
   DBOptions loaded_db_opt;
   std::vector<ColumnFamilyDescriptor> loaded_cf_descs;
@@ -85,7 +85,6 @@ TEST_F(OptionsUtilTest, SaveAndLoad) {
         exact, cf_opts[i], loaded_cf_descs[i].options));
   }
 
-  DestroyDB(dbname_, Options(db_opt, cf_opts[0]));
   for (size_t i = 0; i < kCFCount; ++i) {
     if (cf_opts[i].compaction_filter) {
       delete cf_opts[i].compaction_filter;
@@ -122,8 +121,8 @@ TEST_F(OptionsUtilTest, SaveAndLoadWithCacheCheck) {
   cf_names.push_back("cf_plain_table_sample");
   // Saving DB in file
   const std::string kFileName = "OPTIONS-LOAD_CACHE_123456";
-  ASSERT_OK(PersistRocksDBOptions(db_opt, cf_names, cf_opts, kFileName,
-                                  env_->GetFileSystem().get()));
+  ASSERT_OK(
+      PersistRocksDBOptions(db_opt, cf_names, cf_opts, kFileName, fs_.get()));
   DBOptions loaded_db_opt;
   std::vector<ColumnFamilyDescriptor> loaded_cf_descs;
 
@@ -155,7 +154,6 @@ TEST_F(OptionsUtilTest, SaveAndLoadWithCacheCheck) {
       ASSERT_EQ(loaded_bbt_opt->block_cache.get(), cache.get());
     }
   }
-  DestroyDB(dbname_, Options(loaded_db_opt, cf_opts[0]));
 }
 
 namespace {
@@ -361,280 +359,8 @@ TEST_F(OptionsUtilTest, SanityCheck) {
     ASSERT_OK(
         CheckOptionsCompatibility(config_options, dbname_, db_opt, cf_descs));
   }
-  DestroyDB(dbname_, Options(db_opt, cf_descs[0].options));
 }
 
-TEST_F(OptionsUtilTest, LatestOptionsNotFound) {
-  std::unique_ptr<Env> env(NewMemEnv(Env::Default()));
-  Status s;
-  Options options;
-  ConfigOptions config_opts;
-  std::vector<ColumnFamilyDescriptor> cf_descs;
-
-  options.env = env.get();
-  options.create_if_missing = true;
-  config_opts.env = options.env;
-  config_opts.ignore_unknown_options = false;
-
-  std::vector<std::string> children;
-
-  std::string options_file_name;
-  DestroyDB(dbname_, options);
-  // First, test where the db directory does not exist
-  ASSERT_NOK(options.env->GetChildren(dbname_, &children));
-
-  s = GetLatestOptionsFileName(dbname_, options.env, &options_file_name);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  s = LoadLatestOptions(dbname_, options.env, &options, &cf_descs);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  s = LoadLatestOptions(config_opts, dbname_, &options, &cf_descs);
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  s = GetLatestOptionsFileName(dbname_, options.env, &options_file_name);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  // Second, test where the db directory exists but is empty
-  ASSERT_OK(options.env->CreateDir(dbname_));
-
-  s = GetLatestOptionsFileName(dbname_, options.env, &options_file_name);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  s = LoadLatestOptions(dbname_, options.env, &options, &cf_descs);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  // Finally, test where a file exists but is not an "Options" file
-  std::unique_ptr<WritableFile> file;
-  ASSERT_OK(
-      options.env->NewWritableFile(dbname_ + "/temp.txt", &file, EnvOptions()));
-  ASSERT_OK(file->Close());
-  s = GetLatestOptionsFileName(dbname_, options.env, &options_file_name);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-
-  s = LoadLatestOptions(config_opts, dbname_, &options, &cf_descs);
-  ASSERT_TRUE(s.IsNotFound());
-  ASSERT_TRUE(s.IsPathNotFound());
-  ASSERT_OK(options.env->DeleteFile(dbname_ + "/temp.txt"));
-  ASSERT_OK(options.env->DeleteDir(dbname_));
-}
-
-TEST_F(OptionsUtilTest, LoadLatestOptions) {
-  Options options;
-  options.OptimizeForSmallDb();
-  ColumnFamilyDescriptor cf_desc;
-  ConfigOptions config_opts;
-  DBOptions db_opts;
-  std::vector<ColumnFamilyDescriptor> cf_descs;
-  std::vector<ColumnFamilyHandle*> handles;
-  DB* db;
-  options.create_if_missing = true;
-
-  DestroyDB(dbname_, options);
-
-  cf_descs.emplace_back();
-  cf_descs.back().name = kDefaultColumnFamilyName;
-  cf_descs.back().options.table_factory.reset(NewBlockBasedTableFactory());
-  cf_descs.emplace_back();
-  cf_descs.back().name = "Plain";
-  cf_descs.back().options.table_factory.reset(NewPlainTableFactory());
-  db_opts.create_missing_column_families = true;
-  db_opts.create_if_missing = true;
-
-  // open and persist the options
-  ASSERT_OK(DB::Open(db_opts, dbname_, cf_descs, &handles, &db));
-
-  std::string options_file_name;
-  std::string new_options_file;
-
-  ASSERT_OK(GetLatestOptionsFileName(dbname_, options.env, &options_file_name));
-  ASSERT_OK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  ASSERT_EQ(cf_descs.size(), 2U);
-  ASSERT_OK(RocksDBOptionsParser::VerifyDBOptions(config_opts,
-                                                  db->GetDBOptions(), db_opts));
-  ASSERT_OK(handles[0]->GetDescriptor(&cf_desc));
-  ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_opts, cf_desc.options,
-                                                  cf_descs[0].options));
-  ASSERT_OK(handles[1]->GetDescriptor(&cf_desc));
-  ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_opts, cf_desc.options,
-                                                  cf_descs[1].options));
-
-  // Now change some of the DBOptions
-  ASSERT_OK(db->SetDBOptions(
-      {{"delayed_write_rate", "1234"}, {"bytes_per_sync", "32768"}}));
-  ASSERT_OK(GetLatestOptionsFileName(dbname_, options.env, &new_options_file));
-  ASSERT_NE(options_file_name, new_options_file);
-  ASSERT_OK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  ASSERT_OK(RocksDBOptionsParser::VerifyDBOptions(config_opts,
-                                                  db->GetDBOptions(), db_opts));
-  options_file_name = new_options_file;
-
-  // Now change some of the ColumnFamilyOptions
-  ASSERT_OK(db->SetOptions(handles[1], {{"write_buffer_size", "32768"}}));
-  ASSERT_OK(GetLatestOptionsFileName(dbname_, options.env, &new_options_file));
-  ASSERT_NE(options_file_name, new_options_file);
-  ASSERT_OK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  ASSERT_OK(RocksDBOptionsParser::VerifyDBOptions(config_opts,
-                                                  db->GetDBOptions(), db_opts));
-  ASSERT_OK(handles[0]->GetDescriptor(&cf_desc));
-  ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_opts, cf_desc.options,
-                                                  cf_descs[0].options));
-  ASSERT_OK(handles[1]->GetDescriptor(&cf_desc));
-  ASSERT_OK(RocksDBOptionsParser::VerifyCFOptions(config_opts, cf_desc.options,
-                                                  cf_descs[1].options));
-
-  // close the db
-  for (auto* handle : handles) {
-    delete handle;
-  }
-  delete db;
-  DestroyDB(dbname_, options, cf_descs);
-}
-
-static void WriteOptionsFile(Env* env, const std::string& path,
-                             const std::string& options_file, int major,
-                             int minor, const std::string& db_opts,
-                             const std::string& cf_opts,
-                             const std::string& bbt_opts = "") {
-  std::string options_file_header =
-      "\n"
-      "[Version]\n"
-      "  rocksdb_version=" +
-      ToString(major) + "." + ToString(minor) +
-      ".0\n"
-      "  options_file_version=1\n";
-
-  std::unique_ptr<WritableFile> wf;
-  ASSERT_OK(env->NewWritableFile(path + "/" + options_file, &wf, EnvOptions()));
-  ASSERT_OK(
-      wf->Append(options_file_header + "[ DBOptions ]\n" + db_opts + "\n"));
-  ASSERT_OK(wf->Append(
-      "[CFOptions   \"default\"]  # column family must be specified\n" +
-      cf_opts + "\n"));
-  ASSERT_OK(wf->Append("[TableOptions/BlockBasedTable   \"default\"]\n" +
-                       bbt_opts + "\n"));
-  ASSERT_OK(wf->Close());
-
-  std::string latest_options_file;
-  ASSERT_OK(GetLatestOptionsFileName(path, env, &latest_options_file));
-  ASSERT_EQ(latest_options_file, options_file);
-}
-
-TEST_F(OptionsUtilTest, BadLatestOptions) {
-  Status s;
-  ConfigOptions config_opts;
-  DBOptions db_opts;
-  std::vector<ColumnFamilyDescriptor> cf_descs;
-  Options options;
-  options.env = env_.get();
-  config_opts.env = env_.get();
-  config_opts.ignore_unknown_options = false;
-  config_opts.delimiter = "\n";
-
-  ConfigOptions ignore_opts = config_opts;
-  ignore_opts.ignore_unknown_options = true;
-
-  std::string options_file_name;
-
-  // Test where the db directory exists but is empty
-  ASSERT_OK(options.env->CreateDir(dbname_));
-  ASSERT_NOK(
-      GetLatestOptionsFileName(dbname_, options.env, &options_file_name));
-  ASSERT_NOK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-
-  // Write an options file for a previous major release with an unknown DB
-  // Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0001", ROCKSDB_MAJOR - 1,
-                   ROCKSDB_MINOR, "unknown_db_opt=true", "");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Write an options file for a previous minor release with an unknown CF
-  // Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0002", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR - 1, "", "unknown_cf_opt=true");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Write an options file for a previous minor release with an unknown BBT
-  // Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0003", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR - 1, "", "", "unknown_bbt_opt=true");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
-  // Write an options file for the current release with an unknown DB Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0004", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR, "unknown_db_opt=true", "");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
-  // Write an options file for the current release with an unknown CF Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0005", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR, "", "unknown_cf_opt=true");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
-  // Write an options file for the current release with an invalid DB Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0006", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR, "create_if_missing=hello", "");
-  s = LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-  // Even though ignore_unknown_options=true, we still return an error...
-  s = LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs);
-  ASSERT_NOK(s);
-  ASSERT_TRUE(s.IsInvalidArgument());
-
-  // Write an options file for the next release with an invalid DB Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0007", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR + 1, "create_if_missing=hello", "");
-  ASSERT_NOK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  ASSERT_OK(LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs));
-
-  // Write an options file for the next release with an unknown DB Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0008", ROCKSDB_MAJOR,
-                   ROCKSDB_MINOR + 1, "unknown_db_opt=true", "");
-  ASSERT_NOK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  // Ignore the errors for future releases when ignore_unknown_options=true
-  ASSERT_OK(LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs));
-
-  // Write an options file for the next major release with an unknown CF Option
-  WriteOptionsFile(options.env, dbname_, "OPTIONS-0009", ROCKSDB_MAJOR + 1,
-                   ROCKSDB_MINOR, "", "unknown_cf_opt=true");
-  ASSERT_NOK(LoadLatestOptions(config_opts, dbname_, &db_opts, &cf_descs));
-  // Ignore the errors for future releases when ignore_unknown_options=true
-  ASSERT_OK(LoadLatestOptions(ignore_opts, dbname_, &db_opts, &cf_descs));
-}
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {