Fix the logic of setting read_amp_bytes_per_bit from OPTIONS file (#7680 )

Summary: Instead of using `EncodeFixed32` which always serialize a integer to little endian, we should use the local machine's endianness when populating a native data structure during options parsing. Without this fix, `read_amp_bytes_per_bit` may be populated incorrectly on big-endian machines. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7680 Test Plan: make check Reviewed By: pdillinger Differential Revision: D24999166 Pulled By: riversand963 fbshipit-source-id: dc603cff6e17f8fa32479ce6df93b93082e6b0c4
Bump version and update HISTORY
2026-11-06 19:12:36 -05:00 · 2020-11-17 10:18:19 -08:00 · 2020-11-15 15:04:28 -08:00 · 2020-11-15 14:52:54 -08:00 · 2020-11-11 12:21:58 -08:00 · 2020-10-12 13:15:15 -07:00
120 changed files with 1261 additions and 2677 deletions
@@ -53,21 +53,7 @@ jobs:
    resource_class: 2xlarge
    steps:
      - checkout # check out the code in the project directory
-      - run: sudo apt-get update -y
-      - run: sudo apt-get install -y clang
-      - run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make all -j32
-
-  build-linux-clang10-no-test:
-    machine:
-      image: ubuntu-1604:201903-01
-    resource_class: 2xlarge
-    steps:
-      - checkout # check out the code in the project directory
-      - run: sudo sh -c 'echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" >> /etc/apt/sources.list'
-      - run: sudo sh -c 'echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" >> /etc/apt/sources.list'
-      - run: sudo apt-get update -y
-      - run: sudo apt-get install -y clang-10
-      - run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make all -j32 # aligned new doesn't work for reason we haven't figured out
+      - run: USE_CLANG=1 make all -j32

  build-linux-cmake:
    machine:
@@ -137,9 +123,6 @@ workflows:
  build-linux-clang-no-test:
    jobs:
      - build-linux-clang-no-test
-  build-linux-clang10-no-test:
-    jobs:
-      - build-linux-clang10-no-test
  build-linux-cmake:
    jobs:
      - build-linux-cmake
@@ -95,7 +95,7 @@ if(MSVC)
  option(WITH_XPRESS "build with windows built in compression" OFF)
  include(${CMAKE_CURRENT_SOURCE_DIR}/thirdparty.inc)
 else()
-  if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD" AND NOT CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
+  if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
    # FreeBSD has jemalloc as default malloc
    # but it does not have all the jemalloc files in include/...
    set(WITH_JEMALLOC ON)
@@ -448,8 +448,6 @@ elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
  add_definitions(-DOS_LINUX)
 elseif(CMAKE_SYSTEM_NAME MATCHES "SunOS")
  add_definitions(-DOS_SOLARIS)
-elseif(CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
-  add_definitions(-DOS_GNU_KFREEBSD)
 elseif(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
  add_definitions(-DOS_FREEBSD)
 elseif(CMAKE_SYSTEM_NAME MATCHES "NetBSD")
@@ -976,19 +974,12 @@ if(NOT WIN32 OR ROCKSDB_INSTALL_ON_WINDOWS)
  )
 endif()

-option(WITH_ALL_TESTS "Build all test, rather than a small subset" ON)
-
 if(WITH_TESTS)
  add_subdirectory(third-party/gtest-1.8.1/fused-src/gtest)
  add_library(testharness STATIC
  test_util/testharness.cc)
  target_link_libraries(testharness gtest)
  set(TESTS
-        db/db_basic_test.cc
-        env/env_basic_test.cc
-  )
-  if(WITH_ALL_TESTS)
-    list(APPEND TESTS
        cache/cache_test.cc
        cache/lru_cache_test.cc
        db/blob/blob_file_addition_test.cc
@@ -1003,6 +994,7 @@ if(WITH_TESTS)
        db/comparator_db_test.cc
        db/corruption_test.cc
        db/cuckoo_table_db_test.cc
+        db/db_basic_test.cc
        db/db_with_timestamp_basic_test.cc
        db/db_block_cache_test.cc
        db/db_bloom_filter_test.cc
@@ -1065,6 +1057,7 @@ if(WITH_TESTS)
        db/write_batch_test.cc
        db/write_callback_test.cc
        db/write_controller_test.cc
+        env/env_basic_test.cc
        env/env_test.cc
        env/io_posix_test.cc
        env/mock_env_test.cc
@@ -1147,8 +1140,7 @@ if(WITH_TESTS)
        utilities/transactions/write_unprepared_transaction_test.cc
        utilities/ttl/ttl_test.cc
        utilities/write_batch_with_index/write_batch_with_index_test.cc
-    )
-  endif()
+  )
  if(WITH_LIBRADOS)
    list(APPEND TESTS utilities/env_librados_test.cc)
  endif()
@@ -1269,8 +1261,3 @@ if(WITH_TOOLS)
  add_custom_target(tools
    DEPENDS ${tool_deps})
 endif()
-
-option(WITH_EXAMPLES "build with examples" OFF)
-if(WITH_EXAMPLES)
-  add_subdirectory(examples)
-endif()
@@ -1,23 +1,37 @@
 # Rocksdb Change Log
-## Unreleased
-### Public API Change
-* Encryption file classes now exposed for inheritance in env_encryption.h
+## 6.11.7 (11/15/2020)
+### Bug Fixes
+* Fix a bug of encoding and parsing BlockBasedTableOptions::read_amp_bytes_per_bit as a 64-bit integer.
+* Fixed the logic of populating native data structure for `read_amp_bytes_per_bit` during OPTIONS file parsing on big-endian architecture. Without this fix, original code introduced in PR7659, when running on big-endian machine, can mistakenly store read_amp_bytes_per_bit (an uint32) in little endian format. Future access to `read_amp_bytes_per_bit` will give wrong values. Little endian architecture is not affected.

-### Behavior Changes
+## 6.11.6 (10/12/2020)
+### Bug Fixes
+* Fixed a bug in the following combination of features: indexes with user keys (`format_version >= 3`), indexes are partitioned (`index_type == kTwoLevelIndexSearch`), and some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`). The bug could cause keys to be truncated when read from the index leading to wrong read results or other unexpected behavior.
+* Fixed a bug when indexes are partitioned (`index_type == kTwoLevelIndexSearch`), some index partitions are pinned in memory (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache`), and partitions reads could be mixed between block cache and directly from the file (e.g., with `enable_index_compression == 1` and `mmap_read == 1`, partitions that were stored uncompressed due to poor compression ratio would be read directly from the file via mmap, while partitions that were stored compressed would be read from block cache). The bug could cause index partitions to be mistakenly considered empty during reads leading to wrong read results.
+
+## 6.11.5 (7/23/2020)
+### Bug Fixes
+* Memtable lookup should report unrecognized value_type as corruption (#7121).
+
+## 6.11.4 (7/15/2020)
+### Bug Fixes
+* Make compaction report InternalKey corruption while iterating over the input.
+
+## 6.11.3 (7/9/2020)
+### Bug Fixes
+* Fix a bug when index_type == kTwoLevelIndexSearch in PartitionedIndexBuilder to update FlushPolicy to point to internal key partitioner when it changes from user-key mode to internal-key mode in index partition.
+* Disable file deletion after MANIFEST write/sync failure until db re-open or Resume() so that subsequent re-open will not see MANIFEST referencing deleted SSTs.
+
+## 6.11.1 (6/23/2020)
+### Bug Fixes
 * Best-efforts recovery ignores CURRENT file completely. If CURRENT file is missing during recovery, best-efforts recovery still proceeds with MANIFEST file(s).
 * In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early.
 * Compressed block cache was automatically disabled with read-only DBs by mistake. Now it is fixed: compressed block cache will be in effective with read-only DB too.
+* Fail recovery and report once hitting a physical log record checksum mismatch, while reading MANIFEST. RocksDB should not continue processing the MANIFEST any further.
 * Fix a bug of wrong iterator result if another thread finishes an update and a DB flush between two statement.

 ### Public API Change
-* `DB::GetDbSessionId(std::string& session_id)` is added. `session_id` stores a unique identifier that gets reset every time the DB is opened. This DB session ID should be unique among all open DB instances on all hosts, and should be unique among re-openings of the same or other DBs. This identifier is recorded in the LOG file on the line starting with "DB Session ID:".
-
-### New Features
-* DB identity (`db_id`) and DB session identity (`db_session_id`) are added to table properties and stored in SST files. SST files generated from SstFileWriter and Repairer have DB identity “SST Writer” and “DB Repairer”, respectively. Their DB session IDs are generated in the same way as `DB::GetDbSessionId`. The session ID for SstFileWriter (resp., Repairer) resets every time `SstFileWriter::Open` (resp., `Repairer::Run`) is called.
-* Added experimental option BlockBasedTableOptions::optimize_filters_for_memory for reducing allocated memory size of Bloom filters (~10% savings with Jemalloc) while preserving the same general accuracy. To have an effect, the option requires format_version=5 and malloc_usable_size. Enabling this option is forward and backward compatible with existing format_version=5.
-
-### Bug Fixes
-* Fail recovery and report once hitting a physical log record checksum mismatch, while reading MANIFEST. RocksDB should not continue processing the MANIFEST any further.
+* `DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`.

 ## 6.11 (6/12/2020)
 ### Bug Fixes
@@ -43,7 +57,6 @@
 * Disable delete triggered compaction (NewCompactOnDeletionCollectorFactory) in universal compaction mode and num_levels = 1 in order to avoid a corruption bug.
 * `pin_l0_filter_and_index_blocks_in_cache` no longer applies to L0 files larger than `1.5 * write_buffer_size` to give more predictable memory usage. Such L0 files may exist due to intra-L0 compaction, external file ingestion, or user dynamically changing `write_buffer_size` (note, however, that files that are already pinned will continue being pinned, even after such a dynamic change).
 * In point-in-time wal recovery mode, fail database recovery in case of IOError while reading the WAL to avoid data loss.
-* A new method `Env::LowerThreadPoolCPUPriority(Priority, CpuPriority)` is added to `Env` to be able to lower to a specific priority such as `CpuPriority::kIdle`.

 ### New Features
 * sst_dump to add a new --readahead_size argument. Users can specify read size when scanning the data. Sst_dump also tries to prefetch tail part of the SST files so usually some number of I/Os are saved there too.
@@ -9,7 +9,9 @@
 BASH_EXISTS := $(shell which bash)
 SHELL := $(shell which bash)
 # Default to python3. Some distros like CentOS 8 do not have `python`.
-PYTHON?=$(shell which python3 || which python || echo python3)
+ifeq ($(origin PYTHON), undefined)
+	PYTHON := $(shell which python3 || which python || echo python3)
+endif
 export PYTHON

 CLEAN_FILES = # deliberately empty, so we can append below.
@@ -640,7 +642,6 @@ TESTS = \
 	timer_test \
 	db_with_timestamp_compaction_test \
 	testutil_test \
-	io_tracer_test \

 PARALLEL_TEST = \
 	backupable_db_test \
@@ -675,9 +676,9 @@ ifdef COMPILE_WITH_UBSAN
        TESTS := $(shell echo $(TESTS) | sed 's/\boptions_settable_test\b//g')
 endif
 ifdef ASSERT_STATUS_CHECKED
-	# This is a new check for which we will add support incrementally. This
-	# list can be removed once support is fully added.
-	TESTS_PASSING_ASC = \
+	# This is a new check for which we will add support incrementally. The
+	# whitelist can be removed once support is fully added.
+	TESTS_WHITELIST = \
 		arena_test \
 		autovector_test \
 		blob_file_addition_test \
@@ -723,8 +724,8 @@ ifdef ASSERT_STATUS_CHECKED
 		work_queue_test \
 		write_controller_test \

-	TESTS := $(filter $(TESTS_PASSING_ASC),$(TESTS))
-	PARALLEL_TEST := $(filter $(TESTS_PASSING_ASC),$(PARALLEL_TEST))
+	TESTS := $(filter $(TESTS_WHITELIST),$(TESTS))
+	PARALLEL_TEST := $(filter $(TESTS_WHITELIST),$(PARALLEL_TEST))
 endif
 SUBSET := $(TESTS)
 ifdef ROCKSDBTESTS_START
@@ -845,8 +846,7 @@ endif  # PLATFORM_SHARED_EXT
 	dbg rocksdbjavastatic rocksdbjava install install-static install-shared uninstall \
 	analyze tools tools_lib \
 	blackbox_crash_test_with_atomic_flush whitebox_crash_test_with_atomic_flush  \
-	blackbox_crash_test_with_txn whitebox_crash_test_with_txn \
-	blackbox_crash_test_with_best_efforts_recovery
+	blackbox_crash_test_with_txn whitebox_crash_test_with_txn


 all: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(TESTS)
@@ -978,7 +978,7 @@ J ?= 100%
 # Use this regexp to select the subset of tests whose names match.
 tests-regexp = .

-ifeq ($(PRINT_PARALLEL_OUTPUTS), 1)
+ifeq ($(PRINT_PARALLEL_OUTPUTS), 1)	
 	parallel_com = '{}'
 else
 	parallel_com = '{} >& t/log-{/}'
@@ -1002,7 +1002,7 @@ check_0:
 	awk '{ if ($$7 != 0 || $$8 != 0) { if ($$7 == "Exitval") { h = $$0; } else { if (!f) print h; print; f = 1 } } } END { if(f) exit 1; }' < LOG ; \
 	if [ $$parallel_retcode -ne 0 ] ; then exit 1 ; fi

-valgrind-exclude-regexp = InlineSkipTest.ConcurrentInsert|TransactionStressTest.DeadlockStress|DBCompactionTest.SuggestCompactRangeNoTwoLevel0Compactions|BackupableDBTest.RateLimiting|DBTest.CloseSpeedup|DBTest.ThreadStatusFlush|DBTest.RateLimitingTest|DBTest.EncodeDecompressedBlockSizeTest|FaultInjectionTest.UninstalledCompaction|HarnessTest.Randomized|ExternalSSTFileTest.CompactDuringAddFileRandom|ExternalSSTFileTest.IngestFileWithGlobalSeqnoRandomized|MySQLStyleTransactionTest.TransactionStressTest
+valgrind-blacklist-regexp = InlineSkipTest.ConcurrentInsert|TransactionStressTest.DeadlockStress|DBCompactionTest.SuggestCompactRangeNoTwoLevel0Compactions|BackupableDBTest.RateLimiting|DBTest.CloseSpeedup|DBTest.ThreadStatusFlush|DBTest.RateLimitingTest|DBTest.EncodeDecompressedBlockSizeTest|FaultInjectionTest.UninstalledCompaction|HarnessTest.Randomized|ExternalSSTFileTest.CompactDuringAddFileRandom|ExternalSSTFileTest.IngestFileWithGlobalSeqnoRandomized|MySQLStyleTransactionTest.TransactionStressTest

 .PHONY: valgrind_check_0
 valgrind_check_0:
@@ -1017,7 +1017,7 @@ valgrind_check_0:
 	}								\
 	  | $(prioritize_long_running_tests)				\
 	  | grep -E '$(tests-regexp)'					\
-	  | grep -E -v '$(valgrind-exclude-regexp)'					\
+	  | grep -E -v '$(valgrind-blacklist-regexp)'					\
 	  | build_tools/gnu_parallel -j$(J) --plain --joblog=LOG $$eta --gnu \
 	  '(if [[ "{}" == "./"* ]] ; then $(DRIVER) {}; else {}; fi) ' \
 	  '>& t/valgrind_log-{/}'
@@ -1059,7 +1059,7 @@ ifndef ASSERT_STATUS_CHECKED # not yet working with these tests
 	sh tools/rocksdb_dump_test.sh
 endif
 endif
-endif
+endif	
 ifndef SKIP_FORMAT_BUCK_CHECKS
 	$(MAKE) check-format
 	$(MAKE) check-buck-targets
@@ -1079,8 +1079,6 @@ crash_test_with_atomic_flush: whitebox_crash_test_with_atomic_flush blackbox_cra

 crash_test_with_txn: whitebox_crash_test_with_txn blackbox_crash_test_with_txn

-crash_test_with_best_efforts_recovery: blackbox_crash_test_with_best_efforts_recovery
-
 blackbox_crash_test: db_stress
 	$(PYTHON) -u tools/db_crashtest.py --simple blackbox $(CRASH_TEST_EXT_ARGS)
 	$(PYTHON) -u tools/db_crashtest.py blackbox $(CRASH_TEST_EXT_ARGS)
@@ -1091,9 +1089,6 @@ blackbox_crash_test_with_atomic_flush: db_stress
 blackbox_crash_test_with_txn: db_stress
 	$(PYTHON) -u tools/db_crashtest.py --txn blackbox $(CRASH_TEST_EXT_ARGS)

-blackbox_crash_test_with_best_efforts_recovery: db_stress
-	$(PYTHON) -u tools/db_crashtest.py --test_best_efforts_recovery blackbox $(CRASH_TEST_EXT_ARGS)
-
 ifeq ($(CRASH_TEST_KILL_ODD),)
  CRASH_TEST_KILL_ODD=888887
 endif
@@ -1132,11 +1127,6 @@ asan_crash_test_with_txn:
 	COMPILE_WITH_ASAN=1 $(MAKE) crash_test_with_txn
 	$(MAKE) clean

-asan_crash_test_with_best_efforts_recovery:
-	$(MAKE) clean
-	COMPILE_WITH_ASAN=1 $(MAKE) crash_test_with_best_efforts_recovery
-	$(MAKE) clean
-
 ubsan_check:
 	$(MAKE) clean
 	COMPILE_WITH_UBSAN=1 $(MAKE) check -j32
@@ -1157,11 +1147,6 @@ ubsan_crash_test_with_txn:
 	COMPILE_WITH_UBSAN=1 $(MAKE) crash_test_with_txn
 	$(MAKE) clean

-ubsan_crash_test_with_best_efforts_recovery:
-	$(MAKE) clean
-	COMPILE_WITH_UBSAN=1 $(MAKE) crash_test_with_best_efforts_recovery
-	$(MAKE) clean
-
 valgrind_test:
 	ROCKSDB_VALGRIND_RUN=1 DISABLE_JEMALLOC=1 $(MAKE) valgrind_check

@@ -1897,8 +1882,6 @@ timer_test: util/timer_test.o $(LIBOBJECTS) $(TESTHARNESS)

 testutil_test: test_util/testutil_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)
-io_tracer_test: trace_replay/io_tracer_test.o trace_replay/io_tracer.o $(LIBOBJECTS) $(TESTHARNESS)
-	$(AM_LINK)

 #-------------------------------------------------
 # make install related stuff
@@ -1,8 +1,7 @@
 ## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

-[![CircleCI Status](https://circleci.com/gh/facebook/rocksdb.svg?style=svg)](https://circleci.com/gh/facebook/rocksdb)
-[![TravisCI Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
-[![Appveyor Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/master?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/master)
+[![Linux/Mac Build Status](https://travis-ci.org/facebook/rocksdb.svg?branch=master)](https://travis-ci.org/facebook/rocksdb)
+[![Windows Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/master?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/master)
 [![PPC64le Build Status](http://140.211.168.68:8080/buildStatus/icon?job=Rocksdb)](http://140.211.168.68:8080/job/Rocksdb)

 RocksDB is developed and maintained by Facebook Database Engineering Team.
@@ -294,7 +294,6 @@ cpp_library(
        "tools/ldb_tool.cc",
        "tools/sst_dump_tool.cc",
        "trace_replay/block_cache_tracer.cc",
-        "trace_replay/io_tracer.cc",
        "trace_replay/trace_replay.cc",
        "util/build_version.cc",
        "util/coding.cc",
@@ -448,6 +447,27 @@ cpp_library(
    external_deps = ROCKSDB_EXTERNAL_DEPS,
 )

+if not is_opt_mode:
+    cpp_binary(
+        name = "c_test_bin",
+        srcs = ["db/c_test.c"],
+        arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
+        os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
+        compiler_flags = ROCKSDB_COMPILER_FLAGS,
+        preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+        deps = [":rocksdb_test_lib"],
+    )
+
+if not is_opt_mode:
+    custom_unittest(
+        "c_test",
+        command = [
+            native.package_name() + "/buckifier/rocks_test_runner.sh",
+            "$(location :{})".format("c_test_bin"),
+        ],
+        type = "simple",
+    )
+
 cpp_library(
    name = "env_basic_test_lib",
    srcs = ["env/env_basic_test.cc"],
@@ -561,13 +581,6 @@ ROCKS_TESTS = [
        [],
        [],
    ],
-    [
-        "c_test",
-        "db/c_test.c",
-        "serial",
-        [],
-        [],
-    ],
    [
        "cache_simulator_test",
        "utilities/simulator_cache/cache_simulator_test.cc",
@@ -1156,13 +1169,6 @@ ROCKS_TESTS = [
        [],
        [],
    ],
-    [
-        "io_tracer_test",
-        "trace_replay/io_tracer_test.cc",
-        "serial",
-        [],
-        [],
-    ],
    [
        "iostats_context_test",
        "monitoring/iostats_context_test.cc",
@@ -68,7 +68,7 @@ before_build:
  - md %APPVEYOR_BUILD_FOLDER%\build
  - cd %APPVEYOR_BUILD_FOLDER%\build
  - if DEFINED CMAKE_PLATEFORM_NAME (set "PLATEFORM_OPT=-A %CMAKE_PLATEFORM_NAME%")
-  - cmake .. -G "%CMAKE_GENERATOR%" %PLATEFORM_OPT% %CMAKE_OPT% -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DLZ4=1 -DZSTD=1 -DXPRESS=1 -DJNI=1 -DWITH_ALL_TESTS=0
+  - cmake .. -G "%CMAKE_GENERATOR%" %PLATEFORM_OPT% %CMAKE_OPT% -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DLZ4=1 -DZSTD=1 -DXPRESS=1 -DJNI=1
  - cd ..

 build:
@@ -79,7 +79,7 @@ build:
 test:

 test_script:
-  - ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,env_basic_test -Concurrency 8
+  - ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_with_timestamp_basic_test,db_test2,db_test,env_basic_test,env_test,db_merge_operand_test -Concurrency 8

 on_failure:
  - cmd: 7z a build-failed.zip %APPVEYOR_BUILD_FOLDER%\build\ && appveyor PushArtifact build-failed.zip
@@ -64,8 +64,6 @@ def get_cc_files(repo_path):
            continue
        for filename in fnmatch.filter(filenames, '*.cc'):
            cc_files.append(os.path.join(root, filename))
-        for filename in fnmatch.filter(filenames, '*.c'):
-            cc_files.append(os.path.join(root, filename))
    return cc_files


@@ -178,10 +176,18 @@ def generate_targets(repo_path, deps_map):
        + ["test_util/testutil.cc"])

    print("Extra dependencies:\n{0}".format(json.dumps(deps_map)))
-    # test for every test we found in the Makefile
+
+    # c_test.c is added through TARGETS.add_c_test(). If there
+    # are more than one .c test file, we need to extend
+    # TARGETS.add_c_test() to include other C tests too.
+    TARGETS.add_c_test()
+
+    # test for every .cc test we found in the Makefile
    for target_alias, deps in deps_map.items():
        for test in sorted(tests):
-            match_src = [src for src in cc_files if ("/%s.c" % test) in src]
+            if test == 'c_test':
+                continue
+            match_src = [src for src in cc_files if ("/%s.cc" % test) in src]
            if len(match_src) == 0:
                print(ColorString.warning("Cannot find .cc file for %s" % test))
                continue
@@ -76,6 +76,30 @@ class TARGETSBuilder(object):
            pretty_list(deps)))
        self.total_bin = self.total_bin + 1

+    def add_c_test(self):
+        self.targets_file.write("""
+if not is_opt_mode:
+    cpp_binary(
+        name = "c_test_bin",
+        srcs = ["db/c_test.c"],
+        arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
+        os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
+        compiler_flags = ROCKSDB_COMPILER_FLAGS,
+        preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
+        deps = [":rocksdb_test_lib"],
+    )
+
+if not is_opt_mode:
+    custom_unittest(
+        "c_test",
+        command = [
+            native.package_name() + "/buckifier/rocks_test_runner.sh",
+            "$(location :{})".format("c_test_bin"),
+        ],
+        type = "simple",
+    )
+""")
+
    def register_test(self,
                      test_name,
                      src,
@@ -192,17 +192,6 @@ EOF
        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread"
        # PORT_FILES=port/freebsd/freebsd_specific.cc
        ;;
-    GNU/kFreeBSD)
-        PLATFORM=OS_GNU_KFREEBSD
-        COMMON_FLAGS="$COMMON_FLAGS -DOS_GNU_KFREEBSD"
-        if [ -z "$USE_CLANG" ]; then
-            COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp"
-        else
-            PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -latomic"
-        fi
-        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread -lrt"
-        # PORT_FILES=port/gnu_kfreebsd/gnu_kfreebsd_specific.cc
-        ;;
    NetBSD)
        PLATFORM=OS_NETBSD
        COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_NETBSD"
@@ -51,8 +51,7 @@ TableBuilder* NewTableBuilder(
    uint64_t sample_for_compression, const CompressionOptions& compression_opts,
    int level, const bool skip_filters, const uint64_t creation_time,
    const uint64_t oldest_key_time, const uint64_t target_file_size,
-    const uint64_t file_creation_time, const std::string& db_id,
-    const std::string& db_session_id) {
+    const uint64_t file_creation_time) {
  assert((column_family_id ==
          TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
         column_family_name.empty());
@@ -62,7 +61,7 @@ TableBuilder* NewTableBuilder(
                          sample_for_compression, compression_opts,
                          skip_filters, column_family_name, level,
                          creation_time, oldest_key_time, target_file_size,
-                          file_creation_time, db_id, db_session_id),
+                          file_creation_time),
      column_family_id, file);
 }

@@ -86,8 +85,7 @@ Status BuildTable(
    EventLogger* event_logger, int job_id, const Env::IOPriority io_priority,
    TableProperties* table_properties, int level, const uint64_t creation_time,
    const uint64_t oldest_key_time, Env::WriteLifeTimeHint write_hint,
-    const uint64_t file_creation_time, const std::string& db_id,
-    const std::string& db_session_id) {
+    const uint64_t file_creation_time) {
  assert((column_family_id ==
          TablePropertiesCollectorFactory::Context::kUnknownColumnFamily) ==
         column_family_name.empty());
@@ -144,7 +142,7 @@ Status BuildTable(
          column_family_name, file_writer.get(), compression,
          sample_for_compression, compression_opts_for_flush, level,
          false /* skip_filters */, creation_time, oldest_key_time,
-          0 /*target_file_size*/, file_creation_time, db_id, db_session_id);
+          0 /*target_file_size*/, file_creation_time);
    }

    MergeHelper merge(env, internal_comparator.user_comparator(),
@@ -51,8 +51,7 @@ TableBuilder* NewTableBuilder(
    const CompressionOptions& compression_opts, int level,
    const bool skip_filters = false, const uint64_t creation_time = 0,
    const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0,
-    const uint64_t file_creation_time = 0, const std::string& db_id = "",
-    const std::string& db_session_id = "");
+    const uint64_t file_creation_time = 0);

 // Build a Table file from the contents of *iter.  The generated file
 // will be named according to number specified in meta. On success, the rest of
@@ -84,7 +83,6 @@ extern Status BuildTable(
    TableProperties* table_properties = nullptr, int level = -1,
    const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
    Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET,
-    const uint64_t file_creation_time = 0, const std::string& db_id = "",
-    const std::string& db_session_id = "");
+    const uint64_t file_creation_time = 0);

 }  // namespace ROCKSDB_NAMESPACE
@@ -246,6 +246,7 @@ void CompactionIterator::NextFromInput() {
    iter_stats_.num_input_records++;

    if (!ParseInternalKey(key_, &ikey_)) {
+      iter_stats_.num_input_corrupt_records++;
      // If `expect_valid_internal_key_` is false, return the corrupted key
      // and let the caller decide what to do with it.
      // TODO(noetzli): We should have a more elegant solution for this.
@@ -258,7 +259,6 @@ void CompactionIterator::NextFromInput() {
      has_current_user_key_ = false;
      current_user_key_sequence_ = kMaxSequenceNumber;
      current_user_key_snapshot_ = 0;
-      iter_stats_.num_input_corrupt_records++;
      valid_ = true;
      break;
    }
@@ -305,15 +305,12 @@ CompactionJob::CompactionJob(
    const SnapshotChecker* snapshot_checker, std::shared_ptr<Cache> table_cache,
    EventLogger* event_logger, bool paranoid_file_checks, bool measure_io_stats,
    const std::string& dbname, CompactionJobStats* compaction_job_stats,
-    Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused,
-    const std::string& db_id, const std::string& db_session_id)
+    Env::Priority thread_pri, const std::atomic<bool>* manual_compaction_paused)
    : job_id_(job_id),
      compact_(new CompactionState(compaction)),
      compaction_job_stats_(compaction_job_stats),
      compaction_stats_(compaction->compaction_reason(), 1),
      dbname_(dbname),
-      db_id_(db_id),
-      db_session_id_(db_session_id),
      db_options_(db_options),
      file_options_(file_options),
      env_(db_options.env),
@@ -724,7 +721,6 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
  cfd->internal_stats()->AddCompactionStats(
      compact_->compaction->output_level(), thread_pri_, compaction_stats_);

-  versions_->SetIOStatusOK();
  if (status.ok()) {
    status = InstallCompactionResults(mutable_cf_options);
  }
@@ -899,9 +895,10 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
  sub_compact->c_iter.reset(new CompactionIterator(
      input.get(), cfd->user_comparator(), &merge, versions_->LastSequence(),
      &existing_snapshots_, earliest_write_conflict_snapshot_,
-      snapshot_checker_, env_, ShouldReportDetailedTime(env_, stats_), false,
-      &range_del_agg, sub_compact->compaction, compaction_filter,
-      shutting_down_, preserve_deletes_seqnum_, manual_compaction_paused_,
+      snapshot_checker_, env_, ShouldReportDetailedTime(env_, stats_),
+      /*expect_valid_internal_key=*/true, &range_del_agg,
+      sub_compact->compaction, compaction_filter, shutting_down_,
+      preserve_deletes_seqnum_, manual_compaction_paused_,
      db_options_.info_log));
  auto c_iter = sub_compact->c_iter.get();
  c_iter->SeekToFirst();
@@ -1557,8 +1554,7 @@ Status CompactionJob::OpenCompactionOutputFile(
      sub_compact->compaction->output_compression_opts(),
      sub_compact->compaction->output_level(), skip_filters,
      oldest_ancester_time, 0 /* oldest_key_time */,
-      sub_compact->compaction->max_output_file_size(), current_time, db_id_,
-      db_session_id_));
+      sub_compact->compaction->max_output_file_size(), current_time));
  LogFlush(db_options_.info_log);
  return s;
 }
@@ -78,9 +78,7 @@ class CompactionJob {
                const std::string& dbname,
                CompactionJobStats* compaction_job_stats,
                Env::Priority thread_pri,
-                const std::atomic<bool>* manual_compaction_paused = nullptr,
-                const std::string& db_id = "",
-                const std::string& db_session_id = "");
+                const std::atomic<bool>* manual_compaction_paused = nullptr);

  ~CompactionJob();

@@ -154,8 +152,6 @@ class CompactionJob {

  // DBImpl state
  const std::string& dbname_;
-  const std::string db_id_;
-  const std::string db_session_id_;
  const ImmutableDBOptions& db_options_;
  const FileOptions file_options_;

@@ -395,7 +395,7 @@ TEST_F(CompactionJobTest, Simple) {
  RunCompaction({ files }, expected_results);
 }

-TEST_F(CompactionJobTest, SimpleCorrupted) {
+TEST_F(CompactionJobTest, DISABLED_SimpleCorrupted) {
  NewDB();

  auto expected_results = CreateTwoFiles(true);
@@ -989,7 +989,7 @@ TEST_F(CompactionJobTest, MultiSingleDelete) {
 // single deletion and the (single) deletion gets removed while the corrupt key
 // gets written out. TODO(noetzli): We probably want a better way to treat
 // corrupt keys.
-TEST_F(CompactionJobTest, CorruptionAfterDeletion) {
+TEST_F(CompactionJobTest, DISABLED_CorruptionAfterDeletion) {
  NewDB();

  auto file1 =
@@ -41,56 +41,6 @@ TEST_F(DBBasicTest, OpenWhenOpen) {
  delete db2;
 }

-TEST_F(DBBasicTest, UniqueSession) {
-  Options options = CurrentOptions();
-  std::string sid1, sid2, sid3, sid4;
-
-  db_->GetDbSessionId(sid1);
-  Reopen(options);
-  db_->GetDbSessionId(sid2);
-  ASSERT_OK(Put("foo", "v1"));
-  db_->GetDbSessionId(sid4);
-  Reopen(options);
-  db_->GetDbSessionId(sid3);
-
-  ASSERT_NE(sid1, sid2);
-  ASSERT_NE(sid1, sid3);
-  ASSERT_NE(sid2, sid3);
-
-  ASSERT_EQ(sid2, sid4);
-
-#ifndef ROCKSDB_LITE
-  Close();
-  ASSERT_OK(ReadOnlyReopen(options));
-  db_->GetDbSessionId(sid1);
-  // Test uniqueness between readonly open (sid1) and regular open (sid3)
-  ASSERT_NE(sid1, sid3);
-  Close();
-  ASSERT_OK(ReadOnlyReopen(options));
-  db_->GetDbSessionId(sid2);
-  ASSERT_EQ("v1", Get("foo"));
-  db_->GetDbSessionId(sid3);
-
-  ASSERT_NE(sid1, sid2);
-
-  ASSERT_EQ(sid2, sid3);
-#endif  // ROCKSDB_LITE
-
-  CreateAndReopenWithCF({"goku"}, options);
-  db_->GetDbSessionId(sid1);
-  ASSERT_OK(Put("bar", "e1"));
-  db_->GetDbSessionId(sid2);
-  ASSERT_EQ("e1", Get("bar"));
-  db_->GetDbSessionId(sid3);
-  ReopenWithColumnFamilies({"default", "goku"}, options);
-  db_->GetDbSessionId(sid4);
-
-  ASSERT_EQ(sid1, sid2);
-  ASSERT_EQ(sid2, sid3);
-
-  ASSERT_NE(sid1, sid4);
-}
-
 #ifndef ROCKSDB_LITE
 TEST_F(DBBasicTest, ReadOnlyDB) {
  ASSERT_OK(Put("foo", "v1"));
@@ -3073,6 +3023,33 @@ TEST_F(DBBasicTestMultiGetDeadline, MultiGetDeadlineExceeded) {
  Close();
 }

+TEST_F(DBBasicTest, ManifestWriteFailure) {
+  Options options = GetDefaultOptions();
+  options.create_if_missing = true;
+  options.disable_auto_compactions = true;
+  options.env = env_;
+  DestroyAndReopen(options);
+  ASSERT_OK(Put("foo", "bar"));
+  ASSERT_OK(Flush());
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+  SyncPoint::GetInstance()->SetCallBack(
+      "VersionSet::ProcessManifestWrites:AfterSyncManifest", [&](void* arg) {
+        ASSERT_NE(nullptr, arg);
+        auto* s = reinterpret_cast<Status*>(arg);
+        ASSERT_OK(*s);
+        // Manually overwrite return status
+        *s = Status::IOError();
+      });
+  SyncPoint::GetInstance()->EnableProcessing();
+  ASSERT_OK(Put("key", "value"));
+  ASSERT_NOK(Flush());
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+  SyncPoint::GetInstance()->EnableProcessing();
+  Reopen(options);
+}
+
 }  // namespace ROCKSDB_NAMESPACE

 #ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
@@ -5183,6 +5183,7 @@ TEST_P(DBCompactionTestWithParam,
  // Put one key, to make biggest log sequence number in this memtable is bigger
  // than sst which would be ingested in next step.
  ASSERT_OK(Put(Key(2), "b"));
+  ASSERT_EQ(10, NumTableFilesAtLevel(0));
  dbfull()->TEST_WaitForCompact();
  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
  std::vector<std::vector<FileMetaData>> level_to_files;
@@ -23,57 +23,6 @@

 namespace ROCKSDB_NAMESPACE {

-Status DBImpl::DisableFileDeletions() {
-  InstrumentedMutexLock l(&mutex_);
-  ++disable_delete_obsolete_files_;
-  if (disable_delete_obsolete_files_ == 1) {
-    ROCKS_LOG_INFO(immutable_db_options_.info_log, "File Deletions Disabled");
-  } else {
-    ROCKS_LOG_WARN(immutable_db_options_.info_log,
-                   "File Deletions Disabled, but already disabled. Counter: %d",
-                   disable_delete_obsolete_files_);
-  }
-  return Status::OK();
-}
-
-Status DBImpl::EnableFileDeletions(bool force) {
-  // Job id == 0 means that this is not our background process, but rather
-  // user thread
-  JobContext job_context(0);
-  bool file_deletion_enabled = false;
-  {
-    InstrumentedMutexLock l(&mutex_);
-    if (force) {
-      // if force, we need to enable file deletions right away
-      disable_delete_obsolete_files_ = 0;
-    } else if (disable_delete_obsolete_files_ > 0) {
-      --disable_delete_obsolete_files_;
-    }
-    if (disable_delete_obsolete_files_ == 0)  {
-      file_deletion_enabled = true;
-      FindObsoleteFiles(&job_context, true);
-      bg_cv_.SignalAll();
-    }
-  }
-  if (file_deletion_enabled) {
-    ROCKS_LOG_INFO(immutable_db_options_.info_log, "File Deletions Enabled");
-    if (job_context.HaveSomethingToDelete()) {
-      PurgeObsoleteFiles(job_context);
-    }
-  } else {
-    ROCKS_LOG_WARN(immutable_db_options_.info_log,
-                   "File Deletions Enable, but not really enabled. Counter: %d",
-                   disable_delete_obsolete_files_);
-  }
-  job_context.Clean();
-  LogFlush(immutable_db_options_.info_log);
-  return Status::OK();
-}
-
-int DBImpl::IsFileDeletionsEnabled() const {
-  return !disable_delete_obsolete_files_;
-}
-
 Status DBImpl::GetLiveFiles(std::vector<std::string>& ret,
                            uint64_t* manifest_file_size,
                            bool flush_memtable) {
@@ -249,8 +249,7 @@ DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
      new ColumnFamilyMemTablesImpl(versions_->GetColumnFamilySet()));

  DumpRocksDBBuildVersion(immutable_db_options_.info_log.get());
-  SetDbSessionId();
-  DumpDBFileSummary(immutable_db_options_, dbname_, db_session_id_);
+  DumpDBFileSummary(immutable_db_options_, dbname_);
  immutable_db_options_.Dump(immutable_db_options_.info_log.get());
  mutable_db_options_.Dump(immutable_db_options_.info_log.get());
  DumpSupportInfo(immutable_db_options_.info_log.get());
@@ -313,8 +312,36 @@ Status DBImpl::ResumeImpl() {
  }

  // Make sure the IO Status stored in version set is set to OK.
+  bool file_deletion_disabled = !IsFileDeletionsEnabled();
  if (s.ok()) {
-    versions_->SetIOStatusOK();
+    IOStatus io_s = versions_->io_status();
+    if (io_s.IsIOError()) {
+      // If resuming from IOError resulted from MANIFEST write, then assert
+      // that we must have already set the MANIFEST writer to nullptr during
+      // clean-up phase MANIFEST writing. We must have also disabled file
+      // deletions.
+      assert(!versions_->descriptor_log_);
+      assert(file_deletion_disabled);
+      // Since we are trying to recover from MANIFEST write error, we need to
+      // switch to a new MANIFEST anyway. The old MANIFEST can be corrupted.
+      // Therefore, force writing a dummy version edit because we do not know
+      // whether there are flush jobs with non-empty data to flush, triggering
+      // appends to MANIFEST.
+      VersionEdit edit;
+      auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(default_cf_handle_);
+      assert(cfh);
+      ColumnFamilyData* cfd = cfh->cfd();
+      const MutableCFOptions& cf_opts = *cfd->GetLatestMutableCFOptions();
+      s = versions_->LogAndApply(cfd, cf_opts, &edit, &mutex_,
+                                 directories_.GetDbDir());
+      if (!s.ok()) {
+        io_s = versions_->io_status();
+        if (!io_s.ok()) {
+          s = error_handler_.SetBGError(io_s,
+                                        BackgroundErrorReason::kManifestWrite);
+        }
+      }
+    }
  }

  // We cannot guarantee consistency of the WAL. So force flush Memtables of
@@ -365,6 +392,13 @@ Status DBImpl::ResumeImpl() {
  job_context.Clean();

  if (s.ok()) {
+    assert(versions_->io_status().ok());
+    // If we reach here, we should re-enable file deletions if it was disabled
+    // during previous error handling.
+    if (file_deletion_disabled) {
+      // Always return ok
+      EnableFileDeletions(/*force=*/true);
+    }
    ROCKS_LOG_INFO(immutable_db_options_.info_log, "Successfully resumed DB");
  }
  mutex_.Lock();
@@ -3622,21 +3656,6 @@ Status DBImpl::GetDbIdentityFromIdentityFile(std::string* identity) const {
  return s;
 }

-Status DBImpl::GetDbSessionId(std::string& session_id) const {
-  session_id.assign(db_session_id_);
-  return Status::OK();
-}
-
-void DBImpl::SetDbSessionId() {
-  // GenerateUniqueId() generates an identifier
-  // that has a negligible probability of being duplicated
-  db_session_id_ = env_->GenerateUniqueId();
-  // Remove the extra '\n' at the end if there is one
-  if (!db_session_id_.empty() && db_session_id_.back() == '\n') {
-    db_session_id_.pop_back();
-  }
-}
-
 // Default implementation -- returns not supported status
 Status DB::CreateColumnFamily(const ColumnFamilyOptions& /*cf_options*/,
                              const std::string& /*column_family_name*/,
@@ -4405,6 +4424,14 @@ Status DBImpl::IngestExternalFiles(
 #endif  // !NDEBUG
        }
      }
+    } else if (versions_->io_status().IsIOError()) {
+      // Error while writing to MANIFEST.
+      // In fact, versions_->io_status() can also be the result of renaming
+      // CURRENT file. With current code, it's just difficult to tell. So just
+      // be pessimistic and try write to a new MANIFEST.
+      // TODO: distinguish between MANIFEST write and CURRENT renaming
+      const IOStatus& io_s = versions_->io_status();
+      error_handler_.SetBGError(io_s, BackgroundErrorReason::kManifestWrite);
    }

    // Resume writes to the DB
@@ -350,14 +350,18 @@ class DBImpl : public DB {

  virtual Status GetDbIdentityFromIdentityFile(std::string* identity) const;

-  virtual Status GetDbSessionId(std::string& session_id) const override;
-
  ColumnFamilyHandle* DefaultColumnFamily() const override;

  ColumnFamilyHandle* PersistentStatsColumnFamily() const;

  virtual Status Close() override;

+  virtual Status DisableFileDeletions() override;
+
+  virtual Status EnableFileDeletions(bool force) override;
+
+  virtual bool IsFileDeletionsEnabled() const;
+
  Status GetStatsHistory(
      uint64_t start_time, uint64_t end_time,
      std::unique_ptr<StatsHistoryIterator>* stats_iterator) override;
@@ -365,9 +369,6 @@ class DBImpl : public DB {
 #ifndef ROCKSDB_LITE
  using DB::ResetStats;
  virtual Status ResetStats() override;
-  virtual Status DisableFileDeletions() override;
-  virtual Status EnableFileDeletions(bool force) override;
-  virtual int IsFileDeletionsEnabled() const;
  // All the returned filenames start with "/"
  virtual Status GetLiveFiles(std::vector<std::string>&,
                              uint64_t* manifest_file_size,
@@ -983,9 +984,6 @@ class DBImpl : public DB {
 protected:
  const std::string dbname_;
  std::string db_id_;
-  // db_session_id_ is an identifier that gets reset
-  // every time the DB is opened
-  std::string db_session_id_;
  std::unique_ptr<VersionSet> versions_;
  // Flag to check whether we allocated and own the info log file
  bool own_info_log_;
@@ -1167,10 +1165,6 @@ class DBImpl : public DB {
  // bump up the version set's next_file_number_ to be 1 + largest_file_number.
  Status FinishBestEffortsRecovery();

-  // SetDbSessionId() should be called in the constuctor DBImpl()
-  // to ensure that db_session_id_ gets updated every time the DB is opened
-  void SetDbSessionId();
-
 private:
  friend class DB;
  friend class ErrorHandler;
@@ -1789,6 +1783,8 @@ class DBImpl : public DB {
      SuperVersion* sv, SequenceNumber snap_seqnum, ReadCallback* callback,
      bool* is_blob_index);

+  Status DisableFileDeletionsWithLock();
+
  // table_cache_ provides its own synchronization
  std::shared_ptr<Cache> table_cache_;

@@ -154,8 +154,7 @@ Status DBImpl::FlushMemTableToOutputFile(
      GetDataDir(cfd, 0U),
      GetCompressionFlush(*cfd->ioptions(), mutable_cf_options), stats_,
      &event_logger_, mutable_cf_options.report_bg_io_stats,
-      true /* sync_output_directory */, true /* write_manifest */, thread_pri,
-      db_id_, db_session_id_);
+      true /* sync_output_directory */, true /* write_manifest */, thread_pri);
  FileMetaData file_meta;

  TEST_SYNC_POINT("DBImpl::FlushMemTableToOutputFile:BeforePickMemtables");
@@ -211,7 +210,15 @@ Status DBImpl::FlushMemTableToOutputFile(
  if (!s.ok() && !s.IsShutdownInProgress() && !s.IsColumnFamilyDropped()) {
    if (!io_s.ok() && !io_s.IsShutdownInProgress() &&
        !io_s.IsColumnFamilyDropped()) {
-      error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
+      // Error while writing to MANIFEST.
+      // In fact, versions_->io_status() can also be the result of renaming
+      // CURRENT file. With current code, it's just difficult to tell. So just
+      // be pessimistic and try write to a new MANIFEST.
+      // TODO: distinguish between MANIFEST write and CURRENT renaming
+      auto err_reason = versions_->io_status().ok()
+                            ? BackgroundErrorReason::kFlush
+                            : BackgroundErrorReason::kManifestWrite;
+      error_handler_.SetBGError(io_s, err_reason);
    } else {
      Status new_bg_error = s;
      error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
@@ -346,7 +353,7 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
        data_dir, GetCompressionFlush(*cfd->ioptions(), mutable_cf_options),
        stats_, &event_logger_, mutable_cf_options.report_bg_io_stats,
        false /* sync_output_directory */, false /* write_manifest */,
-        thread_pri, db_id_, db_session_id_));
+        thread_pri));
    jobs.back()->PickMemTable();
  }

@@ -575,7 +582,15 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
  // it is not because of CF drop.
  if (!s.ok() && !s.IsColumnFamilyDropped()) {
    if (!io_s.ok() && !io_s.IsColumnFamilyDropped()) {
-      error_handler_.SetBGError(io_s, BackgroundErrorReason::kFlush);
+      // Error while writing to MANIFEST.
+      // In fact, versions_->io_status() can also be the result of renaming
+      // CURRENT file. With current code, it's just difficult to tell. So just
+      // be pessimistic and try write to a new MANIFEST.
+      // TODO: distinguish between MANIFEST write and CURRENT renaming
+      auto err_reason = versions_->io_status().ok()
+                            ? BackgroundErrorReason::kFlush
+                            : BackgroundErrorReason::kManifestWrite;
+      error_handler_.SetBGError(io_s, err_reason);
    } else {
      Status new_bg_error = s;
      error_handler_.SetBGError(new_bg_error, BackgroundErrorReason::kFlush);
@@ -1032,8 +1047,7 @@ Status DBImpl::CompactFilesImpl(
      snapshot_checker, table_cache_, &event_logger_,
      c->mutable_cf_options()->paranoid_file_checks,
      c->mutable_cf_options()->report_bg_io_stats, dbname_,
-      &compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_,
-      db_id_, db_session_id_);
+      &compaction_job_stats, Env::Priority::USER, &manual_compaction_paused_);

  // Creating a compaction influences the compaction score because the score
  // takes running compactions into account (by skipping files that are already
@@ -2689,7 +2703,6 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
    for (const auto& f : *c->inputs(0)) {
      c->edit()->DeleteFile(c->level(), f->fd.GetNumber());
    }
-    versions_->SetIOStatusOK();
    status = versions_->LogAndApply(c->column_family_data(),
                                    *c->mutable_cf_options(), c->edit(),
                                    &mutex_, directories_.GetDbDir());
@@ -2747,7 +2760,6 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
      }
    }

-    versions_->SetIOStatusOK();
    status = versions_->LogAndApply(c->column_family_data(),
                                    *c->mutable_cf_options(), c->edit(),
                                    &mutex_, directories_.GetDbDir());
@@ -2824,8 +2836,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
        &event_logger_, c->mutable_cf_options()->paranoid_file_checks,
        c->mutable_cf_options()->report_bg_io_stats, dbname_,
        &compaction_job_stats, thread_pri,
-        is_manual ? &manual_compaction_paused_ : nullptr, db_id_,
-        db_session_id_);
+        is_manual ? &manual_compaction_paused_ : nullptr);
    compaction_job.Prepare();

    NotifyOnCompactionBegin(c->column_family_data(), c.get(), status,
@@ -2880,7 +2891,15 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
    ROCKS_LOG_WARN(immutable_db_options_.info_log, "Compaction error: %s",
                   status.ToString().c_str());
    if (!io_s.ok()) {
-      error_handler_.SetBGError(io_s, BackgroundErrorReason::kCompaction);
+      // Error while writing to MANIFEST.
+      // In fact, versions_->io_status() can also be the result of renaming
+      // CURRENT file. With current code, it's just difficult to tell. So just
+      // be pessimistic and try write to a new MANIFEST.
+      // TODO: distinguish between MANIFEST write and CURRENT renaming
+      auto err_reason = versions_->io_status().ok()
+                            ? BackgroundErrorReason::kCompaction
+                            : BackgroundErrorReason::kManifestWrite;
+      error_handler_.SetBGError(io_s, err_reason);
    } else {
      error_handler_.SetBGError(status, BackgroundErrorReason::kCompaction);
    }
@@ -36,6 +36,62 @@ uint64_t DBImpl::MinObsoleteSstNumberToKeep() {
  return std::numeric_limits<uint64_t>::max();
 }

+Status DBImpl::DisableFileDeletions() {
+  InstrumentedMutexLock l(&mutex_);
+  return DisableFileDeletionsWithLock();
+}
+
+Status DBImpl::DisableFileDeletionsWithLock() {
+  mutex_.AssertHeld();
+  ++disable_delete_obsolete_files_;
+  if (disable_delete_obsolete_files_ == 1) {
+    ROCKS_LOG_INFO(immutable_db_options_.info_log, "File Deletions Disabled");
+  } else {
+    ROCKS_LOG_WARN(immutable_db_options_.info_log,
+                   "File Deletions Disabled, but already disabled. Counter: %d",
+                   disable_delete_obsolete_files_);
+  }
+  return Status::OK();
+}
+
+Status DBImpl::EnableFileDeletions(bool force) {
+  // Job id == 0 means that this is not our background process, but rather
+  // user thread
+  JobContext job_context(0);
+  bool file_deletion_enabled = false;
+  {
+    InstrumentedMutexLock l(&mutex_);
+    if (force) {
+      // if force, we need to enable file deletions right away
+      disable_delete_obsolete_files_ = 0;
+    } else if (disable_delete_obsolete_files_ > 0) {
+      --disable_delete_obsolete_files_;
+    }
+    if (disable_delete_obsolete_files_ == 0) {
+      file_deletion_enabled = true;
+      FindObsoleteFiles(&job_context, true);
+      bg_cv_.SignalAll();
+    }
+  }
+  if (file_deletion_enabled) {
+    ROCKS_LOG_INFO(immutable_db_options_.info_log, "File Deletions Enabled");
+    if (job_context.HaveSomethingToDelete()) {
+      PurgeObsoleteFiles(job_context);
+    }
+  } else {
+    ROCKS_LOG_WARN(immutable_db_options_.info_log,
+                   "File Deletions Enable, but not really enabled. Counter: %d",
+                   disable_delete_obsolete_files_);
+  }
+  job_context.Clean();
+  LogFlush(immutable_db_options_.info_log);
+  return Status::OK();
+}
+
+bool DBImpl::IsFileDeletionsEnabled() const {
+  return 0 == disable_delete_obsolete_files_;
+}
+
 // * Returns the list of live files in 'sst_live' and 'blob_live'.
 // If it's doing full scan:
 // * Returns the list of all files in the filesystem in
@@ -1289,8 +1289,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
          mutable_cf_options.compression_opts, paranoid_file_checks,
          cfd->internal_stats(), TableFileCreationReason::kRecovery, &io_s,
          &event_logger_, job_id, Env::IO_HIGH, nullptr /* table_properties */,
-          -1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
-          0 /* file_creation_time */, db_id_, db_session_id_);
+          -1 /* level */, current_time, write_hint);
      LogFlush(immutable_db_options_.info_log);
      ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
                      "[%s] [WriteLevel0TableForRecovery]"
@@ -17,8 +17,7 @@
 namespace ROCKSDB_NAMESPACE {

 void DumpDBFileSummary(const ImmutableDBOptions& options,
-                       const std::string& dbname,
-                       const std::string& session_id) {
+                       const std::string& dbname) {
  if (options.info_log == nullptr) {
    return;
  }
@@ -33,8 +32,6 @@ void DumpDBFileSummary(const ImmutableDBOptions& options,
  std::string file_info, wal_info;

  Header(options.info_log, "DB SUMMARY\n");
-  Header(options.info_log, "DB Session ID:  %s\n", session_id.c_str());
-
  // Get files in dbname dir
  if (!env->GetChildren(dbname, &files).ok()) {
    Error(options.info_log,
@@ -10,6 +10,5 @@

 namespace ROCKSDB_NAMESPACE {
 void DumpDBFileSummary(const ImmutableDBOptions& options,
-                       const std::string& dbname,
-                       const std::string& session_id = "");
+                       const std::string& dbname);
 }  // namespace ROCKSDB_NAMESPACE
@@ -97,8 +97,7 @@ struct StressTestIterator : public InternalIterator {

  bool MaybeFail() {
    if (rnd->Next() >=
-        static_cast<double>(std::numeric_limits<uint64_t>::max()) *
-            error_probability) {
+        std::numeric_limits<uint64_t>::max() * error_probability) {
      return false;
    }
    if (rnd->Next() % 2) {
@@ -115,8 +114,7 @@ struct StressTestIterator : public InternalIterator {

  void MaybeMutate() {
    if (rnd->Next() >=
-        static_cast<double>(std::numeric_limits<uint64_t>::max()) *
-            mutation_probability) {
+        std::numeric_limits<uint64_t>::max() * mutation_probability) {
      return;
    }
    do {
@@ -128,9 +126,8 @@ struct StressTestIterator : public InternalIterator {
      if (data->hidden.empty()) {
        hide_probability = 1;
      }
-      bool do_hide = rnd->Next() <
-                     static_cast<double>(std::numeric_limits<uint64_t>::max()) *
-                         hide_probability;
+      bool do_hide =
+          rnd->Next() < std::numeric_limits<uint64_t>::max() * hide_probability;
      if (do_hide) {
        // Hide a random entry.
        size_t idx = rnd->Next() % data->entries.size();
@@ -303,14 +303,14 @@ TEST_F(DBMemTableTest, InsertWithHint) {
  ASSERT_EQ(hint_bar, rep->last_hint_in());
  ASSERT_EQ(hint_bar, rep->last_hint_out());
  ASSERT_EQ(5, rep->num_insert_with_hint());
-  ASSERT_OK(Put("NotInPrefixDomain", "vvv"));
+  ASSERT_OK(Put("whitelisted", "vvv"));
  ASSERT_EQ(5, rep->num_insert_with_hint());
  ASSERT_EQ("foo_v1", Get("foo_k1"));
  ASSERT_EQ("foo_v2", Get("foo_k2"));
  ASSERT_EQ("foo_v3", Get("foo_k3"));
  ASSERT_EQ("bar_v1", Get("bar_k1"));
  ASSERT_EQ("bar_v2", Get("bar_k2"));
-  ASSERT_EQ("vvv", Get("NotInPrefixDomain"));
+  ASSERT_EQ("vvv", Get("whitelisted"));
 }

 TEST_F(DBMemTableTest, ColumnFamilyId) {
@@ -39,7 +39,7 @@ class DBOptionsTest : public DBTestBase {
    std::unordered_map<std::string, std::string> options_map;
    StringToMap(options_str, &options_map);
    std::unordered_map<std::string, std::string> mutable_map;
-    for (const auto& opt : db_options_type_info) {
+    for (const auto opt : db_options_type_info) {
      if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
        mutable_map[opt.first] = options_map[opt.first];
      }
@@ -57,7 +57,7 @@ class DBOptionsTest : public DBTestBase {
    std::unordered_map<std::string, std::string> options_map;
    StringToMap(options_str, &options_map);
    std::unordered_map<std::string, std::string> mutable_map;
-    for (const auto& opt : cf_options_type_info) {
+    for (const auto opt : cf_options_type_info) {
      if (opt.second.IsMutable() && opt.second.ShouldSerialize()) {
        mutable_map[opt.first] = options_map[opt.first];
      }
@@ -252,26 +252,6 @@ TEST_F(DBTablePropertiesTest, GetColumnFamilyNameProperty) {
  }
 }

-TEST_F(DBTablePropertiesTest, GetDbIdentifiersProperty) {
-  CreateAndReopenWithCF({"goku"}, CurrentOptions());
-
-  for (uint32_t cf = 0; cf < 2; ++cf) {
-    Put(cf, "key", "val");
-    Put(cf, "foo", "bar");
-    Flush(cf);
-
-    TablePropertiesCollection fname_to_props;
-    ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[cf], &fname_to_props));
-    ASSERT_EQ(1U, fname_to_props.size());
-
-    std::string id, sid;
-    db_->GetDbIdentity(id);
-    db_->GetDbSessionId(sid);
-    ASSERT_EQ(id, fname_to_props.begin()->second->db_id);
-    ASSERT_EQ(sid, fname_to_props.begin()->second->db_session_id);
-  }
-}
-
 class DeletionTriggeredCompactionTestListener : public EventListener {
 public:
  void OnCompactionBegin(DB* , const CompactionJobInfo& ci) override {
@@ -2990,10 +2990,11 @@ class ModelDB : public DB {

  Status SyncWAL() override { return Status::OK(); }

-#ifndef ROCKSDB_LITE
  Status DisableFileDeletions() override { return Status::OK(); }

  Status EnableFileDeletions(bool /*force*/) override { return Status::OK(); }
+#ifndef ROCKSDB_LITE
+
  Status GetLiveFiles(std::vector<std::string>&, uint64_t* /*size*/,
                      bool /*flush_memtable*/ = true) override {
    return Status::OK();
@@ -3031,10 +3032,6 @@ class ModelDB : public DB {
    return Status::OK();
  }

-  Status GetDbSessionId(std::string& /*session_id*/) const override {
-    return Status::OK();
-  }
-
  SequenceNumber GetLatestSequenceNumber() const override { return 0; }

  bool SetPreserveDeletesSequenceNumber(SequenceNumber /*seqnum*/) override {
@@ -144,6 +144,38 @@ INSTANTIATE_TEST_CASE_P(TestReadOnlyWithCompressedCache,
                        TestReadOnlyWithCompressedCache,
                        ::testing::Combine(::testing::Values(-1, 100),
                                           ::testing::Bool()));
+
+class PartitionedIndexTestListener : public EventListener {
+ public:
+  void OnFlushCompleted(DB* /*db*/, const FlushJobInfo& info) override {
+    ASSERT_GT(info.table_properties.index_partitions, 1);
+    ASSERT_EQ(info.table_properties.index_key_is_user_key, 0);
+  }
+};
+
+TEST_F(DBTest2, PartitionedIndexUserToInternalKey) {
+  BlockBasedTableOptions table_options;
+  Options options = CurrentOptions();
+  table_options.index_type = BlockBasedTableOptions::kTwoLevelIndexSearch;
+  PartitionedIndexTestListener* listener = new PartitionedIndexTestListener();
+  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
+  options.listeners.emplace_back(listener);
+  std::vector<const Snapshot*> snapshots;
+  Reopen(options);
+  Random rnd(301);
+
+  for (int i = 0; i < 3000; i++) {
+    int j = i % 30;
+    std::string value = RandomString(&rnd, 10500);
+    ASSERT_OK(Put("keykey_" + std::to_string(j), value));
+    snapshots.push_back(db_->GetSnapshot());
+  }
+  Flush();
+  for (auto s : snapshots) {
+    db_->ReleaseSnapshot(s);
+  }
+}
+
 #endif  // ROCKSDB_LITE

 class PrefixFullBloomWithReverseComparator
@@ -2068,10 +2100,6 @@ class MockPersistentCache : public PersistentCache {
    return PersistentCache::StatsType();
  }

-  uint64_t NewId() override {
-    return last_id_.fetch_add(1, std::memory_order_relaxed);
-  }
-
  Status Insert(const Slice& page_key, const char* data,
                const size_t size) override {
    MutexLock _(&lock_);
@@ -2112,7 +2140,6 @@ class MockPersistentCache : public PersistentCache {
  const bool is_compressed_ = true;
  size_t size_ = 0;
  const size_t max_size_ = 10 * 1024;  // 10KiB
-  std::atomic<uint64_t> last_id_{1};
 };

 #ifdef OS_LINUX
@@ -2236,7 +2263,10 @@ TEST_F(DBTest2, TestPerfContextIterCpuTime) {
 }
 #endif  // OS_LINUX

-#if !defined OS_SOLARIS
+// GetUniqueIdFromFile is not implemented on these platforms. Persistent cache
+// breaks when that function is not implemented and no regular block cache is
+// provided.
+#if !defined(OS_SOLARIS) && !defined(OS_WIN)
 TEST_F(DBTest2, PersistentCache) {
  int num_iter = 80;

@@ -2300,7 +2330,7 @@ TEST_F(DBTest2, PersistentCache) {
    }
  }
 }
-#endif  // !defined OS_SOLARIS
+#endif  // !defined(OS_SOLARIS) && !defined(OS_WIN)

 namespace {
 void CountSyncPoint() {
@@ -51,9 +51,19 @@ std::map<std::tuple<BackgroundErrorReason, Status::Code, Status::SubCode, bool>,
                         Status::Code::kIOError, Status::SubCode::kNoSpace,
                         false),
         Status::Severity::kHardError},
+        // Errors during MANIFEST write
+        {std::make_tuple(BackgroundErrorReason::kManifestWrite,
+                         Status::Code::kIOError, Status::SubCode::kNoSpace,
+                         true),
+         Status::Severity::kHardError},
+        {std::make_tuple(BackgroundErrorReason::kManifestWrite,
+                         Status::Code::kIOError, Status::SubCode::kNoSpace,
+                         false),
+         Status::Severity::kHardError},
 };

-std::map<std::tuple<BackgroundErrorReason, Status::Code, bool>, Status::Severity>
+std::map<std::tuple<BackgroundErrorReason, Status::Code, bool>,
+         Status::Severity>
    DefaultErrorSeverityMap = {
        // Errors during BG compaction
        {std::make_tuple(BackgroundErrorReason::kCompaction,
@@ -75,11 +85,11 @@ std::map<std::tuple<BackgroundErrorReason, Status::Code, bool>, Status::Severity
        {std::make_tuple(BackgroundErrorReason::kFlush,
                         Status::Code::kCorruption, false),
         Status::Severity::kNoError},
-        {std::make_tuple(BackgroundErrorReason::kFlush,
-                         Status::Code::kIOError, true),
+        {std::make_tuple(BackgroundErrorReason::kFlush, Status::Code::kIOError,
+                         true),
         Status::Severity::kFatalError},
-        {std::make_tuple(BackgroundErrorReason::kFlush,
-                         Status::Code::kIOError, false),
+        {std::make_tuple(BackgroundErrorReason::kFlush, Status::Code::kIOError,
+                         false),
         Status::Severity::kNoError},
        // Errors during Write
        {std::make_tuple(BackgroundErrorReason::kWriteCallback,
@@ -94,30 +104,36 @@ std::map<std::tuple<BackgroundErrorReason, Status::Code, bool>, Status::Severity
        {std::make_tuple(BackgroundErrorReason::kWriteCallback,
                         Status::Code::kIOError, false),
         Status::Severity::kNoError},
+        {std::make_tuple(BackgroundErrorReason::kManifestWrite,
+                         Status::Code::kIOError, true),
+         Status::Severity::kFatalError},
+        {std::make_tuple(BackgroundErrorReason::kManifestWrite,
+                         Status::Code::kIOError, false),
+         Status::Severity::kFatalError},
 };

 std::map<std::tuple<BackgroundErrorReason, bool>, Status::Severity>
    DefaultReasonMap = {
        // Errors during BG compaction
        {std::make_tuple(BackgroundErrorReason::kCompaction, true),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
        {std::make_tuple(BackgroundErrorReason::kCompaction, false),
-          Status::Severity::kNoError},
+         Status::Severity::kNoError},
        // Errors during BG flush
        {std::make_tuple(BackgroundErrorReason::kFlush, true),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
        {std::make_tuple(BackgroundErrorReason::kFlush, false),
-          Status::Severity::kNoError},
+         Status::Severity::kNoError},
        // Errors during Write
        {std::make_tuple(BackgroundErrorReason::kWriteCallback, true),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
        {std::make_tuple(BackgroundErrorReason::kWriteCallback, false),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
        // Errors during Memtable update
        {std::make_tuple(BackgroundErrorReason::kMemTable, true),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
        {std::make_tuple(BackgroundErrorReason::kMemTable, false),
-          Status::Severity::kFatalError},
+         Status::Severity::kFatalError},
 };

 void ErrorHandler::CancelErrorRecovery() {
@@ -247,6 +263,10 @@ Status ErrorHandler::SetBGError(const IOStatus& bg_io_err,
  if (recovery_in_prog_ && recovery_error_.ok()) {
    recovery_error_ = bg_io_err;
  }
+  if (BackgroundErrorReason::kManifestWrite == reason) {
+    // Always returns ok
+    db_->DisableFileDeletionsWithLock();
+  }
  Status new_bg_io_err = bg_io_err;
  Status s;
  if (bg_io_err.GetDataLoss()) {
@@ -121,9 +121,7 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
              << table_properties.compression_options << "creation_time"
              << table_properties.creation_time << "oldest_key_time"
              << table_properties.oldest_key_time << "file_creation_time"
-              << table_properties.file_creation_time << "db_id"
-              << table_properties.db_id << "db_session_id"
-              << table_properties.db_session_id;
+              << table_properties.file_creation_time;

      // user collected properties
      for (const auto& prop : table_properties.readable_properties) {
@@ -12,7 +12,6 @@
 #include "file/filename.h"
 #include "port/port.h"
 #include "port/stack_trace.h"
-#include "rocksdb/sst_file_reader.h"
 #include "rocksdb/sst_file_writer.h"
 #include "test_util/fault_injection_test_env.h"
 #include "test_util/testutil.h"
@@ -2394,41 +2393,6 @@ TEST_F(ExternalSSTFileTest, IngestFileWrittenWithCompressionDictionary) {
  ASSERT_EQ(1, num_compression_dicts);
 }

-// Very slow, not worth the cost to run regularly
-TEST_F(ExternalSSTFileTest, DISABLED_HugeBlockChecksum) {
-  int max_checksum = static_cast<int>(kxxHash64);
-  for (int i = 0; i <= max_checksum; ++i) {
-    BlockBasedTableOptions table_options;
-    table_options.checksum = static_cast<ChecksumType>(i);
-    Options options = CurrentOptions();
-    options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-
-    SstFileWriter sst_file_writer(EnvOptions(), options);
-
-    // 2^32 - 1, will lead to data block with more than 2^32 bytes
-    size_t huge_size = port::kMaxUint32;
-
-    std::string f = sst_files_dir_ + "f.sst";
-    ASSERT_OK(sst_file_writer.Open(f));
-    {
-      Random64 r(123);
-      std::string huge(huge_size, 0);
-      for (size_t j = 0; j + 7 < huge_size; j += 8) {
-        EncodeFixed64(&huge[j], r.Next());
-      }
-      ASSERT_OK(sst_file_writer.Put("Huge", huge));
-    }
-
-    ExternalSstFileInfo f_info;
-    ASSERT_OK(sst_file_writer.Finish(&f_info));
-    ASSERT_GT(f_info.file_size, uint64_t{huge_size} + 10);
-
-    SstFileReader sst_file_reader(options);
-    ASSERT_OK(sst_file_reader.Open(f));
-    ASSERT_OK(sst_file_reader.VerifyChecksum());
-  }
-}
-
 TEST_P(ExternalSSTFileTest, IngestFilesIntoMultipleColumnFamilies_Success) {
  std::unique_ptr<FaultInjectionTestEnv> fault_injection_env(
      new FaultInjectionTestEnv(env_));
@@ -97,11 +97,8 @@ FlushJob::FlushJob(const std::string& dbname, ColumnFamilyData* cfd,
                   CompressionType output_compression, Statistics* stats,
                   EventLogger* event_logger, bool measure_io_stats,
                   const bool sync_output_directory, const bool write_manifest,
-                   Env::Priority thread_pri, const std::string& db_id,
-                   const std::string& db_session_id)
+                   Env::Priority thread_pri)
    : dbname_(dbname),
-      db_id_(db_id),
-      db_session_id_(db_session_id),
      cfd_(cfd),
      db_options_(db_options),
      mutable_cf_options_(mutable_cf_options),
@@ -396,8 +393,7 @@ Status FlushJob::WriteLevel0Table() {
          mutable_cf_options_.paranoid_file_checks, cfd_->internal_stats(),
          TableFileCreationReason::kFlush, &io_s, event_logger_,
          job_context_->job_id, Env::IO_HIGH, &table_properties_, 0 /* level */,
-          creation_time, oldest_key_time, write_hint, current_time, db_id_,
-          db_session_id_);
+          creation_time, oldest_key_time, write_hint, current_time);
      if (!io_s.ok()) {
        io_status_ = io_s;
      }
@@ -72,8 +72,7 @@ class FlushJob {
           CompressionType output_compression, Statistics* stats,
           EventLogger* event_logger, bool measure_io_stats,
           const bool sync_output_directory, const bool write_manifest,
-           Env::Priority thread_pri, const std::string& db_id = "",
-           const std::string& db_session_id = "");
+           Env::Priority thread_pri);

  ~FlushJob();

@@ -104,8 +103,6 @@ class FlushJob {
 #endif  // !ROCKSDB_LITE

  const std::string& dbname_;
-  const std::string db_id_;
-  const std::string db_session_id_;
  ColumnFamilyData* cfd_;
  const ImmutableDBOptions& db_options_;
  const MutableCFOptions& mutable_cf_options_;
@@ -282,6 +282,7 @@ TEST_F(FlushJobTest, FlushMemTablesSingleColumnFamily) {
  assert(memtable_ids.size() == num_mems);
  uint64_t smallest_memtable_id = memtable_ids.front();
  uint64_t flush_memtable_id = smallest_memtable_id + num_mems_to_flush - 1;
+
  FlushJob flush_job(dbname_, versions_->GetColumnFamilySet()->GetDefault(),
                     db_options_, *cfd->GetLatestMutableCFOptions(),
                     &flush_memtable_id, env_options_, versions_.get(), &mutex_,
@@ -798,7 +798,7 @@ bool InternalStats::HandleCurrentSuperVersionNumber(uint64_t* value,

 bool InternalStats::HandleIsFileDeletionsEnabled(uint64_t* value, DBImpl* db,
                                                 Version* /*version*/) {
-  *value = db->IsFileDeletionsEnabled();
+  *value = db->IsFileDeletionsEnabled() ? 1 : 0;
  return true;
 }

@@ -766,7 +766,13 @@ static bool SaveValue(void* arg, const char* entry) {
        }
        return true;
      }
-      default:
+      default: {
+        std::string msg("Unrecognized value type: " +
+                        std::to_string(static_cast<int>(type)) + ". ");
+        msg.append("User key: " + user_key_slice.ToString(/*hex=*/true) + ". ");
+        msg.append("seq: " + std::to_string(seq) + ".");
+        *(s->status) = Status::Corruption(msg.c_str());
+      }
        assert(false);
        return true;
    }
@@ -470,7 +470,6 @@ Status MemTableList::TryInstallMemtableFlushResults(
      }

      // this can release and reacquire the mutex.
-      vset->SetIOStatusOK();
      s = vset->LogAndApply(cfd, mutable_cf_options, edit_list, mu,
                            db_directory);
      *io_s = vset->io_status();
@@ -183,8 +183,6 @@ class Repairer {
      }
      // Just create a DBImpl temporarily so we can reuse NewDB()
      DBImpl* db_impl = new DBImpl(db_options_, dbname_);
-      // Also use this temp DBImpl to get a session id
-      db_impl->GetDbSessionId(db_session_id_);
      status = db_impl->NewDB();
      delete db_impl;
    }
@@ -231,7 +229,6 @@ class Repairer {
  };

  std::string const dbname_;
-  std::string db_session_id_;
  Env* const env_;
  const EnvOptions env_options_;
  const DBOptions db_options_;
@@ -438,9 +435,7 @@ class Repairer {
          CompressionOptions(), false, nullptr /* internal_stats */,
          TableFileCreationReason::kRecovery, &io_s, nullptr /* event_logger */,
          0 /* job_id */, Env::IO_HIGH, nullptr /* table_properties */,
-          -1 /* level */, current_time, 0 /* oldest_key_time */, write_hint,
-          0 /* file_creation_time */, "DB Repairer" /* db_id */,
-          db_session_id_);
+          -1 /* level */, current_time, write_hint);
      ROCKS_LOG_INFO(db_options_.info_log,
                     "Log #%" PRIu64 ": %d ops saved to Table #%" PRIu64 " %s",
                     log, counter, meta.fd.GetNumber(),
@@ -3878,10 +3878,6 @@ Status VersionSet::ProcessManifestWrites(
  }
 #endif  // NDEBUG

-  uint64_t new_manifest_file_size = 0;
-  Status s;
-  IOStatus io_s;
-
  assert(pending_manifest_file_number_ == 0);
  if (!descriptor_log_ ||
      manifest_file_size_ > db_options_->max_manifest_file_size) {
@@ -3911,6 +3907,9 @@ Status VersionSet::ProcessManifestWrites(
    }
  }

+  uint64_t new_manifest_file_size = 0;
+  Status s;
+  IOStatus io_s;
  {
    FileOptions opt_file_opts = fs_->OptimizeForManifestWrite(file_options_);
    mu->Unlock();
@@ -3947,9 +3946,9 @@ Status VersionSet::ProcessManifestWrites(
      std::string descriptor_fname =
          DescriptorFileName(dbname_, pending_manifest_file_number_);
      std::unique_ptr<FSWritableFile> descriptor_file;
-      s = NewWritableFile(fs_, descriptor_fname, &descriptor_file,
-                          opt_file_opts);
-      if (s.ok()) {
+      io_s = NewWritableFile(fs_, descriptor_fname, &descriptor_file,
+                             opt_file_opts);
+      if (io_s.ok()) {
        descriptor_file->SetPreallocationBlockSize(
            db_options_->manifest_preallocation_size);

@@ -3958,7 +3957,10 @@ Status VersionSet::ProcessManifestWrites(
            nullptr, db_options_->listeners));
        descriptor_log_.reset(
            new log::Writer(std::move(file_writer), 0, false));
-        s = WriteCurrentStateToManifest(curr_state, descriptor_log_.get());
+        s = WriteCurrentStateToManifest(curr_state, descriptor_log_.get(),
+                                        io_s);
+      } else {
+        s = io_s;
      }
    }

@@ -3994,16 +3996,16 @@ Status VersionSet::ProcessManifestWrites(
 #endif /* !NDEBUG */
        io_s = descriptor_log_->AddRecord(record);
        if (!io_s.ok()) {
-          io_status_ = io_s;
          s = io_s;
          break;
        }
      }
      if (s.ok()) {
        io_s = SyncManifest(env_, db_options_, descriptor_log_->file());
+        TEST_SYNC_POINT_CALLBACK(
+            "VersionSet::ProcessManifestWrites:AfterSyncManifest", &io_s);
      }
      if (!io_s.ok()) {
-        io_status_ = io_s;
        s = io_s;
        ROCKS_LOG_ERROR(db_options_->info_log, "MANIFEST write %s\n",
                        s.ToString().c_str());
@@ -4016,7 +4018,6 @@ Status VersionSet::ProcessManifestWrites(
      io_s = SetCurrentFile(fs_, dbname_, pending_manifest_file_number_,
                            db_directory);
      if (!io_s.ok()) {
-        io_status_ = io_s;
        s = io_s;
      }
      TEST_SYNC_POINT("VersionSet::ProcessManifestWrites:AfterNewManifest");
@@ -4038,6 +4039,14 @@ Status VersionSet::ProcessManifestWrites(
    mu->Lock();
  }

+  if (!io_s.ok()) {
+    if (io_status_.ok()) {
+      io_status_ = io_s;
+    }
+  } else if (!io_status_.ok()) {
+    io_status_ = io_s;
+  }
+
  // Append the old manifest file to the obsolete_manifest_ list to be deleted
  // by PurgeObsoleteFiles later.
  if (s.ok() && new_descriptor_log) {
@@ -5291,7 +5300,7 @@ void VersionSet::MarkMinLogNumberToKeep2PC(uint64_t number) {

 Status VersionSet::WriteCurrentStateToManifest(
    const std::unordered_map<uint32_t, MutableCFState>& curr_state,
-    log::Writer* log) {
+    log::Writer* log, IOStatus& io_s) {
  // TODO: Break up into multiple records to reduce memory usage on recovery?

  // WARNING: This method doesn't hold a mutex!!
@@ -5300,6 +5309,7 @@ Status VersionSet::WriteCurrentStateToManifest(
  // LogAndApply. Column family manipulations can only happen within LogAndApply
  // (the same single thread), so we're safe to iterate.

+  assert(io_s.ok());
  if (db_options_->write_dbid_to_manifest) {
    VersionEdit edit_for_db_id;
    assert(!db_id_.empty());
@@ -5309,10 +5319,9 @@ Status VersionSet::WriteCurrentStateToManifest(
      return Status::Corruption("Unable to Encode VersionEdit:" +
                                edit_for_db_id.DebugString(true));
    }
-    IOStatus io_s = log->AddRecord(db_id_record);
+    io_s = log->AddRecord(db_id_record);
    if (!io_s.ok()) {
-      io_status_ = io_s;
-      return std::move(io_s);
+      return io_s;
    }
  }

@@ -5339,10 +5348,9 @@ Status VersionSet::WriteCurrentStateToManifest(
        return Status::Corruption(
            "Unable to Encode VersionEdit:" + edit.DebugString(true));
      }
-      IOStatus io_s = log->AddRecord(record);
+      io_s = log->AddRecord(record);
      if (!io_s.ok()) {
-        io_status_ = io_s;
-        return std::move(io_s);
+        return io_s;
      }
    }

@@ -5392,10 +5400,9 @@ Status VersionSet::WriteCurrentStateToManifest(
        return Status::Corruption(
            "Unable to Encode VersionEdit:" + edit.DebugString(true));
      }
-      IOStatus io_s = log->AddRecord(record);
+      io_s = log->AddRecord(record);
      if (!io_s.ok()) {
-        io_status_ = io_s;
-        return std::move(io_s);
+        return io_s;
      }
    }
  }
@@ -1159,10 +1159,7 @@ class VersionSet {
  static uint64_t GetTotalSstFilesSize(Version* dummy_versions);

  // Get the IO Status returned by written Manifest.
-  IOStatus io_status() const { return io_status_; }
-
-  // Set the IO Status to OK. Called before Manifest write if needed.
-  void SetIOStatusOK() { io_status_ = IOStatus::OK(); }
+  const IOStatus& io_status() const { return io_status_; }

 protected:
  using VersionBuilderMap =
@@ -1205,7 +1202,7 @@ class VersionSet {
  // Save current contents to *log
  Status WriteCurrentStateToManifest(
      const std::unordered_map<uint32_t, MutableCFState>& curr_state,
-      log::Writer* log);
+      log::Writer* log, IOStatus& io_s);

  void AppendVersion(ColumnFamilyData* column_family_data, Version* v);

@@ -146,7 +146,6 @@ DECLARE_int32(reopen);
 DECLARE_double(bloom_bits);
 DECLARE_bool(use_block_based_filter);
 DECLARE_bool(partition_filters);
-DECLARE_bool(optimize_filters_for_memory);
 DECLARE_int32(index_type);
 DECLARE_string(db);
 DECLARE_string(secondaries_base);
@@ -162,7 +161,7 @@ DECLARE_bool(statistics);
 DECLARE_bool(sync);
 DECLARE_bool(use_fsync);
 DECLARE_int32(kill_random_test);
-DECLARE_string(kill_exclude_prefixes);
+DECLARE_string(kill_prefix_blacklist);
 DECLARE_bool(disable_wal);
 DECLARE_uint64(recycle_log_file_num);
 DECLARE_int64(target_file_size_base);
@@ -237,10 +236,6 @@ DECLARE_double(blob_db_gc_cutoff);
 DECLARE_int32(approximate_size_one_in);
 DECLARE_bool(sync_fault_injection);

-DECLARE_bool(best_efforts_recovery);
-DECLARE_bool(skip_verifydb);
-DECLARE_bool(enable_compaction_filter);
-
 const long KB = 1024;
 const int kRandomValueMaxFactor = 3;
 const int kValueMaxLen = 100;
@@ -441,10 +436,19 @@ extern inline bool GetIntVal(std::string big_endian_key, uint64_t* key_p) {

  assert(size_key <= key_gen_ctx.weights.size() * sizeof(uint64_t));

+  // Pad with zeros to make it a multiple of 8. This function may be called
+  // with a prefix, in which case we return the first index that falls
+  // inside or outside that prefix, dependeing on whether the prefix is
+  // the start of upper bound of a scan
+  unsigned int pad = sizeof(uint64_t) - (size_key % sizeof(uint64_t));
+  if (pad < sizeof(uint64_t)) {
+    big_endian_key.append(pad, '\0');
+    size_key += pad;
+  }
+
  std::string little_endian_key;
  little_endian_key.resize(size_key);
-  for (size_t start = 0; start + sizeof(uint64_t) <= size_key;
-       start += sizeof(uint64_t)) {
+  for (size_t start = 0; start < size_key; start += sizeof(uint64_t)) {
    size_t end = start + sizeof(uint64_t);
    for (size_t i = 0; i < sizeof(uint64_t); ++i) {
      little_endian_key[start + i] = big_endian_key[end - 1 - i];
@@ -463,40 +467,17 @@ extern inline bool GetIntVal(std::string big_endian_key, uint64_t* key_p) {
    uint64_t pfx = prefixes[i];
    key += (pfx / key_gen_ctx.weights[i]) * key_gen_ctx.window +
           pfx % key_gen_ctx.weights[i];
-    if (i < prefixes.size() - 1) {
-      // The encoding writes a `key_gen_ctx.weights[i] - 1` that counts for
-      // `key_gen_ctx.weights[i]` when there are more prefixes to come. So we
-      // need to add back the one here as we're at a non-last prefix.
-      ++key;
-    }
  }
  *key_p = key;
  return true;
 }

-// Given a string prefix, map it to the first corresponding index in the
-// expected values buffer.
-inline bool GetFirstIntValInPrefix(std::string big_endian_prefix,
-                                   uint64_t* key_p) {
-  size_t size_key = big_endian_prefix.size();
-  // Pad with zeros to make it a multiple of 8. This function may be called
-  // with a prefix, in which case we return the first index that falls
-  // inside or outside that prefix, dependeing on whether the prefix is
-  // the start of upper bound of a scan
-  unsigned int pad = sizeof(uint64_t) - (size_key % sizeof(uint64_t));
-  if (pad < sizeof(uint64_t)) {
-    big_endian_prefix.append(pad, '\0');
-  }
-  return GetIntVal(std::move(big_endian_prefix), key_p);
-}
-
 extern inline uint64_t GetPrefixKeyCount(const std::string& prefix,
                                         const std::string& ub) {
  uint64_t start = 0;
  uint64_t end = 0;

-  if (!GetFirstIntValInPrefix(prefix, &start) ||
-      !GetFirstIntValInPrefix(ub, &end)) {
+  if (!GetIntVal(prefix, &start) || !GetIntVal(ub, &end)) {
    return 0;
  }

@@ -1,79 +0,0 @@
-//  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under both the GPLv2 (found in the
-//  COPYING file in the root directory) and Apache 2.0 License
-//  (found in the LICENSE.Apache file in the root directory).
-//
-// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
-
-#pragma once
-
-#include "rocksdb/compaction_filter.h"
-
-namespace ROCKSDB_NAMESPACE {
-
-// DbStressCompactionFilter is safe to use with db_stress as it does not perform
-// any mutation. It only makes `kRemove` decisions for keys that are already
-// non-existent according to the `SharedState`.
-class DbStressCompactionFilter : public CompactionFilter {
- public:
-  DbStressCompactionFilter(SharedState* state, int cf_id)
-      : state_(state), cf_id_(cf_id) {}
-
-  Decision FilterV2(int /*level*/, const Slice& key, ValueType /*value_type*/,
-                    const Slice& /*existing_value*/, std::string* /*new_value*/,
-                    std::string* /*skip_until*/) const override {
-    if (state_ == nullptr) {
-      return Decision::kKeep;
-    }
-    if (key.empty() || ('0' <= key[0] && key[0] <= '9')) {
-      // It is likely leftover from a test_batches_snapshots run. Below this
-      // conditional, the test_batches_snapshots key format is not handled
-      // properly. Just keep it to be safe.
-      return Decision::kKeep;
-    }
-    uint64_t key_num = 0;
-    bool ok = GetIntVal(key.ToString(), &key_num);
-    assert(ok);
-    (void)ok;
-    MutexLock key_lock(state_->GetMutexForKey(cf_id_, key_num));
-    if (!state_->Exists(cf_id_, key_num)) {
-      return Decision::kRemove;
-    }
-    return Decision::kKeep;
-  }
-
-  const char* Name() const override { return "DbStressCompactionFilter"; }
-
- private:
-  SharedState* const state_;
-  const int cf_id_;
-};
-
-class DbStressCompactionFilterFactory : public CompactionFilterFactory {
- public:
-  DbStressCompactionFilterFactory() : state_(nullptr) {}
-
-  void SetSharedState(SharedState* state) {
-    MutexLock state_mutex_guard(&state_mutex_);
-    state_ = state;
-  }
-
-  std::unique_ptr<CompactionFilter> CreateCompactionFilter(
-      const CompactionFilter::Context& context) override {
-    MutexLock state_mutex_guard(&state_mutex_);
-    return std::unique_ptr<CompactionFilter>(
-        new DbStressCompactionFilter(state_, context.column_family_id));
-  }
-
-  const char* Name() const override {
-    return "DbStressCompactionFilterFactory";
-  }
-
- private:
-  port::Mutex state_mutex_;
-  SharedState* state_;
-};
-
-}  // namespace ROCKSDB_NAMESPACE
@@ -16,7 +16,7 @@ void ThreadBody(void* v) {
  ThreadState* thread = reinterpret_cast<ThreadState*>(v);
  SharedState* shared = thread->shared;

-  if (!FLAGS_skip_verifydb && shared->ShouldVerifyAtBeginning()) {
+  if (shared->ShouldVerifyAtBeginning()) {
    thread->shared->GetStressTest()->VerifyDb(thread);
  }
  {
@@ -42,9 +42,7 @@ void ThreadBody(void* v) {
    }
  }

-  if (!FLAGS_skip_verifydb) {
-    thread->shared->GetStressTest()->VerifyDb(thread);
-  }
+  thread->shared->GetStressTest()->VerifyDb(thread);

  {
    MutexLock l(shared->GetMutex());
@@ -57,8 +55,11 @@ void ThreadBody(void* v) {

 bool RunStressTest(StressTest* stress) {
  stress->InitDb();
+
  SharedState shared(db_stress_env, stress);
-  stress->FinishInitDb(&shared);
+  if (FLAGS_read_only) {
+    stress->InitReadonlyDb(&shared);
+  }

 #ifndef NDEBUG
  if (FLAGS_sync_fault_injection) {
@@ -117,9 +118,6 @@ bool RunStressTest(StressTest* stress) {
    if (FLAGS_test_batches_snapshots) {
      fprintf(stdout, "%s Limited verification already done during gets\n",
              db_stress_env->TimeToString((uint64_t)now / 1000000).c_str());
-    } else if (FLAGS_skip_verifydb) {
-      fprintf(stdout, "%s Verification skipped\n",
-              db_stress_env->TimeToString((uint64_t)now / 1000000).c_str());
    } else {
      fprintf(stdout, "%s Starting verification\n",
              db_stress_env->TimeToString((uint64_t)now / 1000000).c_str());
@@ -142,8 +140,7 @@ bool RunStressTest(StressTest* stress) {
    threads[i] = nullptr;
  }
  now = db_stress_env->NowMicros();
-  if (!FLAGS_skip_verifydb && !FLAGS_test_batches_snapshots &&
-      !shared.HasVerificationFailedYet()) {
+  if (!FLAGS_test_batches_snapshots && !shared.HasVerificationFailedYet()) {
    fprintf(stdout, "%s Verification successful\n",
            db_stress_env->TimeToString(now / 1000000).c_str());
  }
@@ -361,11 +361,6 @@ DEFINE_bool(partition_filters, false,
            "use partitioned filters "
            "for block-based table");

-DEFINE_bool(
-    optimize_filters_for_memory,
-    ROCKSDB_NAMESPACE::BlockBasedTableOptions().optimize_filters_for_memory,
-    "Minimize memory footprint of filters");
-
 DEFINE_int32(
    index_type,
    static_cast<int32_t>(
@@ -419,10 +414,10 @@ static const bool FLAGS_kill_random_test_dummy __attribute__((__unused__)) =
    RegisterFlagValidator(&FLAGS_kill_random_test, &ValidateInt32Positive);
 extern int rocksdb_kill_odds;

-DEFINE_string(kill_exclude_prefixes, "",
+DEFINE_string(kill_prefix_blacklist, "",
              "If non-empty, kill points with prefix in the list given will be"
              " skipped. Items are comma-separated.");
-extern std::vector<std::string> rocksdb_kill_exclude_prefixes;
+extern std::vector<std::string> rocksdb_kill_prefix_blacklist;

 DEFINE_bool(disable_wal, false, "If true, do not write WAL for write.");

@@ -696,13 +691,4 @@ DEFINE_int32(read_fault_one_in, 1000,
 DEFINE_bool(sync_fault_injection, false,
            "If true, FaultInjectionTestFS will be used for write operations, "
            " and unsynced data in DB will lost after crash.");
-
-DEFINE_bool(best_efforts_recovery, false,
-            "If true, use best efforts recovery.");
-DEFINE_bool(skip_verifydb, false, "If true, skip VerifyDb() calls.");
-
-DEFINE_bool(enable_compaction_filter, false,
-            "If true, configures a compaction filter that returns a kRemove "
-            "decision for deleted keys.");
-
 #endif  // GFLAGS
@@ -10,7 +10,6 @@

 #ifdef GFLAGS
 #include "db_stress_tool/db_stress_common.h"
-#include "db_stress_tool/db_stress_compaction_filter.h"
 #include "db_stress_tool/db_stress_driver.h"
 #include "rocksdb/convenience.h"
 #include "rocksdb/sst_file_manager.h"
@@ -196,18 +195,11 @@ void StressTest::InitDb() {
  BuildOptionsTable();
 }

-void StressTest::FinishInitDb(SharedState* shared) {
-  if (FLAGS_read_only) {
-    uint64_t now = db_stress_env->NowMicros();
-    fprintf(stdout, "%s Preloading db with %" PRIu64 " KVs\n",
-            db_stress_env->TimeToString(now / 1000000).c_str(), FLAGS_max_key);
-    PreloadDbAndReopenAsReadOnly(FLAGS_max_key, shared);
-  }
-  if (FLAGS_enable_compaction_filter) {
-    reinterpret_cast<DbStressCompactionFilterFactory*>(
-        options_.compaction_filter_factory.get())
-        ->SetSharedState(shared);
-  }
+void StressTest::InitReadonlyDb(SharedState* shared) {
+  uint64_t now = db_stress_env->NowMicros();
+  fprintf(stdout, "%s Preloading db with %" PRIu64 " KVs\n",
+          db_stress_env->TimeToString(now / 1000000).c_str(), FLAGS_max_key);
+  PreloadDbAndReopenAsReadOnly(FLAGS_max_key, shared);
 }

 bool StressTest::VerifySecondaries() {
@@ -1717,9 +1709,9 @@ void StressTest::PrintEnv() const {
  fprintf(stdout, "Memtablerep               : %s\n", memtablerep);

  fprintf(stdout, "Test kill odd             : %d\n", rocksdb_kill_odds);
-  if (!rocksdb_kill_exclude_prefixes.empty()) {
+  if (!rocksdb_kill_prefix_blacklist.empty()) {
    fprintf(stdout, "Skipping kill points prefixes:\n");
-    for (auto& p : rocksdb_kill_exclude_prefixes) {
+    for (auto& p : rocksdb_kill_prefix_blacklist) {
      fprintf(stdout, "  %s\n", p.c_str());
    }
  }
@@ -1737,8 +1729,6 @@ void StressTest::PrintEnv() const {
          static_cast<int>(FLAGS_level_compaction_dynamic_level_bytes));
  fprintf(stdout, "Read fault one in         : %d\n", FLAGS_read_fault_one_in);
  fprintf(stdout, "Sync fault injection      : %d\n", FLAGS_sync_fault_injection);
-  fprintf(stdout, "Best efforts recovery     : %d\n",
-          static_cast<int>(FLAGS_best_efforts_recovery));

  fprintf(stdout, "------------------------------------------------\n");
 }
@@ -1762,8 +1752,6 @@ void StressTest::Open() {
        static_cast<int32_t>(FLAGS_index_block_restart_interval);
    block_based_options.filter_policy = filter_policy_;
    block_based_options.partition_filters = FLAGS_partition_filters;
-    block_based_options.optimize_filters_for_memory =
-        FLAGS_optimize_filters_for_memory;
    block_based_options.index_type =
        static_cast<BlockBasedTableOptions::IndexType>(FLAGS_index_type);
    options_.table_factory.reset(
@@ -1924,12 +1912,6 @@ void StressTest::Open() {
  } else {
    options_.merge_operator = MergeOperators::CreatePutOperator();
  }
-  if (FLAGS_enable_compaction_filter) {
-    options_.compaction_filter_factory =
-        std::make_shared<DbStressCompactionFilterFactory>();
-  }
-
-  options_.best_efforts_recovery = FLAGS_best_efforts_recovery;

  fprintf(stdout, "DB path: [%s]\n", FLAGS_db.c_str());

@@ -27,9 +27,7 @@ class StressTest {
  bool BuildOptionsTable();

  void InitDb();
-  // The initialization work is split into two parts to avoid a circular
-  // dependency with `SharedState`.
-  void FinishInitDb(SharedState*);
+  void InitReadonlyDb(SharedState*);

  // Return false if verification fails.
  bool VerifySecondaries();
@@ -216,39 +216,9 @@ int db_stress_tool(int argc, char** argv) {
        "Must set -test_secondary=true if secondary_catch_up_one_in > 0.\n");
    exit(1);
  }
-  if (FLAGS_best_efforts_recovery && !FLAGS_skip_verifydb &&
-      !FLAGS_disable_wal) {
-    fprintf(stderr,
-            "With best-efforts recovery, either skip_verifydb or disable_wal "
-            "should be set to true.\n");
-    exit(1);
-  }
-  if (FLAGS_skip_verifydb) {
-    if (FLAGS_verify_db_one_in > 0) {
-      fprintf(stderr,
-              "Must set -verify_db_one_in=0 if skip_verifydb is true.\n");
-      exit(1);
-    }
-    if (FLAGS_continuous_verification_interval > 0) {
-      fprintf(stderr,
-              "Must set -continuous_verification_interval=0 if skip_verifydb "
-              "is true.\n");
-      exit(1);
-    }
-  }
-  if (FLAGS_enable_compaction_filter &&
-      (FLAGS_acquire_snapshot_one_in > 0 || FLAGS_compact_range_one_in > 0 ||
-       FLAGS_iterpercent > 0 || FLAGS_test_batches_snapshots ||
-       FLAGS_test_cf_consistency)) {
-    fprintf(
-        stderr,
-        "Error: acquire_snapshot_one_in, compact_range_one_in, iterpercent, "
-        "test_batches_snapshots  must all be 0 when using compaction filter\n");
-    exit(1);
-  }

  rocksdb_kill_odds = FLAGS_kill_random_test;
-  rocksdb_kill_exclude_prefixes = SplitString(FLAGS_kill_exclude_prefixes);
+  rocksdb_kill_prefix_blacklist = SplitString(FLAGS_kill_prefix_blacklist);

  unsigned int levels = FLAGS_max_key_len;
  std::vector<std::string> weights;
@@ -555,18 +555,14 @@ class CompositeEnvWrapper : public Env {
    return env_target_->IncBackgroundThreadsIfNeeded(num, pri);
  }

-  void LowerThreadPoolIOPriority(Priority pool) override {
+  void LowerThreadPoolIOPriority(Priority pool = LOW) override {
    env_target_->LowerThreadPoolIOPriority(pool);
  }

-  void LowerThreadPoolCPUPriority(Priority pool) override {
+  void LowerThreadPoolCPUPriority(Priority pool = LOW) override {
    env_target_->LowerThreadPoolCPUPriority(pool);
  }

-  Status LowerThreadPoolCPUPriority(Priority pool, CpuPriority pri) override {
-    return env_target_->LowerThreadPoolCPUPriority(pool, pri);
-  }
-
  std::string TimeToString(uint64_t time) override {
    return env_target_->TimeToString(time);
  }
@@ -4,14 +4,13 @@
 //
 // Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.

-#include <algorithm>
 #include <memory>
 #include <string>
 #include <vector>
+#include <algorithm>

 #include "env/mock_env.h"
 #include "rocksdb/env.h"
-#include "rocksdb/env_encryption.h"
 #include "test_util/testharness.h"

 namespace ROCKSDB_NAMESPACE {
@@ -90,24 +89,6 @@ INSTANTIATE_TEST_CASE_P(EnvDefault, EnvMoreTestWithParam,
 static std::unique_ptr<Env> mock_env(new MockEnv(Env::Default()));
 INSTANTIATE_TEST_CASE_P(MockEnv, EnvBasicTestWithParam,
                        ::testing::Values(mock_env.get()));
-
-#ifndef ROCKSDB_LITE
-// next statements run env test against default encryption code.
-static ROT13BlockCipher encrypt_block_rot13(32);
-
-static CTREncryptionProvider encrypt_provider_ctr(encrypt_block_rot13);
-
-static std::unique_ptr<Env> ecpt_env(NewEncryptedEnv(Env::Default(),
-                                                     &encrypt_provider_ctr));
-
-static std::unique_ptr<Env> encrypt_env(
-    new NormalizingEnvWrapper(ecpt_env.get()));
-INSTANTIATE_TEST_CASE_P(EncryptedEnv, EnvBasicTestWithParam,
-                        ::testing::Values(encrypt_env.get()));
-INSTANTIATE_TEST_CASE_P(EncryptedEnv, EnvMoreTestWithParam,
-                        ::testing::Values(encrypt_env.get()));
-#endif  // ROCKSDB_LITE
-
 #ifndef ROCKSDB_LITE
 static std::unique_ptr<Env> mem_env(NewMemEnv(Env::Default()));
 INSTANTIATE_TEST_CASE_P(MemEnv, EnvBasicTestWithParam,
@@ -23,6 +23,19 @@ namespace ROCKSDB_NAMESPACE {

 #ifndef ROCKSDB_LITE

+class EncryptedSequentialFile : public SequentialFile {
+  private:
+    std::unique_ptr<SequentialFile> file_;
+    std::unique_ptr<BlockAccessCipherStream> stream_;
+    uint64_t offset_;
+    size_t prefixLength_;
+
+     public:
+  // Default ctor. Given underlying sequential file is supposed to be at
+  // offset == prefixLength.
+  EncryptedSequentialFile(SequentialFile* f, BlockAccessCipherStream* s, size_t prefixLength)
+      : file_(f), stream_(s), offset_(prefixLength), prefixLength_(prefixLength) {
+  }

  // Read up to "n" bytes from the file.  "scratch[0..n-1]" may be
  // written by this routine.  Sets "*result" to the data that was
@@ -32,73 +45,83 @@ namespace ROCKSDB_NAMESPACE {
  // If an error was encountered, returns a non-OK status.
  //
  // REQUIRES: External synchronization
-Status EncryptedSequentialFile::Read(size_t n, Slice* result, char* scratch) {
-  assert(scratch);
-  Status status = file_->Read(n, result, scratch);
-  if (!status.ok()) {
+  Status Read(size_t n, Slice* result, char* scratch) override {
+    assert(scratch);
+    Status status = file_->Read(n, result, scratch);
+    if (!status.ok()) {
+      return status;
+    }
+    {
+      PERF_TIMER_GUARD(decrypt_data_nanos);
+      status = stream_->Decrypt(offset_, (char*)result->data(), result->size());
+    }
+    offset_ += result->size();  // We've already ready data from disk, so update
+                                // offset_ even if decryption fails.
    return status;
  }
-  {
-    PERF_TIMER_GUARD(decrypt_data_nanos);
-    status = stream_->Decrypt(offset_, (char*)result->data(), result->size());
-  }
-  offset_ += result->size();  // We've already ready data from disk, so update
-                              // offset_ even if decryption fails.
-  return status;
-}

-// Skip "n" bytes from the file. This is guaranteed to be no
-// slower that reading the same data, but may be faster.
-//
-// If end of file is reached, skipping will stop at the end of the
-// file, and Skip will return OK.
-//
-// REQUIRES: External synchronization
-Status EncryptedSequentialFile::Skip(uint64_t n) {
-  auto status = file_->Skip(n);
-  if (!status.ok()) {
+  // Skip "n" bytes from the file. This is guaranteed to be no
+  // slower that reading the same data, but may be faster.
+  //
+  // If end of file is reached, skipping will stop at the end of the
+  // file, and Skip will return OK.
+  //
+  // REQUIRES: External synchronization
+  Status Skip(uint64_t n) override {
+    auto status = file_->Skip(n);
+    if (!status.ok()) {
+      return status;
+    }
+    offset_ += n;
    return status;
  }
-  offset_ += n;
-  return status;
-}

-// Indicates the upper layers if the current SequentialFile implementation
-// uses direct IO.
-bool EncryptedSequentialFile::use_direct_io() const {
-  return file_->use_direct_io();
-}
+  // Indicates the upper layers if the current SequentialFile implementation
+  // uses direct IO.
+  bool use_direct_io() const override { return file_->use_direct_io(); }

-// Use the returned alignment value to allocate
-// aligned buffer for Direct I/O
-size_t EncryptedSequentialFile::GetRequiredBufferAlignment() const {
-  return file_->GetRequiredBufferAlignment();
-}
+  // Use the returned alignment value to allocate
+  // aligned buffer for Direct I/O
+  size_t GetRequiredBufferAlignment() const override {
+    return file_->GetRequiredBufferAlignment();
+  }

  // Remove any kind of caching of data from the offset to offset+length
  // of this file. If the length is 0, then it refers to the end of file.
  // If the system is not caching the file contents, then this is a noop.
-Status EncryptedSequentialFile::InvalidateCache(size_t offset, size_t length) {
-  return file_->InvalidateCache(offset + prefixLength_, length);
-}
+  Status InvalidateCache(size_t offset, size_t length) override {
+    return file_->InvalidateCache(offset + prefixLength_, length);
+  }

  // Positioned Read for direct I/O
  // If Direct I/O enabled, offset, n, and scratch should be properly aligned
-Status EncryptedSequentialFile::PositionedRead(uint64_t offset, size_t n,
-                                               Slice* result, char* scratch) {
-  assert(scratch);
-  offset += prefixLength_;  // Skip prefix
-  auto status = file_->PositionedRead(offset, n, result, scratch);
-  if (!status.ok()) {
+  Status PositionedRead(uint64_t offset, size_t n, Slice* result,
+                        char* scratch) override {
+    assert(scratch);
+    offset += prefixLength_; // Skip prefix
+    auto status = file_->PositionedRead(offset, n, result, scratch);
+    if (!status.ok()) {
+      return status;
+    }
+    offset_ = offset + result->size();
+    {
+      PERF_TIMER_GUARD(decrypt_data_nanos);
+      status = stream_->Decrypt(offset, (char*)result->data(), result->size());
+    }
    return status;
  }
-  offset_ = offset + result->size();
-  {
-    PERF_TIMER_GUARD(decrypt_data_nanos);
-    status = stream_->Decrypt(offset, (char*)result->data(), result->size());
-  }
-  return status;
-}
+};
+
+// A file abstraction for randomly reading the contents of a file.
+class EncryptedRandomAccessFile : public RandomAccessFile {
+  private:
+    std::unique_ptr<RandomAccessFile> file_;
+    std::unique_ptr<BlockAccessCipherStream> stream_;
+    size_t prefixLength_;
+
+ public:
+  EncryptedRandomAccessFile(RandomAccessFile* f, BlockAccessCipherStream* s, size_t prefixLength)
+    : file_(f), stream_(s), prefixLength_(prefixLength) { }

  // Read up to "n" bytes from the file starting at "offset".
  // "scratch[0..n-1]" may be written by this routine.  Sets "*result"
@@ -110,26 +133,26 @@ Status EncryptedSequentialFile::PositionedRead(uint64_t offset, size_t n,
  //
  // Safe for concurrent use by multiple threads.
  // If Direct I/O enabled, offset, n, and scratch should be aligned properly.
-Status EncryptedRandomAccessFile::Read(uint64_t offset, size_t n, Slice* result,
-                                       char* scratch) const {
-  assert(scratch);
-  offset += prefixLength_;
-  auto status = file_->Read(offset, n, result, scratch);
-  if (!status.ok()) {
+  Status Read(uint64_t offset, size_t n, Slice* result,
+              char* scratch) const override {
+    assert(scratch);
+    offset += prefixLength_;
+    auto status = file_->Read(offset, n, result, scratch);
+    if (!status.ok()) {
+      return status;
+    }
+    {
+      PERF_TIMER_GUARD(decrypt_data_nanos);
+      status = stream_->Decrypt(offset, (char*)result->data(), result->size());
+    }
    return status;
  }
-  {
-    PERF_TIMER_GUARD(decrypt_data_nanos);
-    status = stream_->Decrypt(offset, (char*)result->data(), result->size());
-  }
-  return status;
-}

  // Readahead the file starting from offset by n bytes for caching.
-Status EncryptedRandomAccessFile::Prefetch(uint64_t offset, size_t n) {
-  // return Status::OK();
-  return file_->Prefetch(offset + prefixLength_, n);
-}
+  Status Prefetch(uint64_t offset, size_t n) override {
+    //return Status::OK();
+    return file_->Prefetch(offset + prefixLength_, n);
+  }

  // Tries to get an unique ID for this file that will be the same each time
  // the file is opened (and will stay the same while the file is open).
@@ -146,128 +169,132 @@ Status EncryptedRandomAccessFile::Prefetch(uint64_t offset, size_t n) {
  // a single varint.
  //
  // Note: these IDs are only valid for the duration of the process.
-size_t EncryptedRandomAccessFile::GetUniqueId(char* id, size_t max_size) const {
-  return file_->GetUniqueId(id, max_size);
-};
+  size_t GetUniqueId(char* id, size_t max_size) const override {
+    return file_->GetUniqueId(id, max_size);
+  };

-void EncryptedRandomAccessFile::Hint(AccessPattern pattern) {
-  file_->Hint(pattern);
-}
+  void Hint(AccessPattern pattern) override { file_->Hint(pattern); }

  // Indicates the upper layers if the current RandomAccessFile implementation
  // uses direct IO.
-bool EncryptedRandomAccessFile::use_direct_io() const {
-  return file_->use_direct_io();
-}
+  bool use_direct_io() const override { return file_->use_direct_io(); }

  // Use the returned alignment value to allocate
  // aligned buffer for Direct I/O
-size_t EncryptedRandomAccessFile::GetRequiredBufferAlignment() const {
-  return file_->GetRequiredBufferAlignment();
-}
+  size_t GetRequiredBufferAlignment() const override {
+    return file_->GetRequiredBufferAlignment();
+  }

  // Remove any kind of caching of data from the offset to offset+length
  // of this file. If the length is 0, then it refers to the end of file.
  // If the system is not caching the file contents, then this is a noop.
-Status EncryptedRandomAccessFile::InvalidateCache(size_t offset,
-                                                  size_t length) {
-  return file_->InvalidateCache(offset + prefixLength_, length);
-}
+  Status InvalidateCache(size_t offset, size_t length) override {
+    return file_->InvalidateCache(offset + prefixLength_, length);
+  }
+};

 // A file abstraction for sequential writing.  The implementation
 // must provide buffering since callers may append small fragments
 // at a time to the file.
-Status EncryptedWritableFile::Append(const Slice& data) {
-  AlignedBuffer buf;
-  Status status;
-  Slice dataToAppend(data);
-  if (data.size() > 0) {
-    auto offset = file_->GetFileSize();  // size including prefix
-    // Encrypt in cloned buffer
-    buf.Alignment(GetRequiredBufferAlignment());
-    buf.AllocateNewBuffer(data.size());
-    // TODO (sagar0): Modify AlignedBuffer.Append to allow doing a memmove
-    // so that the next two lines can be replaced with buf.Append().
-    memmove(buf.BufferStart(), data.data(), data.size());
-    buf.Size(data.size());
-    {
-      PERF_TIMER_GUARD(encrypt_data_nanos);
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
-    }
-    if (!status.ok()) {
-      return status;
-    }
-    dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
-  }
-  status = file_->Append(dataToAppend);
-  if (!status.ok()) {
-    return status;
-  }
-  return status;
-}
+class EncryptedWritableFile : public WritableFileWrapper {
+  private:
+    std::unique_ptr<WritableFile> file_;
+    std::unique_ptr<BlockAccessCipherStream> stream_;
+    size_t prefixLength_;

-Status EncryptedWritableFile::PositionedAppend(const Slice& data,
-                                               uint64_t offset) {
-  AlignedBuffer buf;
-  Status status;
-  Slice dataToAppend(data);
-  offset += prefixLength_;
-  if (data.size() > 0) {
-    // Encrypt in cloned buffer
-    buf.Alignment(GetRequiredBufferAlignment());
-    buf.AllocateNewBuffer(data.size());
-    memmove(buf.BufferStart(), data.data(), data.size());
-    buf.Size(data.size());
-    {
-      PERF_TIMER_GUARD(encrypt_data_nanos);
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+ public:
+  // Default ctor. Prefix is assumed to be written already.
+  EncryptedWritableFile(WritableFile* f, BlockAccessCipherStream* s, size_t prefixLength)
+    : WritableFileWrapper(f), file_(f), stream_(s), prefixLength_(prefixLength) { }
+
+  Status Append(const Slice& data) override {
+    AlignedBuffer buf;
+    Status status;
+    Slice dataToAppend(data);
+    if (data.size() > 0) {
+      auto offset = file_->GetFileSize(); // size including prefix
+      // Encrypt in cloned buffer
+      buf.Alignment(GetRequiredBufferAlignment());
+      buf.AllocateNewBuffer(data.size());
+      // TODO (sagar0): Modify AlignedBuffer.Append to allow doing a memmove
+      // so that the next two lines can be replaced with buf.Append().
+      memmove(buf.BufferStart(), data.data(), data.size());
+      buf.Size(data.size());
+      {
+        PERF_TIMER_GUARD(encrypt_data_nanos);
+        status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      }
+      if (!status.ok()) {
+        return status;
+      }
+      dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
    }
+    status = file_->Append(dataToAppend);
+    if (!status.ok()) {
+      return status;
+    }
+    return status;
+  }
+
+  Status PositionedAppend(const Slice& data, uint64_t offset) override {
+    AlignedBuffer buf;
+    Status status;
+    Slice dataToAppend(data);
+    offset += prefixLength_;
+    if (data.size() > 0) {
+      // Encrypt in cloned buffer
+      buf.Alignment(GetRequiredBufferAlignment());
+      buf.AllocateNewBuffer(data.size());
+      memmove(buf.BufferStart(), data.data(), data.size());
+      buf.Size(data.size());
+      {
+        PERF_TIMER_GUARD(encrypt_data_nanos);
+        status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      }
+      if (!status.ok()) {
+        return status;
+      }
+      dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
+    }
+    status = file_->PositionedAppend(dataToAppend, offset);
    if (!status.ok()) {
      return status;
    }
-    dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
-  }
-  status = file_->PositionedAppend(dataToAppend, offset);
-  if (!status.ok()) {
    return status;
  }
-  return status;
-}

  // Indicates the upper layers if the current WritableFile implementation
  // uses direct IO.
-bool EncryptedWritableFile::use_direct_io() const {
-  return file_->use_direct_io();
-}
+  bool use_direct_io() const override { return file_->use_direct_io(); }

  // Use the returned alignment value to allocate
  // aligned buffer for Direct I/O
-size_t EncryptedWritableFile::GetRequiredBufferAlignment() const {
-  return file_->GetRequiredBufferAlignment();
-}
+  size_t GetRequiredBufferAlignment() const override {
+    return file_->GetRequiredBufferAlignment();
+  }

-/*
- * Get the size of valid data in the file.
- */
-uint64_t EncryptedWritableFile::GetFileSize() {
-  return file_->GetFileSize() - prefixLength_;
-}
+    /*
+   * Get the size of valid data in the file.
+   */
+  uint64_t GetFileSize() override {
+    return file_->GetFileSize() - prefixLength_;
+  }

  // Truncate is necessary to trim the file to the correct size
  // before closing. It is not always possible to keep track of the file
  // size due to whole pages writes. The behavior is undefined if called
  // with other writes to follow.
-Status EncryptedWritableFile::Truncate(uint64_t size) {
-  return file_->Truncate(size + prefixLength_);
-}
+  Status Truncate(uint64_t size) override {
+    return file_->Truncate(size + prefixLength_);
+  }

    // Remove any kind of caching of data from the offset to offset+length
  // of this file. If the length is 0, then it refers to the end of file.
  // If the system is not caching the file contents, then this is a noop.
  // This call has no effect on dirty pages in the cache.
-Status EncryptedWritableFile::InvalidateCache(size_t offset, size_t length) {
-  return file_->InvalidateCache(offset + prefixLength_, length);
-}
+  Status InvalidateCache(size_t offset, size_t length) override {
+    return file_->InvalidateCache(offset + prefixLength_, length);
+  }

  // Sync a file range with disk.
  // offset is the starting byte of the file range to be synchronized.
@@ -275,103 +302,111 @@ Status EncryptedWritableFile::InvalidateCache(size_t offset, size_t length) {
  // This asks the OS to initiate flushing the cached data to disk,
  // without waiting for completion.
  // Default implementation does nothing.
-Status EncryptedWritableFile::RangeSync(uint64_t offset, uint64_t nbytes) {
-  return file_->RangeSync(offset + prefixLength_, nbytes);
-}
+  Status RangeSync(uint64_t offset, uint64_t nbytes) override {
+    return file_->RangeSync(offset + prefixLength_, nbytes);
+  }

  // PrepareWrite performs any necessary preparation for a write
  // before the write actually occurs.  This allows for pre-allocation
  // of space on devices where it can result in less file
  // fragmentation and/or less waste from over-zealous filesystem
  // pre-allocation.
-void EncryptedWritableFile::PrepareWrite(size_t offset, size_t len) {
-  file_->PrepareWrite(offset + prefixLength_, len);
-}
+  void PrepareWrite(size_t offset, size_t len) override {
+    file_->PrepareWrite(offset + prefixLength_, len);
+  }

  // Pre-allocates space for a file.
-Status EncryptedWritableFile::Allocate(uint64_t offset, uint64_t len) {
-  return file_->Allocate(offset + prefixLength_, len);
-}
+  Status Allocate(uint64_t offset, uint64_t len) override {
+    return file_->Allocate(offset + prefixLength_, len);
+  }
+};

 // A file abstraction for random reading and writing.
+class EncryptedRandomRWFile : public RandomRWFile {
+  private:
+    std::unique_ptr<RandomRWFile> file_;
+    std::unique_ptr<BlockAccessCipherStream> stream_;
+    size_t prefixLength_;

-// Indicates if the class makes use of direct I/O
-// If false you must pass aligned buffer to Write()
-bool EncryptedRandomRWFile::use_direct_io() const {
-  return file_->use_direct_io();
-}
+ public:
+  EncryptedRandomRWFile(RandomRWFile* f, BlockAccessCipherStream* s, size_t prefixLength)
+    : file_(f), stream_(s), prefixLength_(prefixLength) {}
+
+  // Indicates if the class makes use of direct I/O
+  // If false you must pass aligned buffer to Write()
+  bool use_direct_io() const override { return file_->use_direct_io(); }

  // Use the returned alignment value to allocate
  // aligned buffer for Direct I/O
-size_t EncryptedRandomRWFile::GetRequiredBufferAlignment() const {
-  return file_->GetRequiredBufferAlignment();
-}
+  size_t GetRequiredBufferAlignment() const override {
+    return file_->GetRequiredBufferAlignment();
+  }

  // Write bytes in `data` at  offset `offset`, Returns Status::OK() on success.
  // Pass aligned buffer when use_direct_io() returns true.
-Status EncryptedRandomRWFile::Write(uint64_t offset, const Slice& data) {
-  AlignedBuffer buf;
-  Status status;
-  Slice dataToWrite(data);
-  offset += prefixLength_;
-  if (data.size() > 0) {
-    // Encrypt in cloned buffer
-    buf.Alignment(GetRequiredBufferAlignment());
-    buf.AllocateNewBuffer(data.size());
-    memmove(buf.BufferStart(), data.data(), data.size());
-    buf.Size(data.size());
-    {
-      PERF_TIMER_GUARD(encrypt_data_nanos);
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+  Status Write(uint64_t offset, const Slice& data) override {
+    AlignedBuffer buf;
+    Status status;
+    Slice dataToWrite(data);
+    offset += prefixLength_;
+    if (data.size() > 0) {
+      // Encrypt in cloned buffer
+      buf.Alignment(GetRequiredBufferAlignment());
+      buf.AllocateNewBuffer(data.size());
+      memmove(buf.BufferStart(), data.data(), data.size());
+      buf.Size(data.size());
+      {
+        PERF_TIMER_GUARD(encrypt_data_nanos);
+        status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      }
+      if (!status.ok()) {
+        return status;
+      }
+      dataToWrite = Slice(buf.BufferStart(), buf.CurrentSize());
    }
-    if (!status.ok()) {
-      return status;
-    }
-    dataToWrite = Slice(buf.BufferStart(), buf.CurrentSize());
+    status = file_->Write(offset, dataToWrite);
+    return status;
  }
-  status = file_->Write(offset, dataToWrite);
-  return status;
-}

  // Read up to `n` bytes starting from offset `offset` and store them in
  // result, provided `scratch` size should be at least `n`.
  // Returns Status::OK() on success.
-Status EncryptedRandomRWFile::Read(uint64_t offset, size_t n, Slice* result,
-                                   char* scratch) const {
-  assert(scratch);
-  offset += prefixLength_;
-  auto status = file_->Read(offset, n, result, scratch);
-  if (!status.ok()) {
+  Status Read(uint64_t offset, size_t n, Slice* result,
+              char* scratch) const override {
+    assert(scratch);
+    offset += prefixLength_;
+    auto status = file_->Read(offset, n, result, scratch);
+    if (!status.ok()) {
+      return status;
+    }
+    {
+      PERF_TIMER_GUARD(decrypt_data_nanos);
+      status = stream_->Decrypt(offset, (char*)result->data(), result->size());
+    }
    return status;
  }
-  {
-    PERF_TIMER_GUARD(decrypt_data_nanos);
-    status = stream_->Decrypt(offset, (char*)result->data(), result->size());
-  }
-  return status;
-}

-Status EncryptedRandomRWFile::Flush() { return file_->Flush(); }
+  Status Flush() override { return file_->Flush(); }

-Status EncryptedRandomRWFile::Sync() { return file_->Sync(); }
+  Status Sync() override { return file_->Sync(); }

-Status EncryptedRandomRWFile::Fsync() { return file_->Fsync(); }
+  Status Fsync() override { return file_->Fsync(); }

-Status EncryptedRandomRWFile::Close() { return file_->Close(); }
+  Status Close() override { return file_->Close(); }
+};

-// EncryptedEnv implements an Env wrapper that adds encryption to files stored
-// on disk.
+// EncryptedEnv implements an Env wrapper that adds encryption to files stored on disk.
 class EncryptedEnv : public EnvWrapper {
 public:
-  EncryptedEnv(Env* base_env, EncryptionProvider* provider)
+  EncryptedEnv(Env* base_env, EncryptionProvider *provider)
      : EnvWrapper(base_env) {
    provider_ = provider;
  }

  // NewSequentialFile opens a file for sequential reading.
-  virtual Status NewSequentialFile(const std::string& fname,
-                                   std::unique_ptr<SequentialFile>* result,
-                                   const EnvOptions& options) override {
+  Status NewSequentialFile(const std::string& fname,
+                           std::unique_ptr<SequentialFile>* result,
+                           const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_reads) {
      return Status::InvalidArgument();
@@ -390,8 +425,7 @@ class EncryptedEnv : public EnvWrapper {
      // Read prefix
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
-      status =
-          underlying->Read(prefixLength, &prefixSlice, prefixBuf.BufferStart());
+      status = underlying->Read(prefixLength, &prefixSlice, prefixBuf.BufferStart());
      if (!status.ok()) {
        return status;
      }
@@ -399,20 +433,18 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<SequentialFile>(new EncryptedSequentialFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<SequentialFile>(new EncryptedSequentialFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

  // NewRandomAccessFile opens a file for random read access.
-  virtual Status NewRandomAccessFile(const std::string& fname,
-                                     std::unique_ptr<RandomAccessFile>* result,
-                                     const EnvOptions& options) override {
+  Status NewRandomAccessFile(const std::string& fname,
+                             std::unique_ptr<RandomAccessFile>* result,
+                             const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_reads) {
      return Status::InvalidArgument();
@@ -431,8 +463,7 @@ class EncryptedEnv : public EnvWrapper {
      // Read prefix
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
-      status = underlying->Read(0, prefixLength, &prefixSlice,
-                                prefixBuf.BufferStart());
+      status = underlying->Read(0, prefixLength, &prefixSlice, prefixBuf.BufferStart());
      if (!status.ok()) {
        return status;
      }
@@ -440,20 +471,18 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<RandomAccessFile>(new EncryptedRandomAccessFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<RandomAccessFile>(new EncryptedRandomAccessFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

  // NewWritableFile opens a file for sequential writing.
-  virtual Status NewWritableFile(const std::string& fname,
-                                 std::unique_ptr<WritableFile>* result,
-                                 const EnvOptions& options) override {
+  Status NewWritableFile(const std::string& fname,
+                         std::unique_ptr<WritableFile>* result,
+                         const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_writes) {
      return Status::InvalidArgument();
@@ -483,13 +512,11 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

@@ -500,9 +527,9 @@ class EncryptedEnv : public EnvWrapper {
  // returns non-OK.
  //
  // The returned file will only be accessed by one thread at a time.
-  virtual Status ReopenWritableFile(const std::string& fname,
-                                    std::unique_ptr<WritableFile>* result,
-                                    const EnvOptions& options) override {
+  Status ReopenWritableFile(const std::string& fname,
+                            std::unique_ptr<WritableFile>* result,
+                            const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_writes) {
      return Status::InvalidArgument();
@@ -532,29 +559,26 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

  // Reuse an existing file by renaming it and opening it as writable.
-  virtual Status ReuseWritableFile(const std::string& fname,
-                                   const std::string& old_fname,
-                                   std::unique_ptr<WritableFile>* result,
-                                   const EnvOptions& options) override {
+  Status ReuseWritableFile(const std::string& fname,
+                           const std::string& old_fname,
+                           std::unique_ptr<WritableFile>* result,
+                           const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_writes) {
      return Status::InvalidArgument();
    }
    // Open file using underlying Env implementation
    std::unique_ptr<WritableFile> underlying;
-    Status status =
-        EnvWrapper::ReuseWritableFile(fname, old_fname, &underlying, options);
+    Status status = EnvWrapper::ReuseWritableFile(fname, old_fname, &underlying, options);
    if (!status.ok()) {
      return status;
    }
@@ -577,13 +601,11 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<WritableFile>(new EncryptedWritableFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

@@ -592,9 +614,9 @@ class EncryptedEnv : public EnvWrapper {
  // *result and returns OK.  On failure returns non-OK.
  //
  // The returned file will only be accessed by one thread at a time.
-  virtual Status NewRandomRWFile(const std::string& fname,
-                                 std::unique_ptr<RandomRWFile>* result,
-                                 const EnvOptions& options) override {
+  Status NewRandomRWFile(const std::string& fname,
+                         std::unique_ptr<RandomRWFile>* result,
+                         const EnvOptions& options) override {
    result->reset();
    if (options.use_mmap_reads || options.use_mmap_writes) {
      return Status::InvalidArgument();
@@ -617,16 +639,14 @@ class EncryptedEnv : public EnvWrapper {
      prefixBuf.AllocateNewBuffer(prefixLength);
      if (!isNewFile) {
        // File already exists, read prefix
-        status = underlying->Read(0, prefixLength, &prefixSlice,
-                                  prefixBuf.BufferStart());
+        status = underlying->Read(0, prefixLength, &prefixSlice, prefixBuf.BufferStart());
        if (!status.ok()) {
          return status;
        }
        prefixBuf.Size(prefixLength);
      } else {
        // File is new, initialize & write prefix
-        provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(),
-                                   prefixLength);
+        provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(), prefixLength);
        prefixBuf.Size(prefixLength);
        prefixSlice = Slice(prefixBuf.BufferStart(), prefixBuf.CurrentSize());
        // Write prefix
@@ -638,51 +658,40 @@ class EncryptedEnv : public EnvWrapper {
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
-    status =
-        provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
+    status = provider_->CreateCipherStream(fname, options, prefixSlice, &stream);
    if (!status.ok()) {
      return status;
    }
-    (*result) = std::unique_ptr<RandomRWFile>(new EncryptedRandomRWFile(
-        std::move(underlying), std::move(stream), prefixLength));
+    (*result) = std::unique_ptr<RandomRWFile>(new EncryptedRandomRWFile(underlying.release(), stream.release(), prefixLength));
    return Status::OK();
  }

-  // Store in *result the attributes of the children of the specified
-  // directory.
-  // In case the implementation lists the directory prior to iterating the
-  // files
-  // and files are concurrently deleted, the deleted files will be omitted
-  // from
+  // Store in *result the attributes of the children of the specified directory.
+  // In case the implementation lists the directory prior to iterating the files
+  // and files are concurrently deleted, the deleted files will be omitted from
  // result.
  // The name attributes are relative to "dir".
  // Original contents of *results are dropped.
  // Returns OK if "dir" exists and "*result" contains its children.
-  //         NotFound if "dir" does not exist, the calling process does not
-  //         have
+  //         NotFound if "dir" does not exist, the calling process does not have
  //                  permission to access "dir", or if "dir" is invalid.
  //         IOError if an IO Error was encountered
-  virtual Status GetChildrenFileAttributes(
+  Status GetChildrenFileAttributes(
      const std::string& dir, std::vector<FileAttributes>* result) override {
    auto status = EnvWrapper::GetChildrenFileAttributes(dir, result);
    if (!status.ok()) {
      return status;
    }
    size_t prefixLength = provider_->GetPrefixLength();
-    for (auto it = std::begin(*result); it != std::end(*result); ++it) {
-      // assert(it->size_bytes >= prefixLength);
-      //  breaks env_basic_test when called on directory containing
-      //  directories
-      // which makes subtraction of prefixLength worrisome since
-      // FileAttributes does not identify directories
+    for (auto it = std::begin(*result); it!=std::end(*result); ++it) {
+      assert(it->size_bytes >= prefixLength);
      it->size_bytes -= prefixLength;
    }
    return Status::OK();
  }

  // Store the size of fname in *file_size.
-  virtual Status GetFileSize(const std::string& fname,
-                             uint64_t* file_size) override {
+  Status GetFileSize(const std::string& fname, uint64_t* file_size) override {
    auto status = EnvWrapper::GetFileSize(fname, file_size);
    if (!status.ok()) {
      return status;
@@ -694,7 +703,7 @@ class EncryptedEnv : public EnvWrapper {
  }

 private:
-  EncryptionProvider* provider_;
+  EncryptionProvider *provider_;
 };

 // Returns an Env that encrypts data when stored on disk and decrypts data when
@@ -12,6 +12,7 @@
 #define ROCKSDB_HDFS_FILE_C

 #include <stdio.h>
+#include <sys/time.h>
 #include <time.h>
 #include <algorithm>
 #include <iostream>
@@ -123,9 +124,8 @@ class HdfsReadableFile : virtual public SequentialFile,
    Status s;
    ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile preading %s\n",
                    filename_.c_str());
-    tSize bytes_read =
-        hdfsPread(fileSys_, hfile_, offset, static_cast<void*>(scratch),
-                  static_cast<tSize>(n));
+    ssize_t bytes_read = hdfsPread(fileSys_, hfile_, offset,
+                                   (void*)scratch, (tSize)n);
    ROCKS_LOG_DEBUG(mylog, "[hdfs] HdfsReadableFile pread %s\n",
                    filename_.c_str());
    *result = Slice(scratch, (bytes_read < 0) ? 0 : bytes_read);
@@ -41,7 +41,7 @@
 #include <time.h>
 #include <algorithm>
 // Get nano time includes
-#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_GNU_KFREEBSD)
+#if defined(OS_LINUX) || defined(OS_FREEBSD)
 #elif defined(__MACH__)
 #include <Availability.h>
 #include <mach/clock.h>
@@ -249,8 +249,7 @@ class PosixEnv : public CompositeEnvWrapper {
  }

  uint64_t NowNanos() override {
-#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_GNU_KFREEBSD) || \
-    defined(OS_AIX)
+#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_AIX)
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return static_cast<uint64_t>(ts.tv_sec) * 1000000000 + ts.tv_nsec;
@@ -270,8 +269,8 @@ class PosixEnv : public CompositeEnvWrapper {
  }

  uint64_t NowCPUNanos() override {
-#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_GNU_KFREEBSD) || \
-    defined(OS_AIX) || (defined(__MACH__) && defined(__MAC_10_12))
+#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_AIX) || \
+    (defined(__MACH__) && defined(__MAC_10_12))
    struct timespec ts;
    clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
    return static_cast<uint64_t>(ts.tv_sec) * 1000000000 + ts.tv_nsec;
@@ -330,7 +329,7 @@ class PosixEnv : public CompositeEnvWrapper {
    thread_pools_[pri].IncBackgroundThreadsIfNeeded(num);
  }

-  void LowerThreadPoolIOPriority(Priority pool) override {
+  void LowerThreadPoolIOPriority(Priority pool = LOW) override {
    assert(pool >= Priority::BOTTOM && pool <= Priority::HIGH);
 #ifdef OS_LINUX
    thread_pools_[pool].LowerIOPriority();
@@ -339,15 +338,13 @@ class PosixEnv : public CompositeEnvWrapper {
 #endif
  }

-  void LowerThreadPoolCPUPriority(Priority pool) override {
+  void LowerThreadPoolCPUPriority(Priority pool = LOW) override {
    assert(pool >= Priority::BOTTOM && pool <= Priority::HIGH);
-    thread_pools_[pool].LowerCPUPriority(CpuPriority::kLow);
-  }
-
-  Status LowerThreadPoolCPUPriority(Priority pool, CpuPriority pri) override {
-    assert(pool >= Priority::BOTTOM && pool <= Priority::HIGH);
-    thread_pools_[pool].LowerCPUPriority(pri);
-    return Status::OK();
+#ifdef OS_LINUX
+    thread_pools_[pool].LowerCPUPriority();
+#else
+    (void)pool;
+#endif
  }

  std::string TimeToString(uint64_t secondsSince1970) override {
@@ -202,79 +202,6 @@ TEST_F(EnvPosixTest, DISABLED_FilePermission) {
    }
  }
 }
-
-TEST_F(EnvPosixTest, LowerThreadPoolCpuPriority) {
-  std::atomic<CpuPriority> from_priority(CpuPriority::kNormal);
-  std::atomic<CpuPriority> to_priority(CpuPriority::kNormal);
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
-      "ThreadPoolImpl::BGThread::BeforeSetCpuPriority", [&](void* pri) {
-        from_priority.store(*reinterpret_cast<CpuPriority*>(pri));
-      });
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
-      "ThreadPoolImpl::BGThread::AfterSetCpuPriority", [&](void* pri) {
-        to_priority.store(*reinterpret_cast<CpuPriority*>(pri));
-      });
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
-
-  env_->SetBackgroundThreads(1, Env::BOTTOM);
-  env_->SetBackgroundThreads(1, Env::HIGH);
-
-  auto RunTask = [&](Env::Priority pool) {
-    std::atomic<bool> called(false);
-    env_->Schedule(&SetBool, &called, pool);
-    for (int i = 0; i < kDelayMicros; i++) {
-      if (called.load()) {
-        break;
-      }
-      Env::Default()->SleepForMicroseconds(1);
-    }
-    ASSERT_TRUE(called.load());
-  };
-
-  {
-    // Same priority, no-op.
-    env_->LowerThreadPoolCPUPriority(Env::Priority::BOTTOM,
-                                     CpuPriority::kNormal);
-    RunTask(Env::Priority::BOTTOM);
-    ASSERT_EQ(from_priority, CpuPriority::kNormal);
-    ASSERT_EQ(to_priority, CpuPriority::kNormal);
-  }
-
-  {
-    // Higher priority, no-op.
-    env_->LowerThreadPoolCPUPriority(Env::Priority::BOTTOM, CpuPriority::kHigh);
-    RunTask(Env::Priority::BOTTOM);
-    ASSERT_EQ(from_priority, CpuPriority::kNormal);
-    ASSERT_EQ(to_priority, CpuPriority::kNormal);
-  }
-
-  {
-    // Lower priority from kNormal -> kLow.
-    env_->LowerThreadPoolCPUPriority(Env::Priority::BOTTOM, CpuPriority::kLow);
-    RunTask(Env::Priority::BOTTOM);
-    ASSERT_EQ(from_priority, CpuPriority::kNormal);
-    ASSERT_EQ(to_priority, CpuPriority::kLow);
-  }
-
-  {
-    // Lower priority from kLow -> kIdle.
-    env_->LowerThreadPoolCPUPriority(Env::Priority::BOTTOM, CpuPriority::kIdle);
-    RunTask(Env::Priority::BOTTOM);
-    ASSERT_EQ(from_priority, CpuPriority::kLow);
-    ASSERT_EQ(to_priority, CpuPriority::kIdle);
-  }
-
-  {
-    // Lower priority from kNormal -> kIdle for another pool.
-    env_->LowerThreadPoolCPUPriority(Env::Priority::HIGH, CpuPriority::kIdle);
-    RunTask(Env::Priority::HIGH);
-    ASSERT_EQ(from_priority, CpuPriority::kNormal);
-    ASSERT_EQ(to_priority, CpuPriority::kIdle);
-  }
-
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
-  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
-}
 #endif

 TEST_F(EnvPosixTest, MemoryMappedFileBuffer) {
@@ -1120,7 +1047,7 @@ TEST_P(EnvPosixTestWithParam, RandomAccessUniqueIDConcurrent) {

    // Collect and check whether the IDs are unique.
    std::unordered_set<std::string> ids;
-    for (const std::string& fname : fnames) {
+    for (const std::string fname : fnames) {
      std::unique_ptr<RandomAccessFile> file;
      std::string unique_id;
      ASSERT_OK(env_->NewRandomAccessFile(fname, &file, soptions));
@@ -1134,7 +1061,7 @@ TEST_P(EnvPosixTestWithParam, RandomAccessUniqueIDConcurrent) {
    }

    // Delete the files
-    for (const std::string& fname : fnames) {
+    for (const std::string fname : fnames) {
      ASSERT_OK(env_->DeleteFile(fname));
    }

@@ -150,11 +150,11 @@ bool PosixPositionedWrite(int fd, const char* buf, size_t nbyte, off_t offset) {
 #endif

 bool IsSyncFileRangeSupported(int fd) {
-  // This function tracks and checks for cases where we know `sync_file_range`
-  // definitely will not work properly despite passing the compile-time check
-  // (`ROCKSDB_RANGESYNC_PRESENT`). If we are unsure, or if any of the checks
-  // fail in unexpected ways, we allow `sync_file_range` to be used. This way
-  // should minimize risk of impacting existing use cases.
+  // The approach taken in this function is to build a blacklist of cases where
+  // we know `sync_file_range` definitely will not work properly despite passing
+  // the compile-time check (`ROCKSDB_RANGESYNC_PRESENT`). If we are unsure, or
+  // if any of the checks fail in unexpected ways, we allow `sync_file_range` to
+  // be used. This way should minimize risk of impacting existing use cases.
  struct statfs buf;
  int ret = fstatfs(fd, &buf);
  assert(ret == 0);
@@ -176,7 +176,7 @@ bool IsSyncFileRangeSupported(int fd) {
    // ("Function not implemented").
    return false;
  }
-  // None of the known cases matched, so allow `sync_file_range` use.
+  // None of the cases on the blacklist matched, so allow `sync_file_range` use.
  return true;
 }

@@ -1,45 +0,0 @@
-add_executable(simple_example
-  simple_example.cc)
-target_link_libraries(simple_example
-  ${ROCKSDB_LIB})
-
-add_executable(column_families_example
-  column_families_example.cc)
-target_link_libraries(column_families_example
-  ${ROCKSDB_LIB})
-
-add_executable(compact_files_example
-  compact_files_example.cc)
-target_link_libraries(compact_files_example
-  ${ROCKSDB_LIB})
-
-add_executable(c_simple_example
-  c_simple_example.c)
-target_link_libraries(c_simple_example
-  ${ROCKSDB_LIB})
-
-add_executable(optimistic_transaction_example
-  optimistic_transaction_example.cc)
-target_link_libraries(optimistic_transaction_example
-  ${ROCKSDB_LIB})
-
-add_executable(transaction_example
-  transaction_example.cc)
-target_link_libraries(transaction_example
-  ${ROCKSDB_LIB})
-
-add_executable(compaction_filter_example
-  compaction_filter_example.cc)
-target_link_libraries(compaction_filter_example
-  ${ROCKSDB_LIB})
-
-add_executable(options_file_example
-  options_file_example.cc)
-target_link_libraries(options_file_example
-  ${ROCKSDB_LIB})
-
-add_executable(multi_processes_example
-  EXCLUDE_FROM_ALL
-  multi_processes_example.cc)
-target_link_libraries(multi_processes_example
-  ${ROCKSDB_LIB})
@@ -68,7 +68,7 @@ int main() {
  }

  PinnableSlice pinnable_val;
-  s = db->Get(ReadOptions(), db->DefaultColumnFamily(), "key1", &pinnable_val);
+  db->Get(ReadOptions(), db->DefaultColumnFamily(), "key1", &pinnable_val);
  assert(s.IsNotFound());
  // Reset PinnableSlice after each use and before each reuse
  pinnable_val.Reset();
@@ -132,7 +132,7 @@ class RandomAccessFileReader {

  FSRandomAccessFile* file() { return file_.get(); }

-  const std::string& file_name() const { return file_name_; }
+  std::string file_name() const { return file_name_; }

  bool use_direct_io() const { return file_->use_direct_io(); }

@@ -162,7 +162,10 @@ class HdfsEnv : public Env {
    return posixEnv->TimeToString(number);
  }

-  static uint64_t gettid() { return Env::Default()->GetThreadID(); }
+  static uint64_t gettid() {
+    assert(sizeof(pthread_t) <= sizeof(uint64_t));
+    return (uint64_t)pthread_self();
+  }

  uint64_t GetThreadID() const override { return HdfsEnv::gettid(); }

@@ -206,7 +209,8 @@ class HdfsEnv : public Env {
    std::string portStr = (rem == 0 ? remaining :
                           remaining.substr(0, rem));

-    tPort port = static_cast<tPort>(atoi(portStr.c_str()));
+    tPort port;
+    port = atoi(portStr.c_str());
    if (port == 0) {
      throw HdfsFatalException("Bad host-port for hdfs " + uri);
    }
@@ -1266,8 +1266,6 @@ class DB {
  // updated, false if user attempted to call if with seqnum <= current value.
  virtual bool SetPreserveDeletesSequenceNumber(SequenceNumber seqnum) = 0;

-#ifndef ROCKSDB_LITE
-
  // Prevent file deletions. Compactions will continue to occur,
  // but no obsolete files will be deleted. Calling this multiple
  // times have the same effect as calling it once.
@@ -1284,6 +1282,7 @@ class DB {
  // threads call EnableFileDeletions()
  virtual Status EnableFileDeletions(bool force = true) = 0;

+#ifndef ROCKSDB_LITE
  // GetLiveFiles followed by GetSortedWalFiles can generate a lossless backup

  // Retrieve the list of all files in the database. The files are
@@ -1544,13 +1543,6 @@ class DB {
  // Returns Status::OK if identity could be set properly
  virtual Status GetDbIdentity(std::string& identity) const = 0;

-  // Return a unique identifier for each DB object that is opened
-  // This DB session ID should be unique among all open DB instances on all
-  // hosts, and should be unique among re-openings of the same or other DBs.
-  // (Two open DBs have the same identity from other function GetDbIdentity when
-  // one is physically copied from the other.)
-  virtual Status GetDbSessionId(std::string& session_id) const = 0;
-
  // Returns default column family handle
  virtual ColumnFamilyHandle* DefaultColumnFamily() const = 0;

@@ -61,13 +61,6 @@ class FileSystem;

 const size_t kDefaultPageSize = 4 * 1024;

-enum class CpuPriority {
-  kIdle = 0,
-  kLow = 1,
-  kNormal = 2,
-  kHigh = 3,
-};
-
 // Options while opening a file to read/write
 struct EnvOptions {
  // Construct with default Options
@@ -481,13 +474,6 @@ class Env {
  // Lower IO priority for threads from the specified pool.
  virtual void LowerThreadPoolIOPriority(Priority /*pool*/ = LOW) {}

-  // Lower CPU priority for threads from the specified pool.
-  virtual Status LowerThreadPoolCPUPriority(Priority /*pool*/,
-                                            CpuPriority /*pri*/) {
-    return Status::NotSupported(
-        "Env::LowerThreadPoolCPUPriority(Priority, CpuPriority) not supported");
-  }
-
  // Lower CPU priority for threads from the specified pool.
  virtual void LowerThreadPoolCPUPriority(Priority /*pool*/ = LOW) {}

@@ -1369,18 +1355,14 @@ class EnvWrapper : public Env {
    return target_->IncBackgroundThreadsIfNeeded(num, pri);
  }

-  void LowerThreadPoolIOPriority(Priority pool) override {
+  void LowerThreadPoolIOPriority(Priority pool = LOW) override {
    target_->LowerThreadPoolIOPriority(pool);
  }

-  void LowerThreadPoolCPUPriority(Priority pool) override {
+  void LowerThreadPoolCPUPriority(Priority pool = LOW) override {
    target_->LowerThreadPoolCPUPriority(pool);
  }

-  Status LowerThreadPoolCPUPriority(Priority pool, CpuPriority pri) override {
-    return target_->LowerThreadPoolCPUPriority(pool, pri);
-  }
-
  std::string TimeToString(uint64_t time) override {
    return target_->TimeToString(time);
  }
@@ -169,10 +169,8 @@ class CTREncryptionProvider : public EncryptionProvider {
  virtual ~CTREncryptionProvider() {}

  // GetPrefixLength returns the length of the prefix that is added to every
-  // file
-  // and used for storing encryption options.
-  // For optimal performance, the prefix length should be a multiple of
-  // the page size.
+  // file and used for storing encryption options. For optimal performance, the
+  // prefix length should be a multiple of the page size.
  virtual size_t GetPrefixLength() override;

  // CreateNewPrefix initialized an allocated block of prefix memory
@@ -196,243 +194,13 @@ class CTREncryptionProvider : public EncryptionProvider {
                                          size_t blockSize);

  // CreateCipherStreamFromPrefix creates a block access cipher stream for a
-  // file given
-  // given name and options. The given prefix is already decrypted.
+  // file given given name and options. The given prefix is already decrypted.
  virtual Status CreateCipherStreamFromPrefix(
      const std::string& fname, const EnvOptions& options,
      uint64_t initialCounter, const Slice& iv, const Slice& prefix,
      std::unique_ptr<BlockAccessCipherStream>* result);
 };

-class EncryptedSequentialFile : public SequentialFile {
- protected:
-  std::unique_ptr<SequentialFile> file_;
-  std::unique_ptr<BlockAccessCipherStream> stream_;
-  uint64_t offset_;
-  size_t prefixLength_;
-
- public:
-  // Default ctor. Given underlying sequential file is supposed to be at
-  // offset == prefixLength.
-  EncryptedSequentialFile(std::unique_ptr<SequentialFile>&& f,
-                          std::unique_ptr<BlockAccessCipherStream>&& s,
-                          size_t prefixLength)
-      : file_(std::move(f)),
-        stream_(std::move(s)),
-        offset_(prefixLength),
-        prefixLength_(prefixLength) {}
-
-  // Read up to "n" bytes from the file.  "scratch[0..n-1]" may be
-  // written by this routine.  Sets "*result" to the data that was
-  // read (including if fewer than "n" bytes were successfully read).
-  // May set "*result" to point at data in "scratch[0..n-1]", so
-  // "scratch[0..n-1]" must be live when "*result" is used.
-  // If an error was encountered, returns a non-OK status.
-  //
-  // REQUIRES: External synchronization
-  virtual Status Read(size_t n, Slice* result, char* scratch) override;
-
-  // Skip "n" bytes from the file. This is guaranteed to be no
-  // slower that reading the same data, but may be faster.
-  //
-  // If end of file is reached, skipping will stop at the end of the
-  // file, and Skip will return OK.
-  //
-  // REQUIRES: External synchronization
-  virtual Status Skip(uint64_t n) override;
-
-  // Indicates the upper layers if the current SequentialFile implementation
-  // uses direct IO.
-  virtual bool use_direct_io() const override;
-
-  // Use the returned alignment value to allocate
-  // aligned buffer for Direct I/O
-  virtual size_t GetRequiredBufferAlignment() const override;
-
-  // Remove any kind of caching of data from the offset to offset+length
-  // of this file. If the length is 0, then it refers to the end of file.
-  // If the system is not caching the file contents, then this is a noop.
-  virtual Status InvalidateCache(size_t offset, size_t length) override;
-
-  // Positioned Read for direct I/O
-  // If Direct I/O enabled, offset, n, and scratch should be properly aligned
-  virtual Status PositionedRead(uint64_t offset, size_t n, Slice* result,
-                                char* scratch) override;
-};
-
-// A file abstraction for randomly reading the contents of a file.
-class EncryptedRandomAccessFile : public RandomAccessFile {
- protected:
-  std::unique_ptr<RandomAccessFile> file_;
-  std::unique_ptr<BlockAccessCipherStream> stream_;
-  size_t prefixLength_;
-
- public:
-  EncryptedRandomAccessFile(std::unique_ptr<RandomAccessFile>&& f,
-                            std::unique_ptr<BlockAccessCipherStream>&& s,
-                            size_t prefixLength)
-      : file_(std::move(f)),
-        stream_(std::move(s)),
-        prefixLength_(prefixLength) {}
-
-  // Read up to "n" bytes from the file starting at "offset".
-  // "scratch[0..n-1]" may be written by this routine.  Sets "*result"
-  // to the data that was read (including if fewer than "n" bytes were
-  // successfully read).  May set "*result" to point at data in
-  // "scratch[0..n-1]", so "scratch[0..n-1]" must be live when
-  // "*result" is used.  If an error was encountered, returns a non-OK
-  // status.
-  //
-  // Safe for concurrent use by multiple threads.
-  // If Direct I/O enabled, offset, n, and scratch should be aligned properly.
-  virtual Status Read(uint64_t offset, size_t n, Slice* result,
-                      char* scratch) const override;
-
-  // Readahead the file starting from offset by n bytes for caching.
-  virtual Status Prefetch(uint64_t offset, size_t n) override;
-
-  // Tries to get an unique ID for this file that will be the same each time
-  // the file is opened (and will stay the same while the file is open).
-  // Furthermore, it tries to make this ID at most "max_size" bytes. If such an
-  // ID can be created this function returns the length of the ID and places it
-  // in "id"; otherwise, this function returns 0, in which case "id"
-  // may not have been modified.
-  //
-  // This function guarantees, for IDs from a given environment, two unique ids
-  // cannot be made equal to each other by adding arbitrary bytes to one of
-  // them. That is, no unique ID is the prefix of another.
-  //
-  // This function guarantees that the returned ID will not be interpretable as
-  // a single varint.
-  //
-  // Note: these IDs are only valid for the duration of the process.
-  virtual size_t GetUniqueId(char* id, size_t max_size) const override;
-
-  virtual void Hint(AccessPattern pattern) override;
-
-  // Indicates the upper layers if the current RandomAccessFile implementation
-  // uses direct IO.
-  virtual bool use_direct_io() const override;
-
-  // Use the returned alignment value to allocate
-  // aligned buffer for Direct I/O
-  virtual size_t GetRequiredBufferAlignment() const override;
-
-  // Remove any kind of caching of data from the offset to offset+length
-  // of this file. If the length is 0, then it refers to the end of file.
-  // If the system is not caching the file contents, then this is a noop.
-  virtual Status InvalidateCache(size_t offset, size_t length) override;
-};
-
-// A file abstraction for sequential writing.  The implementation
-// must provide buffering since callers may append small fragments
-// at a time to the file.
-class EncryptedWritableFile : public WritableFileWrapper {
- protected:
-  std::unique_ptr<WritableFile> file_;
-  std::unique_ptr<BlockAccessCipherStream> stream_;
-  size_t prefixLength_;
-
- public:
-  // Default ctor. Prefix is assumed to be written already.
-  EncryptedWritableFile(std::unique_ptr<WritableFile>&& f,
-                        std::unique_ptr<BlockAccessCipherStream>&& s,
-                        size_t prefixLength)
-      : WritableFileWrapper(f.get()),
-        file_(std::move(f)),
-        stream_(std::move(s)),
-        prefixLength_(prefixLength) {}
-
-  Status Append(const Slice& data) override;
-
-  Status PositionedAppend(const Slice& data, uint64_t offset) override;
-
-  // Indicates the upper layers if the current WritableFile implementation
-  // uses direct IO.
-  virtual bool use_direct_io() const override;
-
-  // Use the returned alignment value to allocate
-  // aligned buffer for Direct I/O
-  virtual size_t GetRequiredBufferAlignment() const override;
-
-  /*
-   * Get the size of valid data in the file.
-   */
-  virtual uint64_t GetFileSize() override;
-
-  // Truncate is necessary to trim the file to the correct size
-  // before closing. It is not always possible to keep track of the file
-  // size due to whole pages writes. The behavior is undefined if called
-  // with other writes to follow.
-  virtual Status Truncate(uint64_t size) override;
-
-  // Remove any kind of caching of data from the offset to offset+length
-  // of this file. If the length is 0, then it refers to the end of file.
-  // If the system is not caching the file contents, then this is a noop.
-  // This call has no effect on dirty pages in the cache.
-  virtual Status InvalidateCache(size_t offset, size_t length) override;
-
-  // Sync a file range with disk.
-  // offset is the starting byte of the file range to be synchronized.
-  // nbytes specifies the length of the range to be synchronized.
-  // This asks the OS to initiate flushing the cached data to disk,
-  // without waiting for completion.
-  // Default implementation does nothing.
-  virtual Status RangeSync(uint64_t offset, uint64_t nbytes) override;
-
-  // PrepareWrite performs any necessary preparation for a write
-  // before the write actually occurs.  This allows for pre-allocation
-  // of space on devices where it can result in less file
-  // fragmentation and/or less waste from over-zealous filesystem
-  // pre-allocation.
-  virtual void PrepareWrite(size_t offset, size_t len) override;
-
-  // Pre-allocates space for a file.
-  virtual Status Allocate(uint64_t offset, uint64_t len) override;
-};
-
-// A file abstraction for random reading and writing.
-class EncryptedRandomRWFile : public RandomRWFile {
- protected:
-  std::unique_ptr<RandomRWFile> file_;
-  std::unique_ptr<BlockAccessCipherStream> stream_;
-  size_t prefixLength_;
-
- public:
-  EncryptedRandomRWFile(std::unique_ptr<RandomRWFile>&& f,
-                        std::unique_ptr<BlockAccessCipherStream>&& s,
-                        size_t prefixLength)
-      : file_(std::move(f)),
-        stream_(std::move(s)),
-        prefixLength_(prefixLength) {}
-
-  // Indicates if the class makes use of direct I/O
-  // If false you must pass aligned buffer to Write()
-  virtual bool use_direct_io() const override;
-
-  // Use the returned alignment value to allocate
-  // aligned buffer for Direct I/O
-  virtual size_t GetRequiredBufferAlignment() const override;
-
-  // Write bytes in `data` at  offset `offset`, Returns Status::OK() on success.
-  // Pass aligned buffer when use_direct_io() returns true.
-  virtual Status Write(uint64_t offset, const Slice& data) override;
-
-  // Read up to `n` bytes starting from offset `offset` and store them in
-  // result, provided `scratch` size should be at least `n`.
-  // Returns Status::OK() on success.
-  virtual Status Read(uint64_t offset, size_t n, Slice* result,
-                      char* scratch) const override;
-
-  virtual Status Flush() override;
-
-  virtual Status Sync() override;
-
-  virtual Status Fsync() override;
-
-  virtual Status Close() override;
-};
-
 }  // namespace ROCKSDB_NAMESPACE

 #endif  // !defined(ROCKSDB_LITE)
@@ -117,6 +117,7 @@ enum class BackgroundErrorReason {
  kCompaction,
  kWriteCallback,
  kMemTable,
+  kManifestWrite,
 };

 enum class WriteStallCondition {
@@ -51,6 +51,13 @@ class InternalKeyComparator;
 class WalFilter;
 class FileSystem;

+enum class CpuPriority {
+  kIdle = 0,
+  kLow = 1,
+  kNormal = 2,
+  kHigh = 3,
+};
+
 // DB contents are stored in a set of blocks, each of which holds a
 // sequence of key,value pairs.  Each block may be compressed before
 // being stored in a file.  The following enum describes which
@@ -56,12 +56,6 @@ class PersistentCache {
  virtual StatsType Stats() = 0;

  virtual std::string GetPrintableOptions() const = 0;
-
-  // Return a new numeric id.  May be used by multiple clients who are
-  // sharding the same persistent cache to partition the key space.  Typically
-  // the client will allocate a new id at startup and prepend the id to its
-  // cache keys.
-  virtual uint64_t NewId() = 0;
 };

 // Factor method to create a new persistent cache
@@ -200,40 +200,6 @@ struct BlockBasedTableOptions {
  // incompatible with block-based filters.
  bool partition_filters = false;

-  // EXPERIMENTAL Option to generate Bloom filters that minimize memory
-  // internal fragmentation.
-  //
-  // When false, malloc_usable_size is not available, or format_version < 5,
-  // filters are generated without regard to internal fragmentation when
-  // loaded into memory (historical behavior). When true (and
-  // malloc_usable_size is available and format_version >= 5), then Bloom
-  // filters are generated to "round up" and "round down" their sizes to
-  // minimize internal fragmentation when loaded into memory, assuming the
-  // reading DB has the same memory allocation characteristics as the
-  // generating DB. This option does not break forward or backward
-  // compatibility.
-  //
-  // While individual filters will vary in bits/key and false positive rate
-  // when setting is true, the implementation attempts to maintain a weighted
-  // average FP rate for filters consistent with this option set to false.
-  //
-  // With Jemalloc for example, this setting is expected to save about 10% of
-  // the memory footprint and block cache charge of filters, while increasing
-  // disk usage of filters by about 1-2% due to encoding efficiency losses
-  // with variance in bits/key.
-  //
-  // NOTE: Because some memory counted by block cache might be unmapped pages
-  // within internal fragmentation, this option can increase observed RSS
-  // memory usage. With cache_index_and_filter_blocks=true, this option makes
-  // the block cache better at using space it is allowed.
-  //
-  // NOTE: Do not set to true if you do not trust malloc_usable_size. With
-  // this option, RocksDB might access an allocated memory object beyond its
-  // original size if malloc_usable_size says it is safe to do so. While this
-  // can be considered bad practice, it should not produce undefined behavior
-  // unless malloc_usable_size is buggy or broken.
-  bool optimize_filters_for_memory = false;
-
  // Use delta encoding to compress keys in blocks.
  // ReadOptions::pin_data requires this option to be disabled.
  //
@@ -30,8 +30,6 @@ typedef std::map<std::string, std::string> UserCollectedProperties;

 // table properties' human-readable names in the property block.
 struct TablePropertiesNames {
-  static const std::string kDbId;
-  static const std::string kDbSessionId;
  static const std::string kDataSize;
  static const std::string kIndexSize;
  static const std::string kIndexPartitions;
@@ -195,17 +193,6 @@ struct TableProperties {
  // Actual SST file creation time. 0 means unknown.
  uint64_t file_creation_time = 0;

-  // DB identity
-  // db_id is an identifier generated the first time the DB is created
-  // If DB identity is unset or unassigned, `db_id` will be an empty string.
-  std::string db_id;
-
-  // DB session identity
-  // db_session_id is an identifier that gets reset every time the DB is opened
-  // If DB session identity is unset or unassigned, `db_session_id` will be an
-  // empty string.
-  std::string db_session_id;
-
  // Name of the column family with which this SST file is associated.
  // If column family is unknown, `column_family_name` will be an empty string.
  std::string column_family_name;
@@ -400,10 +400,6 @@ class StackableDB : public DB {
    return db_->GetDbIdentity(identity);
  }

-  virtual Status GetDbSessionId(std::string& session_id) const override {
-    return db_->GetDbSessionId(session_id);
-  }
-
  using DB::SetOptions;
  virtual Status SetOptions(ColumnFamilyHandle* column_family_handle,
                            const std::unordered_map<std::string, std::string>&
@@ -6,7 +6,7 @@

 #define ROCKSDB_MAJOR 6
 #define ROCKSDB_MINOR 11
-#define ROCKSDB_PATCH 0
+#define ROCKSDB_PATCH 7

 // Do not use these. We made the mistake of declaring macros starting with
 // double underscore. Now we have to live with our choice. We'll deprecate these
@@ -656,7 +656,7 @@ Status GetStringFromStruct(
    std::string* opt_string) {
  assert(opt_string);
  opt_string->clear();
-  for (const auto& iter : type_info) {
+  for (const auto iter : type_info) {
    const auto& opt_info = iter.second;
    // If the option is no longer used in rocksdb and marked as deprecated,
    // we skip it in the serialization.
@@ -42,10 +42,10 @@ const char kSpecialChar = 'z';
 typedef std::vector<std::pair<size_t, size_t>> OffsetGap;

 void FillWithSpecialChar(char* start_ptr, size_t total_size,
-                         const OffsetGap& excluded,
+                         const OffsetGap& blacklist,
                         char special_char = kSpecialChar) {
  size_t offset = 0;
-  for (auto& pair : excluded) {
+  for (auto& pair : blacklist) {
    std::memset(start_ptr + offset, special_char, pair.first - offset);
    offset = pair.first + pair.second;
  }
@@ -53,10 +53,10 @@ void FillWithSpecialChar(char* start_ptr, size_t total_size,
 }

 int NumUnsetBytes(char* start_ptr, size_t total_size,
-                  const OffsetGap& excluded) {
+                  const OffsetGap& blacklist) {
  int total_unset_bytes_base = 0;
  size_t offset = 0;
-  for (auto& pair : excluded) {
+  for (auto& pair : blacklist) {
    for (char* ptr = start_ptr + offset; ptr < start_ptr + pair.first; ptr++) {
      if (*ptr == kSpecialChar) {
        total_unset_bytes_base++;
@@ -72,11 +72,11 @@ int NumUnsetBytes(char* start_ptr, size_t total_size,
  return total_unset_bytes_base;
 }

-// Return true iff two structs are the same except excluded fields.
+// Return true iff two structs are the same except blacklist fields.
 bool CompareBytes(char* start_ptr1, char* start_ptr2, size_t total_size,
-                  const OffsetGap& excluded) {
+                  const OffsetGap& blacklist) {
  size_t offset = 0;
-  for (auto& pair : excluded) {
+  for (auto& pair : blacklist) {
    for (; offset < pair.first; offset++) {
      if (*(start_ptr1 + offset) != *(start_ptr2 + offset)) {
        return false;
@@ -99,11 +99,11 @@ bool CompareBytes(char* start_ptr1, char* start_ptr2, size_t total_size,
 // GetBlockBasedTableOptionsFromString() and add the option to the input string
 // passed to the GetBlockBasedTableOptionsFromString() in this test.
 // If it is a complicated type, you also need to add the field to
-// kBbtoExcluded, and maybe add customized verification for it.
+// kBbtoBlacklist, and maybe add customized verification for it.
 TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {
  // Items in the form of <offset, size>. Need to be in ascending order
  // and not overlapping. Need to updated if new pointer-option is added.
-  const OffsetGap kBbtoExcluded = {
+  const OffsetGap kBbtoBlacklist = {
      {offsetof(struct BlockBasedTableOptions, flush_block_policy_factory),
       sizeof(std::shared_ptr<FlushBlockPolicyFactory>)},
      {offsetof(struct BlockBasedTableOptions, block_cache),
@@ -128,20 +128,20 @@ TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {
  // copy a well constructed struct to this memory and see how many special
  // bytes left.
  BlockBasedTableOptions* bbto = new (bbto_ptr) BlockBasedTableOptions();
-  FillWithSpecialChar(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoExcluded);
+  FillWithSpecialChar(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoBlacklist);
  // It based on the behavior of compiler that padding bytes are not changed
  // when copying the struct. It's prone to failure when compiler behavior
  // changes. We verify there is unset bytes to detect the case.
  *bbto = BlockBasedTableOptions();
  int unset_bytes_base =
-      NumUnsetBytes(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoExcluded);
+      NumUnsetBytes(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoBlacklist);
  ASSERT_GT(unset_bytes_base, 0);
  bbto->~BlockBasedTableOptions();

  // Construct the base option passed into
  // GetBlockBasedTableOptionsFromString().
  bbto = new (bbto_ptr) BlockBasedTableOptions();
-  FillWithSpecialChar(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoExcluded);
+  FillWithSpecialChar(bbto_ptr, sizeof(BlockBasedTableOptions), kBbtoBlacklist);
  // This option is not setable:
  bbto->use_delta_encoding = true;

@@ -149,7 +149,7 @@ TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {
  BlockBasedTableOptions* new_bbto =
      new (new_bbto_ptr) BlockBasedTableOptions();
  FillWithSpecialChar(new_bbto_ptr, sizeof(BlockBasedTableOptions),
-                      kBbtoExcluded);
+                      kBbtoBlacklist);

  // Need to update the option string if a new option is added.
  ASSERT_OK(GetBlockBasedTableOptionsFromString(
@@ -167,7 +167,6 @@ TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {
      "block_size_deviation=8;block_restart_interval=4; "
      "metadata_block_size=1024;"
      "partition_filters=false;"
-      "optimize_filters_for_memory=true;"
      "index_block_restart_interval=4;"
      "filter_policy=bloomfilter:4:true;whole_key_filtering=1;"
      "format_version=1;"
@@ -179,7 +178,7 @@ TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {

  ASSERT_EQ(unset_bytes_base,
            NumUnsetBytes(new_bbto_ptr, sizeof(BlockBasedTableOptions),
-                          kBbtoExcluded));
+                          kBbtoBlacklist));

  ASSERT_TRUE(new_bbto->block_cache.get() != nullptr);
  ASSERT_TRUE(new_bbto->block_cache_compressed.get() != nullptr);
@@ -199,9 +198,9 @@ TEST_F(OptionsSettableTest, BlockBasedTableOptionsAllFieldsSettable) {
 // GetDBOptionsFromString() and add the option to the input string passed to
 // DBOptionsFromString()in this test.
 // If it is a complicated type, you also need to add the field to
-// kDBOptionsExcluded, and maybe add customized verification for it.
+// kDBOptionsBlacklist, and maybe add customized verification for it.
 TEST_F(OptionsSettableTest, DBOptionsAllFieldsSettable) {
-  const OffsetGap kDBOptionsExcluded = {
+  const OffsetGap kDBOptionsBlacklist = {
      {offsetof(struct DBOptions, env), sizeof(Env*)},
      {offsetof(struct DBOptions, rate_limiter),
       sizeof(std::shared_ptr<RateLimiter>)},
@@ -229,22 +228,22 @@ TEST_F(OptionsSettableTest, DBOptionsAllFieldsSettable) {
  // copy a well constructed struct to this memory and see how many special
  // bytes left.
  DBOptions* options = new (options_ptr) DBOptions();
-  FillWithSpecialChar(options_ptr, sizeof(DBOptions), kDBOptionsExcluded);
+  FillWithSpecialChar(options_ptr, sizeof(DBOptions), kDBOptionsBlacklist);
  // It based on the behavior of compiler that padding bytes are not changed
  // when copying the struct. It's prone to failure when compiler behavior
  // changes. We verify there is unset bytes to detect the case.
  *options = DBOptions();
  int unset_bytes_base =
-      NumUnsetBytes(options_ptr, sizeof(DBOptions), kDBOptionsExcluded);
+      NumUnsetBytes(options_ptr, sizeof(DBOptions), kDBOptionsBlacklist);
  ASSERT_GT(unset_bytes_base, 0);
  options->~DBOptions();

  options = new (options_ptr) DBOptions();
-  FillWithSpecialChar(options_ptr, sizeof(DBOptions), kDBOptionsExcluded);
+  FillWithSpecialChar(options_ptr, sizeof(DBOptions), kDBOptionsBlacklist);

  char* new_options_ptr = new char[sizeof(DBOptions)];
  DBOptions* new_options = new (new_options_ptr) DBOptions();
-  FillWithSpecialChar(new_options_ptr, sizeof(DBOptions), kDBOptionsExcluded);
+  FillWithSpecialChar(new_options_ptr, sizeof(DBOptions), kDBOptionsBlacklist);

  // Need to update the option string if a new option is added.
  ASSERT_OK(
@@ -328,7 +327,7 @@ TEST_F(OptionsSettableTest, DBOptionsAllFieldsSettable) {
                             new_options));

  ASSERT_EQ(unset_bytes_base, NumUnsetBytes(new_options_ptr, sizeof(DBOptions),
-                                            kDBOptionsExcluded));
+                                            kDBOptionsBlacklist));

  options->~DBOptions();
  new_options->~DBOptions();
@@ -350,12 +349,12 @@ inline int offset_of(T1 T2::*member) {
 // GetColumnFamilyOptionsFromString() and add the option to the input
 // string passed to GetColumnFamilyOptionsFromString()in this test.
 // If it is a complicated type, you also need to add the field to
-// kColumnFamilyOptionsExcluded, and maybe add customized verification
+// kColumnFamilyOptionsBlacklist, and maybe add customized verification
 // for it.
 TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
-  // options in the excluded set need to appear in the same order as in
+  // options in the blacklist need to appear in the same order as in
  // ColumnFamilyOptions.
-  const OffsetGap kColumnFamilyOptionsExcluded = {
+  const OffsetGap kColumnFamilyOptionsBlacklist = {
      {offset_of(&ColumnFamilyOptions::inplace_callback),
       sizeof(UpdateStatus(*)(char*, uint32_t*, Slice, std::string*))},
      {offset_of(
@@ -394,7 +393,7 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
  // bytes left.
  ColumnFamilyOptions* options = new (options_ptr) ColumnFamilyOptions();
  FillWithSpecialChar(options_ptr, sizeof(ColumnFamilyOptions),
-                      kColumnFamilyOptionsExcluded);
+                      kColumnFamilyOptionsBlacklist);

  // It based on the behavior of compiler that padding bytes are not changed
  // when copying the struct. It's prone to failure when compiler behavior
@@ -406,13 +405,13 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
  options->max_mem_compaction_level = 0;

  int unset_bytes_base = NumUnsetBytes(options_ptr, sizeof(ColumnFamilyOptions),
-                                       kColumnFamilyOptionsExcluded);
+                                       kColumnFamilyOptionsBlacklist);
  ASSERT_GT(unset_bytes_base, 0);
  options->~ColumnFamilyOptions();

  options = new (options_ptr) ColumnFamilyOptions();
  FillWithSpecialChar(options_ptr, sizeof(ColumnFamilyOptions),
-                      kColumnFamilyOptionsExcluded);
+                      kColumnFamilyOptionsBlacklist);

  // Following options are not settable through
  // GetColumnFamilyOptionsFromString():
@@ -428,7 +427,7 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
  ColumnFamilyOptions* new_options =
      new (new_options_ptr) ColumnFamilyOptions();
  FillWithSpecialChar(new_options_ptr, sizeof(ColumnFamilyOptions),
-                      kColumnFamilyOptionsExcluded);
+                      kColumnFamilyOptionsBlacklist);

  // Need to update the option string if a new option is added.
  ASSERT_OK(GetColumnFamilyOptionsFromString(
@@ -491,7 +490,7 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {

  ASSERT_EQ(unset_bytes_base,
            NumUnsetBytes(new_options_ptr, sizeof(ColumnFamilyOptions),
-                          kColumnFamilyOptionsExcluded));
+                          kColumnFamilyOptionsBlacklist));

  ColumnFamilyOptions rnd_filled_options = *new_options;

@@ -503,7 +502,7 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {

  // Test copying to mutabable and immutable options and copy back the mutable
  // part.
-  const OffsetGap kMutableCFOptionsExcluded = {
+  const OffsetGap kMutableCFOptionsBlacklist = {
      {offset_of(&MutableCFOptions::prefix_extractor),
       sizeof(std::shared_ptr<const SliceTransform>)},
      {offset_of(&MutableCFOptions::max_bytes_for_level_multiplier_additional),
@@ -518,16 +517,16 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
  const char kMySpecialChar = 'x';
  char* mcfo1_ptr = new char[sizeof(MutableCFOptions)];
  FillWithSpecialChar(mcfo1_ptr, sizeof(MutableCFOptions),
-                      kMutableCFOptionsExcluded, kMySpecialChar);
+                      kMutableCFOptionsBlacklist, kMySpecialChar);
  char* mcfo2_ptr = new char[sizeof(MutableCFOptions)];
  FillWithSpecialChar(mcfo2_ptr, sizeof(MutableCFOptions),
-                      kMutableCFOptionsExcluded, kMySpecialChar);
+                      kMutableCFOptionsBlacklist, kMySpecialChar);

  // A clean column family options is constructed after filling the same special
  // char as the initial one. So that the padding bytes are the same.
  char* cfo_clean_ptr = new char[sizeof(ColumnFamilyOptions)];
  FillWithSpecialChar(cfo_clean_ptr, sizeof(ColumnFamilyOptions),
-                      kColumnFamilyOptionsExcluded);
+                      kColumnFamilyOptionsBlacklist);
  rnd_filled_options.num_levels = 66;
  ColumnFamilyOptions* cfo_clean = new (cfo_clean_ptr) ColumnFamilyOptions();

@@ -537,7 +536,7 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
  MutableCFOptions* mcfo2 = new (mcfo2_ptr) MutableCFOptions(cfo_back);

  ASSERT_TRUE(CompareBytes(mcfo1_ptr, mcfo2_ptr, sizeof(MutableCFOptions),
-                           kMutableCFOptionsExcluded));
+                           kMutableCFOptionsBlacklist));

  cfo_clean->~ColumnFamilyOptions();
  mcfo1->~MutableCFOptions();
@@ -647,7 +647,12 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
      "block_cache=1M;block_cache_compressed=1k;block_size=1024;"
      "block_size_deviation=8;block_restart_interval=4;"
      "format_version=5;whole_key_filtering=1;"
-      "filter_policy=bloomfilter:4.567:false;",
+      "filter_policy=bloomfilter:4.567:false;"
+      // A bug caused read_amp_bytes_per_bit to be a large integer in OPTIONS
+      // file generated by 6.10 to 6.14. Though bug is fixed in these releases,
+      // we need to handle the case of loading OPTIONS file generated before the
+      // fix.
+      "read_amp_bytes_per_bit=17179869185;",
      &new_opt));
  ASSERT_TRUE(new_opt.cache_index_and_filter_blocks);
  ASSERT_EQ(new_opt.index_type, BlockBasedTableOptions::kHashSearch);
@@ -668,6 +673,9 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
      dynamic_cast<const BloomFilterPolicy&>(*new_opt.filter_policy);
  EXPECT_EQ(bfp.GetMillibitsPerKey(), 4567);
  EXPECT_EQ(bfp.GetWholeBitsPerKey(), 5);
+  // Verify that only the lower 32bits are stored in
+  // new_opt.read_amp_bytes_per_bit.
+  EXPECT_EQ(1U, new_opt.read_amp_bytes_per_bit);

  // unknown option
  ASSERT_NOK(GetBlockBasedTableOptionsFromString(
@@ -1279,7 +1287,7 @@ TEST_F(OptionsTest, ConvertOptionsTest) {
 // This test suite tests the old APIs into the Configure options methods.
 // Once those APIs are officially deprecated, this test suite can be deleted.
 class OptionsOldApiTest : public testing::Test {};
-  
+
 TEST_F(OptionsOldApiTest, GetOptionsFromMapTest) {
  std::unordered_map<std::string, std::string> cf_options_map = {
      {"write_buffer_size", "1"},
@@ -1744,7 +1752,7 @@ TEST_F(OptionsOldApiTest, GetColumnFamilyOptionsFromStringTest) {
  ASSERT_TRUE(new_cf_opt.memtable_factory != nullptr);
  ASSERT_EQ(std::string(new_cf_opt.memtable_factory->Name()), "SkipListFactory");
 }
-  
+
 TEST_F(OptionsOldApiTest, GetBlockBasedTableOptionsFromString) {
  BlockBasedTableOptions table_opt;
  BlockBasedTableOptions new_opt;
@@ -1919,7 +1927,7 @@ TEST_F(OptionsOldApiTest, GetBlockBasedTableOptionsFromString) {
                ->GetHighPriPoolRatio(),
            0.5);
 }
-  
+
 TEST_F(OptionsOldApiTest, GetPlainTableOptionsFromString) {
  PlainTableOptions table_opt;
  PlainTableOptions new_opt;
@@ -1950,7 +1958,7 @@ TEST_F(OptionsOldApiTest, GetPlainTableOptionsFromString) {
             "encoding_type=kPrefixXX",
             &new_opt));
 }
-  
+
 TEST_F(OptionsOldApiTest, GetOptionsFromStringTest) {
  Options base_options, new_options;
  base_options.write_buffer_size = 20;
@@ -2508,7 +2516,7 @@ TEST_F(OptionsParserTest, Readahead) {
  uint64_t file_size = 0;
  ASSERT_OK(env_->GetFileSize(kOptionsFileName, &file_size));
  assert(file_size > 0);
-  
+
  RocksDBOptionsParser parser;

  env_->num_seq_file_read_ = 0;
@@ -36,7 +36,7 @@ namespace port {

 namespace {

-#if defined(OS_LINUX) || defined(OS_FREEBSD) || defined(OS_GNU_KFREEBSD)
+#if defined(OS_LINUX) || defined(OS_FREEBSD)
 const char* GetExecutableName() {
  static char name[1024];

@@ -176,7 +176,6 @@ LIB_SOURCES =                                                   \
  tools/dump/db_dump_tool.cc                                    \
  trace_replay/trace_replay.cc                                  \
  trace_replay/block_cache_tracer.cc                            \
-  trace_replay/io_tracer.cc                                     \
  util/build_version.cc                                         \
  util/coding.cc                                                \
  util/compaction_job_stats_impl.cc                             \
@@ -438,9 +437,8 @@ MAIN_SOURCES =                                                          \
  tools/ldb_cmd_test.cc                                                 \
  tools/reduce_levels_test.cc                                           \
  tools/sst_dump_test.cc                                                \
-  tools/trace_analyzer_test.cc				                \
+  tools/trace_analyzer_test.cc				             	                    \
  trace_replay/block_cache_tracer_test.cc                               \
-  trace_replay/io_tracer_test.cc                                        \
  util/autovector_test.cc                                               \
  util/bloom_test.cc                                                    \
  util/coding_test.cc                                                   \
@@ -374,10 +374,6 @@ struct BlockBasedTableBuilder::Rep {
  const uint64_t target_file_size;
  uint64_t file_creation_time = 0;

-  // DB IDs
-  const std::string db_id;
-  const std::string db_session_id;
-
  std::vector<std::unique_ptr<IntTblPropCollector>> table_properties_collectors;

  std::unique_ptr<ParallelCompressionRep> pc_rep;
@@ -451,8 +447,7 @@ struct BlockBasedTableBuilder::Rep {
      const CompressionOptions& _compression_opts, const bool skip_filters,
      const int _level_at_creation, const std::string& _column_family_name,
      const uint64_t _creation_time, const uint64_t _oldest_key_time,
-      const uint64_t _target_file_size, const uint64_t _file_creation_time,
-      const std::string& _db_id, const std::string& _db_session_id)
+      const uint64_t _target_file_size, const uint64_t _file_creation_time)
      : ioptions(_ioptions),
        moptions(_moptions),
        table_options(table_opt),
@@ -493,9 +488,7 @@ struct BlockBasedTableBuilder::Rep {
        creation_time(_creation_time),
        oldest_key_time(_oldest_key_time),
        target_file_size(_target_file_size),
-        file_creation_time(_file_creation_time),
-        db_id(_db_id),
-        db_session_id(_db_session_id) {
+        file_creation_time(_file_creation_time) {
    for (uint32_t i = 0; i < compression_opts.parallel_threads; i++) {
      compression_ctxs[i].reset(new CompressionContext(compression_type));
    }
@@ -705,8 +698,7 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
    const CompressionOptions& compression_opts, const bool skip_filters,
    const std::string& column_family_name, const int level_at_creation,
    const uint64_t creation_time, const uint64_t oldest_key_time,
-    const uint64_t target_file_size, const uint64_t file_creation_time,
-    const std::string& db_id, const std::string& db_session_id) {
+    const uint64_t target_file_size, const uint64_t file_creation_time) {
  BlockBasedTableOptions sanitized_table_options(table_options);
  if (sanitized_table_options.format_version == 0 &&
      sanitized_table_options.checksum != kCRC32c) {
@@ -719,18 +711,18 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
    sanitized_table_options.format_version = 1;
  }

-  rep_ = new Rep(
-      ioptions, moptions, sanitized_table_options, internal_comparator,
-      int_tbl_prop_collector_factories, column_family_id, file,
-      compression_type, sample_for_compression, compression_opts, skip_filters,
-      level_at_creation, column_family_name, creation_time, oldest_key_time,
-      target_file_size, file_creation_time, db_id, db_session_id);
+  rep_ = new Rep(ioptions, moptions, sanitized_table_options,
+                 internal_comparator, int_tbl_prop_collector_factories,
+                 column_family_id, file, compression_type,
+                 sample_for_compression, compression_opts, skip_filters,
+                 level_at_creation, column_family_name, creation_time,
+                 oldest_key_time, target_file_size, file_creation_time);

  if (rep_->filter_builder != nullptr) {
    rep_->filter_builder->StartBlock(0);
  }
  if (table_options.block_cache_compressed.get() != nullptr) {
-    BlockBasedTable::GenerateCachePrefix<Cache, FSWritableFile>(
+    BlockBasedTable::GenerateCachePrefix(
        table_options.block_cache_compressed.get(), file->writable_file(),
        &rep_->compressed_cache_key_prefix[0],
        &rep_->compressed_cache_key_prefix_size);
@@ -1093,43 +1085,42 @@ void BlockBasedTableBuilder::WriteRawBlock(const Slice& block_contents,
  if (io_s.ok()) {
    char trailer[kBlockTrailerSize];
    trailer[0] = type;
-    uint32_t checksum = 0;
+    char* trailer_without_type = trailer + 1;
    switch (r->table_options.checksum) {
      case kNoChecksum:
+        EncodeFixed32(trailer_without_type, 0);
        break;
      case kCRC32c: {
-        uint32_t crc =
-            crc32c::Value(block_contents.data(), block_contents.size());
-        // Extend to cover compression type
-        crc = crc32c::Extend(crc, trailer, 1);
-        checksum = crc32c::Mask(crc);
+        auto crc = crc32c::Value(block_contents.data(), block_contents.size());
+        crc = crc32c::Extend(crc, trailer, 1);  // Extend to cover block type
+        EncodeFixed32(trailer_without_type, crc32c::Mask(crc));
        break;
      }
      case kxxHash: {
        XXH32_state_t* const state = XXH32_createState();
        XXH32_reset(state, 0);
-        XXH32_update(state, block_contents.data(), block_contents.size());
-        // Extend to cover compression type
-        XXH32_update(state, trailer, 1);
-        checksum = XXH32_digest(state);
+        XXH32_update(state, block_contents.data(),
+                     static_cast<uint32_t>(block_contents.size()));
+        XXH32_update(state, trailer, 1);  // Extend  to cover block type
+        EncodeFixed32(trailer_without_type, XXH32_digest(state));
        XXH32_freeState(state);
        break;
      }
      case kxxHash64: {
        XXH64_state_t* const state = XXH64_createState();
        XXH64_reset(state, 0);
-        XXH64_update(state, block_contents.data(), block_contents.size());
-        // Extend to cover compression type
-        XXH64_update(state, trailer, 1);
-        checksum = Lower32of64(XXH64_digest(state));
+        XXH64_update(state, block_contents.data(),
+                     static_cast<uint32_t>(block_contents.size()));
+        XXH64_update(state, trailer, 1);  // Extend  to cover block type
+        EncodeFixed32(
+            trailer_without_type,
+            static_cast<uint32_t>(XXH64_digest(state) &  // lower 32 bits
+                                  uint64_t{0xffffffff}));
        XXH64_freeState(state);
        break;
      }
-      default:
-        assert(false);
-        break;
    }
-    EncodeFixed32(trailer + 1, checksum);
+
    assert(io_s.ok());
    TEST_SYNC_POINT_CALLBACK(
        "BlockBasedTableBuilder::WriteRawBlock:TamperWithChecksum",
@@ -1454,8 +1445,6 @@ void BlockBasedTableBuilder::WritePropertiesBlock(
    rep_->props.creation_time = rep_->creation_time;
    rep_->props.oldest_key_time = rep_->oldest_key_time;
    rep_->props.file_creation_time = rep_->file_creation_time;
-    rep_->props.db_id = rep_->db_id;
-    rep_->props.db_session_id = rep_->db_session_id;

    // Add basic properties
    property_block_builder.AddTableProperty(rep_->props);
@@ -51,8 +51,7 @@ class BlockBasedTableBuilder : public TableBuilder {
      const std::string& column_family_name, const int level_at_creation,
      const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
      const uint64_t target_file_size = 0,
-      const uint64_t file_creation_time = 0, const std::string& db_id = "",
-      const std::string& db_session_id = "");
+      const uint64_t file_creation_time = 0);

  // No copying allowed
  BlockBasedTableBuilder(const BlockBasedTableBuilder&) = delete;
@@ -268,10 +268,6 @@ static std::unordered_map<std::string, OptionTypeInfo>
         {offsetof(struct BlockBasedTableOptions, partition_filters),
          OptionType::kBoolean, OptionVerificationType::kNormal,
          OptionTypeFlags::kNone, 0}},
-        {"optimize_filters_for_memory",
-         {offsetof(struct BlockBasedTableOptions, optimize_filters_for_memory),
-          OptionType::kBoolean, OptionVerificationType::kNormal,
-          OptionTypeFlags::kNone, 0}},
        {"filter_policy",
         {offsetof(struct BlockBasedTableOptions, filter_policy),
          OptionType::kUnknown, OptionVerificationType::kByNameAllowFromNull,
@@ -331,8 +327,24 @@ static std::unordered_map<std::string, OptionTypeInfo>
          OptionTypeFlags::kNone, 0}},
        {"read_amp_bytes_per_bit",
         {offsetof(struct BlockBasedTableOptions, read_amp_bytes_per_bit),
-          OptionType::kSizeT, OptionVerificationType::kNormal,
-          OptionTypeFlags::kNone, 0}},
+          OptionType::kUInt32T, OptionVerificationType::kNormal,
+          OptionTypeFlags::kNone, 0,
+          [](const ConfigOptions& /*opts*/, const std::string& /*name*/,
+             const std::string& value, char* addr) {
+            // A workaround to fix a bug in 6.10, 6.11, 6.12, 6.13
+            // and 6.14. The bug will write out 8 bytes to OPTIONS file from the
+            // starting address of BlockBasedTableOptions.read_amp_bytes_per_bit
+            // which is actually a uint32. Consequently, the value of
+            // read_amp_bytes_per_bit written in the OPTIONS file is wrong.
+            // From 6.15, RocksDB will try to parse the read_amp_bytes_per_bit
+            // from OPTIONS file as a uint32. To be able to load OPTIONS file
+            // generated by affected releases before the fix, we need to
+            // manually parse read_amp_bytes_per_bit with this special hack.
+            uint64_t read_amp_bytes_per_bit = ParseUint64(value);
+            *(reinterpret_cast<uint32_t*>(addr)) =
+                static_cast<uint32_t>(read_amp_bytes_per_bit);
+            return Status::OK();
+          }}},
        {"enable_index_compression",
         {offsetof(struct BlockBasedTableOptions, enable_index_compression),
          OptionType::kBoolean, OptionVerificationType::kNormal,
@@ -443,8 +455,7 @@ TableBuilder* BlockBasedTableFactory::NewTableBuilder(
      table_builder_options.creation_time,
      table_builder_options.oldest_key_time,
      table_builder_options.target_file_size,
-      table_builder_options.file_creation_time, table_builder_options.db_id,
-      table_builder_options.db_session_id);
+      table_builder_options.file_creation_time);

  return table_builder;
 }
@@ -452,21 +452,44 @@ void BlockBasedTable::SetupCacheKeyPrefix(Rep* rep) {
  rep->cache_key_prefix_size = 0;
  rep->compressed_cache_key_prefix_size = 0;
  if (rep->table_options.block_cache != nullptr) {
-    GenerateCachePrefix<Cache, FSRandomAccessFile>(
-        rep->table_options.block_cache.get(), rep->file->file(),
-        &rep->cache_key_prefix[0], &rep->cache_key_prefix_size);
+    GenerateCachePrefix(rep->table_options.block_cache.get(), rep->file->file(),
+                        &rep->cache_key_prefix[0], &rep->cache_key_prefix_size);
  }
  if (rep->table_options.persistent_cache != nullptr) {
-    GenerateCachePrefix<PersistentCache, FSRandomAccessFile>(
-        rep->table_options.persistent_cache.get(), rep->file->file(),
-        &rep->persistent_cache_key_prefix[0],
-        &rep->persistent_cache_key_prefix_size);
+    GenerateCachePrefix(/*cache=*/nullptr, rep->file->file(),
+                        &rep->persistent_cache_key_prefix[0],
+                        &rep->persistent_cache_key_prefix_size);
  }
  if (rep->table_options.block_cache_compressed != nullptr) {
-    GenerateCachePrefix<Cache, FSRandomAccessFile>(
-        rep->table_options.block_cache_compressed.get(), rep->file->file(),
-        &rep->compressed_cache_key_prefix[0],
-        &rep->compressed_cache_key_prefix_size);
+    GenerateCachePrefix(rep->table_options.block_cache_compressed.get(),
+                        rep->file->file(), &rep->compressed_cache_key_prefix[0],
+                        &rep->compressed_cache_key_prefix_size);
+  }
+}
+
+void BlockBasedTable::GenerateCachePrefix(Cache* cc, FSRandomAccessFile* file,
+                                          char* buffer, size_t* size) {
+  // generate an id from the file
+  *size = file->GetUniqueId(buffer, kMaxCacheKeyPrefixSize);
+
+  // If the prefix wasn't generated or was too long,
+  // create one from the cache.
+  if (cc != nullptr && *size == 0) {
+    char* end = EncodeVarint64(buffer, cc->NewId());
+    *size = static_cast<size_t>(end - buffer);
+  }
+}
+
+void BlockBasedTable::GenerateCachePrefix(Cache* cc, FSWritableFile* file,
+                                          char* buffer, size_t* size) {
+  // generate an id from the file
+  *size = file->GetUniqueId(buffer, kMaxCacheKeyPrefixSize);
+
+  // If the prefix wasn't generated or was too long,
+  // create one from the cache.
+  if (cc != nullptr && *size == 0) {
+    char* end = EncodeVarint64(buffer, cc->NewId());
+    *size = static_cast<size_t>(end - buffer);
  }
 }

@@ -776,9 +799,9 @@ Status BlockBasedTable::TryReadPropertiesWithGlobalSeqno(
      EncodeFixed64(
          tmp_buf.get() + global_seqno_offset - props_block_handle.offset(), 0);
    }
-    s = ROCKSDB_NAMESPACE::VerifyBlockChecksum(
-        rep_->footer.checksum(), tmp_buf.get(), block_size,
-        rep_->file->file_name(), props_block_handle.offset());
+    uint32_t value = DecodeFixed32(tmp_buf.get() + block_size + 1);
+    s = ROCKSDB_NAMESPACE::VerifyChecksum(rep_->footer.checksum(),
+                                          tmp_buf.get(), block_size + 1, value);
  }
  return s;
 }
@@ -1728,14 +1751,15 @@ void BlockBasedTable::RetrieveMultipleBlocks(
      if (options.verify_checksums) {
        PERF_TIMER_GUARD(block_checksum_time);
        const char* data = req.result.data();
-        // Since the scratch might be shared, the offset of the data block in
+        uint32_t expected =
+            DecodeFixed32(data + req_offset + handle.size() + 1);
+        // Since the scratch might be shared. the offset of the data block in
        // the buffer might not be 0. req.result.data() only point to the
        // begin address of each read request, we need to add the offset
        // in each read request. Checksum is stored in the block trailer,
-        // beyond the payload size.
-        s = ROCKSDB_NAMESPACE::VerifyBlockChecksum(
-            footer.checksum(), data + req_offset, handle.size(),
-            rep_->file->file_name(), handle.offset());
+        // which is handle.size() + 1.
+        s = ROCKSDB_NAMESPACE::VerifyChecksum(
+            footer.checksum(), data + req_offset, handle.size() + 1, expected);
        TEST_SYNC_POINT_CALLBACK("RetrieveMultipleBlocks:VerifyChecksum", &s);
      }
    } else if (!use_shared_buffer) {
@@ -439,19 +439,10 @@ class BlockBasedTable : public TableReader {
  static void SetupCacheKeyPrefix(Rep* rep);

  // Generate a cache key prefix from the file
-  template <typename TCache, typename TFile>
-  static void GenerateCachePrefix(TCache* cc, TFile* file, char* buffer,
-                                  size_t* size) {
-    // generate an id from the file
-    *size = file->GetUniqueId(buffer, kMaxCacheKeyPrefixSize);
-
-    // If the prefix wasn't generated or was too long,
-    // create one from the cache.
-    if (cc != nullptr && *size == 0) {
-      char* end = EncodeVarint64(buffer, cc->NewId());
-      *size = static_cast<size_t>(end - buffer);
-    }
-  }
+  static void GenerateCachePrefix(Cache* cc, FSRandomAccessFile* file,
+                                  char* buffer, size_t* size);
+  static void GenerateCachePrefix(Cache* cc, FSWritableFile* file, char* buffer,
+                                  size_t* size);

  // Size of all data blocks, maybe approximate
  uint64_t GetApproximateDataSize();
@@ -28,12 +28,9 @@ namespace {
 // See description in FastLocalBloomImpl
 class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {
 public:
-  // Non-null aggregate_rounding_balance implies optimize_filters_for_memory
-  explicit FastLocalBloomBitsBuilder(
-      const int millibits_per_key,
-      std::atomic<int64_t>* aggregate_rounding_balance)
+  explicit FastLocalBloomBitsBuilder(const int millibits_per_key)
      : millibits_per_key_(millibits_per_key),
-        aggregate_rounding_balance_(aggregate_rounding_balance) {
+        num_probes_(FastLocalBloomImpl::ChooseNumProbes(millibits_per_key_)) {
    assert(millibits_per_key >= 1000);
  }

@@ -51,36 +48,33 @@ class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {
  }

  virtual Slice Finish(std::unique_ptr<const char[]>* buf) override {
-    size_t num_entry = hash_entries_.size();
-    std::unique_ptr<char[]> mutable_buf;
    uint32_t len_with_metadata =
-        CalculateAndAllocate(num_entry, &mutable_buf, /*update_balance*/ true);
+        CalculateSpace(static_cast<uint32_t>(hash_entries_.size()));
+    char* data = new char[len_with_metadata];
+    memset(data, 0, len_with_metadata);

-    assert(mutable_buf);
+    assert(data);
    assert(len_with_metadata >= 5);

-    // Compute num_probes after any rounding / adjustments
-    int num_probes = GetNumProbes(num_entry, len_with_metadata);
-
    uint32_t len = len_with_metadata - 5;
    if (len > 0) {
-      AddAllEntries(mutable_buf.get(), len, num_probes);
+      AddAllEntries(data, len);
    }

-    assert(hash_entries_.empty());
-
    // See BloomFilterPolicy::GetBloomBitsReader re: metadata
    // -1 = Marker for newer Bloom implementations
-    mutable_buf[len] = static_cast<char>(-1);
+    data[len] = static_cast<char>(-1);
    // 0 = Marker for this sub-implementation
-    mutable_buf[len + 1] = static_cast<char>(0);
+    data[len + 1] = static_cast<char>(0);
    // num_probes (and 0 in upper bits for 64-byte block size)
-    mutable_buf[len + 2] = static_cast<char>(num_probes);
+    data[len + 2] = static_cast<char>(num_probes_);
    // rest of metadata stays zero

-    Slice rv(mutable_buf.get(), len_with_metadata);
-    *buf = std::move(mutable_buf);
-    return rv;
+    const char* const_data = data;
+    buf->reset(const_data);
+    assert(hash_entries_.empty());
+
+    return Slice(data, len_with_metadata);
  }

  int CalculateNumEntry(const uint32_t bytes) override {
@@ -90,163 +84,26 @@ class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {
  }

  uint32_t CalculateSpace(const int num_entry) override {
-    // NB: the BuiltinFilterBitsBuilder API presumes len fits in uint32_t.
-    return static_cast<uint32_t>(
-        CalculateAndAllocate(static_cast<size_t>(num_entry),
-                             /* buf */ nullptr,
-                             /*update_balance*/ false));
+    uint32_t num_cache_lines = 0;
+    if (millibits_per_key_ > 0 && num_entry > 0) {
+      num_cache_lines = static_cast<uint32_t>(
+          (int64_t{num_entry} * millibits_per_key_ + 511999) / 512000);
+    }
+    return num_cache_lines * 64 + /*metadata*/ 5;
  }

-  // To choose size using malloc_usable_size, we have to actually allocate.
-  uint32_t CalculateAndAllocate(size_t num_entry, std::unique_ptr<char[]>* buf,
-                                bool update_balance) {
-    std::unique_ptr<char[]> tmpbuf;
-
-    // If not for cache line blocks in the filter, what would the target
-    // length in bytes be?
-    size_t raw_target_len = static_cast<size_t>(
-        (uint64_t{num_entry} * millibits_per_key_ + 7999) / 8000);
-
-    if (raw_target_len >= size_t{0xffffffc0}) {
-      // Max supported for this data structure implementation
-      raw_target_len = size_t{0xffffffc0};
-    }
-
-    // Round up to nearest multiple of 64 (block size). This adjustment is
-    // used for target FP rate only so that we don't receive complaints about
-    // lower FP rate vs. historic Bloom filter behavior.
-    uint32_t target_len =
-        static_cast<uint32_t>(raw_target_len + 63) & ~uint32_t{63};
-
-    // Return value set to a default; overwritten in some cases
-    uint32_t rv = target_len + /* metadata */ 5;
-#ifdef ROCKSDB_MALLOC_USABLE_SIZE
-    if (aggregate_rounding_balance_ != nullptr) {
-      // Do optimize_filters_for_memory, using malloc_usable_size.
-      // Approach: try to keep FP rate balance better than or on
-      // target (negative aggregate_rounding_balance_). We can then select a
-      // lower bound filter size (within reasonable limits) that gets us as
-      // close to on target as possible. We request allocation for that filter
-      // size and use malloc_usable_size to "round up" to the actual
-      // allocation size.
-
-      // Although it can be considered bad practice to use malloc_usable_size
-      // to access an object beyond its original size, this approach should
-      // quite general: working for all allocators that properly support
-      // malloc_usable_size.
-
-      // Race condition on balance is OK because it can only cause temporary
-      // skew in rounding up vs. rounding down, as long as updates are atomic
-      // and relative.
-      int64_t balance = aggregate_rounding_balance_->load();
-
-      double target_fp_rate = EstimatedFpRate(num_entry, target_len + 5);
-      double rv_fp_rate = target_fp_rate;
-
-      if (balance < 0) {
-        // See formula for BloomFilterPolicy::aggregate_rounding_balance_
-        double for_balance_fp_rate =
-            -balance / double{0x100000000} + target_fp_rate;
-
-        // To simplify, we just try a few modified smaller sizes. This also
-        // caps how much we vary filter size vs. target, to avoid outlier
-        // behavior from excessive variance.
-        for (uint64_t maybe_len64 :
-             {uint64_t{3} * target_len / 4, uint64_t{13} * target_len / 16,
-              uint64_t{7} * target_len / 8, uint64_t{15} * target_len / 16}) {
-          uint32_t maybe_len =
-              static_cast<uint32_t>(maybe_len64) & ~uint32_t{63};
-          double maybe_fp_rate = EstimatedFpRate(num_entry, maybe_len + 5);
-          if (maybe_fp_rate <= for_balance_fp_rate) {
-            rv = maybe_len + /* metadata */ 5;
-            rv_fp_rate = maybe_fp_rate;
-            break;
-          }
-        }
-      }
-
-      // Filter blocks are loaded into block cache with their block trailer.
-      // We need to make sure that's accounted for in choosing a
-      // fragmentation-friendly size.
-      const uint32_t kExtraPadding = kBlockTrailerSize;
-      size_t requested = rv + kExtraPadding;
-
-      // Allocate and get usable size
-      tmpbuf.reset(new char[requested]);
-      size_t usable = malloc_usable_size(tmpbuf.get());
-
-      if (usable - usable / 4 > requested) {
-        // Ratio greater than 4/3 is too much for utilizing, if it's
-        // not a buggy or mislinked malloc_usable_size implementation.
-        // Non-linearity of FP rates with bits/key means rapidly
-        // diminishing returns in overall accuracy for additional
-        // storage on disk.
-        // Nothing to do, except assert that the result is accurate about
-        // the usable size. (Assignment never used.)
-        assert(tmpbuf[usable - 1] = 'x');
-      } else if (usable > requested) {
-        // Adjust for reasonably larger usable size
-        size_t usable_len = (usable - kExtraPadding - /* metadata */ 5);
-        if (usable_len >= size_t{0xffffffc0}) {
-          // Max supported for this data structure implementation
-          usable_len = size_t{0xffffffc0};
-        }
-
-        rv = (static_cast<uint32_t>(usable_len) & ~uint32_t{63}) +
-             /* metadata */ 5;
-        rv_fp_rate = EstimatedFpRate(num_entry, rv);
-      } else {
-        // Too small means bad malloc_usable_size
-        assert(usable == requested);
-      }
-      memset(tmpbuf.get(), 0, rv);
-
-      if (update_balance) {
-        int64_t diff = static_cast<int64_t>((rv_fp_rate - target_fp_rate) *
-                                            double{0x100000000});
-        *aggregate_rounding_balance_ += diff;
-      }
-    }
-#else
-    (void)update_balance;
-#endif  // ROCKSDB_MALLOC_USABLE_SIZE
-    if (buf) {
-      if (tmpbuf) {
-        *buf = std::move(tmpbuf);
-      } else {
-        buf->reset(new char[rv]());
-      }
-    }
-    return rv;
-  }
-
-  double EstimatedFpRate(size_t keys, size_t len_with_metadata) override {
-    int num_probes = GetNumProbes(keys, len_with_metadata);
-    return FastLocalBloomImpl::EstimatedFpRate(
-        keys, len_with_metadata - /*metadata*/ 5, num_probes, /*hash bits*/ 64);
+  double EstimatedFpRate(size_t keys, size_t bytes) override {
+    return FastLocalBloomImpl::EstimatedFpRate(keys, bytes - /*metadata*/ 5,
+                                               num_probes_, /*hash bits*/ 64);
  }

 private:
-  // Compute num_probes after any rounding / adjustments
-  int GetNumProbes(size_t keys, size_t len_with_metadata) {
-    uint64_t millibits = uint64_t{len_with_metadata - 5} * 8000;
-    int actual_millibits_per_key =
-        static_cast<int>(millibits / std::max(keys, size_t{1}));
-    // BEGIN XXX/TODO(peterd): preserving old/default behavior for now to
-    // minimize unit test churn. Remove this some time.
-    if (!aggregate_rounding_balance_) {
-      actual_millibits_per_key = millibits_per_key_;
-    }
-    // END XXX/TODO
-    return FastLocalBloomImpl::ChooseNumProbes(actual_millibits_per_key);
-  }
-
-  void AddAllEntries(char* data, uint32_t len, int num_probes) {
+  void AddAllEntries(char* data, uint32_t len) {
    // Simple version without prefetching:
    //
    // for (auto h : hash_entries_) {
    //   FastLocalBloomImpl::AddHash(Lower32of64(h), Upper32of64(h), len,
-    //                               num_probes, data);
+    //                               num_probes_, data);
    // }

    const size_t num_entries = hash_entries_.size();
@@ -272,7 +129,7 @@ class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {
      uint32_t& hash_ref = hashes[i & kBufferMask];
      uint32_t& byte_offset_ref = byte_offsets[i & kBufferMask];
      // Process (add)
-      FastLocalBloomImpl::AddHashPrepared(hash_ref, num_probes,
+      FastLocalBloomImpl::AddHashPrepared(hash_ref, num_probes_,
                                          data + byte_offset_ref);
      // And buffer
      uint64_t h = hash_entries_.front();
@@ -284,16 +141,13 @@ class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {

    // Finish processing
    for (i = 0; i <= kBufferMask && i < num_entries; ++i) {
-      FastLocalBloomImpl::AddHashPrepared(hashes[i], num_probes,
+      FastLocalBloomImpl::AddHashPrepared(hashes[i], num_probes_,
                                          data + byte_offsets[i]);
    }
  }

-  // Target allocation per added key, in thousandths of a bit.
  int millibits_per_key_;
-  // See BloomFilterPolicy::aggregate_rounding_balance_. If nullptr,
-  // always "round up" like historic behavior.
-  std::atomic<int64_t>* aggregate_rounding_balance_;
+  int num_probes_;
  // A deque avoids unnecessary copying of already-saved values
  // and has near-minimal peak memory use.
  std::deque<uint64_t> hash_entries_;
@@ -603,7 +457,7 @@ const std::vector<BloomFilterPolicy::Mode> BloomFilterPolicy::kAllUserModes = {
 };

 BloomFilterPolicy::BloomFilterPolicy(double bits_per_key, Mode mode)
-    : mode_(mode), warned_(false), aggregate_rounding_balance_(0) {
+    : mode_(mode), warned_(false) {
  // Sanitize bits_per_key
  if (bits_per_key < 1.0) {
    bits_per_key = 1.0;
@@ -695,7 +549,6 @@ FilterBitsBuilder* BloomFilterPolicy::GetFilterBitsBuilder() const {
 FilterBitsBuilder* BloomFilterPolicy::GetBuilderWithContext(
    const FilterBuildingContext& context) const {
  Mode cur = mode_;
-  bool offm = context.table_options.optimize_filters_for_memory;
  // Unusual code construction so that we can have just
  // one exhaustive switch without (risky) recursion
  for (int i = 0; i < 2; ++i) {
@@ -710,8 +563,7 @@ FilterBitsBuilder* BloomFilterPolicy::GetBuilderWithContext(
      case kDeprecatedBlock:
        return nullptr;
      case kFastLocalBloom:
-        return new FastLocalBloomBitsBuilder(
-            millibits_per_key_, offm ? &aggregate_rounding_balance_ : nullptr);
+        return new FastLocalBloomBitsBuilder(millibits_per_key_);
      case kLegacyBloom:
        if (whole_bits_per_key_ >= 14 && context.info_log &&
            !warned_.load(std::memory_order_relaxed)) {
@@ -135,16 +135,6 @@ class BloomFilterPolicy : public FilterPolicy {
  // only report once per BloomFilterPolicy instance, to keep the noise down.)
  mutable std::atomic<bool> warned_;

-  // State for implementing optimize_filters_for_memory. Essentially, this
-  // tracks a surplus or deficit in total FP rate of filters generated by
-  // builders under this policy vs. what would have been generated without
-  // optimize_filters_for_memory.
-  //
-  // To avoid floating point weirdness, the actual value is
-  //  Sum over all generated filters f:
-  //   (predicted_fp_rate(f) - predicted_fp_rate(f|o_f_f_m=false)) * 2^32
-  mutable std::atomic<int64_t> aggregate_rounding_balance_;
-
  // For newer Bloom filter implementation(s)
  FilterBitsReader* GetBloomBitsReader(const Slice& contents) const;
 };
@@ -104,6 +104,15 @@ void PartitionedIndexBuilder::MakeNewSubIndexBuilder() {
      comparator_, table_opt_.index_block_restart_interval,
      table_opt_.format_version, use_value_delta_encoding_,
      table_opt_.index_shortening, /* include_first_key */ false);
+
+  // Set sub_index_builder_->seperator_is_key_plus_seq_ to true if
+  // seperator_is_key_plus_seq_ is true (internal-key mode) (set to false by
+  // default on Creation) so that flush policy can point to
+  // sub_index_builder_->index_block_builder_
+  if (seperator_is_key_plus_seq_) {
+    sub_index_builder_->seperator_is_key_plus_seq_ = true;
+  }
+
  flush_policy_.reset(FlushBlockBySizePolicyFactory::NewFlushBlockPolicy(
      table_opt_.metadata_block_size, table_opt_.block_size_deviation,
      // Note: this is sub-optimal since sub_index_builder_ could later reset
@@ -129,9 +138,15 @@ void PartitionedIndexBuilder::AddIndexEntry(
    }
    sub_index_builder_->AddIndexEntry(last_key_in_current_block,
                                      first_key_in_next_block, block_handle);
-    if (sub_index_builder_->seperator_is_key_plus_seq_) {
-      // then we need to apply it to all sub-index builders
+    if (!seperator_is_key_plus_seq_ &&
+        sub_index_builder_->seperator_is_key_plus_seq_) {
+      // then we need to apply it to all sub-index builders and reset
+      // flush_policy to point to Block Builder of sub_index_builder_ that store
+      // internal keys.
      seperator_is_key_plus_seq_ = true;
+      flush_policy_.reset(FlushBlockBySizePolicyFactory::NewFlushBlockPolicy(
+          table_opt_.metadata_block_size, table_opt_.block_size_deviation,
+          sub_index_builder_->index_block_builder_));
    }
    sub_index_last_key_ = std::string(*last_key_in_current_block);
    entries_.push_back(
@@ -161,9 +176,15 @@ void PartitionedIndexBuilder::AddIndexEntry(
    sub_index_builder_->AddIndexEntry(last_key_in_current_block,
                                      first_key_in_next_block, block_handle);
    sub_index_last_key_ = std::string(*last_key_in_current_block);
-    if (sub_index_builder_->seperator_is_key_plus_seq_) {
-      // then we need to apply it to all sub-index builders
+    if (!seperator_is_key_plus_seq_ &&
+        sub_index_builder_->seperator_is_key_plus_seq_) {
+      // then we need to apply it to all sub-index builders and reset
+      // flush_policy to point to Block Builder of sub_index_builder_ that store
+      // internal keys.
      seperator_is_key_plus_seq_ = true;
+      flush_policy_.reset(FlushBlockBySizePolicyFactory::NewFlushBlockPolicy(
+          table_opt_.metadata_block_size, table_opt_.block_size_deviation,
+          sub_index_builder_->index_block_builder_));
    }
  }
 }
@@ -167,7 +167,7 @@ void PartitionIndexReader::CacheDependencies(bool pin) {

    assert(s.ok() || block.GetValue() == nullptr);
    if (s.ok() && block.GetValue() != nullptr) {
-      if (block.IsCached()) {
+      if (block.IsCached() || block.GetOwnValue()) {
        if (pin) {
          partition_map_[handle.offset()] = std::move(block);
        }
@@ -8,11 +8,7 @@
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 #include "table/block_based/reader_common.h"

-#include "monitoring/perf_context_imp.h"
-#include "util/coding.h"
 #include "util/crc32c.h"
-#include "util/hash.h"
-#include "util/string_util.h"
 #include "util/xxhash.h"

 namespace ROCKSDB_NAMESPACE {
@@ -22,42 +18,29 @@ void ForceReleaseCachedEntry(void* arg, void* h) {
  cache->Release(handle, true /* force_erase */);
 }

-Status VerifyBlockChecksum(ChecksumType type, const char* data,
-                           size_t block_size, const std::string& file_name,
-                           uint64_t offset) {
-  PERF_TIMER_GUARD(block_checksum_time);
-  // After block_size bytes is compression type (1 byte), which is part of
-  // the checksummed section.
-  size_t len = block_size + 1;
-  // And then the stored checksum value (4 bytes).
-  uint32_t stored = DecodeFixed32(data + len);
-
+Status VerifyChecksum(const ChecksumType type, const char* buf, size_t len,
+                      uint32_t expected) {
  Status s;
-  uint32_t computed = 0;
+  uint32_t actual = 0;
  switch (type) {
    case kNoChecksum:
      break;
    case kCRC32c:
-      stored = crc32c::Unmask(stored);
-      computed = crc32c::Value(data, len);
+      expected = crc32c::Unmask(expected);
+      actual = crc32c::Value(buf, len);
      break;
    case kxxHash:
-      computed = XXH32(data, len, 0);
+      actual = XXH32(buf, static_cast<int>(len), 0);
      break;
    case kxxHash64:
-      computed = Lower32of64(XXH64(data, len, 0));
+      actual = static_cast<uint32_t>(XXH64(buf, static_cast<int>(len), 0) &
+                                     uint64_t{0xffffffff});
      break;
    default:
-      s = Status::Corruption(
-          "unknown checksum type " + ToString(type) + " from footer of " +
-          file_name + ", while checking block at offset " + ToString(offset) +
-          " size " + ToString(block_size));
+      s = Status::Corruption("unknown checksum type");
  }
-  if (s.ok() && stored != computed) {
-    s = Status::Corruption(
-        "block checksum mismatch: stored = " + ToString(stored) +
-        ", computed = " + ToString(computed) + "  in " + file_name +
-        " offset " + ToString(offset) + " size " + ToString(block_size));
+  if (s.ok() && actual != expected) {
+    s = Status::Corruption("properties block checksum mismatched");
  }
  return s;
 }
@@ -28,10 +28,6 @@ inline MemoryAllocator* GetMemoryAllocatorForCompressedBlock(
             : nullptr;
 }

-// Assumes block has a trailer as in format.h. file_name and offset provided
-// for generating a diagnostic message in returned status.
-extern Status VerifyBlockChecksum(ChecksumType type, const char* data,
-                                  size_t block_size,
-                                  const std::string& file_name,
-                                  uint64_t offset);
+extern Status VerifyChecksum(const ChecksumType type, const char* buf,
+                             size_t len, uint32_t expected);
 }  // namespace ROCKSDB_NAMESPACE
@@ -19,20 +19,51 @@
 #include "rocksdb/env.h"
 #include "table/block_based/block.h"
 #include "table/block_based/block_based_table_reader.h"
-#include "table/block_based/reader_common.h"
 #include "table/format.h"
 #include "table/persistent_cache_helper.h"
+#include "util/coding.h"
 #include "util/compression.h"
+#include "util/crc32c.h"
 #include "util/stop_watch.h"
+#include "util/string_util.h"
+#include "util/xxhash.h"

 namespace ROCKSDB_NAMESPACE {

 inline void BlockFetcher::CheckBlockChecksum() {
  // Check the crc of the type and the block contents
  if (read_options_.verify_checksums) {
-    status_ = ROCKSDB_NAMESPACE::VerifyBlockChecksum(
-        footer_.checksum(), slice_.data(), block_size_, file_->file_name(),
-        handle_.offset());
+    const char* data = slice_.data();  // Pointer to where Read put the data
+    PERF_TIMER_GUARD(block_checksum_time);
+    uint32_t value = DecodeFixed32(data + block_size_ + 1);
+    uint32_t actual = 0;
+    switch (footer_.checksum()) {
+      case kNoChecksum:
+        break;
+      case kCRC32c:
+        value = crc32c::Unmask(value);
+        actual = crc32c::Value(data, block_size_ + 1);
+        break;
+      case kxxHash:
+        actual = XXH32(data, static_cast<int>(block_size_) + 1, 0);
+        break;
+      case kxxHash64:
+        actual = static_cast<uint32_t>(
+            XXH64(data, static_cast<int>(block_size_) + 1, 0) &
+            uint64_t{0xffffffff});
+        break;
+      default:
+        status_ = Status::Corruption(
+            "unknown checksum type " + ToString(footer_.checksum()) + " in " +
+            file_->file_name() + " offset " + ToString(handle_.offset()) +
+            " size " + ToString(block_size_));
+    }
+    if (status_.ok() && actual != value) {
+      status_ = Status::Corruption(
+          "block checksum mismatch: expected " + ToString(actual) + ", got " +
+          ToString(value) + "  in " + file_->file_name() + " offset " +
+          ToString(handle_.offset()) + " size " + ToString(block_size_));
+    }
  }
 }

@@ -53,8 +53,7 @@ CuckooTableBuilder::CuckooTableBuilder(
    const Comparator* user_comparator, uint32_t cuckoo_block_size,
    bool use_module_hash, bool identity_as_first_hash,
    uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint64_t),
-    uint32_t column_family_id, const std::string& column_family_name,
-    const std::string& db_id, const std::string& db_session_id)
+    uint32_t column_family_id, const std::string& column_family_name)
    : num_hash_func_(2),
      file_(file),
      max_hash_table_ratio_(max_hash_table_ratio),
@@ -80,8 +79,6 @@ CuckooTableBuilder::CuckooTableBuilder(
  properties_.filter_size = 0;
  properties_.column_family_id = column_family_id;
  properties_.column_family_name = column_family_name;
-  properties_.db_id = db_id;
-  properties_.db_session_id = db_session_id;
 }

 void CuckooTableBuilder::Add(const Slice& key, const Slice& value) {
@@ -22,14 +22,15 @@ namespace ROCKSDB_NAMESPACE {

 class CuckooTableBuilder: public TableBuilder {
 public:
-  CuckooTableBuilder(
-      WritableFileWriter* file, double max_hash_table_ratio,
-      uint32_t max_num_hash_func, uint32_t max_search_depth,
-      const Comparator* user_comparator, uint32_t cuckoo_block_size,
-      bool use_module_hash, bool identity_as_first_hash,
-      uint64_t (*get_slice_hash)(const Slice&, uint32_t, uint64_t),
-      uint32_t column_family_id, const std::string& column_family_name,
-      const std::string& db_id = "", const std::string& db_session_id = "");
+  CuckooTableBuilder(WritableFileWriter* file, double max_hash_table_ratio,
+                     uint32_t max_num_hash_func, uint32_t max_search_depth,
+                     const Comparator* user_comparator,
+                     uint32_t cuckoo_block_size, bool use_module_hash,
+                     bool identity_as_first_hash,
+                     uint64_t (*get_slice_hash)(const Slice&, uint32_t,
+                                                uint64_t),
+                     uint32_t column_family_id,
+                     const std::string& column_family_name);
  // No copying allowed
  CuckooTableBuilder(const CuckooTableBuilder&) = delete;
  void operator=(const CuckooTableBuilder&) = delete;
@@ -40,8 +40,7 @@ TableBuilder* CuckooTableFactory::NewTableBuilder(
      table_builder_options.internal_comparator.user_comparator(),
      table_options_.cuckoo_block_size, table_options_.use_module_hash,
      table_options_.identity_as_first_hash, nullptr /* get_slice_hash */,
-      column_family_id, table_builder_options.column_family_name,
-      table_builder_options.db_id, table_builder_options.db_session_id);
+      column_family_id, table_builder_options.column_family_name);
 }

 std::string CuckooTableFactory::GetPrintableTableOptions() const {
@@ -220,7 +220,7 @@ Status ReadFooterFromFile(RandomAccessFileReader* file,
                          uint64_t file_size, Footer* footer,
                          uint64_t enforce_table_magic_number = 0);

-// 1-byte compression type + 32-bit checksum
+// 1-byte type + 32-bit crc
 static const size_t kBlockTrailerSize = 5;

 // Make block size calculation for IO less error prone
@@ -142,6 +142,11 @@ class IteratorWrapperBase {
    return result_.value_prepared;
  }

+  Slice user_key() const {
+    assert(Valid());
+    return iter_->user_key();
+  }
+
 private:
  void Update() {
    valid_ = iter_->Valid();
@@ -96,8 +96,6 @@ void PropertyBlockBuilder::AddTableProperty(const TableProperties& props) {
  if (props.file_creation_time > 0) {
    Add(TablePropertiesNames::kFileCreationTime, props.file_creation_time);
  }
-  Add(TablePropertiesNames::kDbId, props.db_id);
-  Add(TablePropertiesNames::kDbSessionId, props.db_session_id);

  if (!props.filter_policy_name.empty()) {
    Add(TablePropertiesNames::kFilterPolicy, props.filter_policy_name);
@@ -313,10 +311,6 @@ Status ReadProperties(const Slice& handle_value, RandomAccessFileReader* file,
        continue;
      }
      *(pos->second) = val;
-    } else if (key == TablePropertiesNames::kDbId) {
-      new_table_properties->db_id = raw_val.ToString();
-    } else if (key == TablePropertiesNames::kDbSessionId) {
-      new_table_properties->db_session_id = raw_val.ToString();
    } else if (key == TablePropertiesNames::kFilterPolicy) {
      new_table_properties->filter_policy_name = raw_val.ToString();
    } else if (key == TablePropertiesNames::kColumnFamilyName) {
@@ -64,8 +64,7 @@ PlainTableBuilder::PlainTableBuilder(
    EncodingType encoding_type, size_t index_sparseness,
    uint32_t bloom_bits_per_key, const std::string& column_family_name,
    uint32_t num_probes, size_t huge_page_tlb_size, double hash_table_ratio,
-    bool store_index_in_file, const std::string& db_id,
-    const std::string& db_session_id)
+    bool store_index_in_file)
    : ioptions_(ioptions),
      moptions_(moptions),
      bloom_block_(num_probes),
@@ -98,8 +97,6 @@ PlainTableBuilder::PlainTableBuilder(
  properties_.format_version = (encoding_type == kPlain) ? 0 : 1;
  properties_.column_family_id = column_family_id;
  properties_.column_family_name = column_family_name;
-  properties_.db_id = db_id;
-  properties_.db_session_id = db_session_id;
  properties_.prefix_extractor_name = moptions_.prefix_extractor != nullptr
                                          ? moptions_.prefix_extractor->Name()
                                          : "nullptr";
@@ -45,8 +45,7 @@ class PlainTableBuilder: public TableBuilder {
      size_t index_sparseness, uint32_t bloom_bits_per_key,
      const std::string& column_family_name, uint32_t num_probes = 6,
      size_t huge_page_tlb_size = 0, double hash_table_ratio = 0,
-      bool store_index_in_file = false, const std::string& db_id = "",
-      const std::string& db_session_id = "");
+      bool store_index_in_file = false);
  // No copying allowed
  PlainTableBuilder(const PlainTableBuilder&) = delete;
  void operator=(const PlainTableBuilder&) = delete;
@@ -76,8 +76,7 @@ TableBuilder* PlainTableFactory::NewTableBuilder(
      table_options_.index_sparseness, table_options_.bloom_bits_per_key,
      table_builder_options.column_family_name, 6,
      table_options_.huge_page_tlb_size, table_options_.hash_table_ratio,
-      table_options_.store_index_in_file, table_builder_options.db_id,
-      table_builder_options.db_session_id);
+      table_options_.store_index_in_file);
 }

 std::string PlainTableFactory::GetPrintableTableOptions() const {
@@ -237,22 +237,12 @@ Status SstFileWriter::Open(const std::string& file_path) {
    r->column_family_name = "";
    cf_id = TablePropertiesCollectorFactory::Context::kUnknownColumnFamily;
  }
-  // SstFileWriter is used to create sst files that can be added to database
-  // later. Therefore, no real db_id and db_session_id are associated with it.
-  // Here we mimic the way db_session_id behaves by resetting the db_session_id
-  // every time SstFileWriter is used, and in this case db_id is set to be "SST
-  // Writer".
-  std::string db_session_id = r->ioptions.env->GenerateUniqueId();
-  if (!db_session_id.empty() && db_session_id.back() == '\n') {
-    db_session_id.pop_back();
-  }
+
  TableBuilderOptions table_builder_options(
      r->ioptions, r->mutable_cf_options, r->internal_comparator,
      &int_tbl_prop_collector_factories, compression_type,
      sample_for_compression, compression_opts, r->skip_filters,
-      r->column_family_name, unknown_level, 0 /* creation_time */,
-      0 /* oldest_key_time */, 0 /* target_file_size */,
-      0 /* file_creation_time */, "SST Writer" /* db_id */, db_session_id);
+      r->column_family_name, unknown_level);
  r->file_writer.reset(new WritableFileWriter(
      NewLegacyWritableFileWrapper(std::move(sst_file)), file_path,
      r->env_options, r->ioptions.env, nullptr /* stats */,
@@ -97,8 +97,7 @@ struct TableBuilderOptions {
      const std::string& _column_family_name, int _level,
      const uint64_t _creation_time = 0, const int64_t _oldest_key_time = 0,
      const uint64_t _target_file_size = 0,
-      const uint64_t _file_creation_time = 0, const std::string& _db_id = "",
-      const std::string& _db_session_id = "")
+      const uint64_t _file_creation_time = 0)
      : ioptions(_ioptions),
        moptions(_moptions),
        internal_comparator(_internal_comparator),
@@ -112,10 +111,7 @@ struct TableBuilderOptions {
        creation_time(_creation_time),
        oldest_key_time(_oldest_key_time),
        target_file_size(_target_file_size),
-        file_creation_time(_file_creation_time),
-        db_id(_db_id),
-        db_session_id(_db_session_id) {}
-
+        file_creation_time(_file_creation_time) {}
  const ImmutableCFOptions& ioptions;
  const MutableCFOptions& moptions;
  const InternalKeyComparator& internal_comparator;
@@ -131,8 +127,6 @@ struct TableBuilderOptions {
  const int64_t oldest_key_time;
  const uint64_t target_file_size;
  const uint64_t file_creation_time;
-  const std::string db_id;
-  const std::string db_session_id;
 };

 // TableBuilder provides the interface used to build a Table
Author	SHA1	Message	Date
Yanqin Jin	c509dd3249	Fix the logic of setting read_amp_bytes_per_bit from OPTIONS file (#7680 ) Summary: Instead of using `EncodeFixed32` which always serialize a integer to little endian, we should use the local machine's endianness when populating a native data structure during options parsing. Without this fix, `read_amp_bytes_per_bit` may be populated incorrectly on big-endian machines. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7680 Test Plan: make check Reviewed By: pdillinger Differential Revision: D24999166 Pulled By: riversand963 fbshipit-source-id: dc603cff6e17f8fa32479ce6df93b93082e6b0c4	2020-11-17 10:18:19 -08:00
Yanqin Jin	19c8025386	Bump version and update HISTORY	2020-11-15 15:04:28 -08:00
Yanqin Jin	63462c3a94	Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659 ) Summary: A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and 6.14. The bug will write out 8 bytes to OPTIONS file from the starting address of BlockBasedTableOptions.read_amp_bytes_per_bit which is actually a uint32. Consequently, the value of read_amp_bytes_per_bit written in the OPTIONS file is wrong. From 6.15, RocksDB will try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32. To be able to load OPTIONS file generated by affected releases before the fix, we need to manually parse read_amp_bytes_per_bit with this hack. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7659 Test Plan: Generate a db with current 6.14.fb (head at https://github.com/facebook/rocksdb/commit/b6db05dbb5364c658c5401a8078d73697bb5f31d). Maybe use db_stress. Checkout this PR, run ``` ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only ``` Expect success, and should not see ``` Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184 ``` Also make check Reviewed By: anand1976 Differential Revision: D24954752 Pulled By: riversand963 fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394	2020-11-15 14:52:54 -08:00
Huisheng Liu	62f110f527	fix read_amp_bytes_per_bit field size (#7651 ) Summary: The field in BlockBasedTableOptions is 4 bytes: // Default: 0 (disabled) uint32_t read_amp_bytes_per_bit = 0; Pull Request resolved: https://github.com/facebook/rocksdb/pull/7651 Reviewed By: ltamasi Differential Revision: D24844994 Pulled By: riversand963 fbshipit-source-id: e2695e55532256ef8996dd6939cad06987a80293	2020-11-11 12:21:58 -08:00
Andrew Kryczka	878b3a41de	update HISTORY.md and version.h for 6.11.6	2020-10-12 13:15:15 -07:00
Andrew Kryczka	f36f44c5e6	Fix bug in pinned partitioned indexes with some reads bypassing block cache Backports part of `75d3b6fdf0`.	2020-10-12 13:14:45 -07:00
Andrew Kryczka	8d70e53114	Fix bug in pinned partitioned user key indexes Backports part of `75d3b6fdf0`.	2020-10-12 13:14:45 -07:00
Yanqin Jin	5a4b8005a7	Update HISTORY and version	2020-07-23 14:42:25 -07:00
Yanqin Jin	4cd964946a	Report corruption on unrecognized value type (#7121 ) Summary: During memtable lookup, an unrecognized value type should be reported as Status::Corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7121 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D22512124 Pulled By: riversand963 fbshipit-source-id: 9b97be7d9b230c5aae9205f96054420e5ea09066	2020-07-23 14:37:30 -07:00
Andrew Kryczka	48bfca38f6	update HISTORY.md and version.h for 6.11.4	2020-07-15 17:54:15 -07:00
Adam Retter	c8b9556cb6	Only check for python location once (#7123 ) Summary: This fixes an issue introduced in `0c56fc4` whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6. The location is now only checked once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123 Reviewed By: zhichao-cao Differential Revision: D22532274 Pulled By: ajkr fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970	2020-07-15 17:51:52 -07:00
Yanqin Jin	30bfa2a44e	Report corrupted keys during compaction (#7124 ) Summary: Currently, RocksDB lets compaction to go through even in case of corrupted keys, the number of which is reported in CompactionJobStats. However, RocksDB does not check this value. We should let compaction run in a stricter mode. Temporarily disable two tests that allow corrupted keys in compaction. With this PR, the two tests will assert(false) and terminate. Still need to investigate what is the recommended google-test way of doing it. Death test (EXPECT_DEATH) in gtest has warnings now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7124 Test Plan: make check Reviewed By: ajkr Differential Revision: D22530722 Pulled By: riversand963 fbshipit-source-id: 6a5a6a992028c6d4f92cb74693c92db462ae4ad6	2020-07-15 17:51:08 -07:00
Andrew Kryczka	3e1eb99ab7	bump version to 6.11.3	2020-07-09 15:51:37 -07:00
Yanqin Jin	3514cf51fa	Fix data race to VersionSet::io_status_ (#7034 ) Summary: After https://github.com/facebook/rocksdb/issues/6949 , VersionSet::io_status_ can be concurrently accessed by multiple threads without lock, causing tsan test to fail. For example, a bg flush thread resets io_status_ before calling LogAndApply(), while another thread already in the process of LogAndApply() reads io_status_. This is a bug. We do not have to reset io_status_ each time we call LogAndApply(). io_status_ is part of the state of VersionSet, and it indicates the outcome of preceding MANIFEST/CURRENT files IO operations. Its value should be updated only when: 1. MANIFEST/CURRENT files IO fail for the first time. 2. MANIFEST/CURRENT files IO succeed as part of recovering from a prior failure without process restart, e.g. calling Resume(). Test Plan (devserver): COMPILE_WITH_TSAN=1 make check COMPILE_WITH_TSAN=1 make db_test2 ./db_test2 --gtest_filter=DBTest2.CompactionStall Pull Request resolved: https://github.com/facebook/rocksdb/pull/7034 Reviewed By: zhichao-cao Differential Revision: D22247137 Pulled By: riversand963 fbshipit-source-id: 77b83e05390f3ee3cd2d96d3fdd6fe4f225e3216	2020-07-09 15:51:01 -07:00
Yanqin Jin	dd63f04c83	First step towards handling MANIFEST write error (#6949 ) Summary: This PR provides preliminary support for handling IO error during MANIFEST write. File write/sync is not guaranteed to be atomic. If we encounter an IOError while writing/syncing to the MANIFEST file, we cannot be sure about the state of the MANIFEST file. The version edits may or may not have reached the file. During cleanup, if we delete the newly-generated SST files referenced by the pending version edit(s), but the version edit(s) actually are persistent in the MANIFEST, then next recovery attempt will process the version edits(s) and then fail since the SST files have already been deleted. One approach is to truncate the MANIFEST after write/sync error, so that it is safe to delete the SST files. However, file truncation may not be supported on certain file systems. Therefore, we take the following approach. If an IOError is detected during MANIFEST write/sync, we disable file deletions for the faulty database. Depending on whether the IOError is retryable (set by underlying file system), either RocksDB or application can call `DB::Resume()`, or simply shutdown and restart. During `Resume()`, RocksDB will try to switch to a new MANIFEST and write all existing in-memory version storage in the new file. If this succeeds, then RocksDB may proceed. If all recovery is completed, then file deletions will be re-enabled. Note that multiple threads can call `LogAndApply()` at the same time, though only one of them will be going through the process MANIFEST write, possibly batching the version edits of other threads. When the leading MANIFEST writer finishes, all of the MANIFEST writing threads in this batch will have the same IOError. They will all call `ErrorHandler::SetBGError()` in which file deletion will be disabled. Possible future directions: - Add an `ErrorContext` structure so that it is easier to pass more info to `ErrorHandler`. Currently, as in this example, a new `BackgroundErrorReason` has to be added. Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6949 Reviewed By: anand1976 Differential Revision: D22026020 Pulled By: riversand963 fbshipit-source-id: f3c68a2ef45d9b505d0d625c7c5e0c88495b91c8	2020-07-09 15:50:33 -07:00
Akanksha Mahajan	115c9113ca	Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7096 ) Summary: When format_version is high enough to support user-key and there are index entries for same user key that spans multiple data blocks then it changes from user-key mode to internal-key mode. But the flush policy is not reset to point to Block Builder of internal-keys. After this switch, no entries are added to user key index partition result, thus it never triggers flushing the block. Fix: 1. After adding the entry in sub_builder_index_, if there is a switch from user-key to internal-key, then flush policy is updated to point to Block Builder of internal-keys index partition. 2. Set sub_builder_index_->seperator_is_key_plus_seq_ = true if seperator_is_key_plus_seq_ is set to true so that subsequent partitions can also use internal key mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7096 Test Plan: make check -j64 Reviewed By: ajkr Differential Revision: D22416598 Pulled By: akankshamahajan15 fbshipit-source-id: 01fc2dc07ea1b32f8fb803995ebe6e9a3fbe67ac	2020-07-09 15:48:54 -07:00
Andrew Kryczka	07e2ca100a	bump patch version to 6.11.2	2020-07-07 09:34:16 -07:00
Peter Dillinger	9912ebe1b5	Exclude c_test from buck build opt mode Summary: Fix a Facebook internal build Test Plan: buck build @mode/opt :c_test :c_test_bin (was compilation failure, now "not found") buck build @mode/dev :c_test :c_test_bin (still passes)	2020-07-07 09:31:44 -07:00
sdong	0fdd019f62	Fix test in buck test (#7076 ) Summary: This is to fix special logic to run tests inside FB. Buck test is broken after moving to cpp_unittest(). Move c_test back to the previous approach. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7076 Test Plan: Watch the Sandcastle run Reviewed By: ajkr Differential Revision: D22370096 fbshipit-source-id: 4a464d0903f2c76ae2de3a8ad373ffc9bedec64c	2020-07-07 09:12:18 -07:00
Levi Tamasi	43107e0a9a	Move kNoExpiration to blob_db.h (#7018 ) Summary: The constant `kNoExpiration` is currently defined in an internal/implementation header (`blob_log_format.h`); the patch moves it to the public header `blob_db.h` so it is accessible to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7018 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D22191354 Pulled By: ltamasi fbshipit-source-id: 98c8012a83b999a3f1a30e955ce6bb71ba29dc5c	2020-06-23 14:13:48 -07:00
Andrew Kryczka	c0a8d89c27	add missing release note	2020-06-23 13:45:02 -07:00
Andrew Kryczka	85189fd64f	skip 6.11.0 and go straight to 6.11.1 for clarity of HISTORY.md	2020-06-23 13:04:41 -07:00
sdong	1910560c2c	Fix a bug that causes iterator to return wrong result in a rare data race (#6973 ) Summary: The bug fixed in https://github.com/facebook/rocksdb/pull/1816/ is now applicable to iterator too. This was not an issue but https://github.com/facebook/rocksdb/pull/2886 caused the regression. If a put and DB flush happens just between iterator to get latest sequence number and getting super version, empty result for the key or an older value can be returned, which is wrong. Fix it in the same way as the fix in https://github.com/facebook/rocksdb/issues/1816, that is to get the sequence number after referencing the super version. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6973 Test Plan: Will run stress tests for a while to make sure there is no general regression. Reviewed By: ajkr Differential Revision: D22029348 fbshipit-source-id: 94390f93630906796d6e2fec321f44a920953fd1	2020-06-18 10:43:00 -07:00
Yanqin Jin	61cc9ef76f	Fail recovery when MANIFEST record checksum mismatch (#6996 ) Summary: https://github.com/facebook/rocksdb/issues/5411 refactored `VersionSet::Recover` but introduced a bug, explained as follows. Before, once a checksum mismatch happens, `reporter` will set `s` to be non-ok. Therefore, Recover will stop processing the MANIFEST any further. ``` // Correct // Inside Recover LogReporter reporter; reporter.status = &s; log::Reader reader(..., reporter); while (reader.ReadRecord() && s.ok()) { ... } ``` The bug is that, the local variable `s` in `ReadAndRecover` won't be updated by `reporter` while reading the MANIFEST. It is possible that the reader sees a checksum mismatch in a record, but `ReadRecord` retries internally read and finds the next valid record. The mismatched record will be ignored and no error is reported. ``` // Incorrect // Inside Recover LogReporter reporter; reporter.status = &s; log::Reader reader(..., reporter); s = ReadAndRecover(reader, ...); // Inside ReadAndRecover Status s; // Shadows the s in Recover. while (reader.ReadRecord() && s.ok()) { ... } ``` `LogReporter` can use a separate `log_read_status` to track the errors while reading the MANIFEST. RocksDB can process more MANIFEST entries only if `log_read_status.ok()`. Test plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6996 Reviewed By: ajkr Differential Revision: D22105746 Pulled By: riversand963 fbshipit-source-id: b22f717a423457a41ca152a242abbb64cf91fc38	2020-06-18 10:38:16 -07:00
sdong	f5d4dbbeef	Fix the bug that compressed cache is disabled in read-only DBs (#6990 ) Summary: Compressed block cache is disabled in https://github.com/facebook/rocksdb/pull/4650 for no good reason. Re-enable it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6990 Test Plan: Add a unit test to make sure a general function works with read-only DB + compressed block cache. Reviewed By: ltamasi Differential Revision: D22072755 fbshipit-source-id: 2a55df6363de23a78979cf6c747526359e5dc7a1	2020-06-18 10:30:53 -07:00
Andrew Kryczka	84dcfe1a5f	update minor version for 6.11 release (#6994 ) Summary: The 6.11.fb branch is already cut so I will also backport this PR to that branch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6994 Reviewed By: riversand963 Differential Revision: D22084532 Pulled By: ajkr fbshipit-source-id: 0b025f738cc31c65c673cbf89302359e88a34d19	2020-06-17 09:41:11 -07:00
Yanqin Jin	a818699f2f	Fix a bug of overwriting return code (#6989 ) Summary: In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6989 Test Plan: make check Reviewed By: ajkr Differential Revision: D22071418 Pulled By: riversand963 fbshipit-source-id: 5a4ea5dfb1a41f41c7a3fdaf62b163007b42f04b	2020-06-16 14:00:15 -07:00
Yanqin Jin	97a69f4372	Let best-efforts recovery ignore CURRENT file (#6970 ) Summary: Best-efforts recovery does not check the content of CURRENT file to determine which MANIFEST to recover from. However, it still checks the presence of CURRENT file to determine whether to create a new DB during `open()`. Therefore, we can tweak the logic in `open()` a little bit so that best-efforts recovery does not rely on CURRENT file at all. Test plan (dev server): make check ./db_basic_test --gtest_filter=DBBasicTest.RecoverWithNoCurrentFile Pull Request resolved: https://github.com/facebook/rocksdb/pull/6970 Reviewed By: anand1976 Differential Revision: D22013990 Pulled By: riversand963 fbshipit-source-id: db552a1868c60ed70e1f7cd252a3a076eb8ea58f	2020-06-16 09:35:52 -07:00