Making platform 007 (gcc 7) default in build_detect_platform.sh (#5947 )

Summary: Making platform 007 (gcc 7) default in build_detect_platform.sh. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5947 Differential Revision: D18038837 Pulled By: vjnadimpalli fbshipit-source-id: 9ac2ddaa93bf328a416faec028970e039886378e
Fixes for building RocksJava releases on arm64v8
2026-11-06 19:12:36 -05:00 · 2019-10-30 10:30:21 -07:00 · 2019-10-23 15:19:39 -07:00 · 2019-10-01 17:04:43 -07:00 · 2019-10-01 16:48:35 -07:00 · 2019-09-17 17:32:10 -07:00
228 changed files with 4822 additions and 16538 deletions
@@ -24,7 +24,6 @@ make_config.mk
 *.vcxproj.filters
 *.sln
 *.cmake
-.watchmanconfig
 CMakeCache.txt
 CMakeFiles/
 build/
@@ -33,8 +32,6 @@ ldb
 manifest_dump
 sst_dump
 blob_dump
-block_cache_trace_analyzer
-tools/block_cache_analyzer/*.pyc
 column_aware_encoding_exp
 util/build_version.cc
 build_tools/VALGRIND_LOGS/
@@ -50,7 +47,6 @@ rocksdb_undump
 db_test2
 trace_analyzer
 trace_analyzer_test
-block_cache_trace_analyzer
 .DS_Store

 java/out
@@ -1,6 +0,0 @@
-{
-  "content_hash_warming": true,
-  "content_hash_max_items": 333333,
-  "hint_num_files_per_dir": 8,
-  "fsevents_latency": 0.05
-}
@@ -520,7 +520,6 @@ set(SOURCES
        db/flush_job.cc
        db/flush_scheduler.cc
        db/forward_iterator.cc
-        db/import_column_family_job.cc
        db/internal_stats.cc
        db/logs_with_prep_tracker.cc
        db/log_reader.cc
@@ -596,12 +595,10 @@ set(SOURCES
        table/block_based/block_prefix_index.cc
        table/block_based/data_block_hash_index.cc
        table/block_based/data_block_footer.cc
-        table/block_based/filter_block_reader_common.cc
        table/block_based/flush_block_policy.cc
        table/block_based/full_filter_block.cc
        table/block_based/index_builder.cc
        table/block_based/partitioned_filter_block.cc
-        table/block_based/uncompression_dict_reader.cc
        table/block_fetcher.cc
        table/bloom_block.cc
        table/cuckoo/cuckoo_table_builder.cc
@@ -626,7 +623,7 @@ set(SOURCES
        test_util/sync_point_impl.cc
        test_util/testutil.cc
        test_util/transaction_test_util.cc
-        tools/block_cache_analyzer/block_cache_trace_analyzer.cc
+        tools/block_cache_trace_analyzer.cc
        tools/db_bench_tool.cc
        tools/dump/db_dump_tool.cc
        tools/ldb_cmd.cc
@@ -681,7 +678,6 @@ set(SOURCES
        utilities/merge_operators/string_append/stringappend.cc
        utilities/merge_operators/string_append/stringappend2.cc
        utilities/merge_operators/uint64add.cc
-        utilities/object_registry.cc
        utilities/option_change_migration/option_change_migration.cc
        utilities/options/options_util.cc
        utilities/persistent_cache/block_cache_tier.cc
@@ -689,7 +685,6 @@ set(SOURCES
        utilities/persistent_cache/block_cache_tier_metadata.cc
        utilities/persistent_cache/persistent_cache_tier.cc
        utilities/persistent_cache/volatile_tier_impl.cc
-        utilities/simulator_cache/cache_simulator.cc
        utilities/simulator_cache/sim_cache.cc
        utilities/table_properties_collectors/compact_on_deletion_collector.cc
        utilities/trace/file_trace_reader_writer.cc
@@ -953,7 +948,6 @@ if(WITH_TESTS)
        env/mock_env_test.cc
        file/delete_scheduler_test.cc
        logging/auto_roll_logger_test.cc
-        logging/env_logger_test.cc
        logging/event_logger_test.cc
        memory/arena_test.cc
        memtable/inlineskiplist_test.cc
@@ -976,7 +970,7 @@ if(WITH_TESTS)
        table/merger_test.cc
        table/sst_file_reader_test.cc
        table/table_test.cc
-        tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc
+        tools/block_cache_trace_analyzer_test.cc
        tools/ldb_cmd_test.cc
        tools/reduce_levels_test.cc
        tools/sst_dump_test.cc
@@ -1010,7 +1004,6 @@ if(WITH_TESTS)
        utilities/options/options_util_test.cc
        utilities/persistent_cache/hash_table_test.cc
        utilities/persistent_cache/persistent_cache_test.cc
-        utilities/simulator_cache/cache_simulator_test.cc
        utilities/simulator_cache/sim_cache_test.cc
        utilities/table_properties_collectors/compact_on_deletion_collector_test.cc
        utilities/transactions/optimistic_transaction_test.cc
@@ -1,57 +1,25 @@
 # Rocksdb Change Log
-## 6.4.6 (10/16/2019)
-### Bug Fixes
-* Fix a bug when partitioned filters and prefix search are used in conjunction, ::SeekForPrev could return invalid for an existing prefix. ::SeekForPrev might be called by the user, or internally on ::Prev, or within ::Seek if the return value involves Delete or a Merge operand.
+## 6.3.6 (10/1/2019)
+* Revert the feature "Merging iterator to avoid child iterator reseek for some cases (#5286)" since it might cause strong results when reseek happens with a different iterator upper bound.

-## 6.4.5 (10/1/2019)
-### Bug Fixes
-* Revert the feature "Merging iterator to avoid child iterator reseek for some cases (#5286)" since it might cause strange results when reseek happens with a different iterator upper bound.
-* Fix a bug in BlockBasedTableIterator that might return incorrect results when reseek happens with a different iterator upper bound.
-
-## 6.4.4 (9/17/2019)
+## 6.3.5 (9/17/2019)
 * Fix a bug introduced 6.3 which could cause wrong results in a corner case when prefix bloom filter is used and the iterator is reseeked.

-## 6.4.2 (9/3/2019)
+## 6.3.4 (9/3/2019)
 ### Bug Fixes
 * Fix a bug in file ingestion caused by incorrect file number allocation when the number of column families involved in the ingestion exceeds 2.

-## 6.4.1 (8/20/2019)
+## 6.3.3 (8/20/2019)
 ### Bug Fixes
 * Fix a bug where the compaction snapshot refresh feature is not disabled as advertised when `snap_refresh_nanos` is set to 0..

-## 6.4.0 (7/30/2019)
-### Default Option Change
-* LRUCacheOptions.high_pri_pool_ratio is set to 0.5 (previously 0.0) by default, which means that by default midpoint insertion is enabled. The same change is made for the default value of high_pri_pool_ratio argument in NewLRUCache(). When block cache is not explictly created, the small block cache created by BlockBasedTable will still has this option to be 0.0.
-* Change BlockBasedTableOptions.cache_index_and_filter_blocks_with_high_priority's default value from false to true.
-
+## 6.3.2 (8/15/2019)
 ### Public API Change
-* Filter and compression dictionary blocks are now handled similarly to data blocks with regards to the block cache: instead of storing objects in the cache, only the blocks themselves are cached. In addition, filter and compression dictionary blocks (as well as filter partitions) no longer get evicted from the cache when a table is closed.
-* Due to the above refactoring, block cache eviction statistics for filter and compression dictionary blocks are temporarily broken. We plan to reintroduce them in a later phase.
 * The semantics of the per-block-type block read counts in the performance context now match those of the generic block_read_count.
-* Errors related to the retrieval of the compression dictionary are now propagated to the user.
-* db_bench adds a "benchmark" stats_history, which prints out the whole stats history.
-* Overload GetAllKeyVersions() to support non-default column family.
-* Added new APIs ExportColumnFamily() and CreateColumnFamilyWithImport() to support export and import of a Column Family. https://github.com/facebook/rocksdb/issues/3469
-* ldb sometimes uses a string-append merge operator if no merge operator is passed in. This is to allow users to print keys from a DB with a merge operator.
-* Replaces old Registra with ObjectRegistry to allow user to create custom object from string, also add LoadEnv() to Env.
-* Added new overload of GetApproximateSizes which gets SizeApproximationOptions object and returns a Status. The older overloads are redirecting their calls to this new method and no longer assert if the include_flags doesn't have either of INCLUDE_MEMTABLES or INCLUDE_FILES bits set. It's recommended to use the new method only, as it is more type safe and returns a meaningful status in case of errors.
-
-### New Features
-* Add argument `--secondary_path` to ldb to open the database as the secondary instance. This would keep the original DB intact.
-* Compression dictionary blocks are now prefetched and pinned in the cache (based on the customer's settings) the same way as index and filter blocks.
-* Added DBOptions::log_readahead_size which specifies the number of bytes to prefetch when reading the log. This is mostly useful for reading a remotely located log, as it can save the number of round-trips. If 0 (default), then the prefetching is disabled.
-* Support loading custom objects in unit tests. In the affected unit tests, RocksDB will create custom Env objects based on environment variable TEST_ENV_URI. Users need to make sure custom object types are properly registered. For example, a static library should expose a `RegisterCustomObjects` function. By linking the unit test binary with the static library, the unit test can execute this function.
-
-### Performance Improvements
-* Reduce iterator key comparision for upper/lower bound check.
-* Improve performance of row_cache: make reads with newer snapshots than data in an SST file share the same cache key, except in some transaction cases.
-* The compression dictionary is no longer copied to a new object upon retrieval.

 ### Bug Fixes
-* Fix ingested file and directory not being fsync.
-* Return TryAgain status in place of Corruption when new tail is not visible to TransactionLogIterator.
 * Fixed a regression where the fill_cache read option also affected index blocks.
-* Fixed an issue where using cache_index_and_filter_blocks==false affected partitions of partitioned indexes/filters as well.
+* Fixed an issue where using cache_index_and_filter_blocks==false affected partitions of partitioned indexes as well.

 ## 6.3.1 (7/24/2019)
 ### Bug Fixes
@@ -101,7 +69,6 @@
 * Block-based table index now contains exact highest key in the file, rather than an upper bound. This may improve Get() and iterator Seek() performance in some situations, especially when direct IO is enabled and block cache is disabled. A setting BlockBasedTableOptions::index_shortening is introduced to control this behavior. Set it to kShortenSeparatorsAndSuccessor to get the old behavior.
 * When reading from option file/string/map, customized envs can be filled according to object registry.
 * Improve range scan performance when using explicit user readahead by not creating new table readers for every iterator.
-* Add index type BlockBasedTableOptions::IndexType::kBinarySearchWithFirstKey. It significantly reduces read amplification in some setups, especially for iterator seeks. It's not fully implemented yet: IO errors are not handled right.

 ### Public API Change
 * Change the behavior of OptimizeForPointLookup(): move away from hash-based block-based-table index, and use whole key memtable filtering.
@@ -144,8 +144,8 @@ HAVE_POWER8=1
 endif

 ifeq (,$(shell $(CXX) -fsyntax-only -march=armv8-a+crc -xc /dev/null 2>&1))
-CXXFLAGS += -march=armv8-a+crc+crypto
-CFLAGS += -march=armv8-a+crc+crypto
+CXXFLAGS += -march=armv8-a+crc
+CFLAGS += -march=armv8-a+crc
 ARMCRC_SOURCE=1
 endif

@@ -432,7 +432,6 @@ TESTS = \
 	inlineskiplist_test \
 	env_basic_test \
 	env_test \
-	env_logger_test \
 	hash_test \
 	thread_local_test \
 	rate_limiter_test \
@@ -442,6 +441,7 @@ TESTS = \
 	db_block_cache_test \
 	db_test \
 	db_blob_index_test \
+	db_bloom_filter_test \
 	db_iter_test \
 	db_iter_stress_test \
 	db_log_iter_test \
@@ -479,6 +479,7 @@ TESTS = \
 	fault_injection_test \
 	filelock_test \
 	filename_test \
+	file_reader_writer_test \
 	block_based_filter_block_test \
 	full_filter_block_test \
 	partitioned_filter_block_test \
@@ -498,7 +499,6 @@ TESTS = \
 	plain_table_db_test \
 	comparator_db_test \
 	external_sst_file_test \
-	import_column_family_test \
 	prefix_test \
 	skiplist_test \
 	write_buffer_manager_test \
@@ -509,7 +509,6 @@ TESTS = \
 	cassandra_serialize_test \
 	ttl_test \
 	backupable_db_test \
-	cache_simulator_test \
 	sim_cache_test \
 	version_edit_test \
 	version_set_test \
@@ -568,7 +567,6 @@ TESTS = \

 PARALLEL_TEST = \
 	backupable_db_test \
-	db_bloom_filter_test \
 	db_compaction_filter_test \
 	db_compaction_test \
 	db_merge_operator_test \
@@ -577,9 +575,7 @@ PARALLEL_TEST = \
 	db_universal_compaction_test \
 	db_wal_test \
 	external_sst_file_test \
-	import_column_family_test \
 	fault_injection_test \
-	file_reader_writer_test \
 	inlineskiplist_test \
 	manual_compaction_test \
 	persistent_cache_test \
@@ -1068,7 +1064,7 @@ rocksdb.h rocksdb.cc: build_tools/amalgamate.py Makefile $(LIB_SOURCES) unity.cc
 	build_tools/amalgamate.py -I. -i./include unity.cc -x include/rocksdb/c.h -H rocksdb.h -o rocksdb.cc

 clean:
-	rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(PARALLEL_TEST) $(LIBRARY) $(SHARED)
+	rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(LIBRARY) $(SHARED)
 	rm -rf $(CLEAN_FILES) ios-x86 ios-arm scan_build_report
 	$(FIND) . -name "*.[oda]" -exec rm -f {} \;
 	$(FIND) . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
@@ -1114,7 +1110,7 @@ db_bench: tools/db_bench.o $(BENCHTOOLOBJECTS)
 trace_analyzer: tools/trace_analyzer.o $(ANALYZETOOLOBJECTS) $(LIBOBJECTS)
 	$(AM_LINK)

-block_cache_trace_analyzer: tools/block_cache_analyzer/block_cache_trace_analyzer_tool.o $(ANALYZETOOLOBJECTS) $(LIBOBJECTS)
+block_cache_trace_analyzer: tools/block_cache_trace_analyzer_tool.o $(ANALYZETOOLOBJECTS) $(LIBOBJECTS)
 	$(AM_LINK)

 cache_bench: cache/cache_bench.o $(LIBOBJECTS) $(TESTUTIL)
@@ -1276,9 +1272,6 @@ external_sst_file_basic_test: db/external_sst_file_basic_test.o db/db_test_util.
 external_sst_file_test: db/external_sst_file_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

-import_column_family_test: db/import_column_family_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
-	$(AM_LINK)
-
 db_tailing_iter_test: db/db_tailing_iter_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

@@ -1327,9 +1320,6 @@ backupable_db_test: utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TE
 checkpoint_test: utilities/checkpoint/checkpoint_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

-cache_simulator_test: utilities/simulator_cache/cache_simulator_test.o $(LIBOBJECTS) $(TESTHARNESS)
-	$(AM_LINK)
-
 sim_cache_test: utilities/simulator_cache/sim_cache_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

@@ -1539,9 +1529,6 @@ filelock_test: util/filelock_test.o $(LIBOBJECTS) $(TESTHARNESS)
 auto_roll_logger_test: logging/auto_roll_logger_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

-env_logger_test: logging/env_logger_test.o $(LIBOBJECTS) $(TESTHARNESS)
-	$(AM_LINK)
-
 memtable_list_test: db/memtable_list_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

@@ -1614,7 +1601,7 @@ db_secondary_test: db/db_impl/db_secondary_test.o db/db_test_util.o $(LIBOBJECTS
 block_cache_tracer_test: trace_replay/block_cache_tracer_test.o trace_replay/block_cache_tracer.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

-block_cache_trace_analyzer_test: tools/block_cache_analyzer/block_cache_trace_analyzer_test.o tools/block_cache_analyzer/block_cache_trace_analyzer.o $(LIBOBJECTS) $(TESTHARNESS)
+block_cache_trace_analyzer_test: tools/block_cache_trace_analyzer_test.o tools/block_cache_trace_analyzer.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(AM_LINK)

 #-------------------------------------------------
@@ -1,67 +1,15 @@
 version: 1.0.{build}
-
 image: Visual Studio 2017
-
-environment:
-  JAVA_HOME: C:\Program Files\Java\jdk1.8.0
-  THIRDPARTY_HOME: $(APPVEYOR_BUILD_FOLDER)\thirdparty
-  SNAPPY_HOME: $(THIRDPARTY_HOME)\snappy-1.1.7
-  SNAPPY_INCLUDE: $(SNAPPY_HOME);$(SNAPPY_HOME)\build
-  SNAPPY_LIB_DEBUG: $(SNAPPY_HOME)\build\Debug\snappy.lib
-  SNAPPY_LIB_RELEASE: $(SNAPPY_HOME)\build\Release\snappy.lib
-  LZ4_HOME: $(THIRDPARTY_HOME)\lz4-1.8.3
-  LZ4_INCLUDE: $(LZ4_HOME)\lib
-  LZ4_LIB_DEBUG: $(LZ4_HOME)\visual\VS2010\bin\x64_Debug\liblz4_static.lib
-  LZ4_LIB_RELEASE: $(LZ4_HOME)\visual\VS2010\bin\x64_Release\liblz4_static.lib
-  ZSTD_HOME: $(THIRDPARTY_HOME)\zstd-1.4.0
-  ZSTD_INCLUDE: $(ZSTD_HOME)\lib;$(ZSTD_HOME)\lib\dictBuilder
-  ZSTD_LIB_DEBUG: $(ZSTD_HOME)\build\VS2010\bin\x64_Debug\libzstd_static.lib
-  ZSTD_LIB_RELEASE: $(ZSTD_HOME)\build\VS2010\bin\x64_Release\libzstd_static.lib
-
-install:
-  - md %THIRDPARTY_HOME%
-  - echo "Building Snappy dependency..."
-  - cd %THIRDPARTY_HOME%
-  - curl -fsSL -o snappy-1.1.7.zip https://github.com/google/snappy/archive/1.1.7.zip
-  - unzip snappy-1.1.7.zip
-  - cd snappy-1.1.7
-  - mkdir build
-  - cd build
-  - cmake -DCMAKE_GENERATOR_PLATFORM=x64 ..
-  - msbuild Snappy.sln /p:Configuration=Debug /p:Platform=x64
-  - msbuild Snappy.sln /p:Configuration=Release /p:Platform=x64
-  - echo "Building LZ4 dependency..."
-  - cd %THIRDPARTY_HOME%
-  - curl -fsSL -o lz4-1.8.3.zip https://github.com/lz4/lz4/archive/v1.8.3.zip
-  - unzip lz4-1.8.3.zip
-  - cd lz4-1.8.3\visual\VS2010
-  - ps: $CMD="C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\devenv.com"; & $CMD lz4.sln /upgrade
-  - msbuild lz4.sln /p:Configuration=Debug /p:Platform=x64
-  - msbuild lz4.sln /p:Configuration=Release /p:Platform=x64
-  - echo "Building ZStd dependency..."
-  - cd %THIRDPARTY_HOME%
-  - curl -fsSL -o zstd-1.4.0.zip https://github.com/facebook/zstd/archive/v1.4.0.zip
-  - unzip zstd-1.4.0.zip
-  - cd zstd-1.4.0\build\VS2010
-  - ps: $CMD="C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\devenv.com"; & $CMD zstd.sln /upgrade
-  - msbuild zstd.sln /p:Configuration=Debug /p:Platform=x64
-  - msbuild zstd.sln /p:Configuration=Release /p:Platform=x64
-
 before_build:
-  - md %APPVEYOR_BUILD_FOLDER%\build
-  - cd %APPVEYOR_BUILD_FOLDER%\build
-  - cmake -G "Visual Studio 15 Win64" -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DLZ4=1 -DZSTD=1 -DXPRESS=1 -DJNI=1 ..
-  - cd ..
+- md %APPVEYOR_BUILD_FOLDER%\build
+- cd %APPVEYOR_BUILD_FOLDER%\build
+- cmake -G "Visual Studio 15 Win64" -DOPTDBG=1 -DWITH_XPRESS=1 -DPORTABLE=1 -DJNI=1 ..
+- cd ..
 build:
  project: build\rocksdb.sln
  parallel: true
  verbosity: normal
-
 test:
-
 test_script:
-  - ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test2,db_test,env_basic_test,env_test -Concurrency 8
-
-on_failure:
-  - cmd: 7z a build-failed.zip %APPVEYOR_BUILD_FOLDER%\build\ && appveyor PushArtifact build-failed.zip
+- ps: build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test2,db_test,env_basic_test,env_test -Concurrency 8

@@ -4,31 +4,12 @@ from __future__ import division
 from __future__ import print_function
 from __future__ import unicode_literals
 from targets_builder import TARGETSBuilder
-import json
 import os
 import fnmatch
 import sys

 from util import ColorString

-# This script generates TARGETS file for Buck.
-# Buck is a build tool specifying dependencies among different build targets.
-# User can pass extra dependencies as a JSON object via command line, and this
-# script can include these dependencies in the generate TARGETS file.
-# Usage:
-# $python buckifier/buckify_rocksdb.py
-# (This generates a TARGET file without user-specified dependency for unit
-# tests.)
-# $python buckifier/buckify_rocksdb.py \
-#        '{"fake": { \
-#                      "extra_deps": [":test_dep", "//fakes/module:mock1"],  \
-#                      "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"], \
-#                  } \
-#         }'
-# (Generated TARGETS file has test_dep and mock1 as dependencies for RocksDB
-# unit tests, and will use the extra_compiler_flags to compile the unit test
-# source.)
-
 # tests to export as libraries for inclusion in other projects
 _EXPORTED_TEST_LIBS = ["env_basic_test"]

@@ -105,38 +86,8 @@ def get_tests(repo_path):
    return tests


-# Parse extra dependencies passed by user from command line
-def get_dependencies():
-    deps_map = {
-        ''.encode('ascii'): {
-            'extra_deps'.encode('ascii'): [],
-            'extra_compiler_flags'.encode('ascii'): []
-        }
-    }
-    if len(sys.argv) < 2:
-        return deps_map
-
-    def encode_dict(data):
-        rv = {}
-        for k, v in data.items():
-            if isinstance(k, unicode):
-                k = k.encode('ascii')
-            if isinstance(v, unicode):
-                v = v.encode('ascii')
-            elif isinstance(v, list):
-                v = [x.encode('ascii') for x in v]
-            elif isinstance(v, dict):
-                v = encode_dict(v)
-            rv[k] = v
-        return rv
-    extra_deps = json.loads(sys.argv[1], object_hook=encode_dict)
-    for target_alias, deps in extra_deps.items():
-        deps_map[target_alias] = deps
-    return deps_map
-
-
 # Prepare TARGETS file for buck
-def generate_targets(repo_path, deps_map):
+def generate_targets(repo_path):
    print(ColorString.info("Generating TARGETS"))
    # parsed src.mk file
    src_mk = parse_src_mk(repo_path)
@@ -170,33 +121,24 @@ def generate_targets(repo_path, deps_map):
        ["test_util/testutil.cc"],
        [":rocksdb_lib"])

-    print("Extra dependencies:\n{0}".format(str(deps_map)))
    # test for every test we found in the Makefile
-    for target_alias, deps in deps_map.items():
-        for test in sorted(tests):
-            match_src = [src for src in cc_files if ("/%s.c" % test) in src]
-            if len(match_src) == 0:
-                print(ColorString.warning("Cannot find .cc file for %s" % test))
-                continue
-            elif len(match_src) > 1:
-                print(ColorString.warning("Found more than one .cc for %s" % test))
-                print(match_src)
-                continue
+    for test in sorted(tests):
+        match_src = [src for src in cc_files if ("/%s.c" % test) in src]
+        if len(match_src) == 0:
+            print(ColorString.warning("Cannot find .cc file for %s" % test))
+            continue
+        elif len(match_src) > 1:
+            print(ColorString.warning("Found more than one .cc for %s" % test))
+            print(match_src)
+            continue

-            assert(len(match_src) == 1)
-            is_parallel = tests[test]
-            test_target_name = \
-                test if not target_alias else test + "_" + target_alias
-            TARGETS.register_test(
-                test_target_name,
-                match_src[0],
-                is_parallel,
-                deps['extra_deps'],
-                deps['extra_compiler_flags'])
+        assert(len(match_src) == 1)
+        is_parallel = tests[test]
+        TARGETS.register_test(test, match_src[0], is_parallel)

-            if test in _EXPORTED_TEST_LIBS:
-                test_library = "%s_lib" % test_target_name
-                TARGETS.add_library(test_library, match_src, [":rocksdb_test_lib"])
+        if test in _EXPORTED_TEST_LIBS:
+            test_library = "%s_lib" % test
+            TARGETS.add_library(test_library, match_src, [":rocksdb_test_lib"])
    TARGETS.flush_tests()

    print(ColorString.info("Generated TARGETS Summary:"))
@@ -221,9 +163,8 @@ def exit_with_error(msg):


 def main():
-    deps_map = get_dependencies()
    # Generate TARGETS file for buck
-    ok = generate_targets(get_rocksdb_path(), deps_map)
+    ok = generate_targets(get_rocksdb_path())
    if not ok:
        exit_with_error("Failed to generate TARGETS files")

@@ -51,21 +51,14 @@ class TARGETSBuilder:
            pretty_list(deps)))
        self.total_bin = self.total_bin + 1

-    def register_test(self,
-                      test_name,
-                      src,
-                      is_parallel,
-                      extra_deps,
-                      extra_compiler_flags):
+    def register_test(self, test_name, src, is_parallel):
        exec_mode = "serial"
        if is_parallel:
            exec_mode = "parallel"
        self.tests_cfg += targets_cfg.test_cfg_template % (
            test_name,
            str(src),
-            str(exec_mode),
-            extra_deps,
-            extra_compiler_flags)
+            str(exec_mode))

        self.total_test = self.total_test + 1

@@ -11,9 +11,29 @@ REPO_PATH = package_name() + "/"

 ROCKSDB_COMPILER_FLAGS = [
    "-fno-builtin-memcmp",
+    "-DROCKSDB_PLATFORM_POSIX",
+    "-DROCKSDB_LIB_IO_POSIX",
+    "-DROCKSDB_FALLOCATE_PRESENT",
+    "-DROCKSDB_MALLOC_USABLE_SIZE",
+    "-DROCKSDB_RANGESYNC_PRESENT",
+    "-DROCKSDB_SCHED_GETCPU_PRESENT",
+    "-DROCKSDB_SUPPORT_THREAD_LOCAL",
+    "-DOS_LINUX",
+    # Flags to enable libs we include
+    "-DSNAPPY",
+    "-DZLIB",
+    "-DBZIP2",
+    "-DLZ4",
+    "-DZSTD",
+    "-DZSTD_STATIC_LINKING_ONLY",
+    "-DGFLAGS=gflags",
+    "-DNUMA",
+    "-DTBB",
    # Needed to compile in fbcode
    "-Wno-expansion-to-defined",
    # Added missing flags from output of build_detect_platform
+    "-DROCKSDB_PTHREAD_ADAPTIVE_MUTEX",
+    "-DROCKSDB_BACKTRACE",
    "-Wnarrowing",
    "-DROCKSDB_NO_DYNAMIC_EXTENSION",
 ]
@@ -26,54 +46,11 @@ ROCKSDB_EXTERNAL_DEPS = [
    ("lz4", None, "lz4"),
    ("zstd", None),
    ("tbb", None),
+    ("numa", None, "numa"),
    ("googletest", None, "gtest"),
 ]

-ROCKSDB_OS_DEPS = [
-    (
-        "linux",
-        ["third-party//numa:numa"],
-    ),
-]
-
-ROCKSDB_OS_PREPROCESSOR_FLAGS = [
-    (
-        "linux",
-        [
-            "-DOS_LINUX",
-            "-DROCKSDB_FALLOCATE_PRESENT",
-            "-DROCKSDB_MALLOC_USABLE_SIZE",
-            "-DROCKSDB_PTHREAD_ADAPTIVE_MUTEX",
-            "-DROCKSDB_RANGESYNC_PRESENT",
-            "-DROCKSDB_SCHED_GETCPU_PRESENT",
-            "-DHAVE_SSE42",
-            "-DNUMA",
-        ],
-    ),
-    (
-        "macos",
-        ["-DOS_MACOSX"],
-    ),
-]
-
 ROCKSDB_PREPROCESSOR_FLAGS = [
-    "-DROCKSDB_PLATFORM_POSIX",
-    "-DROCKSDB_LIB_IO_POSIX",
-    "-DROCKSDB_SUPPORT_THREAD_LOCAL",
-
-    # Flags to enable libs we include
-    "-DSNAPPY",
-    "-DZLIB",
-    "-DBZIP2",
-    "-DLZ4",
-    "-DZSTD",
-    "-DZSTD_STATIC_LINKING_ONLY",
-    "-DGFLAGS=gflags",
-    "-DTBB",
-
-    # Added missing flags from output of build_detect_platform
-    "-DROCKSDB_BACKTRACE",
-
    # Directories with files for #include
    "-I" + REPO_PATH + "include/",
    "-I" + REPO_PATH,
@@ -81,6 +58,7 @@ ROCKSDB_PREPROCESSOR_FLAGS = [

 ROCKSDB_ARCH_PREPROCESSOR_FLAGS = {
    "x86_64": [
+        "-DHAVE_SSE42",
        "-DHAVE_PCLMUL",
    ],
 }
@@ -97,15 +75,9 @@ sanitizer = read_config("fbcode", "sanitizer")

 # Do not enable jemalloc if sanitizer presents. RocksDB will further detect
 # whether the binary is linked with jemalloc at runtime.
-ROCKSDB_OS_PREPROCESSOR_FLAGS += ([(
-    "linux",
-    ["-DROCKSDB_JEMALLOC"],
-)] if sanitizer == "" else [])
+ROCKSDB_COMPILER_FLAGS += (["-DROCKSDB_JEMALLOC"] if sanitizer == "" else [])

-ROCKSDB_OS_DEPS += ([(
-    "linux",
-    ["third-party//jemalloc:headers"],
-)] if sanitizer == "" else [])
+ROCKSDB_EXTERNAL_DEPS += ([("jemalloc", None, "headers")] if sanitizer == "" else [])
 """


@@ -116,8 +88,6 @@ cpp_library(
    {headers_attr_prefix}headers = {headers},
    arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
    compiler_flags = ROCKSDB_COMPILER_FLAGS,
-    os_deps = ROCKSDB_OS_DEPS,
-    os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
    preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
    deps = [{deps}],
    external_deps = ROCKSDB_EXTERNAL_DEPS,
@@ -140,13 +110,11 @@ test_cfg_template = """    [
        "%s",
        "%s",
        "%s",
-        %s,
-        %s,
    ],
 """

 unittests_template = """
-# [test_name, test_src, test_type, extra_deps, extra_compiler_flags]
+# [test_name, test_src, test_type]
 ROCKS_TESTS = [
 %s]

@@ -155,19 +123,15 @@ ROCKS_TESTS = [
 # will not be included.
 [
    test_binary(
-        extra_compiler_flags = extra_compiler_flags,
-        extra_deps = extra_deps,
        parallelism = parallelism,
        rocksdb_arch_preprocessor_flags = ROCKSDB_ARCH_PREPROCESSOR_FLAGS,
        rocksdb_compiler_flags = ROCKSDB_COMPILER_FLAGS,
        rocksdb_external_deps = ROCKSDB_EXTERNAL_DEPS,
-        rocksdb_os_deps = ROCKSDB_OS_DEPS,
-        rocksdb_os_preprocessor_flags = ROCKSDB_OS_PREPROCESSOR_FLAGS,
        rocksdb_preprocessor_flags = ROCKSDB_PREPROCESSOR_FLAGS,
        test_cc = test_cc,
        test_name = test_name,
    )
-    for test_name, test_cc, parallelism, extra_deps, extra_compiler_flags in ROCKS_TESTS
+    for test_name, test_cc, parallelism in ROCKS_TESTS
    if not is_opt_mode
 ]
 """
@@ -63,21 +63,6 @@ CLEANUP_ENV="
    'user':'root'
 }"

-UPLOAD_DB_DIR="
-{
-    'name':'Upload database directory',
-    'shell':'tar -cvzf rocksdb_db.tar.gz /dev/shm/rocksdb/',
-    'user':'root',
-    'cleanup':true,
-    'provide_artifacts': [
-    {
-      'name':'rocksdb_db_dir',
-      'paths': ['rocksdb_db.tar.gz'],
-      'bundle': false,
-    },
-    ],
-}"
-
 # We will eventually set the RATIO to 1, but we want do this
 # in steps. RATIO=$(nproc) will make it work as J=1
 if [ -z $RATIO ]; then
@@ -443,8 +428,7 @@ STRESS_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS="[
                'shell':'$SHM $DEBUG $NON_TSAN_CRASH make J=1 crash_test_with_atomic_flush || $CONTRUN_NAME=crash_test_with_atomic_flush $TASK_CREATION_TOOL',
                'user':'root',
                $PARSER
-            },
-            $UPLOAD_DB_DIR,
+            }
        ],
        $REPORT
    }
@@ -535,7 +519,6 @@ ASAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS="[
                'user':'root',
                $PARSER
            },
-            $UPLOAD_DB_DIR,
        ],
        $REPORT
    }
@@ -603,7 +586,6 @@ UBSAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS="[
                'user':'root',
                $PARSER
            },
-            $UPLOAD_DB_DIR,
        ],
        $REPORT
    }
@@ -696,7 +678,6 @@ TSAN_CRASH_TEST_WITH_ATOMIC_FLUSH_COMMANDS="[
                'user':'root',
                $PARSER
            },
-            $UPLOAD_DB_DIR,
        ],
        $REPORT
    }
@@ -90,7 +90,7 @@ class CacheTest : public testing::TestWithParam<std::string> {
                                  bool strict_capacity_limit) {
    auto type = GetParam();
    if (type == kLRU) {
-      return NewLRUCache(capacity, num_shard_bits, strict_capacity_limit, 0.0);
+      return NewLRUCache(capacity, num_shard_bits, strict_capacity_limit);
    }
    if (type == kClock) {
      return NewClockCache(capacity, num_shard_bits, strict_capacity_limit);
@@ -562,7 +562,6 @@ TEST_P(CacheTest, SetStrictCapacityLimit) {
    ASSERT_OK(s);
    ASSERT_NE(nullptr, handles[i]);
  }
-  ASSERT_EQ(10, cache->GetUsage());

  // test2: set the flag to true. Insert and check if it fails.
  std::string extra_key = "extra";
@@ -572,7 +571,6 @@ TEST_P(CacheTest, SetStrictCapacityLimit) {
  s = cache->Insert(extra_key, extra_value, 1, &deleter, &handle);
  ASSERT_TRUE(s.IsIncomplete());
  ASSERT_EQ(nullptr, handle);
-  ASSERT_EQ(10, cache->GetUsage());

  for (size_t i = 0; i < 10; i++) {
    cache->Release(handles[i]);
@@ -593,7 +591,7 @@ TEST_P(CacheTest, SetStrictCapacityLimit) {
  s = cache2->Insert(extra_key, extra_value, 1, &deleter);
  // AS if the key have been inserted into cache but get evicted immediately.
  ASSERT_OK(s);
-  ASSERT_EQ(5, cache2->GetUsage());
+  ASSERT_EQ(5, cache->GetUsage());
  ASSERT_EQ(nullptr, cache2->Lookup(extra_key));

  for (size_t i = 0; i < 5; i++) {
@@ -24,7 +24,7 @@ LRUHandleTable::LRUHandleTable() : list_(nullptr), length_(0), elems_(0) {

 LRUHandleTable::~LRUHandleTable() {
  ApplyToAllCacheEntries([](LRUHandle* h) {
-    if (!h->HasRefs()) {
+    if (h->refs == 1) {
      h->Free();
    }
  });
@@ -113,17 +113,29 @@ LRUCacheShard::LRUCacheShard(size_t capacity, bool strict_capacity_limit,
  SetCapacity(capacity);
 }

+LRUCacheShard::~LRUCacheShard() {}
+
+bool LRUCacheShard::Unref(LRUHandle* e) {
+  assert(e->refs > 0);
+  e->refs--;
+  return e->refs == 0;
+}
+
+// Call deleter and free
+
 void LRUCacheShard::EraseUnRefEntries() {
  autovector<LRUHandle*> last_reference_list;
  {
    MutexLock l(&mutex_);
    while (lru_.next != &lru_) {
      LRUHandle* old = lru_.next;
-      // LRU list contains only elements which can be evicted
-      assert(old->InCache() && !old->HasRefs());
+      assert(old->InCache());
+      assert(old->refs ==
+             1);  // LRU list contains elements which may be evicted
      LRU_Remove(old);
      table_.Remove(old->key(), old->hash);
      old->SetInCache(false);
+      Unref(old);
      usage_ -= old->charge;
      last_reference_list.push_back(old);
    }
@@ -136,27 +148,22 @@ void LRUCacheShard::EraseUnRefEntries() {

 void LRUCacheShard::ApplyToAllCacheEntries(void (*callback)(void*, size_t),
                                           bool thread_safe) {
-  const auto applyCallback = [&]() {
-    table_.ApplyToAllCacheEntries(
-        [callback](LRUHandle* h) { callback(h->value, h->charge); });
-  };
-
  if (thread_safe) {
-    MutexLock l(&mutex_);
-    applyCallback();
-  } else {
-    applyCallback();
+    mutex_.Lock();
+  }
+  table_.ApplyToAllCacheEntries(
+      [callback](LRUHandle* h) { callback(h->value, h->charge); });
+  if (thread_safe) {
+    mutex_.Unlock();
  }
 }

 void LRUCacheShard::TEST_GetLRUList(LRUHandle** lru, LRUHandle** lru_low_pri) {
-  MutexLock l(&mutex_);
  *lru = &lru_;
  *lru_low_pri = lru_low_pri_;
 }

 size_t LRUCacheShard::TEST_GetLRUSize() {
-  MutexLock l(&mutex_);
  LRUHandle* lru_handle = lru_.next;
  size_t lru_size = 0;
  while (lru_handle != &lru_) {
@@ -224,13 +231,14 @@ void LRUCacheShard::MaintainPoolSize() {

 void LRUCacheShard::EvictFromLRU(size_t charge,
                                 autovector<LRUHandle*>* deleted) {
-  while ((usage_ + charge) > capacity_ && lru_.next != &lru_) {
+  while (usage_ + charge > capacity_ && lru_.next != &lru_) {
    LRUHandle* old = lru_.next;
-    // LRU list contains only elements which can be evicted
-    assert(old->InCache() && !old->HasRefs());
+    assert(old->InCache());
+    assert(old->refs == 1);  // LRU list contains elements which may be evicted
    LRU_Remove(old);
    table_.Remove(old->key(), old->hash);
    old->SetInCache(false);
+    Unref(old);
    usage_ -= old->charge;
    deleted->push_back(old);
  }
@@ -244,8 +252,8 @@ void LRUCacheShard::SetCapacity(size_t capacity) {
    high_pri_pool_capacity_ = capacity_ * high_pri_pool_ratio_;
    EvictFromLRU(0, &last_reference_list);
  }
-
-  // Free the entries outside of mutex for performance reasons
+  // we free the entries here outside of mutex for
+  // performance reasons
  for (auto entry : last_reference_list) {
    entry->Free();
  }
@@ -261,22 +269,22 @@ Cache::Handle* LRUCacheShard::Lookup(const Slice& key, uint32_t hash) {
  LRUHandle* e = table_.Lookup(key, hash);
  if (e != nullptr) {
    assert(e->InCache());
-    if (!e->HasRefs()) {
-      // The entry is in LRU since it's in hash and has no external references
+    if (e->refs == 1) {
      LRU_Remove(e);
    }
-    e->Ref();
+    e->refs++;
    e->SetHit();
  }
  return reinterpret_cast<Cache::Handle*>(e);
 }

 bool LRUCacheShard::Ref(Cache::Handle* h) {
-  LRUHandle* e = reinterpret_cast<LRUHandle*>(h);
+  LRUHandle* handle = reinterpret_cast<LRUHandle*>(h);
  MutexLock l(&mutex_);
-  // To create another reference - entry must be already externally referenced
-  assert(e->HasRefs());
-  e->Ref();
+  if (handle->InCache() && handle->refs == 1) {
+    LRU_Remove(handle);
+  }
+  handle->refs++;
  return true;
 }

@@ -295,27 +303,30 @@ bool LRUCacheShard::Release(Cache::Handle* handle, bool force_erase) {
  bool last_reference = false;
  {
    MutexLock l(&mutex_);
-    last_reference = e->Unref();
-    if (last_reference && e->InCache()) {
-      // The item is still in cache, and nobody else holds a reference to it
-      if (usage_ > capacity_ || force_erase) {
-        // The LRU list must be empty since the cache is full
-        assert(lru_.next == &lru_ || force_erase);
-        // Take this opportunity and remove the item
-        table_.Remove(e->key(), e->hash);
-        e->SetInCache(false);
-      } else {
-        // Put the item back on the LRU list, and don't free it
-        LRU_Insert(e);
-        last_reference = false;
-      }
-    }
+    last_reference = Unref(e);
    if (last_reference) {
      usage_ -= e->charge;
    }
+    if (e->refs == 1 && e->InCache()) {
+      // The item is still in cache, and nobody else holds a reference to it
+      if (usage_ > capacity_ || force_erase) {
+        // the cache is full
+        // The LRU list must be empty since the cache is full
+        assert(!(usage_ > capacity_) || lru_.next == &lru_);
+        // take this opportunity and remove the item
+        table_.Remove(e->key(), e->hash);
+        e->SetInCache(false);
+        Unref(e);
+        usage_ -= e->charge;
+        last_reference = true;
+      } else {
+        // put the item on the list to be potentially freed
+        LRU_Insert(e);
+      }
+    }
  }

-  // Free the entry here outside of mutex for performance reasons
+  // free outside of mutex
  if (last_reference) {
    e->Free();
  }
@@ -331,7 +342,7 @@ Status LRUCacheShard::Insert(const Slice& key, uint32_t hash, void* value,
  // It shouldn't happen very often though.
  LRUHandle* e = reinterpret_cast<LRUHandle*>(
      new char[sizeof(LRUHandle) - 1 + key.size()]);
-  Status s = Status::OK();
+  Status s;
  autovector<LRUHandle*> last_reference_list;

  e->value = value;
@@ -340,7 +351,9 @@ Status LRUCacheShard::Insert(const Slice& key, uint32_t hash, void* value,
  e->key_length = key.size();
  e->flags = 0;
  e->hash = hash;
-  e->refs = 0;
+  e->refs = (handle == nullptr
+                 ? 1
+                 : 2);  // One from LRUCache, one for the returned handle
  e->next = e->prev = nullptr;
  e->SetInCache(true);
  e->SetPriority(priority);
@@ -353,12 +366,11 @@ Status LRUCacheShard::Insert(const Slice& key, uint32_t hash, void* value,
    // is freed or the lru list is empty
    EvictFromLRU(charge, &last_reference_list);

-    if ((usage_ + charge) > capacity_ &&
+    if (usage_ - lru_usage_ + charge > capacity_ &&
        (strict_capacity_limit_ || handle == nullptr)) {
      if (handle == nullptr) {
        // Don't insert the entry but still return ok, as if the entry inserted
        // into cache and get evicted immediately.
-        e->SetInCache(false);
        last_reference_list.push_back(e);
      } else {
        delete[] reinterpret_cast<char*>(e);
@@ -366,30 +378,32 @@ Status LRUCacheShard::Insert(const Slice& key, uint32_t hash, void* value,
        s = Status::Incomplete("Insert failed due to LRU cache being full.");
      }
    } else {
-      // Insert into the cache. Note that the cache might get larger than its
-      // capacity if not enough space was freed up.
+      // insert into the cache
+      // note that the cache might get larger than its capacity if not enough
+      // space was freed
      LRUHandle* old = table_.Insert(e);
      usage_ += e->charge;
      if (old != nullptr) {
-        assert(old->InCache());
        old->SetInCache(false);
-        if (!old->HasRefs()) {
-          // old is on LRU because it's in cache and its reference count is 0
-          LRU_Remove(old);
+        if (Unref(old)) {
          usage_ -= old->charge;
+          // old is on LRU because it's in cache and its reference count
+          // was just 1 (Unref returned 0)
+          LRU_Remove(old);
          last_reference_list.push_back(old);
        }
      }
      if (handle == nullptr) {
        LRU_Insert(e);
      } else {
-        e->Ref();
        *handle = reinterpret_cast<Cache::Handle*>(e);
      }
+      s = Status::OK();
    }
  }

-  // Free the entries here outside of mutex for performance reasons
+  // we free the entries here outside of mutex for
+  // performance reasons
  for (auto entry : last_reference_list) {
    entry->Free();
  }
@@ -404,18 +418,18 @@ void LRUCacheShard::Erase(const Slice& key, uint32_t hash) {
    MutexLock l(&mutex_);
    e = table_.Remove(key, hash);
    if (e != nullptr) {
-      assert(e->InCache());
-      e->SetInCache(false);
-      if (!e->HasRefs()) {
-        // The entry is in LRU since it's in hash and has no external references
-        LRU_Remove(e);
+      last_reference = Unref(e);
+      if (last_reference) {
        usage_ -= e->charge;
-        last_reference = true;
      }
+      if (last_reference && e->InCache()) {
+        LRU_Remove(e);
+      }
+      e->SetInCache(false);
    }
  }

-  // Free the entry here outside of mutex for performance reasons
+  // mutex not held here
  // last_reference will only be true if e != nullptr
  if (last_reference) {
    e->Free();
@@ -17,34 +17,31 @@

 namespace rocksdb {

-// LRU cache implementation. This class is not thread-safe.
+// LRU cache implementation

 // An entry is a variable length heap-allocated structure.
 // Entries are referenced by cache and/or by any external entity.
-// The cache keeps all its entries in a hash table. Some elements
+// The cache keeps all its entries in table. Some elements
 // are also stored on LRU list.
 //
 // LRUHandle can be in these states:
 // 1. Referenced externally AND in hash table.
-//    In that case the entry is *not* in the LRU list
-//    (refs >= 1 && in_cache == true)
-// 2. Not referenced externally AND in hash table.
-//    In that case the entry is in the LRU list and can be freed.
-//    (refs == 0 && in_cache == true)
-// 3. Referenced externally AND not in hash table.
-//    In that case the entry is not in the LRU list and not in hash table.
-//    The entry can be freed when refs becomes 0.
-//    (refs >= 1 && in_cache == false)
+//  In that case the entry is *not* in the LRU. (refs > 1 && in_cache == true)
+// 2. Not referenced externally and in hash table. In that case the entry is
+// in the LRU and can be freed. (refs == 1 && in_cache == true)
+// 3. Referenced externally and not in hash table. In that case the entry is
+// in not on LRU and not in table. (refs >= 1 && in_cache == false)
 //
 // All newly created LRUHandles are in state 1. If you call
-// LRUCacheShard::Release on entry in state 1, it will go into state 2.
-// To move from state 1 to state 3, either call LRUCacheShard::Erase or
-// LRUCacheShard::Insert with the same key (but possibly different value).
+// LRUCacheShard::Release
+// on entry in state 1, it will go into state 2. To move from state 1 to
+// state 3, either call LRUCacheShard::Erase or LRUCacheShard::Insert with the
+// same key.
 // To move from state 2 to state 1, use LRUCacheShard::Lookup.
 // Before destruction, make sure that no handles are in state 1. This means
 // that any successful LRUCacheShard::Lookup/LRUCacheShard::Insert have a
-// matching LRUCache::Release (to move into state 2) or LRUCacheShard::Erase
-// (to move into state 3).
+// matching
+// RUCache::Release (to move into state 2) or LRUCacheShard::Erase (for state 3)

 struct LRUHandle {
  void* value;
@@ -54,42 +51,37 @@ struct LRUHandle {
  LRUHandle* prev;
  size_t charge;  // TODO(opt): Only allow uint32_t?
  size_t key_length;
-  // The hash of key(). Used for fast sharding and comparisons.
-  uint32_t hash;
-  // The number of external refs to this entry. The cache itself is not counted.
-  uint32_t refs;
+  uint32_t refs;     // a number of refs to this entry
+                     // cache itself is counted as 1

+  // Include the following flags:
+  //   IN_CACHE:         whether this entry is referenced by the hash table.
+  //   IS_HIGH_PRI:      whether this entry is high priority entry.
+  //   IN_HIGH_PRI_POOL: whether this entry is in high-pri pool.
+  //   HAS_HIT:          whether this entry has had any lookups (hits).
  enum Flags : uint8_t {
-    // Whether this entry is referenced by the hash table.
    IN_CACHE = (1 << 0),
-    // Whether this entry is high priority entry.
    IS_HIGH_PRI = (1 << 1),
-    // Whether this entry is in high-pri pool.
    IN_HIGH_PRI_POOL = (1 << 2),
-    // Wwhether this entry has had any lookups (hits).
    HAS_HIT = (1 << 3),
  };

  uint8_t flags;

-  // Beginning of the key (MUST BE THE LAST FIELD IN THIS STRUCT!)
-  char key_data[1];
+  uint32_t hash;     // Hash of key(); used for fast sharding and comparisons

-  Slice key() const { return Slice(key_data, key_length); }
+  char key_data[1];  // Beginning of key

-  // Increase the reference count by 1.
-  void Ref() { refs++; }
-
-  // Just reduce the reference count by 1. Return true if it was last reference.
-  bool Unref() {
-    assert(refs > 0);
-    refs--;
-    return refs == 0;
+  Slice key() const {
+    // For cheaper lookups, we allow a temporary Handle object
+    // to store a pointer to a key in "value".
+    if (next == this) {
+      return *(reinterpret_cast<Slice*>(value));
+    } else {
+      return Slice(key_data, key_length);
+    }
  }

-  // Return true if there are external refs, false otherwise.
-  bool HasRefs() const { return refs > 0; }
-
  bool InCache() const { return flags & IN_CACHE; }
  bool IsHighPri() const { return flags & IS_HIGH_PRI; }
  bool InHighPriPool() const { return flags & IN_HIGH_PRI_POOL; }
@@ -122,7 +114,7 @@ struct LRUHandle {
  void SetHit() { flags |= HAS_HIT; }

  void Free() {
-    assert(refs == 0);
+    assert((refs == 1 && InCache()) || (refs == 0 && !InCache()));
    if (deleter) {
      (*deleter)(key(), value);
    }
@@ -177,7 +169,7 @@ class ALIGN_AS(CACHE_LINE_SIZE) LRUCacheShard final : public CacheShard {
 public:
  LRUCacheShard(size_t capacity, bool strict_capacity_limit,
                double high_pri_pool_ratio, bool use_adaptive_mutex);
-  virtual ~LRUCacheShard() override = default;
+  virtual ~LRUCacheShard();

  // Separate from constructor so caller can easily make an array of LRUCache
  // if current usage is more than new capacity, the function will attempt to
@@ -233,6 +225,10 @@ class ALIGN_AS(CACHE_LINE_SIZE) LRUCacheShard final : public CacheShard {
  // high-pri pool is no larger than the size specify by high_pri_pool_pct.
  void MaintainPoolSize();

+  // Just reduce the reference count by 1.
+  // Return true if last reference
+  bool Unref(LRUHandle* e);
+
  // Free some space following strict LRU policy until enough space
  // to hold (usage_ + charge) is freed or the lru list is empty
  // This function is not thread safe - it needs to be executed while
@@ -1,4 +1,3 @@
-#!/usr/bin/env python2
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
 import re
 import sys
@@ -221,9 +221,8 @@ Status BuildTable(
          mutable_cf_options.prefix_extractor.get(), nullptr,
          (internal_stats == nullptr) ? nullptr
                                      : internal_stats->GetFileReadHist(0),
-          TableReaderCaller::kFlush, /*arena=*/nullptr,
-          /*skip_filter=*/false, level, /*smallest_compaction_key=*/nullptr,
-          /*largest_compaction_key*/ nullptr));
+          false /* for_compaction */, nullptr /* arena */,
+          false /* skip_filter */, level));
      s = it->status();
      if (s.ok() && paranoid_file_checks) {
        for (it->SeekToFirst(); it->Valid(); it->Next()) {
@@ -1034,7 +1034,7 @@ void rocksdb_wal_iter_destroy (const rocksdb_wal_iterator_t* iter) {
 rocksdb_writebatch_t* rocksdb_wal_iter_get_batch (const rocksdb_wal_iterator_t* iter, uint64_t* seq) {
  rocksdb_writebatch_t* result = rocksdb_writebatch_create();
  BatchResult wal_batch = iter->rep->GetBatch();
-  result->rep = std::move(*wal_batch.writeBatchPtr);
+  result->rep = * wal_batch.writeBatchPtr.release();
  if (seq != nullptr) {
    *seq = wal_batch.sequence;
  }
@@ -17,11 +17,9 @@
 #include "memtable/hash_skiplist_rep.h"
 #include "options/options_parser.h"
 #include "port/port.h"
-#include "port/stack_trace.h"
 #include "rocksdb/db.h"
 #include "rocksdb/env.h"
 #include "rocksdb/iterator.h"
-#include "rocksdb/utilities/object_registry.h"
 #include "test_util/fault_injection_test_env.h"
 #include "test_util/sync_point.h"
 #include "test_util/testharness.h"
@@ -62,20 +60,8 @@ class EnvCounter : public EnvWrapper {

 class ColumnFamilyTestBase : public testing::Test {
 public:
-  explicit ColumnFamilyTestBase(uint32_t format) : rnd_(139), format_(format) {
-    Env* base_env = Env::Default();
-#ifndef ROCKSDB_LITE
-    const char* test_env_uri = getenv("TEST_ENV_URI");
-    if (test_env_uri) {
-      Status s = ObjectRegistry::NewInstance()->NewSharedObject(test_env_uri,
-                                                                &env_guard_);
-      base_env = env_guard_.get();
-      EXPECT_OK(s);
-      EXPECT_NE(Env::Default(), base_env);
-    }
-#endif  // !ROCKSDB_LITE
-    EXPECT_NE(nullptr, base_env);
-    env_ = new EnvCounter(base_env);
+  ColumnFamilyTestBase(uint32_t format) : rnd_(139), format_(format) {
+    env_ = new EnvCounter(Env::Default());
    dbname_ = test::PerThreadDBPath("column_family_test");
    db_options_.create_if_missing = true;
    db_options_.fail_if_options_file_error = true;
@@ -546,7 +532,6 @@ class ColumnFamilyTestBase : public testing::Test {
  std::string dbname_;
  DB* db_ = nullptr;
  EnvCounter* env_;
-  std::shared_ptr<Env> env_guard_;
  Random rnd_;
  uint32_t format_;
 };
@@ -3327,17 +3312,7 @@ TEST_P(ColumnFamilyTest, MultipleCFPathsTest) {

 }  // namespace rocksdb

-#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-extern "C" {
-void RegisterCustomObjects(int argc, char** argv);
-}
-#else
-void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
-#endif  // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-
 int main(int argc, char** argv) {
-  rocksdb::port::InstallStackTraceHandler();
  ::testing::InitGoogleTest(&argc, argv);
-  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
@@ -85,15 +85,6 @@ class CompactedDBImpl : public DBImpl {
      const IngestExternalFileOptions& /*ingestion_options*/) override {
    return Status::NotSupported("Not supported in compacted db mode.");
  }
-  using DB::CreateColumnFamilyWithImport;
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& /*options*/,
-      const std::string& /*column_family_name*/,
-      const ImportColumnFamilyOptions& /*import_options*/,
-      const ExportImportFilesMetaData& /*metadata*/,
-      ColumnFamilyHandle** /*handle*/) override {
-    return Status::NotSupported("Not supported in compacted db mode.");
-  }

 private:
  friend class DB;
@@ -521,7 +521,7 @@ void CompactionJob::GenSubcompactionBoundaries() {
    // mutex to reduce contention
    db_mutex_->Unlock();
    uint64_t size = versions_->ApproximateSize(v, a, b, start_lvl, out_lvl + 1,
-                                               TableReaderCaller::kCompaction);
+                                               /*for_compaction*/ true);
    db_mutex_->Lock();
    ranges.emplace_back(a, b, size);
    sum += size;
@@ -646,14 +646,12 @@ Status CompactionJob::Run() {
        // to cache it here for further user reads
        InternalIterator* iter = cfd->table_cache()->NewIterator(
            ReadOptions(), env_options_, cfd->internal_comparator(),
-            *files_meta[file_idx], /*range_del_agg=*/nullptr, prefix_extractor,
-            /*table_reader_ptr=*/nullptr,
+            *files_meta[file_idx], nullptr /* range_del_agg */,
+            prefix_extractor, nullptr,
            cfd->internal_stats()->GetFileReadHist(
                compact_->compaction->output_level()),
-            TableReaderCaller::kCompactionRefill, /*arena=*/nullptr,
-            /*skip_filters=*/false, compact_->compaction->output_level(),
-            /*smallest_compaction_key=*/nullptr,
-            /*largest_compaction_key=*/nullptr);
+            false, nullptr /* arena */, false /* skip_filters */,
+            compact_->compaction->output_level());
        auto s = iter->status();

        if (s.ok() && paranoid_file_checks_) {
@@ -59,7 +59,7 @@ Status VerifySstFileChecksum(const Options& options,
  if (!s.ok()) {
    return s;
  }
-  s = table_reader->VerifyChecksum(TableReaderCaller::kUserVerifyChecksum);
+  s = table_reader->VerifyChecksum();
  return s;
 }

@@ -10,7 +10,6 @@
 #include "db/db_test_util.h"
 #include "port/stack_trace.h"
 #include "rocksdb/perf_context.h"
-#include "rocksdb/utilities/debug.h"
 #include "table/block_based/block_builder.h"
 #include "test_util/fault_injection_test_env.h"
 #if !defined(ROCKSDB_LITE)
@@ -1287,55 +1286,6 @@ TEST_F(DBBasicTest, MultiGetBatchedMultiLevel) {
  }
 }

-#ifndef ROCKSDB_LITE
-TEST_F(DBBasicTest, GetAllKeyVersions) {
-  Options options = CurrentOptions();
-  options.env = env_;
-  options.create_if_missing = true;
-  options.disable_auto_compactions = true;
-  CreateAndReopenWithCF({"pikachu"}, options);
-  ASSERT_EQ(2, handles_.size());
-  const size_t kNumInserts = 4;
-  const size_t kNumDeletes = 4;
-  const size_t kNumUpdates = 4;
-
-  // Check default column family
-  for (size_t i = 0; i != kNumInserts; ++i) {
-    ASSERT_OK(Put(std::to_string(i), "value"));
-  }
-  for (size_t i = 0; i != kNumUpdates; ++i) {
-    ASSERT_OK(Put(std::to_string(i), "value1"));
-  }
-  for (size_t i = 0; i != kNumDeletes; ++i) {
-    ASSERT_OK(Delete(std::to_string(i)));
-  }
-  std::vector<KeyVersion> key_versions;
-  ASSERT_OK(rocksdb::GetAllKeyVersions(db_, Slice(), Slice(),
-                                       std::numeric_limits<size_t>::max(),
-                                       &key_versions));
-  ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates, key_versions.size());
-  ASSERT_OK(rocksdb::GetAllKeyVersions(db_, handles_[0], Slice(), Slice(),
-                                       std::numeric_limits<size_t>::max(),
-                                       &key_versions));
-  ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates, key_versions.size());
-
-  // Check non-default column family
-  for (size_t i = 0; i != kNumInserts - 1; ++i) {
-    ASSERT_OK(Put(1, std::to_string(i), "value"));
-  }
-  for (size_t i = 0; i != kNumUpdates - 1; ++i) {
-    ASSERT_OK(Put(1, std::to_string(i), "value1"));
-  }
-  for (size_t i = 0; i != kNumDeletes - 1; ++i) {
-    ASSERT_OK(Delete(1, std::to_string(i)));
-  }
-  ASSERT_OK(rocksdb::GetAllKeyVersions(db_, handles_[1], Slice(), Slice(),
-                                       std::numeric_limits<size_t>::max(),
-                                       &key_versions));
-  ASSERT_EQ(kNumInserts + kNumDeletes + kNumUpdates - 3, key_versions.size());
-}
-#endif  // !ROCKSDB_LITE
-
 class DBBasicTestWithParallelIO
    : public DBTestBase,
      public testing::WithParamInterface<std::tuple<bool,bool,bool,bool>> {
@@ -1776,17 +1726,8 @@ INSTANTIATE_TEST_CASE_P(Timestamp, DBBasicTestWithTimestampWithParam,

 }  // namespace rocksdb

-#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-extern "C" {
-void RegisterCustomObjects(int argc, char** argv);
-}
-#else
-void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
-#endif  // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-
 int main(int argc, char** argv) {
  rocksdb::port::InstallStackTraceHandler();
  ::testing::InitGoogleTest(&argc, argv);
-  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
@@ -19,9 +19,6 @@ class DBBlockCacheTest : public DBTestBase {
  size_t hit_count_ = 0;
  size_t insert_count_ = 0;
  size_t failure_count_ = 0;
-  size_t compression_dict_miss_count_ = 0;
-  size_t compression_dict_hit_count_ = 0;
-  size_t compression_dict_insert_count_ = 0;
  size_t compressed_miss_count_ = 0;
  size_t compressed_hit_count_ = 0;
  size_t compressed_insert_count_ = 0;
@@ -72,15 +69,6 @@ class DBBlockCacheTest : public DBTestBase {
        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSED_ADD_FAILURES);
  }

-  void RecordCacheCountersForCompressionDict(const Options& options) {
-    compression_dict_miss_count_ =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
-    compression_dict_hit_count_ =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_HIT);
-    compression_dict_insert_count_ =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_ADD);
-  }
-
  void CheckCacheCounters(const Options& options, size_t expected_misses,
                          size_t expected_hits, size_t expected_inserts,
                          size_t expected_failures) {
@@ -99,28 +87,6 @@ class DBBlockCacheTest : public DBTestBase {
    failure_count_ = new_failure_count;
  }

-  void CheckCacheCountersForCompressionDict(
-      const Options& options, size_t expected_compression_dict_misses,
-      size_t expected_compression_dict_hits,
-      size_t expected_compression_dict_inserts) {
-    size_t new_compression_dict_miss_count =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS);
-    size_t new_compression_dict_hit_count =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_HIT);
-    size_t new_compression_dict_insert_count =
-        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_ADD);
-    ASSERT_EQ(compression_dict_miss_count_ + expected_compression_dict_misses,
-              new_compression_dict_miss_count);
-    ASSERT_EQ(compression_dict_hit_count_ + expected_compression_dict_hits,
-              new_compression_dict_hit_count);
-    ASSERT_EQ(
-        compression_dict_insert_count_ + expected_compression_dict_inserts,
-        new_compression_dict_insert_count);
-    compression_dict_miss_count_ = new_compression_dict_miss_count;
-    compression_dict_hit_count_ = new_compression_dict_hit_count;
-    compression_dict_insert_count_ = new_compression_dict_insert_count;
-  }
-
  void CheckCompressedCacheCounters(const Options& options,
                                    size_t expected_misses,
                                    size_t expected_hits,
@@ -399,11 +365,11 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
  ASSERT_EQ(cache->GetUsage(), index_bytes_insert + filter_bytes_insert);
  // set the cache capacity to the current usage
  cache->SetCapacity(index_bytes_insert + filter_bytes_insert);
-  // The index and filter eviction statistics were broken by the refactoring
-  // that moved the readers out of the block cache. Disabling these until we can
+  // The index eviction statistics were broken by the refactoring that moved
+  // the index readers out of the block cache. Disabling these until we can
  // bring the stats back.
  // ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT), 0);
-  // ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT), 0);
+  ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT), 0);
  // Note that the second key needs to be no longer than the first one.
  // Otherwise the second index block may not fit in cache.
  ASSERT_OK(Put(1, "key", "val"));
@@ -414,13 +380,13 @@ TEST_F(DBBlockCacheTest, IndexAndFilterBlocksStats) {
            index_bytes_insert);
  ASSERT_GT(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_INSERT),
            filter_bytes_insert);
-  // The index and filter eviction statistics were broken by the refactoring
-  // that moved the readers out of the block cache. Disabling these until we can
+  // The index eviction statistics were broken by the refactoring that moved
+  // the index readers out of the block cache. Disabling these until we can
  // bring the stats back.
  // ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_INDEX_BYTES_EVICT),
  //           index_bytes_insert);
-  // ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT),
-  //           filter_bytes_insert);
+  ASSERT_EQ(TestGetTickerCount(options, BLOCK_CACHE_FILTER_BYTES_EVICT),
+            filter_bytes_insert);
 }

 namespace {
@@ -705,8 +671,6 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
    options.table_factory.reset(new BlockBasedTableFactory(table_options));
    DestroyAndReopen(options);

-    RecordCacheCountersForCompressionDict(options);
-
    for (int i = 0; i < kNumFiles; ++i) {
      ASSERT_EQ(i, NumTableFilesAtLevel(0, 0));
      for (int j = 0; j < kNumEntriesPerFile; ++j) {
@@ -719,26 +683,27 @@ TEST_F(DBBlockCacheTest, CacheCompressionDict) {
    ASSERT_EQ(0, NumTableFilesAtLevel(0));
    ASSERT_EQ(kNumFiles, NumTableFilesAtLevel(1));

-    // Compression dictionary blocks are preloaded.
-    CheckCacheCountersForCompressionDict(
-        options, kNumFiles /* expected_compression_dict_misses */,
-        0 /* expected_compression_dict_hits */,
-        kNumFiles /* expected_compression_dict_inserts */);
-
    // Seek to a key in a file. It should cause the SST's dictionary meta-block
    // to be read.
    RecordCacheCounters(options);
-    RecordCacheCountersForCompressionDict(options);
+    ASSERT_EQ(0,
+              TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
+    ASSERT_EQ(0, TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_ADD));
+    ASSERT_EQ(
+        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT),
+        0);
    ReadOptions read_options;
    ASSERT_NE("NOT_FOUND", Get(Key(kNumFiles * kNumEntriesPerFile - 1)));
-    // Two block hits: index and dictionary since they are prefetched
-    // One block missed/added: data block
-    CheckCacheCounters(options, 1 /* expected_misses */, 2 /* expected_hits */,
-                       1 /* expected_inserts */, 0 /* expected_failures */);
-    CheckCacheCountersForCompressionDict(
-        options, 0 /* expected_compression_dict_misses */,
-        1 /* expected_compression_dict_hits */,
-        0 /* expected_compression_dict_inserts */);
+    // Two blocks missed/added: dictionary and data block
+    // One block hit: index since it's prefetched
+    CheckCacheCounters(options, 2 /* expected_misses */, 1 /* expected_hits */,
+                       2 /* expected_inserts */, 0 /* expected_failures */);
+    ASSERT_EQ(1,
+              TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_MISS));
+    ASSERT_EQ(1, TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_ADD));
+    ASSERT_GT(
+        TestGetTickerCount(options, BLOCK_CACHE_COMPRESSION_DICT_BYTES_INSERT),
+        0);
  }
 }

@@ -497,14 +497,14 @@ TEST_F(DBCompactionTest, TestTableReaderForCompaction) {

  // Create new iterator for:
  // (1) 1 for verifying flush results
-  // (2) 1 for verifying compaction results.
-  // (3) New TableReaders will not be created for compaction inputs
-  ASSERT_EQ(num_new_table_reader, 2);
+  // (2) 3 for compaction input files
+  // (3) 1 for verifying compaction results.
+  ASSERT_EQ(num_new_table_reader, 5);

  num_table_cache_lookup = 0;
  num_new_table_reader = 0;
  ASSERT_EQ(Key(1), Get(Key(1)));
-  ASSERT_EQ(num_table_cache_lookup + old_num_table_cache_lookup2, 5);
+  ASSERT_EQ(num_table_cache_lookup + old_num_table_cache_lookup2, 3);
  ASSERT_EQ(num_new_table_reader, 0);

  num_table_cache_lookup = 0;
@@ -519,14 +519,13 @@ TEST_F(DBCompactionTest, TestTableReaderForCompaction) {
  // May preload table cache too.
  ASSERT_GE(num_table_cache_lookup, 1);
  old_num_table_cache_lookup2 = num_table_cache_lookup;
-  // One for verifying compaction results.
-  // No new iterator created for compaction.
-  ASSERT_EQ(num_new_table_reader, 1);
+  // One for compaction input, one for verifying compaction results.
+  ASSERT_EQ(num_new_table_reader, 2);

  num_table_cache_lookup = 0;
  num_new_table_reader = 0;
  ASSERT_EQ(Key(1), Get(Key(1)));
-  ASSERT_EQ(num_table_cache_lookup + old_num_table_cache_lookup2, 3);
+  ASSERT_EQ(num_table_cache_lookup + old_num_table_cache_lookup2, 2);
  ASSERT_EQ(num_new_table_reader, 0);

  rocksdb::SyncPoint::GetInstance()->ClearAllCallBacks();
@@ -4340,6 +4339,12 @@ TEST_P(DBCompactionDirectIOTest, DirectIO) {
  options.env = new MockEnv(Env::Default());
  Reopen(options);
  bool readahead = false;
+  SyncPoint::GetInstance()->SetCallBack(
+      "TableCache::NewIterator:for_compaction", [&](void* arg) {
+        bool* use_direct_reads = static_cast<bool*>(arg);
+        ASSERT_EQ(*use_direct_reads,
+                  options.use_direct_reads);
+      });
  SyncPoint::GetInstance()->SetCallBack(
      "CompactionJob::OpenCompactionOutputFile", [&](void* arg) {
        bool* use_direct_writes = static_cast<bool*>(arg);
@@ -14,7 +14,6 @@
 #endif

 #include <algorithm>
-#include <cinttypes>
 #include <cstdio>
 #include <map>
 #include <set>
@@ -33,7 +32,6 @@
 #include "db/error_handler.h"
 #include "db/event_helpers.h"
 #include "db/external_sst_file_ingestion_job.h"
-#include "db/import_column_family_job.h"
 #include "db/flush_job.h"
 #include "db/forward_iterator.h"
 #include "db/job_context.h"
@@ -103,6 +101,7 @@ namespace rocksdb {
 const std::string kDefaultColumnFamilyName("default");
 const std::string kPersistentStatsColumnFamilyName(
    "___rocksdb_stats_history___");
+const int kMicrosInSecond = 1000 * 1000;
 void DumpRocksDBBuildVersion(Logger* log);

 CompressionType GetCompressionFlush(
@@ -699,15 +698,10 @@ void DBImpl::PersistStats() {
  if (!statistics->getTickerMap(&stats_map)) {
    return;
  }
-  ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                 "------- PERSISTING STATS -------");

  if (immutable_db_options_.persist_stats_to_disk) {
    WriteBatch batch;
    if (stats_slice_initialized_) {
-      ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                     "Reading %" ROCKSDB_PRIszt " stats from statistics\n",
-                     stats_slice_.size());
      for (const auto& stat : stats_map) {
        char key[100];
        int length =
@@ -729,13 +723,8 @@ void DBImpl::PersistStats() {
    Status s = Write(wo, &batch);
    if (!s.ok()) {
      ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                     "Writing to persistent stats CF failed -- %s",
+                     "Writing to persistent stats CF failed -- %s\n",
                     s.ToString().c_str());
-    } else {
-      ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                     "Writing %" ROCKSDB_PRIszt " stats with timestamp %" PRIu64
-                     " to persistent stats CF succeeded",
-                     stats_slice_.size(), now_seconds);
    }
    // TODO(Zhongyi): add purging for persisted data
  } else {
@@ -748,10 +737,6 @@ void DBImpl::PersistStats() {
          stats_delta[stat.first] = stat.second - stats_slice_[stat.first];
        }
      }
-      ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                     "Storing %" ROCKSDB_PRIszt " stats with timestamp %" PRIu64
-                     " to in-memory stats history",
-                     stats_slice_.size(), now_seconds);
      stats_history_[now_seconds] = stats_delta;
    }
    stats_slice_initialized_ = true;
@@ -759,22 +744,15 @@ void DBImpl::PersistStats() {
    TEST_SYNC_POINT("DBImpl::PersistStats:StatsCopied");

    // delete older stats snapshots to control memory consumption
-    size_t stats_history_size = EstimateInMemoryStatsHistorySize();
-    bool purge_needed = stats_history_size > stats_history_size_limit;
-    ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                   "[Pre-GC] In-memory stats history size: %" ROCKSDB_PRIszt
-                   " bytes, slice count: %" ROCKSDB_PRIszt,
-                   stats_history_size, stats_history_.size());
+    bool purge_needed =
+        EstimateInMemoryStatsHistorySize() > stats_history_size_limit;
    while (purge_needed && !stats_history_.empty()) {
      stats_history_.erase(stats_history_.begin());
      purge_needed =
          EstimateInMemoryStatsHistorySize() > stats_history_size_limit;
    }
-    ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                   "[Post-GC] In-memory stats history size: %" ROCKSDB_PRIszt
-                   " bytes, slice count: %" ROCKSDB_PRIszt,
-                   stats_history_size, stats_history_.size());
  }
+  // TODO: persist stats to disk
 #endif  // !ROCKSDB_LITE
 }

@@ -883,6 +861,16 @@ Directory* DBImpl::GetDataDir(ColumnFamilyData* cfd, size_t path_id) const {
  return ret_dir;
 }

+Directory* DBImpl::Directories::GetDataDir(size_t path_id) const {
+  assert(path_id < data_dirs_.size());
+  Directory* ret_dir = data_dirs_[path_id].get();
+  if (ret_dir == nullptr) {
+    // Should use db_dir_
+    return db_dir_.get();
+  }
+  return ret_dir;
+}
+
 Status DBImpl::SetOptions(
    ColumnFamilyHandle* column_family,
    const std::unordered_map<std::string, std::string>& options_map) {
@@ -1441,7 +1429,16 @@ ColumnFamilyHandle* DBImpl::PersistentStatsColumnFamily() const {
 Status DBImpl::Get(const ReadOptions& read_options,
                   ColumnFamilyHandle* column_family, const Slice& key,
                   PinnableSlice* value) {
-  return GetImpl(read_options, column_family, key, value);
+  if (nullptr == read_options.timestamp) {
+    return GetImpl(read_options, column_family, key, value);
+  }
+  Slice akey;
+  std::string buf;
+  Status s = AppendTimestamp(key, *(read_options.timestamp), &akey, &buf);
+  if (s.ok()) {
+    s = GetImpl(read_options, column_family, akey, value);
+  }
+  return s;
 }

 Status DBImpl::GetImpl(const ReadOptions& read_options,
@@ -1490,22 +1487,7 @@ Status DBImpl::GetImpl(const ReadOptions& read_options,
                   ? versions_->LastSequence()
                   : versions_->LastPublishedSequence();
    if (callback) {
-      // The unprep_seqs are not published for write unprepared, so it could be
-      // that max_visible_seq is larger. Seek to the std::max of the two.
-      // However, we still want our callback to contain the actual snapshot so
-      // that it can do the correct visibility filtering.
      callback->Refresh(snapshot);
-
-      // Internally, WriteUnpreparedTxnReadCallback::Refresh would set
-      // max_visible_seq = max(max_visible_seq, snapshot)
-      //
-      // Currently, the commented out assert is broken by
-      // InvalidSnapshotReadCallback, but if write unprepared recovery followed
-      // the regular transaction flow, then this special read callback would not
-      // be needed.
-      //
-      // assert(callback->max_visible_seq() >= snapshot);
-      snapshot = callback->max_visible_seq();
    }
  }
  TEST_SYNC_POINT("DBImpl::GetImpl:3");
@@ -1519,7 +1501,7 @@ Status DBImpl::GetImpl(const ReadOptions& read_options,
  // First look in the memtable, then in the immutable memtable (if any).
  // s is both in/out. When in, s could either be OK or MergeInProgress.
  // merge_operands will contain the sequence of merges in the latter case.
-  LookupKey lkey(key, snapshot, read_options.timestamp);
+  LookupKey lkey(key, snapshot);
  PERF_TIMER_STOP(get_snapshot_time);

  bool skip_memtable = (read_options.read_tier == kPersistedTier &&
@@ -1857,24 +1839,6 @@ void DBImpl::MultiGetImpl(
    snapshot = last_seq_same_as_publish_seq_
                   ? versions_->LastSequence()
                   : versions_->LastPublishedSequence();
-    if (callback) {
-      // The unprep_seqs are not published for write unprepared, so it could be
-      // that max_visible_seq is larger. Seek to the std::max of the two.
-      // However, we still want our callback to contain the actual snapshot so
-      // that it can do the correct visibility filtering.
-      callback->Refresh(snapshot);
-
-      // Internally, WriteUnpreparedTxnReadCallback::Refresh would set
-      // max_visible_seq = max(max_visible_seq, snapshot)
-      //
-      // Currently, the commented out assert is broken by
-      // InvalidSnapshotReadCallback, but if write unprepared recovery followed
-      // the regular transaction flow, then this special read callback would not
-      // be needed.
-      //
-      // assert(callback->max_visible_seq() >= snapshot);
-      snapshot = callback->max_visible_seq();
-    }
  }

  // For each of the given keys, apply the entire "get" process as follows:
@@ -2788,13 +2752,11 @@ void DBImpl::GetApproximateMemTableStats(ColumnFamilyHandle* column_family,
  ReturnAndCleanupSuperVersion(cfd, sv);
 }

-Status DBImpl::GetApproximateSizes(const SizeApproximationOptions& options,
-                                   ColumnFamilyHandle* column_family,
-                                   const Range* range, int n, uint64_t* sizes) {
-  if (!options.include_memtabtles && !options.include_files) {
-    return Status::InvalidArgument("Invalid options");
-  }
-
+void DBImpl::GetApproximateSizes(ColumnFamilyHandle* column_family,
+                                 const Range* range, int n, uint64_t* sizes,
+                                 uint8_t include_flags) {
+  assert(include_flags & DB::SizeApproximationFlags::INCLUDE_FILES ||
+         include_flags & DB::SizeApproximationFlags::INCLUDE_MEMTABLES);
  Version* v;
  auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
  auto cfd = cfh->cfd();
@@ -2806,19 +2768,18 @@ Status DBImpl::GetApproximateSizes(const SizeApproximationOptions& options,
    InternalKey k1(range[i].start, kMaxSequenceNumber, kValueTypeForSeek);
    InternalKey k2(range[i].limit, kMaxSequenceNumber, kValueTypeForSeek);
    sizes[i] = 0;
-    if (options.include_files) {
+    if (include_flags & DB::SizeApproximationFlags::INCLUDE_FILES) {
      sizes[i] += versions_->ApproximateSize(
          v, k1.Encode(), k2.Encode(), /*start_level=*/0, /*end_level=*/-1,
-          TableReaderCaller::kUserApproximateSize);
+          /*for_compaction=*/false);
    }
-    if (options.include_memtabtles) {
+    if (include_flags & DB::SizeApproximationFlags::INCLUDE_MEMTABLES) {
      sizes[i] += sv->mem->ApproximateStats(k1.Encode(), k2.Encode()).size;
      sizes[i] += sv->imm->ApproximateStats(k1.Encode(), k2.Encode()).size;
    }
  }

  ReturnAndCleanupSuperVersion(cfd, sv);
-  return Status::OK();
 }

 std::list<uint64_t>::iterator
@@ -3689,7 +3650,7 @@ Status DBImpl::IngestExternalFiles(
    auto* cfd = static_cast<ColumnFamilyHandleImpl*>(arg.column_family)->cfd();
    ingestion_jobs.emplace_back(env_, versions_.get(), cfd,
                                immutable_db_options_, env_options_,
-                                &snapshots_, arg.options, &directories_);
+                                &snapshots_, arg.options);
  }
  std::vector<std::pair<bool, Status>> exec_results;
  for (size_t i = 0; i != num_cfs; ++i) {
@@ -3922,127 +3883,6 @@ Status DBImpl::IngestExternalFiles(
  return status;
 }

-Status DBImpl::CreateColumnFamilyWithImport(
-    const ColumnFamilyOptions& options, const std::string& column_family_name,
-    const ImportColumnFamilyOptions& import_options,
-    const ExportImportFilesMetaData& metadata,
-    ColumnFamilyHandle** handle) {
-  assert(handle != nullptr);
-  assert(*handle == nullptr);
-  std::string cf_comparator_name = options.comparator->Name();
-  if (cf_comparator_name != metadata.db_comparator_name) {
-    return Status::InvalidArgument("Comparator name mismatch");
-  }
-
-  // Create column family.
-  auto status = CreateColumnFamily(options, column_family_name, handle);
-  if (!status.ok()) {
-    return status;
-  }
-
-  // Import sst files from metadata.
-  auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(*handle);
-  auto cfd = cfh->cfd();
-  ImportColumnFamilyJob import_job(env_, versions_.get(), cfd,
-                                   immutable_db_options_, env_options_,
-                                   import_options, metadata.files);
-
-  SuperVersionContext dummy_sv_ctx(/* create_superversion */ true);
-  VersionEdit dummy_edit;
-  uint64_t next_file_number = 0;
-  std::list<uint64_t>::iterator pending_output_elem;
-  {
-    // Lock db mutex
-    InstrumentedMutexLock l(&mutex_);
-    if (error_handler_.IsDBStopped()) {
-      // Don't import files when there is a bg_error
-      status = error_handler_.GetBGError();
-    }
-
-    // Make sure that bg cleanup wont delete the files that we are importing
-    pending_output_elem = CaptureCurrentFileNumberInPendingOutputs();
-
-    if (status.ok()) {
-      // If crash happen after a hard link established, Recover function may
-      // reuse the file number that has already assigned to the internal file,
-      // and this will overwrite the external file. To protect the external
-      // file, we have to make sure the file number will never being reused.
-      next_file_number =
-          versions_->FetchAddFileNumber(metadata.files.size());
-      auto cf_options = cfd->GetLatestMutableCFOptions();
-      status = versions_->LogAndApply(cfd, *cf_options, &dummy_edit, &mutex_,
-                                      directories_.GetDbDir());
-      if (status.ok()) {
-        InstallSuperVersionAndScheduleWork(cfd, &dummy_sv_ctx, *cf_options);
-      }
-    }
-  }
-  dummy_sv_ctx.Clean();
-
-  if (status.ok()) {
-    SuperVersion* sv = cfd->GetReferencedSuperVersion(&mutex_);
-    status = import_job.Prepare(next_file_number, sv);
-    CleanupSuperVersion(sv);
-  }
-
-  if (status.ok()) {
-    SuperVersionContext sv_context(true /*create_superversion*/);
-    {
-      // Lock db mutex
-      InstrumentedMutexLock l(&mutex_);
-
-      // Stop writes to the DB by entering both write threads
-      WriteThread::Writer w;
-      write_thread_.EnterUnbatched(&w, &mutex_);
-      WriteThread::Writer nonmem_w;
-      if (two_write_queues_) {
-        nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
-      }
-
-      num_running_ingest_file_++;
-      assert(!cfd->IsDropped());
-      status = import_job.Run();
-
-      // Install job edit [Mutex will be unlocked here]
-      if (status.ok()) {
-        auto cf_options = cfd->GetLatestMutableCFOptions();
-        status = versions_->LogAndApply(cfd, *cf_options, import_job.edit(),
-                                        &mutex_, directories_.GetDbDir());
-        if (status.ok()) {
-          InstallSuperVersionAndScheduleWork(cfd, &sv_context, *cf_options);
-        }
-      }
-
-      // Resume writes to the DB
-      if (two_write_queues_) {
-        nonmem_write_thread_.ExitUnbatched(&nonmem_w);
-      }
-      write_thread_.ExitUnbatched(&w);
-
-      num_running_ingest_file_--;
-      if (num_running_ingest_file_ == 0) {
-        bg_cv_.SignalAll();
-      }
-    }
-    // mutex_ is unlocked here
-
-    sv_context.Clean();
-  }
-
-  {
-    InstrumentedMutexLock l(&mutex_);
-    ReleaseFileNumberFromPendingOutputs(pending_output_elem);
-  }
-
-  import_job.Cleanup(status);
-  if (!status.ok()) {
-    DropColumnFamily(*handle);
-    DestroyColumnFamilyHandle(*handle);
-    *handle = nullptr;
-  }
-  return status;
-}
-
 Status DBImpl::VerifyChecksum() {
  Status s;
  std::vector<ColumnFamilyData*> cfd_list;
@@ -27,7 +27,6 @@
 #include "db/external_sst_file_ingestion_job.h"
 #include "db/flush_job.h"
 #include "db/flush_scheduler.h"
-#include "db/import_column_family_job.h"
 #include "db/internal_stats.h"
 #include "db/log_writer.h"
 #include "db/logs_with_prep_tracker.h"
@@ -78,38 +77,6 @@ struct JobContext;
 struct ExternalSstFileInfo;
 struct MemTableInfo;

-// Class to maintain directories for all database paths other than main one.
-class Directories {
- public:
-  Status SetDirectories(Env* env, const std::string& dbname,
-                        const std::string& wal_dir,
-                        const std::vector<DbPath>& data_paths);
-
-  Directory* GetDataDir(size_t path_id) const {
-    assert(path_id < data_dirs_.size());
-    Directory* ret_dir = data_dirs_[path_id].get();
-    if (ret_dir == nullptr) {
-      // Should use db_dir_
-      return db_dir_.get();
-    }
-    return ret_dir;
-  }
-
-  Directory* GetWalDir() {
-    if (wal_dir_) {
-      return wal_dir_.get();
-    }
-    return db_dir_.get();
-  }
-
-  Directory* GetDbDir() { return db_dir_.get(); }
-
- private:
-  std::unique_ptr<Directory> db_dir_;
-  std::vector<std::unique_ptr<Directory>> data_dirs_;
-  std::unique_ptr<Directory> wal_dir_;
-};
-
 // While DB is the public interface of RocksDB, and DBImpl is the actual
 // class implementing it. It's the entrance of the core RocksdB engine.
 // All other DB implementations, e.g. TransactionDB, BlobDB, etc, wrap a
@@ -233,10 +200,9 @@ class DBImpl : public DB {
  virtual bool GetAggregatedIntProperty(const Slice& property,
                                        uint64_t* aggregated_value) override;
  using DB::GetApproximateSizes;
-  virtual Status GetApproximateSizes(const SizeApproximationOptions& options,
-                                     ColumnFamilyHandle* column_family,
-                                     const Range* range, int n,
-                                     uint64_t* sizes) override;
+  virtual void GetApproximateSizes(
+      ColumnFamilyHandle* column_family, const Range* range, int n,
+      uint64_t* sizes, uint8_t include_flags = INCLUDE_FILES) override;
  using DB::GetApproximateMemTableStats;
  virtual void GetApproximateMemTableStats(ColumnFamilyHandle* column_family,
                                           const Range& range,
@@ -358,13 +324,6 @@ class DBImpl : public DB {
  virtual Status IngestExternalFiles(
      const std::vector<IngestExternalFileArg>& args) override;

-  using DB::CreateColumnFamilyWithImport;
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& options, const std::string& column_family_name,
-      const ImportColumnFamilyOptions& import_options,
-      const ExportImportFilesMetaData& metadata,
-      ColumnFamilyHandle** handle) override;
-
  virtual Status VerifyChecksum() override;

  using DB::StartTrace;
@@ -1098,6 +1057,30 @@ class DBImpl : public DB {
    }
  };

+  // Class to maintain directories for all database paths other than main one.
+  class Directories {
+   public:
+    Status SetDirectories(Env* env, const std::string& dbname,
+                          const std::string& wal_dir,
+                          const std::vector<DbPath>& data_paths);
+
+    Directory* GetDataDir(size_t path_id) const;
+
+    Directory* GetWalDir() {
+      if (wal_dir_) {
+        return wal_dir_.get();
+      }
+      return db_dir_.get();
+    }
+
+    Directory* GetDbDir() { return db_dir_.get(); }
+
+   private:
+    std::unique_ptr<Directory> db_dir_;
+    std::vector<std::unique_ptr<Directory>> data_dirs_;
+    std::unique_ptr<Directory> wal_dir_;
+  };
+
  struct LogFileNumberSize {
    explicit LogFileNumberSize(uint64_t _number) : number(_number) {}
    void AddSize(uint64_t new_size) { size += new_size; }
@@ -1311,8 +1294,6 @@ class DBImpl : public DB {

  Status ScheduleFlushes(WriteContext* context);

-  void MaybeFlushStatsCF(autovector<ColumnFamilyData*>* cfds);
-
  Status SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context);

  void SelectColumnFamiliesForAtomicFlush(autovector<ColumnFamilyData*>* cfds);
@@ -1812,8 +1793,7 @@ class DBImpl : public DB {

  std::string db_absolute_path_;

-  // Number of running IngestExternalFile() or CreateColumnFamilyWithImport()
-  // calls.
+  // Number of running IngestExternalFile() calls.
  // REQUIRES: mutex held
  int num_running_ingest_file_;

@@ -1553,39 +1553,13 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
    if (!cfd->mem()->IsEmpty() || !cached_recoverable_state_empty_.load()) {
      s = SwitchMemtable(cfd, &context);
    }
+
    if (s.ok()) {
      if (cfd->imm()->NumNotFlushed() != 0 || !cfd->mem()->IsEmpty() ||
          !cached_recoverable_state_empty_.load()) {
        flush_memtable_id = cfd->imm()->GetLatestMemTableID();
        flush_req.emplace_back(cfd, flush_memtable_id);
      }
-      if (immutable_db_options_.persist_stats_to_disk) {
-        ColumnFamilyData* cfd_stats =
-            versions_->GetColumnFamilySet()->GetColumnFamily(
-                kPersistentStatsColumnFamilyName);
-        if (cfd_stats != nullptr && cfd_stats != cfd &&
-            !cfd_stats->mem()->IsEmpty()) {
-          // only force flush stats CF when it will be the only CF lagging
-          // behind after the current flush
-          bool stats_cf_flush_needed = true;
-          for (auto* loop_cfd : *versions_->GetColumnFamilySet()) {
-            if (loop_cfd == cfd_stats || loop_cfd == cfd) {
-              continue;
-            }
-            if (loop_cfd->GetLogNumber() <= cfd_stats->GetLogNumber()) {
-              stats_cf_flush_needed = false;
-            }
-          }
-          if (stats_cf_flush_needed) {
-            ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                           "Force flushing stats CF with manual flush of %s "
-                           "to avoid holding old logs", cfd->GetName().c_str());
-            s = SwitchMemtable(cfd_stats, &context);
-            flush_memtable_id = cfd_stats->imm()->GetLatestMemTableID();
-            flush_req.emplace_back(cfd_stats, flush_memtable_id);
-          }
-        }
-      }
    }

    if (s.ok() && !flush_req.empty()) {
@@ -284,9 +284,9 @@ Status DBImpl::CreateAndNewDirectory(Env* env, const std::string& dirname,
  return env->NewDirectory(dirname, directory);
 }

-Status Directories::SetDirectories(Env* env, const std::string& dbname,
-                                   const std::string& wal_dir,
-                                   const std::vector<DbPath>& data_paths) {
+Status DBImpl::Directories::SetDirectories(
+    Env* env, const std::string& dbname, const std::string& wal_dir,
+    const std::vector<DbPath>& data_paths) {
  Status s = DBImpl::CreateAndNewDirectory(env, dbname, &db_dir_);
  if (!s.ok()) {
    return s;
@@ -721,8 +721,7 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& log_numbers,
          continue;
        }
      }
-      file_reader.reset(new SequentialFileReader(
-          std::move(file), fname, immutable_db_options_.log_readahead_size));
+      file_reader.reset(new SequentialFileReader(std::move(file), fname));
    }

    // Create the log reader.
@@ -115,16 +115,6 @@ class DBImplReadOnly : public DBImpl {
    return Status::NotSupported("Not supported operation in read only mode.");
  }

-  using DB::CreateColumnFamilyWithImport;
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& /*options*/,
-      const std::string& /*column_family_name*/,
-      const ImportColumnFamilyOptions& /*import_options*/,
-      const ExportImportFilesMetaData& /*metadata*/,
-      ColumnFamilyHandle** /*handle*/) override {
-    return Status::NotSupported("Not supported operation in read only mode.");
-  }
-
 private:
  friend class DB;

@@ -150,8 +150,7 @@ Status DBImplSecondary::MaybeInitLogReader(
        *log_reader = nullptr;
        return status;
      }
-      file_reader.reset(new SequentialFileReader(
-          std::move(file), fname, immutable_db_options_.log_readahead_size));
+      file_reader.reset(new SequentialFileReader(std::move(file), fname));
    }

    // Create the log reader.
@@ -588,9 +587,6 @@ Status DB::OpenAsSecondary(
      &impl->write_controller_));
  impl->column_family_memtables_.reset(
      new ColumnFamilyMemTablesImpl(impl->versions_->GetColumnFamilySet()));
-  impl->wal_in_db_path_ =
-      IsWalDirSameAsDBPath(&impl->immutable_db_options_);
-
  impl->mutex_.Lock();
  s = impl->Recover(column_families, true, false, false);
  if (s.ok()) {
@@ -172,8 +172,7 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
      w.status = WriteBatchInternal::InsertInto(
          &w, w.sequence, &column_family_memtables, &flush_scheduler_,
          write_options.ignore_missing_column_families, 0 /*log_number*/, this,
-          true /*concurrent_memtable_writes*/, seq_per_batch_, w.batch_cnt,
-          batch_per_txn_);
+          true /*concurrent_memtable_writes*/, seq_per_batch_, w.batch_cnt);

      PERF_TIMER_START(write_pre_and_post_process_time);
    }
@@ -1229,7 +1228,6 @@ Status DBImpl::SwitchWAL(WriteContext* write_context) {
        cfds.push_back(cfd);
      }
    }
-    MaybeFlushStatsCF(&cfds);
  }
  for (const auto cfd : cfds) {
    cfd->Ref();
@@ -1296,7 +1294,6 @@ Status DBImpl::HandleWriteBufferFull(WriteContext* write_context) {
    if (cfd_picked != nullptr) {
      cfds.push_back(cfd_picked);
    }
-    MaybeFlushStatsCF(&cfds);
  }

  for (const auto cfd : cfds) {
@@ -1440,40 +1437,6 @@ Status DBImpl::ThrottleLowPriWritesIfNeeded(const WriteOptions& write_options,
  return Status::OK();
 }

-void DBImpl::MaybeFlushStatsCF(autovector<ColumnFamilyData*>* cfds) {
-  assert(cfds != nullptr);
-  if (!cfds->empty() && immutable_db_options_.persist_stats_to_disk) {
-    ColumnFamilyData* cfd_stats =
-        versions_->GetColumnFamilySet()->GetColumnFamily(
-            kPersistentStatsColumnFamilyName);
-    if (cfd_stats != nullptr && !cfd_stats->mem()->IsEmpty()) {
-      for (ColumnFamilyData* cfd : *cfds) {
-        if (cfd == cfd_stats) {
-          // stats CF already included in cfds
-          return;
-        }
-      }
-      // force flush stats CF when its log number is less than all other CF's
-      // log numbers
-      bool force_flush_stats_cf = true;
-      for (auto* loop_cfd : *versions_->GetColumnFamilySet()) {
-        if (loop_cfd == cfd_stats) {
-          continue;
-        }
-        if (loop_cfd->GetLogNumber() <= cfd_stats->GetLogNumber()) {
-          force_flush_stats_cf = false;
-        }
-      }
-      if (force_flush_stats_cf) {
-        cfds->push_back(cfd_stats);
-        ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                       "Force flushing stats CF with automated flush "
-                       "to avoid holding old logs");
-      }
-    }
-  }
-}
-
 Status DBImpl::ScheduleFlushes(WriteContext* context) {
  autovector<ColumnFamilyData*> cfds;
  if (immutable_db_options_.atomic_flush) {
@@ -1487,7 +1450,6 @@ Status DBImpl::ScheduleFlushes(WriteContext* context) {
    while ((tmp_cfd = flush_scheduler_.TakeNextColumnFamily()) != nullptr) {
      cfds.push_back(tmp_cfd);
    }
-    MaybeFlushStatsCF(&cfds);
  }
  Status status;
  for (auto& cfd : cfds) {
@@ -1734,16 +1696,14 @@ Status DB::Put(const WriteOptions& opt, ColumnFamilyHandle* column_family,
    }
    return Write(opt, &batch);
  }
-  const Slice* ts = opt.timestamp;
-  assert(nullptr != ts);
-  size_t ts_sz = ts->size();
-  WriteBatch batch(key.size() + ts_sz + value.size() + 24, /*max_bytes=*/0,
-                   ts_sz);
-  Status s = batch.Put(column_family, key, value);
+  Slice akey;
+  std::string buf;
+  Status s = AppendTimestamp(key, *(opt.timestamp), &akey, &buf);
  if (!s.ok()) {
    return s;
  }
-  s = batch.AssignTimestamp(*ts);
+  WriteBatch batch(akey.size() + value.size() + 24);
+  s = batch.Put(column_family, akey, value);
  if (!s.ok()) {
    return s;
  }
@@ -263,6 +263,12 @@ class DBIter final: public Iterator {
  bool TooManyInternalKeysSkipped(bool increment = true);
  inline bool IsVisible(SequenceNumber sequence);

+  // CanReseekToSkip() returns whether the iterator can use the optimization
+  // where it reseek by sequence number to get the next key when there are too
+  // many versions. This is disabled for write unprepared because seeking to
+  // sequence number does not guarantee that it is visible.
+  inline bool CanReseekToSkip();
+
  // Temporarily pin the blocks that we encounter until ReleaseTempPinnedData()
  // is called
  void TempPinData() {
@@ -447,11 +453,6 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)
  //                         greater than that,
  //  - none of the above  : saved_key_ can contain anything, it doesn't matter.
  uint64_t num_skipped = 0;
-  // For write unprepared, the target sequence number in reseek could be larger
-  // than the snapshot, and thus needs to be skipped again. This could result in
-  // an infinite loop of reseeks. To avoid that, we limit the number of reseeks
-  // to one.
-  bool reseek_done = false;

  is_blob_ = false;

@@ -466,9 +467,7 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)

    is_key_seqnum_zero_ = (ikey_.sequence == 0);

-    assert(iterate_upper_bound_ == nullptr || iter_.MayBeOutOfUpperBound() ||
-           user_comparator_.Compare(ikey_.user_key, *iterate_upper_bound_) < 0);
-    if (iterate_upper_bound_ != nullptr && iter_.MayBeOutOfUpperBound() &&
+    if (iterate_upper_bound_ != nullptr &&
        user_comparator_.Compare(ikey_.user_key, *iterate_upper_bound_) >= 0) {
      break;
    }
@@ -497,7 +496,6 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)
        assert(!skipping || user_comparator_.Compare(
                                ikey_.user_key, saved_key_.GetUserKey()) > 0);
        num_skipped = 0;
-        reseek_done = false;
        switch (ikey_.type) {
          case kTypeDeletion:
          case kTypeSingleDeletion:
@@ -551,7 +549,6 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)
                // they are hidden by this deletion.
                skipping = true;
                num_skipped = 0;
-                reseek_done = false;
                PERF_COUNTER_ADD(internal_delete_skipped_count, 1);
              } else if (ikey_.type == kTypeBlobIndex) {
                if (!allow_blob_) {
@@ -582,7 +579,6 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)
              // they are hidden by this deletion.
              skipping = true;
              num_skipped = 0;
-              reseek_done = false;
              PERF_COUNTER_ADD(internal_delete_skipped_count, 1);
            } else {
              // By now, we are sure the current ikey is going to yield a
@@ -613,23 +609,14 @@ inline bool DBIter::FindNextUserEntryInternal(bool skipping, bool prefix_check)
            !iter_.iter()->IsKeyPinned() || !pin_thru_lifetime_ /* copy */);
        skipping = false;
        num_skipped = 0;
-        reseek_done = false;
      }
    }

    // If we have sequentially iterated via numerous equal keys, then it's
    // better to seek so that we can avoid too many key comparisons.
-    //
-    // To avoid infinite loops, do not reseek if we have already attempted to
-    // reseek previously.
-    //
-    // TODO(lth): If we reseek to sequence number greater than ikey_.sequence,
-    // than it does not make sense to reseek as we would actually land further
-    // away from the desired key. There is opportunity for optimization here.
-    if (num_skipped > max_skip_ && !reseek_done) {
+    if (num_skipped > max_skip_ && CanReseekToSkip()) {
      is_key_seqnum_zero_ = false;
      num_skipped = 0;
-      reseek_done = true;
      std::string last_key;
      if (skipping) {
        // We're looking for the next user-key but all we see are the same
@@ -872,10 +859,7 @@ void DBIter::PrevInternal() {
      return;
    }

-    assert(iterate_lower_bound_ == nullptr || iter_.MayBeOutOfLowerBound() ||
-           user_comparator_.Compare(saved_key_.GetUserKey(),
-                                    *iterate_lower_bound_) >= 0);
-    if (iterate_lower_bound_ != nullptr && iter_.MayBeOutOfLowerBound() &&
+    if (iterate_lower_bound_ != nullptr &&
        user_comparator_.Compare(saved_key_.GetUserKey(),
                                 *iterate_lower_bound_) < 0) {
      // We've iterated earlier than the user-specified lower bound.
@@ -948,7 +932,7 @@ bool DBIter::FindValueForCurrentKey() {
    // This user key has lots of entries.
    // We're going from old to new, and it's taking too long. Let's do a Seek()
    // and go from new to old. This helps when a key was overwritten many times.
-    if (num_skipped >= max_skip_) {
+    if (num_skipped >= max_skip_ && CanReseekToSkip()) {
      return FindValueForCurrentKeyUsingSeek();
    }

@@ -1245,7 +1229,7 @@ bool DBIter::FindUserKeyBeforeSavedKey() {
      PERF_COUNTER_ADD(internal_key_skipped_count, 1);
    }

-    if (num_skipped >= max_skip_) {
+    if (num_skipped >= max_skip_ && CanReseekToSkip()) {
      num_skipped = 0;
      IterKey last_key;
      last_key.SetInternalKey(ParsedInternalKey(
@@ -1292,6 +1276,10 @@ bool DBIter::IsVisible(SequenceNumber sequence) {
  }
 }

+bool DBIter::CanReseekToSkip() {
+  return read_callback_ == nullptr || read_callback_->CanReseekToSkip();
+}
+
 void DBIter::Seek(const Slice& target) {
  PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, env_);
  StopWatch sw(env_, statistics_, DB_SEEK);
@@ -182,33 +182,6 @@ TEST_P(DBIteratorTest, IterSeekBeforePrev) {
  delete iter;
 }

-TEST_P(DBIteratorTest, IterReseekNewUpperBound) {
-  Random rnd(301);
-  Options options = CurrentOptions();
-  BlockBasedTableOptions table_options;
-  table_options.block_size = 1024;
-  table_options.block_size_deviation = 50;
-  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-  options.compression = kNoCompression;
-  Reopen(options);
-
-  ASSERT_OK(Put("a", RandomString(&rnd, 400)));
-  ASSERT_OK(Put("aabb", RandomString(&rnd, 400)));
-  ASSERT_OK(Put("aaef", RandomString(&rnd, 400)));
-  ASSERT_OK(Put("b", RandomString(&rnd, 400)));
-  dbfull()->Flush(FlushOptions());
-  ReadOptions opts;
-  Slice ub = Slice("aa");
-  opts.iterate_upper_bound = &ub;
-  auto iter = NewIterator(opts);
-  iter->Seek(Slice("a"));
-  ub = Slice("b");
-  iter->Seek(Slice("aabc"));
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_EQ(iter->key().ToString(), "aaef");
-  delete iter;
-}
-
 TEST_P(DBIteratorTest, IterSeekForPrevBeforeNext) {
  ASSERT_OK(Put("a", "b"));
  ASSERT_OK(Put("c", "d"));
@@ -1076,148 +1049,6 @@ TEST_P(DBIteratorTest, DBIteratorBoundOptimizationTest) {
    ASSERT_EQ(upper_bound_hits, 1);
  }
 }
-
-// Enable kBinarySearchWithFirstKey, do some iterator operations and check that
-// they don't do unnecessary block reads.
-TEST_P(DBIteratorTest, IndexWithFirstKey) {
-  for (int tailing = 0; tailing < 2; ++tailing) {
-    SCOPED_TRACE("tailing = " + std::to_string(tailing));
-    Options options = CurrentOptions();
-    options.env = env_;
-    options.create_if_missing = true;
-    options.prefix_extractor = nullptr;
-    options.merge_operator = MergeOperators::CreateStringAppendOperator();
-    options.statistics = rocksdb::CreateDBStatistics();
-    Statistics* stats = options.statistics.get();
-    BlockBasedTableOptions table_options;
-    table_options.index_type =
-        BlockBasedTableOptions::IndexType::kBinarySearchWithFirstKey;
-    table_options.index_shortening =
-        BlockBasedTableOptions::IndexShorteningMode::kNoShortening;
-    table_options.flush_block_policy_factory =
-        std::make_shared<FlushBlockEveryKeyPolicyFactory>();
-    table_options.block_cache = NewLRUCache(1000);  // fits all blocks
-    options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-
-    DestroyAndReopen(options);
-    ASSERT_OK(Merge("a1", "x1"));
-    ASSERT_OK(Merge("b1", "y1"));
-    ASSERT_OK(Merge("c0", "z1"));
-    ASSERT_OK(Flush());
-    ASSERT_OK(Merge("a2", "x2"));
-    ASSERT_OK(Merge("b2", "y2"));
-    ASSERT_OK(Merge("c0", "z2"));
-    ASSERT_OK(Flush());
-    ASSERT_OK(Merge("a3", "x3"));
-    ASSERT_OK(Merge("b3", "y3"));
-    ASSERT_OK(Merge("c3", "z3"));
-    ASSERT_OK(Flush());
-
-    // Block cache is not important for this test.
-    // We use BLOCK_CACHE_DATA_* counters just because they're the most readily
-    // available way of counting block accesses.
-
-    ReadOptions ropt;
-    ropt.tailing = tailing;
-    std::unique_ptr<Iterator> iter(NewIterator(ropt));
-
-    iter->Seek("b10");
-    ASSERT_TRUE(iter->Valid());
-    EXPECT_EQ("b2", iter->key().ToString());
-    EXPECT_EQ("y2", iter->value().ToString());
-    EXPECT_EQ(1, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-
-    iter->Next();
-    ASSERT_TRUE(iter->Valid());
-    EXPECT_EQ("b3", iter->key().ToString());
-    EXPECT_EQ("y3", iter->value().ToString());
-    EXPECT_EQ(2, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-    EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-
-    iter->Seek("c0");
-    ASSERT_TRUE(iter->Valid());
-    EXPECT_EQ("c0", iter->key().ToString());
-    EXPECT_EQ("z1,z2", iter->value().ToString());
-    EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-    EXPECT_EQ(4, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-
-    iter->Next();
-    ASSERT_TRUE(iter->Valid());
-    EXPECT_EQ("c3", iter->key().ToString());
-    EXPECT_EQ("z3", iter->value().ToString());
-    EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-    EXPECT_EQ(5, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-
-    iter.reset();
-
-    // Enable iterate_upper_bound and check that iterator is not trying to read
-    // blocks that are fully above upper bound.
-    std::string ub = "b3";
-    Slice ub_slice(ub);
-    ropt.iterate_upper_bound = &ub_slice;
-    iter.reset(NewIterator(ropt));
-
-    iter->Seek("b2");
-    ASSERT_TRUE(iter->Valid());
-    EXPECT_EQ("b2", iter->key().ToString());
-    EXPECT_EQ("y2", iter->value().ToString());
-    EXPECT_EQ(1, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-    EXPECT_EQ(5, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-
-    iter->Next();
-    ASSERT_FALSE(iter->Valid());
-    EXPECT_EQ(1, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-    EXPECT_EQ(5, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-  }
-}
-
-TEST_P(DBIteratorTest, IndexWithFirstKeyGet) {
-  Options options = CurrentOptions();
-  options.env = env_;
-  options.create_if_missing = true;
-  options.prefix_extractor = nullptr;
-  options.merge_operator = MergeOperators::CreateStringAppendOperator();
-  options.statistics = rocksdb::CreateDBStatistics();
-  Statistics* stats = options.statistics.get();
-  BlockBasedTableOptions table_options;
-  table_options.index_type =
-      BlockBasedTableOptions::IndexType::kBinarySearchWithFirstKey;
-  table_options.index_shortening =
-      BlockBasedTableOptions::IndexShorteningMode::kNoShortening;
-  table_options.flush_block_policy_factory =
-      std::make_shared<FlushBlockEveryKeyPolicyFactory>();
-  table_options.block_cache = NewLRUCache(1000);  // fits all blocks
-  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-
-  DestroyAndReopen(options);
-  ASSERT_OK(Merge("a", "x1"));
-  ASSERT_OK(Merge("c", "y1"));
-  ASSERT_OK(Merge("e", "z1"));
-  ASSERT_OK(Flush());
-  ASSERT_OK(Merge("c", "y2"));
-  ASSERT_OK(Merge("e", "z2"));
-  ASSERT_OK(Flush());
-
-  // Get() between blocks shouldn't read any blocks.
-  ASSERT_EQ("NOT_FOUND", Get("b"));
-  EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-  EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-
-  // Get() of an existing key shouldn't read any unnecessary blocks when there's
-  // only one key per block.
-
-  ASSERT_EQ("y1,y2", Get("c"));
-  EXPECT_EQ(2, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-  EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-
-  ASSERT_EQ("x1", Get("a"));
-  EXPECT_EQ(3, stats->getTickerCount(BLOCK_CACHE_DATA_MISS));
-  EXPECT_EQ(0, stats->getTickerCount(BLOCK_CACHE_DATA_HIT));
-
-  EXPECT_EQ(std::vector<std::string>({"NOT_FOUND", "z1,z2"}),
-            MultiGet({"b", "e"}));
-}
-
 // TODO(3.13): fix the issue of Seek() + Prev() which might not necessary
 //             return the biggest key which is smaller than the seek key.
 TEST_P(DBIteratorTest, PrevAfterAndNextAfterMerge) {
@@ -2717,94 +2548,6 @@ TEST_P(DBIteratorTest, AvoidReseekLevelIterator) {
  SyncPoint::GetInstance()->DisableProcessing();
 }

-// MyRocks may change iterate bounds before seek. Simply test to make sure such
-// usage doesn't break iterator.
-TEST_P(DBIteratorTest, IterateBoundChangedBeforeSeek) {
-  Options options = CurrentOptions();
-  options.compression = CompressionType::kNoCompression;
-  BlockBasedTableOptions table_options;
-  table_options.block_size = 100;
-  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-  std::string value(50, 'v');
-  Reopen(options);
-  ASSERT_OK(Put("aaa", value));
-  ASSERT_OK(Flush());
-  ASSERT_OK(Put("bbb", "v"));
-  ASSERT_OK(Put("ccc", "v"));
-  ASSERT_OK(Put("ddd", "v"));
-  ASSERT_OK(Flush());
-  ASSERT_OK(Put("eee", "v"));
-  ASSERT_OK(Flush());
-  ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
-
-  std::string ub1 = "e";
-  std::string ub2 = "c";
-  Slice ub(ub1);
-  ReadOptions read_opts1;
-  read_opts1.iterate_upper_bound = &ub;
-  Iterator* iter = NewIterator(read_opts1);
-  // Seek and iterate accross block boundary.
-  iter->Seek("b");
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("bbb", iter->key());
-  ub = Slice(ub2);
-  iter->Seek("b");
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("bbb", iter->key());
-  iter->Next();
-  ASSERT_FALSE(iter->Valid());
-  ASSERT_OK(iter->status());
-  delete iter;
-
-  std::string lb1 = "a";
-  std::string lb2 = "c";
-  Slice lb(lb1);
-  ReadOptions read_opts2;
-  read_opts2.iterate_lower_bound = &lb;
-  iter = NewIterator(read_opts2);
-  iter->SeekForPrev("d");
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("ccc", iter->key());
-  lb = Slice(lb2);
-  iter->SeekForPrev("d");
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("ccc", iter->key());
-  iter->Prev();
-  ASSERT_FALSE(iter->Valid());
-  ASSERT_OK(iter->status());
-  delete iter;
-}
-
-TEST_P(DBIteratorTest, IterateWithLowerBoundAcrossFileBoundary) {
-  ASSERT_OK(Put("aaa", "v"));
-  ASSERT_OK(Put("bbb", "v"));
-  ASSERT_OK(Flush());
-  ASSERT_OK(Put("ccc", "v"));
-  ASSERT_OK(Put("ddd", "v"));
-  ASSERT_OK(Flush());
-  // Move both files to bottom level.
-  ASSERT_OK(dbfull()->CompactRange(CompactRangeOptions(), nullptr, nullptr));
-  Slice lower_bound("b");
-  ReadOptions read_opts;
-  read_opts.iterate_lower_bound = &lower_bound;
-  std::unique_ptr<Iterator> iter(NewIterator(read_opts));
-  iter->SeekForPrev("d");
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("ccc", iter->key());
-  iter->Prev();
-  ASSERT_TRUE(iter->Valid());
-  ASSERT_OK(iter->status());
-  ASSERT_EQ("bbb", iter->key());
-  iter->Prev();
-  ASSERT_FALSE(iter->Valid());
-  ASSERT_OK(iter->status());
-}
-
 INSTANTIATE_TEST_CASE_P(DBIteratorTestInstance, DBIteratorTest,
                        testing::Values(true, false));

@@ -2492,16 +2492,6 @@ class ModelDB : public DB {
    return Status::NotSupported("Not implemented");
  }

-  using DB::CreateColumnFamilyWithImport;
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& /*options*/,
-      const std::string& /*column_family_name*/,
-      const ImportColumnFamilyOptions& /*import_options*/,
-      const ExportImportFilesMetaData& /*metadata*/,
-      ColumnFamilyHandle** /*handle*/) override {
-    return Status::NotSupported("Not implemented.");
-  }
-
  Status VerifyChecksum() override {
    return Status::NotSupported("Not implemented.");
  }
@@ -2598,14 +2588,13 @@ class ModelDB : public DB {
    return false;
  }
  using DB::GetApproximateSizes;
-  Status GetApproximateSizes(const SizeApproximationOptions& /*options*/,
-                             ColumnFamilyHandle* /*column_family*/,
-                             const Range* /*range*/, int n,
-                             uint64_t* sizes) override {
+  void GetApproximateSizes(ColumnFamilyHandle* /*column_family*/,
+                           const Range* /*range*/, int n, uint64_t* sizes,
+                           uint8_t /*include_flags*/
+                           = INCLUDE_FILES) override {
    for (int i = 0; i < n; i++) {
      sizes[i] = 0;
    }
-    return Status::OK();
  }
  using DB::GetApproximateMemTableStats;
  void GetApproximateMemTableStats(ColumnFamilyHandle* /*column_family*/,
@@ -6167,30 +6156,10 @@ TEST_F(DBTest, ThreadLocalPtrDeadlock) {
  fprintf(stderr, "Done. Flushed %d times, destroyed %d threads\n",
          flushes_done.load(), threads_destroyed.load());
 }
-
-TEST_F(DBTest, LargeBlockSizeTest) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"pikachu"}, options);
-  ASSERT_OK(Put(0, "foo", "bar"));
-  BlockBasedTableOptions table_options;
-  table_options.block_size = 8LL*1024*1024*1024LL;
-  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
-  ASSERT_NOK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
-}
-
 }  // namespace rocksdb

-#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-extern "C" {
-void RegisterCustomObjects(int argc, char** argv);
-}
-#else
-void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
-#endif  // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-
 int main(int argc, char** argv) {
  rocksdb::port::InstallStackTraceHandler();
  ::testing::InitGoogleTest(&argc, argv);
-  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
@@ -3811,65 +3811,10 @@ TEST_F(DBTest2, PrefixBloomReseek) {

  delete iter;
 }
-
-#ifndef ROCKSDB_LITE
-TEST_F(DBTest2, RowCacheSnapshot) {
-  Options options = CurrentOptions();
-  options.statistics = rocksdb::CreateDBStatistics();
-  options.row_cache = NewLRUCache(8192);
-  DestroyAndReopen(options);
-
-  ASSERT_OK(Put("foo", "bar1"));
-
-  const Snapshot* s1 = db_->GetSnapshot();
-
-  ASSERT_OK(Put("foo", "bar2"));
-  ASSERT_OK(Flush());
-
-  ASSERT_OK(Put("foo2", "bar"));
-  const Snapshot* s2 = db_->GetSnapshot();
-  ASSERT_OK(Put("foo3", "bar"));
-  const Snapshot* s3 = db_->GetSnapshot();
-
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 0);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 0);
-  ASSERT_EQ(Get("foo"), "bar2");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 0);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 1);
-  ASSERT_EQ(Get("foo"), "bar2");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 1);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 1);
-  ASSERT_EQ(Get("foo", s1), "bar1");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 1);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 2);
-  ASSERT_EQ(Get("foo", s2), "bar2");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 2);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 2);
-  ASSERT_EQ(Get("foo", s1), "bar1");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 3);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 2);
-  ASSERT_EQ(Get("foo", s3), "bar2");
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_HIT), 4);
-  ASSERT_EQ(TestGetTickerCount(options, ROW_CACHE_MISS), 2);
-
-  db_->ReleaseSnapshot(s1);
-  db_->ReleaseSnapshot(s2);
-  db_->ReleaseSnapshot(s3);
-}
-#endif  // ROCKSDB_LITE
 }  // namespace rocksdb

-#ifdef ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-extern "C" {
-void RegisterCustomObjects(int argc, char** argv);
-}
-#else
-void RegisterCustomObjects(int /*argc*/, char** /*argv*/) {}
-#endif  // !ROCKSDB_UNITTESTS_WITH_CUSTOM_OBJECTS_FROM_STATIC_LIBS
-
 int main(int argc, char** argv) {
  rocksdb::port::InstallStackTraceHandler();
  ::testing::InitGoogleTest(&argc, argv);
-  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
@@ -10,7 +10,6 @@
 #include "db/db_test_util.h"
 #include "db/forward_iterator.h"
 #include "rocksdb/env_encryption.h"
-#include "rocksdb/utilities/object_registry.h"

 namespace rocksdb {

@@ -48,32 +47,20 @@ ROT13BlockCipher rot13Cipher_(16);
 #endif  // ROCKSDB_LITE

 DBTestBase::DBTestBase(const std::string path)
-    : mem_env_(nullptr),
+    : mem_env_(!getenv("MEM_ENV") ? nullptr : new MockEnv(Env::Default())),
+#ifndef ROCKSDB_LITE
+      encrypted_env_(
+          !getenv("ENCRYPTED_ENV")
+              ? nullptr
+              : NewEncryptedEnv(mem_env_ ? mem_env_ : Env::Default(),
+                                new CTREncryptionProvider(rot13Cipher_))),
+#else
      encrypted_env_(nullptr),
+#endif  // ROCKSDB_LITE
+      env_(new SpecialEnv(encrypted_env_
+                              ? encrypted_env_
+                              : (mem_env_ ? mem_env_ : Env::Default()))),
      option_config_(kDefault) {
-  Env* base_env = Env::Default();
-#ifndef ROCKSDB_LITE
-  const char* test_env_uri = getenv("TEST_ENV_URI");
-  if (test_env_uri) {
-    Status s = ObjectRegistry::NewInstance()->NewSharedObject(test_env_uri,
-                                                              &env_guard_);
-    base_env = env_guard_.get();
-    EXPECT_OK(s);
-    EXPECT_NE(Env::Default(), base_env);
-  }
-#endif  // !ROCKSDB_LITE
-  EXPECT_NE(nullptr, base_env);
-  if (getenv("MEM_ENV")) {
-    mem_env_ = new MockEnv(base_env);
-  }
-#ifndef ROCKSDB_LITE
-  if (getenv("ENCRYPTED_ENV")) {
-    encrypted_env_ = NewEncryptedEnv(mem_env_ ? mem_env_ : base_env,
-                                     new CTREncryptionProvider(rot13Cipher_));
-  }
-#endif  // !ROCKSDB_LITE
-  env_ = new SpecialEnv(encrypted_env_ ? encrypted_env_
-                                       : (mem_env_ ? mem_env_ : base_env));
  env_->SetBackgroundThreads(1, Env::LOW);
  env_->SetBackgroundThreads(1, Env::HIGH);
  dbname_ = test::PerThreadDBPath(env_, path);
@@ -702,7 +702,6 @@ class DBTestBase : public testing::Test {
  MockEnv* mem_env_;
  Env* encrypted_env_;
  SpecialEnv* env_;
-  std::shared_ptr<Env> env_guard_;
  DB* db_;
  std::vector<ColumnFamilyHandle*> handles_;

@@ -824,9 +824,7 @@ class RecoveryTestHelper {
  // Create WAL files with values filled in
  static void FillData(DBWALTest* test, const Options& options,
                       const size_t wal_count, size_t* count) {
-    // Calling internal functions requires sanitized options.
-    Options sanitized_options = SanitizeOptions(test->dbname_, options);
-    const ImmutableDBOptions db_options(sanitized_options);
+    const ImmutableDBOptions db_options(options);

    *count = 0;

@@ -159,11 +159,9 @@ void InternalKeyComparator::FindShortSuccessor(std::string* key) const {
  }
 }

-LookupKey::LookupKey(const Slice& _user_key, SequenceNumber s,
-                     const Slice* ts) {
+LookupKey::LookupKey(const Slice& _user_key, SequenceNumber s) {
  size_t usize = _user_key.size();
-  size_t ts_sz = (nullptr == ts) ? 0 : ts->size();
-  size_t needed = usize + ts_sz + 13;  // A conservative estimate
+  size_t needed = usize + 13;  // A conservative estimate
  char* dst;
  if (needed <= sizeof(space_)) {
    dst = space_;
@@ -172,14 +170,10 @@ LookupKey::LookupKey(const Slice& _user_key, SequenceNumber s,
  }
  start_ = dst;
  // NOTE: We don't support users keys of more than 2GB :)
-  dst = EncodeVarint32(dst, static_cast<uint32_t>(usize + ts_sz + 8));
+  dst = EncodeVarint32(dst, static_cast<uint32_t>(usize + 8));
  kstart_ = dst;
  memcpy(dst, _user_key.data(), usize);
  dst += usize;
-  if (nullptr != ts) {
-    memcpy(dst, ts->data(), ts_sz);
-    dst += ts_sz;
-  }
  EncodeFixed64(dst, PackSequenceAndType(s, kValueTypeForSeek));
  dst += 8;
  end_ = dst;
@@ -669,4 +669,20 @@ struct ParsedInternalKeyComparator {
  const InternalKeyComparator* cmp;
 };

+// TODO (yanqin): this causes extra memory allocation and copy. Should be
+// addressed in the future.
+inline Status AppendTimestamp(const Slice& key, const Slice& timestamp,
+                              Slice* ret_key, std::string* ret_buf) {
+  assert(ret_key != nullptr);
+  assert(ret_buf != nullptr);
+  if (key.data() + key.size() == timestamp.data()) {
+    *ret_key = Slice(key.data(), key.size() + timestamp.size());
+  } else {
+    ret_buf->assign(key.data(), key.size());
+    ret_buf->append(timestamp.data(), timestamp.size());
+    *ret_key = Slice(*ret_buf);
+  }
+  return Status::OK();
+}
+
 }  // namespace rocksdb
@@ -106,7 +106,7 @@ void EventHelpers::LogAndNotifyTableFileCreationFinished(
              << "num_entries" << table_properties.num_entries
              << "num_deletions" << table_properties.num_deletions
              << "num_merge_operands" << table_properties.num_merge_operands
-              << "num_range_deletions" << table_properties.num_range_deletions
+              << "num_range_deletions" << table_properties.num_merge_operands
              << "format_version" << table_properties.format_version
              << "fixed_key_len" << table_properties.fixed_key_len
              << "filter_policy" << table_properties.filter_policy_name
@@ -9,7 +9,6 @@
 #include "port/port.h"
 #include "port/stack_trace.h"
 #include "rocksdb/sst_file_writer.h"
-#include "test_util/fault_injection_test_env.h"
 #include "test_util/testutil.h"

 namespace rocksdb {
@@ -21,7 +20,6 @@ class ExternalSSTFileBasicTest
 public:
  ExternalSSTFileBasicTest() : DBTestBase("/external_sst_file_basic_test") {
    sst_files_dir_ = dbname_ + "/sst_files/";
-    fault_injection_test_env_.reset(new FaultInjectionTestEnv(Env::Default()));
    DestroyAndRecreateExternalSSTFilesDir();
  }

@@ -142,7 +140,6 @@ class ExternalSSTFileBasicTest

 protected:
  std::string sst_files_dir_;
-  std::unique_ptr<FaultInjectionTestEnv> fault_injection_test_env_;
 };

 TEST_F(ExternalSSTFileBasicTest, Basic) {
@@ -692,59 +689,6 @@ TEST_F(ExternalSSTFileBasicTest, FadviseTrigger) {
  rocksdb::SyncPoint::GetInstance()->DisableProcessing();
 }

-TEST_F(ExternalSSTFileBasicTest, SyncFailure) {
-  Options options;
-  options.create_if_missing = true;
-  options.env = fault_injection_test_env_.get();
-
-  std::vector<std::pair<std::string, std::string>> test_cases = {
-      {"ExternalSstFileIngestionJob::BeforeSyncIngestedFile",
-       "ExternalSstFileIngestionJob::AfterSyncIngestedFile"},
-      {"ExternalSstFileIngestionJob::BeforeSyncDir",
-       "ExternalSstFileIngestionJob::AfterSyncDir"},
-      {"ExternalSstFileIngestionJob::BeforeSyncGlobalSeqno",
-       "ExternalSstFileIngestionJob::AfterSyncGlobalSeqno"}};
-
-  for (size_t i = 0; i < test_cases.size(); i++) {
-    SyncPoint::GetInstance()->SetCallBack(test_cases[i].first, [&](void*) {
-      fault_injection_test_env_->SetFilesystemActive(false);
-    });
-    SyncPoint::GetInstance()->SetCallBack(test_cases[i].second, [&](void*) {
-      fault_injection_test_env_->SetFilesystemActive(true);
-    });
-    SyncPoint::GetInstance()->EnableProcessing();
-
-    DestroyAndReopen(options);
-    if (i == 2) {
-      ASSERT_OK(Put("foo", "v1"));
-    }
-
-    Options sst_file_writer_options;
-    std::unique_ptr<SstFileWriter> sst_file_writer(
-        new SstFileWriter(EnvOptions(), sst_file_writer_options));
-    std::string file_name =
-        sst_files_dir_ + "sync_failure_test_" + ToString(i) + ".sst";
-    ASSERT_OK(sst_file_writer->Open(file_name));
-    ASSERT_OK(sst_file_writer->Put("bar", "v2"));
-    ASSERT_OK(sst_file_writer->Finish());
-
-    IngestExternalFileOptions ingest_opt;
-    if (i == 0) {
-      ingest_opt.move_files = true;
-    }
-    const Snapshot* snapshot = db_->GetSnapshot();
-    if (i == 2) {
-      ingest_opt.write_global_seqno = true;
-    }
-    ASSERT_FALSE(db_->IngestExternalFile({file_name}, ingest_opt).ok());
-    db_->ReleaseSnapshot(snapshot);
-
-    SyncPoint::GetInstance()->DisableProcessing();
-    SyncPoint::GetInstance()->ClearAllCallBacks();
-    Destroy(options);
-  }
-}
-
 TEST_P(ExternalSSTFileBasicTest, IngestionWithRangeDeletions) {
  int kNumLevels = 7;
  Options options = CurrentOptions();
@@ -7,13 +7,11 @@

 #include "db/external_sst_file_ingestion_job.h"

-#include <algorithm>
 #include <cinttypes>
+#include <algorithm>
 #include <string>
-#include <unordered_set>
 #include <vector>

-#include "db/db_impl/db_impl.h"
 #include "db/version_edit.h"
 #include "file/file_util.h"
 #include "table/merging_iterator.h"
@@ -88,7 +86,6 @@ Status ExternalSstFileIngestionJob::Prepare(
  }

  // Copy/Move external files into DB
-  std::unordered_set<size_t> ingestion_path_ids;
  for (IngestedFileInfo& f : files_to_ingest_) {
    f.fd = FileDescriptor(next_file_number++, 0, f.file_size);
    f.copy_file = false;
@@ -98,26 +95,8 @@ Status ExternalSstFileIngestionJob::Prepare(
                      f.fd.GetPathId());
    if (ingestion_options_.move_files) {
      status = env_->LinkFile(path_outside_db, path_inside_db);
-      if (status.ok()) {
-        // It is unsafe to assume application had sync the file and file
-        // directory before ingest the file. For integrity of RocksDB we need
-        // to sync the file.
-        std::unique_ptr<WritableFile> file_to_sync;
-        status = env_->ReopenWritableFile(path_inside_db, &file_to_sync,
-                                          env_options_);
-        if (status.ok()) {
-          TEST_SYNC_POINT(
-              "ExternalSstFileIngestionJob::BeforeSyncIngestedFile");
-          status = SyncIngestedFile(file_to_sync.get());
-          TEST_SYNC_POINT("ExternalSstFileIngestionJob::AfterSyncIngestedFile");
-          if (!status.ok()) {
-            ROCKS_LOG_WARN(db_options_.info_log,
-                           "Failed to sync ingested file %s: %s",
-                           path_inside_db.c_str(), status.ToString().c_str());
-          }
-        }
-      } else if (status.IsNotSupported() &&
-                 ingestion_options_.failed_move_fall_back_to_copy) {
+      if (status.IsNotSupported() &&
+          ingestion_options_.failed_move_fall_back_to_copy) {
        // Original file is on a different FS, use copy instead of hard linking.
        f.copy_file = true;
      }
@@ -128,7 +107,6 @@ Status ExternalSstFileIngestionJob::Prepare(
    if (f.copy_file) {
      TEST_SYNC_POINT_CALLBACK("ExternalSstFileIngestionJob::Prepare:CopyFile",
                               nullptr);
-      // CopyFile also sync the new file.
      status = CopyFile(env_, path_outside_db, path_inside_db, 0,
                        db_options_.use_fsync);
    }
@@ -137,25 +115,8 @@ Status ExternalSstFileIngestionJob::Prepare(
      break;
    }
    f.internal_file_path = path_inside_db;
-    ingestion_path_ids.insert(f.fd.GetPathId());
  }

-  TEST_SYNC_POINT("ExternalSstFileIngestionJob::BeforeSyncDir");
-  if (status.ok()) {
-    for (auto path_id : ingestion_path_ids) {
-      status = directories_->GetDataDir(path_id)->Fsync();
-      if (!status.ok()) {
-        ROCKS_LOG_WARN(db_options_.info_log,
-                       "Failed to sync directory %" ROCKSDB_PRIszt
-                       " while ingest file: %s",
-                       path_id, status.ToString().c_str());
-        break;
-      }
-    }
-  }
-  TEST_SYNC_POINT("ExternalSstFileIngestionJob::AfterSyncDir");
-
-  // TODO: The following is duplicated with Cleanup().
  if (!status.ok()) {
    // We failed, remove all files that we copied into the db
    for (IngestedFileInfo& f : files_to_ingest_) {
@@ -350,8 +311,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
  }

  if (ingestion_options_.verify_checksums_before_ingest) {
-    status =
-        table_reader->VerifyChecksum(TableReaderCaller::kExternalSSTIngestion);
+    status = table_reader->VerifyChecksum();
  }
  if (!status.ok()) {
    return status;
@@ -411,8 +371,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
  // updating the block cache.
  ro.fill_cache = false;
  std::unique_ptr<InternalIterator> iter(table_reader->NewIterator(
-      ro, sv->mutable_cf_options.prefix_extractor.get(), /*arena=*/nullptr,
-      /*skip_filters=*/false, TableReaderCaller::kExternalSSTIngestion));
+      ro, sv->mutable_cf_options.prefix_extractor.get()));
  std::unique_ptr<InternalIterator> range_del_iter(
      table_reader->NewRangeTombstoneIterator(ro));

@@ -601,18 +560,6 @@ Status ExternalSstFileIngestionJob::AssignGlobalSeqnoForIngestedFile(
      std::string seqno_val;
      PutFixed64(&seqno_val, seqno);
      status = rwfile->Write(file_to_ingest->global_seqno_offset, seqno_val);
-      if (status.ok()) {
-        TEST_SYNC_POINT("ExternalSstFileIngestionJob::BeforeSyncGlobalSeqno");
-        status = SyncIngestedFile(rwfile.get());
-        TEST_SYNC_POINT("ExternalSstFileIngestionJob::AfterSyncGlobalSeqno");
-        if (!status.ok()) {
-          ROCKS_LOG_WARN(db_options_.info_log,
-                         "Failed to sync ingested file %s after writing global "
-                         "sequence number: %s",
-                         file_to_ingest->internal_file_path.c_str(),
-                         status.ToString().c_str());
-        }
-      }
      if (!status.ok()) {
        return status;
      }
@@ -653,16 +600,6 @@ bool ExternalSstFileIngestionJob::IngestedFileFitInLevel(
  return true;
 }

-template <typename TWritableFile>
-Status ExternalSstFileIngestionJob::SyncIngestedFile(TWritableFile* file) {
-  assert(file != nullptr);
-  if (db_options_.use_fsync) {
-    return file->Fsync();
-  } else {
-    return file->Sync();
-  }
-}
-
 }  // namespace rocksdb

 #endif  // !ROCKSDB_LITE
@@ -20,8 +20,6 @@

 namespace rocksdb {

-class Directories;
-
 struct IngestedFileInfo {
  // External file path
  std::string external_file_path;
@@ -79,8 +77,7 @@ class ExternalSstFileIngestionJob {
      Env* env, VersionSet* versions, ColumnFamilyData* cfd,
      const ImmutableDBOptions& db_options, const EnvOptions& env_options,
      SnapshotList* db_snapshots,
-      const IngestExternalFileOptions& ingestion_options,
-      Directories* directories)
+      const IngestExternalFileOptions& ingestion_options)
      : env_(env),
        versions_(versions),
        cfd_(cfd),
@@ -88,11 +85,8 @@ class ExternalSstFileIngestionJob {
        env_options_(env_options),
        db_snapshots_(db_snapshots),
        ingestion_options_(ingestion_options),
-        directories_(directories),
        job_start_time_(env_->NowMicros()),
-        consumed_seqno_(false) {
-    assert(directories != nullptr);
-  }
+        consumed_seqno_(false) {}

  // Prepare the job by copying external files into the DB.
  Status Prepare(const std::vector<std::string>& external_files_paths,
@@ -159,10 +153,6 @@ class ExternalSstFileIngestionJob {
  bool IngestedFileFitInLevel(const IngestedFileInfo* file_to_ingest,
                              int level);

-  // Helper method to sync given file.
-  template <typename TWritableFile>
-  Status SyncIngestedFile(TWritableFile* file);
-
  Env* env_;
  VersionSet* versions_;
  ColumnFamilyData* cfd_;
@@ -171,7 +161,6 @@ class ExternalSstFileIngestionJob {
  SnapshotList* db_snapshots_;
  autovector<IngestedFileInfo> files_to_ingest_;
  const IngestExternalFileOptions& ingestion_options_;
-  Directories* directories_;
  VersionEdit edit_;
  uint64_t job_start_time_;
  bool consumed_seqno_;
@@ -79,11 +79,7 @@ class ForwardLevelIterator : public InternalIterator {
        read_options_, *(cfd_->soptions()), cfd_->internal_comparator(),
        *files_[file_index_],
        read_options_.ignore_range_deletions ? nullptr : &range_del_agg,
-        prefix_extractor_, /*table_reader_ptr=*/nullptr,
-        /*file_read_hist=*/nullptr, TableReaderCaller::kUserIterator,
-        /*arena=*/nullptr, /*skip_filters=*/false, /*level=*/-1,
-        /*smallest_compaction_key=*/nullptr,
-        /*largest_compaction_key=*/nullptr);
+        prefix_extractor_, nullptr /* table_reader_ptr */, nullptr, false);
    file_iter_->SetPinnedItersMgr(pinned_iters_mgr_);
    valid_ = false;
    if (!range_del_agg.IsEmpty()) {
@@ -646,12 +642,7 @@ void ForwardIterator::RebuildIterators(bool refresh_sv) {
    l0_iters_.push_back(cfd_->table_cache()->NewIterator(
        read_options_, *cfd_->soptions(), cfd_->internal_comparator(), *l0,
        read_options_.ignore_range_deletions ? nullptr : &range_del_agg,
-        sv_->mutable_cf_options.prefix_extractor.get(),
-        /*table_reader_ptr=*/nullptr, /*file_read_hist=*/nullptr,
-        TableReaderCaller::kUserIterator, /*arena=*/nullptr,
-        /*skip_filters=*/false, /*level=*/-1,
-        /*smallest_compaction_key=*/nullptr,
-        /*largest_compaction_key=*/nullptr));
+        sv_->mutable_cf_options.prefix_extractor.get()));
  }
  BuildLevelIterators(vstorage);
  current_ = nullptr;
@@ -723,12 +714,7 @@ void ForwardIterator::RenewIterators() {
        read_options_, *cfd_->soptions(), cfd_->internal_comparator(),
        *l0_files_new[inew],
        read_options_.ignore_range_deletions ? nullptr : &range_del_agg,
-        svnew->mutable_cf_options.prefix_extractor.get(),
-        /*table_reader_ptr=*/nullptr, /*file_read_hist=*/nullptr,
-        TableReaderCaller::kUserIterator, /*arena=*/nullptr,
-        /*skip_filters=*/false, /*level=*/-1,
-        /*smallest_compaction_key=*/nullptr,
-        /*largest_compaction_key=*/nullptr));
+        svnew->mutable_cf_options.prefix_extractor.get()));
  }

  for (auto* f : l0_iters_) {
@@ -786,13 +772,8 @@ void ForwardIterator::ResetIncompleteIterators() {
    DeleteIterator(l0_iters_[i]);
    l0_iters_[i] = cfd_->table_cache()->NewIterator(
        read_options_, *cfd_->soptions(), cfd_->internal_comparator(),
-        *l0_files[i], /*range_del_agg=*/nullptr,
-        sv_->mutable_cf_options.prefix_extractor.get(),
-        /*table_reader_ptr=*/nullptr, /*file_read_hist=*/nullptr,
-        TableReaderCaller::kUserIterator, /*arena=*/nullptr,
-        /*skip_filters=*/false, /*level=*/-1,
-        /*smallest_compaction_key=*/nullptr,
-        /*largest_compaction_key=*/nullptr);
+        *l0_files[i], nullptr /* range_del_agg */,
+        sv_->mutable_cf_options.prefix_extractor.get());
    l0_iters_[i]->SetPinnedItersMgr(pinned_iters_mgr_);
  }

@@ -1,257 +0,0 @@
-#ifndef ROCKSDB_LITE
-
-#include "db/import_column_family_job.h"
-
-#include <cinttypes>
-#include <algorithm>
-#include <string>
-#include <vector>
-
-#include "db/version_edit.h"
-#include "file/file_util.h"
-#include "table/merging_iterator.h"
-#include "table/scoped_arena_iterator.h"
-#include "table/sst_file_writer_collectors.h"
-#include "table/table_builder.h"
-#include "util/file_reader_writer.h"
-#include "util/stop_watch.h"
-
-namespace rocksdb {
-
-Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,
-                                      SuperVersion* sv) {
-  Status status;
-
-  // Read the information of files we are importing
-  for (const auto& file_metadata : metadata_) {
-    const auto file_path = file_metadata.db_path + "/" + file_metadata.name;
-    IngestedFileInfo file_to_import;
-    status = GetIngestedFileInfo(file_path, &file_to_import, sv);
-    if (!status.ok()) {
-      return status;
-    }
-    files_to_import_.push_back(file_to_import);
-  }
-
-  const auto ucmp = cfd_->internal_comparator().user_comparator();
-  auto num_files = files_to_import_.size();
-  if (num_files == 0) {
-    return Status::InvalidArgument("The list of files is empty");
-  } else if (num_files > 1) {
-    // Verify that passed files don't have overlapping ranges in any particular
-    // level.
-    int min_level = 1;  // Check for overlaps in Level 1 and above.
-    int max_level = -1;
-    for (const auto& file_metadata : metadata_) {
-      if (file_metadata.level > max_level) {
-        max_level = file_metadata.level;
-      }
-    }
-    for (int level = min_level; level <= max_level; ++level) {
-      autovector<const IngestedFileInfo*> sorted_files;
-      for (size_t i = 0; i < num_files; i++) {
-        if (metadata_[i].level == level) {
-          sorted_files.push_back(&files_to_import_[i]);
-        }
-      }
-
-      std::sort(sorted_files.begin(), sorted_files.end(),
-                [&ucmp](const IngestedFileInfo* info1,
-                        const IngestedFileInfo* info2) {
-                  return ucmp->Compare(info1->smallest_user_key,
-                                       info2->smallest_user_key) < 0;
-                });
-
-      for (size_t i = 0; i < sorted_files.size() - 1; i++) {
-        if (ucmp->Compare(sorted_files[i]->largest_user_key,
-                          sorted_files[i + 1]->smallest_user_key) >= 0) {
-          return Status::InvalidArgument("Files have overlapping ranges");
-        }
-      }
-    }
-  }
-
-  for (const auto& f : files_to_import_) {
-    if (f.num_entries == 0) {
-      return Status::InvalidArgument("File contain no entries");
-    }
-
-    if (!f.smallest_internal_key().Valid() ||
-        !f.largest_internal_key().Valid()) {
-      return Status::Corruption("File has corrupted keys");
-    }
-  }
-
-  // Copy/Move external files into DB
-  auto hardlink_files = import_options_.move_files;
-  for (auto& f : files_to_import_) {
-    f.fd = FileDescriptor(next_file_number++, 0, f.file_size);
-
-    const auto path_outside_db = f.external_file_path;
-    const auto path_inside_db = TableFileName(
-        cfd_->ioptions()->cf_paths, f.fd.GetNumber(), f.fd.GetPathId());
-
-    if (hardlink_files) {
-      status = env_->LinkFile(path_outside_db, path_inside_db);
-      if (status.IsNotSupported()) {
-        // Original file is on a different FS, use copy instead of hard linking
-        hardlink_files = false;
-      }
-    }
-    if (!hardlink_files) {
-      status = CopyFile(env_, path_outside_db, path_inside_db, 0,
-                        db_options_.use_fsync);
-    }
-    if (!status.ok()) {
-      break;
-    }
-    f.copy_file = !hardlink_files;
-    f.internal_file_path = path_inside_db;
-  }
-
-  if (!status.ok()) {
-    // We failed, remove all files that we copied into the db
-    for (const auto& f : files_to_import_) {
-      if (f.internal_file_path.empty()) {
-        break;
-      }
-      const auto s = env_->DeleteFile(f.internal_file_path);
-      if (!s.ok()) {
-        ROCKS_LOG_WARN(db_options_.info_log,
-                       "AddFile() clean up for file %s failed : %s",
-                       f.internal_file_path.c_str(), s.ToString().c_str());
-      }
-    }
-  }
-
-  return status;
-}
-
-// REQUIRES: we have become the only writer by entering both write_thread_ and
-// nonmem_write_thread_
-Status ImportColumnFamilyJob::Run() {
-  Status status;
-  edit_.SetColumnFamily(cfd_->GetID());
-
-  for (size_t i = 0; i < files_to_import_.size(); ++i) {
-    const auto& f = files_to_import_[i];
-    const auto& file_metadata = metadata_[i];
-    edit_.AddFile(file_metadata.level, f.fd.GetNumber(), f.fd.GetPathId(),
-                  f.fd.GetFileSize(), f.smallest_internal_key(),
-                  f.largest_internal_key(), file_metadata.smallest_seqno,
-                  file_metadata.largest_seqno, false);
-
-    // If incoming sequence number is higher, update local sequence number.
-    if (file_metadata.largest_seqno > versions_->LastSequence()) {
-      versions_->SetLastAllocatedSequence(file_metadata.largest_seqno);
-      versions_->SetLastPublishedSequence(file_metadata.largest_seqno);
-      versions_->SetLastSequence(file_metadata.largest_seqno);
-    }
-  }
-
-  return status;
-}
-
-void ImportColumnFamilyJob::Cleanup(const Status& status) {
-  if (!status.ok()) {
-    // We failed to add files to the database remove all the files we copied.
-    for (const auto& f : files_to_import_) {
-      const auto s = env_->DeleteFile(f.internal_file_path);
-      if (!s.ok()) {
-        ROCKS_LOG_WARN(db_options_.info_log,
-                       "AddFile() clean up for file %s failed : %s",
-                       f.internal_file_path.c_str(), s.ToString().c_str());
-      }
-    }
-  } else if (status.ok() && import_options_.move_files) {
-    // The files were moved and added successfully, remove original file links
-    for (IngestedFileInfo& f : files_to_import_) {
-      const auto s = env_->DeleteFile(f.external_file_path);
-      if (!s.ok()) {
-        ROCKS_LOG_WARN(
-            db_options_.info_log,
-            "%s was added to DB successfully but failed to remove original "
-            "file link : %s",
-            f.external_file_path.c_str(), s.ToString().c_str());
-      }
-    }
-  }
-}
-
-Status ImportColumnFamilyJob::GetIngestedFileInfo(
-    const std::string& external_file, IngestedFileInfo* file_to_import,
-    SuperVersion* sv) {
-  file_to_import->external_file_path = external_file;
-
-  // Get external file size
-  auto status = env_->GetFileSize(external_file, &file_to_import->file_size);
-  if (!status.ok()) {
-    return status;
-  }
-
-  // Create TableReader for external file
-  std::unique_ptr<TableReader> table_reader;
-  std::unique_ptr<RandomAccessFile> sst_file;
-  std::unique_ptr<RandomAccessFileReader> sst_file_reader;
-
-  status = env_->NewRandomAccessFile(external_file, &sst_file, env_options_);
-  if (!status.ok()) {
-    return status;
-  }
-  sst_file_reader.reset(new RandomAccessFileReader(std::move(sst_file),
-                                                   external_file));
-
-  status = cfd_->ioptions()->table_factory->NewTableReader(
-      TableReaderOptions(*cfd_->ioptions(),
-                         sv->mutable_cf_options.prefix_extractor.get(),
-                         env_options_, cfd_->internal_comparator()),
-      std::move(sst_file_reader), file_to_import->file_size, &table_reader);
-  if (!status.ok()) {
-    return status;
-  }
-
-  // Get the external file properties
-  auto props = table_reader->GetTableProperties();
-
-  // Set original_seqno to 0.
-  file_to_import->original_seqno = 0;
-
-  // Get number of entries in table
-  file_to_import->num_entries = props->num_entries;
-
-  ParsedInternalKey key;
-  ReadOptions ro;
-  // During reading the external file we can cache blocks that we read into
-  // the block cache, if we later change the global seqno of this file, we will
-  // have block in cache that will include keys with wrong seqno.
-  // We need to disable fill_cache so that we read from the file without
-  // updating the block cache.
-  ro.fill_cache = false;
-  std::unique_ptr<InternalIterator> iter(table_reader->NewIterator(
-      ro, sv->mutable_cf_options.prefix_extractor.get(), /*arena=*/nullptr,
-      /*skip_filters=*/false, TableReaderCaller::kExternalSSTIngestion));
-
-  // Get first (smallest) key from file
-  iter->SeekToFirst();
-  if (!ParseInternalKey(iter->key(), &key)) {
-    return Status::Corruption("external file have corrupted keys");
-  }
-  file_to_import->smallest_user_key = key.user_key.ToString();
-
-  // Get last (largest) key from file
-  iter->SeekToLast();
-  if (!ParseInternalKey(iter->key(), &key)) {
-    return Status::Corruption("external file have corrupted keys");
-  }
-  file_to_import->largest_user_key = key.user_key.ToString();
-
-  file_to_import->cf_id = static_cast<uint32_t>(props->column_family_id);
-
-  file_to_import->table_properties = *props;
-
-  return status;
-}
-
-}  // namespace rocksdb
-
-#endif  // !ROCKSDB_LITE
@@ -1,70 +0,0 @@
-#pragma once
-#include <string>
-#include <unordered_set>
-#include <vector>
-
-#include "db/column_family.h"
-#include "db/dbformat.h"
-#include "db/external_sst_file_ingestion_job.h"
-#include "db/snapshot_impl.h"
-#include "options/db_options.h"
-#include "rocksdb/db.h"
-#include "rocksdb/env.h"
-#include "rocksdb/metadata.h"
-#include "rocksdb/sst_file_writer.h"
-#include "util/autovector.h"
-
-namespace rocksdb {
-
-// Imports a set of sst files as is into a new column family. Logic is similar
-// to ExternalSstFileIngestionJob.
-class ImportColumnFamilyJob {
- public:
-  ImportColumnFamilyJob(
-      Env* env, VersionSet* versions, ColumnFamilyData* cfd,
-      const ImmutableDBOptions& db_options, const EnvOptions& env_options,
-      const ImportColumnFamilyOptions& import_options,
-      const std::vector<LiveFileMetaData>& metadata)
-      : env_(env),
-        versions_(versions),
-        cfd_(cfd),
-        db_options_(db_options),
-        env_options_(env_options),
-        import_options_(import_options),
-        metadata_(metadata) {}
-
-  // Prepare the job by copying external files into the DB.
-  Status Prepare(uint64_t next_file_number, SuperVersion* sv);
-
-  // Will execute the import job and prepare edit() to be applied.
-  // REQUIRES: Mutex held
-  Status Run();
-
-  // Cleanup after successful/failed job
-  void Cleanup(const Status& status);
-
-  VersionEdit* edit() { return &edit_; }
-
-  const autovector<IngestedFileInfo>& files_to_import() const {
-    return files_to_import_;
-  }
-
- private:
-  // Open the external file and populate `file_to_import` with all the
-  // external information we need to import this file.
-  Status GetIngestedFileInfo(const std::string& external_file,
-                             IngestedFileInfo* file_to_import,
-                             SuperVersion* sv);
-
-  Env* env_;
-  VersionSet* versions_;
-  ColumnFamilyData* cfd_;
-  const ImmutableDBOptions& db_options_;
-  const EnvOptions& env_options_;
-  autovector<IngestedFileInfo> files_to_import_;
-  VersionEdit edit_;
-  const ImportColumnFamilyOptions& import_options_;
-  std::vector<LiveFileMetaData> metadata_;
-};
-
-}  // namespace rocksdb
@@ -1,569 +0,0 @@
-#ifndef ROCKSDB_LITE
-
-#include <functional>
-#include "db/db_test_util.h"
-#include "port/port.h"
-#include "port/stack_trace.h"
-#include "rocksdb/sst_file_writer.h"
-#include "test_util/testutil.h"
-
-namespace rocksdb {
-
-class ImportColumnFamilyTest : public DBTestBase {
- public:
-  ImportColumnFamilyTest() : DBTestBase("/import_column_family_test") {
-    sst_files_dir_ = dbname_ + "/sst_files/";
-    DestroyAndRecreateExternalSSTFilesDir();
-    export_files_dir_ = test::TmpDir(env_) + "/export";
-    import_cfh_ = nullptr;
-    import_cfh2_ = nullptr;
-    metadata_ptr_ = nullptr;
-  }
-
-  ~ImportColumnFamilyTest() {
-    if (import_cfh_) {
-      db_->DropColumnFamily(import_cfh_);
-      db_->DestroyColumnFamilyHandle(import_cfh_);
-      import_cfh_ = nullptr;
-    }
-    if (import_cfh2_) {
-      db_->DropColumnFamily(import_cfh2_);
-      db_->DestroyColumnFamilyHandle(import_cfh2_);
-      import_cfh2_ = nullptr;
-    }
-    if (metadata_ptr_) {
-      delete metadata_ptr_;
-      metadata_ptr_ = nullptr;
-    }
-    test::DestroyDir(env_, sst_files_dir_);
-    test::DestroyDir(env_, export_files_dir_);
-  }
-
-  void DestroyAndRecreateExternalSSTFilesDir() {
-    test::DestroyDir(env_, sst_files_dir_);
-    env_->CreateDir(sst_files_dir_);
-    test::DestroyDir(env_, export_files_dir_);
-  }
-
-  LiveFileMetaData LiveFileMetaDataInit(std::string name,
-                                        std::string path,
-                                        int level,
-                                        SequenceNumber smallest_seqno,
-                                        SequenceNumber largest_seqno) {
-    LiveFileMetaData metadata;
-    metadata.name = name;
-    metadata.db_path = path;
-    metadata.smallest_seqno = smallest_seqno;
-    metadata.largest_seqno = largest_seqno;
-    metadata.level = level;
-    return metadata;
-  }
-
- protected:
-  std::string sst_files_dir_;
-  std::string export_files_dir_;
-  ColumnFamilyHandle* import_cfh_;
-  ColumnFamilyHandle* import_cfh2_;
-  ExportImportFilesMetaData *metadata_ptr_;
-};
-
-TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFiles) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"koko"}, options);
-
-  SstFileWriter sfw_cf1(EnvOptions(), options, handles_[1]);
-  SstFileWriter sfw_unknown(EnvOptions(), options);
-
-  // cf1.sst
-  const std::string cf1_sst_name = "cf1.sst";
-  const std::string cf1_sst = sst_files_dir_ + cf1_sst_name;
-  ASSERT_OK(sfw_cf1.Open(cf1_sst));
-  ASSERT_OK(sfw_cf1.Put("K1", "V1"));
-  ASSERT_OK(sfw_cf1.Put("K2", "V2"));
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // cf_unknown.sst
-  const std::string unknown_sst_name = "cf_unknown.sst";
-  const std::string unknown_sst = sst_files_dir_ + unknown_sst_name;
-  ASSERT_OK(sfw_unknown.Open(unknown_sst));
-  ASSERT_OK(sfw_unknown.Put("K3", "V1"));
-  ASSERT_OK(sfw_unknown.Put("K4", "V2"));
-  ASSERT_OK(sfw_unknown.Finish());
-
-  {
-    // Import sst file corresponding to cf1 onto a new cf and verify
-    ExportImportFilesMetaData metadata;
-    metadata.files.push_back(
-        LiveFileMetaDataInit(cf1_sst_name, sst_files_dir_, 0, 10, 19));
-    metadata.db_comparator_name = options.comparator->Name();
-
-    ASSERT_OK(db_->CreateColumnFamilyWithImport(
-        options, "toto", ImportColumnFamilyOptions(), metadata, &import_cfh_));
-    ASSERT_NE(import_cfh_, nullptr);
-
-    std::string value;
-    db_->Get(ReadOptions(), import_cfh_, "K1", &value);
-    ASSERT_EQ(value, "V1");
-    db_->Get(ReadOptions(), import_cfh_, "K2", &value);
-    ASSERT_EQ(value, "V2");
-    ASSERT_OK(db_->DropColumnFamily(import_cfh_));
-    ASSERT_OK(db_->DestroyColumnFamilyHandle(import_cfh_));
-    import_cfh_ = nullptr;
-  }
-
-  {
-    // Import sst file corresponding to unknown cf onto a new cf and verify
-    ExportImportFilesMetaData metadata;
-    metadata.files.push_back(
-        LiveFileMetaDataInit(unknown_sst_name, sst_files_dir_, 0, 20, 29));
-    metadata.db_comparator_name = options.comparator->Name();
-
-    ASSERT_OK(db_->CreateColumnFamilyWithImport(
-        options, "yoyo", ImportColumnFamilyOptions(), metadata, &import_cfh_));
-    ASSERT_NE(import_cfh_, nullptr);
-
-    std::string value;
-    db_->Get(ReadOptions(), import_cfh_, "K3", &value);
-    ASSERT_EQ(value, "V1");
-    db_->Get(ReadOptions(), import_cfh_, "K4", &value);
-    ASSERT_EQ(value, "V2");
-  }
-}
-
-TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithOverlap) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"koko"}, options);
-
-  SstFileWriter sfw_cf1(EnvOptions(), options, handles_[1]);
-
-  // file3.sst
-  const std::string file3_sst_name = "file3.sst";
-  const std::string file3_sst = sst_files_dir_ + file3_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file3_sst));
-  for (int i = 0; i < 100; ++i) {
-    sfw_cf1.Put(Key(i), Key(i) + "_val");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // file2.sst
-  const std::string file2_sst_name = "file2.sst";
-  const std::string file2_sst = sst_files_dir_ + file2_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file2_sst));
-  for (int i = 0; i < 100; i += 2) {
-    sfw_cf1.Put(Key(i), Key(i) + "_overwrite1");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // file1a.sst
-  const std::string file1a_sst_name = "file1a.sst";
-  const std::string file1a_sst = sst_files_dir_ + file1a_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file1a_sst));
-  for (int i = 0; i < 52; i += 4) {
-    sfw_cf1.Put(Key(i), Key(i) + "_overwrite2");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // file1b.sst
-  const std::string file1b_sst_name = "file1b.sst";
-  const std::string file1b_sst = sst_files_dir_ + file1b_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file1b_sst));
-  for (int i = 52; i < 100; i += 4) {
-    sfw_cf1.Put(Key(i), Key(i) + "_overwrite2");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // file0a.sst
-  const std::string file0a_sst_name = "file0a.sst";
-  const std::string file0a_sst = sst_files_dir_ + file0a_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file0a_sst));
-  for (int i = 0; i < 100; i += 16) {
-    sfw_cf1.Put(Key(i), Key(i) + "_overwrite3");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // file0b.sst
-  const std::string file0b_sst_name = "file0b.sst";
-  const std::string file0b_sst = sst_files_dir_ + file0b_sst_name;
-  ASSERT_OK(sfw_cf1.Open(file0b_sst));
-  for (int i = 0; i < 100; i += 16) {
-    sfw_cf1.Put(Key(i), Key(i) + "_overwrite4");
-  }
-  ASSERT_OK(sfw_cf1.Finish());
-
-  // Import sst files and verify
-  ExportImportFilesMetaData metadata;
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file3_sst_name, sst_files_dir_, 3, 10, 19));
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file2_sst_name, sst_files_dir_, 2, 20, 29));
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file1a_sst_name, sst_files_dir_, 1, 30, 34));
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file1b_sst_name, sst_files_dir_, 1, 35, 39));
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file0a_sst_name, sst_files_dir_, 0, 40, 49));
-  metadata.files.push_back(
-      LiveFileMetaDataInit(file0b_sst_name, sst_files_dir_, 0, 50, 59));
-  metadata.db_comparator_name = options.comparator->Name();
-
-  ASSERT_OK(db_->CreateColumnFamilyWithImport(
-      options, "toto", ImportColumnFamilyOptions(), metadata, &import_cfh_));
-  ASSERT_NE(import_cfh_, nullptr);
-
-  for (int i = 0; i < 100; i++) {
-    std::string value;
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
-    if (i % 16 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite4");
-    } else if (i % 4 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite2");
-    } else if (i % 2 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite1");
-    } else {
-      ASSERT_EQ(value, Key(i) + "_val");
-    }
-  }
-
-  for (int i = 0; i < 100; i += 5) {
-    ASSERT_OK(
-        db_->Put(WriteOptions(), import_cfh_, Key(i), Key(i) + "_overwrite5"));
-  }
-
-  // Flush and check again
-  ASSERT_OK(db_->Flush(FlushOptions(), import_cfh_));
-  for (int i = 0; i < 100; i++) {
-    std::string value;
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
-    if (i % 5 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite5");
-    } else if (i % 16 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite4");
-    } else if (i % 4 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite2");
-    } else if (i % 2 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite1");
-    } else {
-      ASSERT_EQ(value, Key(i) + "_val");
-    }
-  }
-
-  // Compact and check again.
-  ASSERT_OK(
-      db_->CompactRange(CompactRangeOptions(), import_cfh_, nullptr, nullptr));
-  for (int i = 0; i < 100; i++) {
-    std::string value;
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value);
-    if (i % 5 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite5");
-    } else if (i % 16 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite4");
-    } else if (i % 4 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite2");
-    } else if (i % 2 == 0) {
-      ASSERT_EQ(value, Key(i) + "_overwrite1");
-    } else {
-      ASSERT_EQ(value, Key(i) + "_val");
-    }
-  }
-}
-
-TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherCF) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"koko"}, options);
-
-  for (int i = 0; i < 100; ++i) {
-    Put(1, Key(i), Key(i) + "_val");
-  }
-  ASSERT_OK(Flush(1));
-
-  ASSERT_OK(
-      db_->CompactRange(CompactRangeOptions(), handles_[1], nullptr, nullptr));
-
-  // Overwrite the value in the same set of keys.
-  for (int i = 0; i < 100; ++i) {
-    Put(1, Key(i), Key(i) + "_overwrite");
-  }
-
-  // Flush to create L0 file.
-  ASSERT_OK(Flush(1));
-  for (int i = 0; i < 100; ++i) {
-    Put(1, Key(i), Key(i) + "_overwrite2");
-  }
-
-  // Flush again to create another L0 file. It should have higher sequencer.
-  ASSERT_OK(Flush(1));
-
-  Checkpoint* checkpoint;
-  ASSERT_OK(Checkpoint::Create(db_, &checkpoint));
-  ASSERT_OK(checkpoint->ExportColumnFamily(handles_[1], export_files_dir_,
-                                           &metadata_ptr_));
-  ASSERT_NE(metadata_ptr_, nullptr);
-  delete checkpoint;
-
-  ImportColumnFamilyOptions import_options;
-  import_options.move_files = false;
-  ASSERT_OK(db_->CreateColumnFamilyWithImport(options, "toto", import_options,
-                                              *metadata_ptr_, &import_cfh_));
-  ASSERT_NE(import_cfh_, nullptr);
-
-  import_options.move_files = true;
-  ASSERT_OK(db_->CreateColumnFamilyWithImport(options, "yoyo", import_options,
-                                              *metadata_ptr_, &import_cfh2_));
-  ASSERT_NE(import_cfh2_, nullptr);
-  delete metadata_ptr_;
-  metadata_ptr_ = NULL;
-
-  std::string value1, value2;
-
-  for (int i = 0; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
-    ASSERT_EQ(Get(1, Key(i)), value1);
-  }
-
-  for (int i = 0; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
-    ASSERT_EQ(Get(1, Key(i)), value2);
-  }
-
-  // Modify keys in cf1 and verify.
-  for (int i = 0; i < 25; i++) {
-    ASSERT_OK(db_->Delete(WriteOptions(), import_cfh_, Key(i)));
-  }
-  for (int i = 25; i < 50; i++) {
-    ASSERT_OK(
-        db_->Put(WriteOptions(), import_cfh_, Key(i), Key(i) + "_overwrite3"));
-  }
-  for (int i = 0; i < 25; ++i) {
-    ASSERT_TRUE(
-        db_->Get(ReadOptions(), import_cfh_, Key(i), &value1).IsNotFound());
-  }
-  for (int i = 25; i < 50; ++i) {
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
-    ASSERT_EQ(Key(i) + "_overwrite3", value1);
-  }
-  for (int i = 50; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
-    ASSERT_EQ(Key(i) + "_overwrite2", value1);
-  }
-
-  for (int i = 0; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
-    ASSERT_EQ(Get(1, Key(i)), value2);
-  }
-
-  // Compact and check again.
-  ASSERT_OK(db_->Flush(FlushOptions(), import_cfh_));
-  ASSERT_OK(
-      db_->CompactRange(CompactRangeOptions(), import_cfh_, nullptr, nullptr));
-
-  for (int i = 0; i < 25; ++i) {
-    ASSERT_TRUE(
-        db_->Get(ReadOptions(), import_cfh_, Key(i), &value1).IsNotFound());
-  }
-  for (int i = 25; i < 50; ++i) {
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
-    ASSERT_EQ(Key(i) + "_overwrite3", value1);
-  }
-  for (int i = 50; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh_, Key(i), &value1);
-    ASSERT_EQ(Key(i) + "_overwrite2", value1);
-  }
-
-  for (int i = 0; i < 100; ++i) {
-    db_->Get(ReadOptions(), import_cfh2_, Key(i), &value2);
-    ASSERT_EQ(Get(1, Key(i)), value2);
-  }
-}
-
-TEST_F(ImportColumnFamilyTest, ImportExportedSSTFromAnotherDB) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"koko"}, options);
-
-  for (int i = 0; i < 100; ++i) {
-    Put(1, Key(i), Key(i) + "_val");
-  }
-  ASSERT_OK(Flush(1));
-
-  // Compact to create a L1 file.
-  ASSERT_OK(
-      db_->CompactRange(CompactRangeOptions(), handles_[1], nullptr, nullptr));
-
-  // Overwrite the value in the same set of keys.
-  for (int i = 0; i < 50; ++i) {
-    Put(1, Key(i), Key(i) + "_overwrite");
-  }
-
-  // Flush to create L0 file.
-  ASSERT_OK(Flush(1));
-
-  for (int i = 0; i < 25; ++i) {
-    Put(1, Key(i), Key(i) + "_overwrite2");
-  }
-
-  // Flush again to create another L0 file. It should have higher sequencer.
-  ASSERT_OK(Flush(1));
-
-  Checkpoint* checkpoint;
-  ASSERT_OK(Checkpoint::Create(db_, &checkpoint));
-  ASSERT_OK(checkpoint->ExportColumnFamily(handles_[1], export_files_dir_,
-                                           &metadata_ptr_));
-  ASSERT_NE(metadata_ptr_, nullptr);
-  delete checkpoint;
-
-  // Create a new db and import the files.
-  DB* db_copy;
-  test::DestroyDir(env_, dbname_ + "/db_copy");
-  ASSERT_OK(DB::Open(options, dbname_ + "/db_copy", &db_copy));
-  ColumnFamilyHandle* cfh = nullptr;
-  ASSERT_OK(db_copy->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                  ImportColumnFamilyOptions(),
-                                                  *metadata_ptr_, &cfh));
-  ASSERT_NE(cfh, nullptr);
-
-  for (int i = 0; i < 100; ++i) {
-    std::string value;
-    db_copy->Get(ReadOptions(), cfh, Key(i), &value);
-    ASSERT_EQ(Get(1, Key(i)), value);
-  }
-  db_copy->DropColumnFamily(cfh);
-  db_copy->DestroyColumnFamilyHandle(cfh);
-  delete db_copy;
-  test::DestroyDir(env_, dbname_ + "/db_copy");
-}
-
-TEST_F(ImportColumnFamilyTest, ImportColumnFamilyNegativeTest) {
-  Options options = CurrentOptions();
-  CreateAndReopenWithCF({"koko"}, options);
-
-  {
-    // Create column family with existing cf name.
-    ExportImportFilesMetaData metadata;
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "koko",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("Column family already exists"));
-    ASSERT_EQ(import_cfh_, nullptr);
-  }
-
-  {
-    // Import with no files specified.
-    ExportImportFilesMetaData metadata;
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("The list of files is empty"));
-    ASSERT_EQ(import_cfh_, nullptr);
-  }
-
-  {
-    // Import with overlapping keys in sst files.
-    ExportImportFilesMetaData metadata;
-    SstFileWriter sfw_cf1(EnvOptions(), options, handles_[1]);
-    const std::string file1_sst_name = "file1.sst";
-    const std::string file1_sst = sst_files_dir_ + file1_sst_name;
-    ASSERT_OK(sfw_cf1.Open(file1_sst));
-    ASSERT_OK(sfw_cf1.Put("K1", "V1"));
-    ASSERT_OK(sfw_cf1.Put("K2", "V2"));
-    ASSERT_OK(sfw_cf1.Finish());
-    const std::string file2_sst_name = "file2.sst";
-    const std::string file2_sst = sst_files_dir_ + file2_sst_name;
-    ASSERT_OK(sfw_cf1.Open(file2_sst));
-    ASSERT_OK(sfw_cf1.Put("K2", "V2"));
-    ASSERT_OK(sfw_cf1.Put("K3", "V3"));
-    ASSERT_OK(sfw_cf1.Finish());
-
-    metadata.files.push_back(
-        LiveFileMetaDataInit(file1_sst_name, sst_files_dir_, 1, 10, 19));
-    metadata.files.push_back(
-        LiveFileMetaDataInit(file2_sst_name, sst_files_dir_, 1, 10, 19));
-    metadata.db_comparator_name = options.comparator->Name();
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("Files have overlapping ranges"));
-    ASSERT_EQ(import_cfh_, nullptr);
-  }
-
-  {
-    // Import with a mismatching comparator, should fail with appropriate error.
-    ExportImportFilesMetaData metadata;
-    Options mismatch_options = CurrentOptions();
-    mismatch_options.comparator = ReverseBytewiseComparator();
-    SstFileWriter sfw_cf1(EnvOptions(), mismatch_options, handles_[1]);
-    const std::string file1_sst_name = "file1.sst";
-    const std::string file1_sst = sst_files_dir_ + file1_sst_name;
-    ASSERT_OK(sfw_cf1.Open(file1_sst));
-    ASSERT_OK(sfw_cf1.Put("K2", "V2"));
-    ASSERT_OK(sfw_cf1.Put("K1", "V1"));
-    ASSERT_OK(sfw_cf1.Finish());
-
-    metadata.files.push_back(
-        LiveFileMetaDataInit(file1_sst_name, sst_files_dir_, 1, 10, 19));
-    metadata.db_comparator_name = mismatch_options.comparator->Name();
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "coco",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("Comparator name mismatch"));
-    ASSERT_EQ(import_cfh_, nullptr);
-  }
-
-  {
-    // Import with non existent sst file should fail with appropriate error
-    ExportImportFilesMetaData metadata;
-    SstFileWriter sfw_cf1(EnvOptions(), options, handles_[1]);
-    const std::string file1_sst_name = "file1.sst";
-    const std::string file1_sst = sst_files_dir_ + file1_sst_name;
-    ASSERT_OK(sfw_cf1.Open(file1_sst));
-    ASSERT_OK(sfw_cf1.Put("K1", "V1"));
-    ASSERT_OK(sfw_cf1.Put("K2", "V2"));
-    ASSERT_OK(sfw_cf1.Finish());
-    const std::string file3_sst_name = "file3.sst";
-
-    metadata.files.push_back(
-        LiveFileMetaDataInit(file1_sst_name, sst_files_dir_, 1, 10, 19));
-    metadata.files.push_back(
-        LiveFileMetaDataInit(file3_sst_name, sst_files_dir_, 1, 10, 19));
-    metadata.db_comparator_name = options.comparator->Name();
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::IOError("No such file or directory"));
-    ASSERT_EQ(import_cfh_, nullptr);
-
-    // Test successful import after a failure with the same CF name. Ensures
-    // there is no side effect with CF when there is a failed import
-    metadata.files.pop_back();
-    metadata.db_comparator_name = options.comparator->Name();
-
-    ASSERT_OK(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_));
-    ASSERT_NE(import_cfh_, nullptr);
-  }
-
-}
-
-}  // namespace rocksdb
-
-int main(int argc, char** argv) {
-  rocksdb::port::InstallStackTraceHandler();
-  ::testing::InitGoogleTest(&argc, argv);
-  return RUN_ALL_TESTS();
-}
-
-#else
-#include <stdio.h>
-
-int main(int /*argc*/, char** /*argv*/) {
-  fprintf(stderr,
-          "SKIPPED as External SST File Writer and Import are not supported "
-          "in ROCKSDB_LITE\n");
-  return 0;
-}
-
-#endif  // !ROCKSDB_LITE
@@ -237,28 +237,6 @@ class InternalStats {
      }
    }

-    CompactionStats& operator=(const CompactionStats& c) {
-      micros = c.micros;
-      cpu_micros = c.cpu_micros;
-      bytes_read_non_output_levels = c.bytes_read_non_output_levels;
-      bytes_read_output_level = c.bytes_read_output_level;
-      bytes_written = c.bytes_written;
-      bytes_moved = c.bytes_moved;
-      num_input_files_in_non_output_levels =
-          c.num_input_files_in_non_output_levels;
-      num_input_files_in_output_level = c.num_input_files_in_output_level;
-      num_output_files = c.num_output_files;
-      num_input_records = c.num_input_records;
-      num_dropped_records = c.num_dropped_records;
-      count = c.count;
-
-      int num_of_reasons = static_cast<int>(CompactionReason::kNumOfReasons);
-      for (int i = 0; i < num_of_reasons; i++) {
-        counts[i] = c.counts[i];
-      }
-      return *this;
-    }
-
    void Clear() {
      this->micros = 0;
      this->cpu_micros = 0;
@@ -21,8 +21,7 @@ class LookupKey {
 public:
  // Initialize *this for looking up user_key at a snapshot with
  // the specified sequence number.
-  LookupKey(const Slice& _user_key, SequenceNumber sequence,
-            const Slice* ts = nullptr);
+  LookupKey(const Slice& _user_key, SequenceNumber sequence);

  ~LookupKey();

@@ -20,6 +20,10 @@ namespace rocksdb {

 #ifdef ROCKSDB_JEMALLOC

+#ifdef JEMALLOC_NO_RENAME
+#define malloc_stats_print je_malloc_stats_print
+#endif
+
 typedef struct {
  char* cur;
  char* end;
@@ -42,6 +42,9 @@ class ReadCallback {
  // Refresh to a more recent visible seq
  virtual void Refresh(SequenceNumber seq) { max_visible_seq_ = seq; }

+  // Refer to DBIter::CanReseekToSkip
+  virtual bool CanReseekToSkip() { return true; }
+
 protected:
  // The max visible seq, it is usually the snapshot but could be larger if
  // transaction has its own writes written to db.
@@ -520,11 +520,7 @@ class Repairer {
      InternalIterator* iter = table_cache_->NewIterator(
          ropts, env_options_, cfd->internal_comparator(), t->meta,
          nullptr /* range_del_agg */,
-          cfd->GetLatestMutableCFOptions()->prefix_extractor.get(),
-          /*table_reader_ptr=*/nullptr, /*file_read_hist=*/nullptr,
-          TableReaderCaller::kRepair, /*arena=*/nullptr, /*skip_filters=*/false,
-          /*level=*/-1, /*smallest_compaction_key=*/nullptr,
-          /*largest_compaction_key=*/nullptr);
+          cfd->GetLatestMutableCFOptions()->prefix_extractor.get());
      bool empty = true;
      ParsedInternalKey parsed;
      t->min_sequence = 0;
@@ -11,13 +11,11 @@

 #include "db/dbformat.h"
 #include "db/range_tombstone_fragmenter.h"
-#include "db/snapshot_impl.h"
 #include "db/version_edit.h"
 #include "file/filename.h"

 #include "monitoring/perf_context_imp.h"
 #include "rocksdb/statistics.h"
-#include "table/block_based/block_based_table_reader.h"
 #include "table/get_context.h"
 #include "table/internal_iterator.h"
 #include "table/iterator_wrapper.h"
@@ -25,7 +23,6 @@
 #include "table/table_builder.h"
 #include "table/table_reader.h"
 #include "test_util/sync_point.h"
-#include "util/cast_util.h"
 #include "util/coding.h"
 #include "util/file_reader_writer.h"
 #include "util/stop_watch.h"
@@ -46,6 +43,13 @@ static void UnrefEntry(void* arg1, void* arg2) {
  cache->Release(h);
 }

+static void DeleteTableReader(void* arg1, void* arg2) {
+  TableReader* table_reader = reinterpret_cast<TableReader*>(arg1);
+  Statistics* stats = reinterpret_cast<Statistics*>(arg2);
+  RecordTick(stats, NO_FILE_CLOSES);
+  delete table_reader;
+}
+
 static Slice GetSliceForFileNumber(const uint64_t* file_number) {
  return Slice(reinterpret_cast<const char*>(file_number),
               sizeof(*file_number));
@@ -92,10 +96,10 @@ void TableCache::ReleaseHandle(Cache::Handle* handle) {
 Status TableCache::GetTableReader(
    const EnvOptions& env_options,
    const InternalKeyComparator& internal_comparator, const FileDescriptor& fd,
-    bool sequential_mode, bool record_read_stats, HistogramImpl* file_read_hist,
-    std::unique_ptr<TableReader>* table_reader,
+    bool sequential_mode, size_t readahead, bool record_read_stats,
+    HistogramImpl* file_read_hist, std::unique_ptr<TableReader>* table_reader,
    const SliceTransform* prefix_extractor, bool skip_filters, int level,
-    bool prefetch_index_and_filter_in_cache) {
+    bool prefetch_index_and_filter_in_cache, bool for_compaction) {
  std::string fname =
      TableFileName(ioptions_.cf_paths, fd.GetNumber(), fd.GetPathId());
  std::unique_ptr<RandomAccessFile> file;
@@ -103,6 +107,13 @@ Status TableCache::GetTableReader(

  RecordTick(ioptions_.statistics, NO_FILE_OPENS);
  if (s.ok()) {
+    if (readahead > 0 && !env_options.use_mmap_reads) {
+      // Not compatible with mmap files since ReadaheadRandomAccessFile requires
+      // its wrapped file's Read() to copy data into the provided scratch
+      // buffer, which mmap files don't use.
+      // TODO(ajkr): try madvise for mmap files in place of buffered readahead.
+      file = NewReadaheadRandomAccessFile(std::move(file), readahead);
+    }
    if (!sequential_mode && ioptions_.advise_random_on_open) {
      file->Hint(RandomAccessFile::RANDOM);
    }
@@ -111,7 +122,8 @@ Status TableCache::GetTableReader(
        new RandomAccessFileReader(
            std::move(file), fname, ioptions_.env,
            record_read_stats ? ioptions_.statistics : nullptr, SST_READ_MICROS,
-            file_read_hist, ioptions_.rate_limiter, ioptions_.listeners));
+            file_read_hist, ioptions_.rate_limiter, for_compaction,
+            ioptions_.listeners));
    s = ioptions_.table_factory->NewTableReader(
        TableReaderOptions(ioptions_, prefix_extractor, env_options,
                           internal_comparator, skip_filters, immortal_tables_,
@@ -152,9 +164,10 @@ Status TableCache::FindTable(const EnvOptions& env_options,
    }
    std::unique_ptr<TableReader> table_reader;
    s = GetTableReader(env_options, internal_comparator, fd,
-                       false /* sequential mode */, record_read_stats,
-                       file_read_hist, &table_reader, prefix_extractor,
-                       skip_filters, level, prefetch_index_and_filter_in_cache);
+                       false /* sequential mode */, 0 /* readahead */,
+                       record_read_stats, file_read_hist, &table_reader,
+                       prefix_extractor, skip_filters, level,
+                       prefetch_index_and_filter_in_cache);
    if (!s.ok()) {
      assert(table_reader == nullptr);
      RecordTick(ioptions_.statistics, NO_FILE_ERRORS);
@@ -177,27 +190,54 @@ InternalIterator* TableCache::NewIterator(
    const InternalKeyComparator& icomparator, const FileMetaData& file_meta,
    RangeDelAggregator* range_del_agg, const SliceTransform* prefix_extractor,
    TableReader** table_reader_ptr, HistogramImpl* file_read_hist,
-    TableReaderCaller caller, Arena* arena, bool skip_filters, int level,
+    bool for_compaction, Arena* arena, bool skip_filters, int level,
    const InternalKey* smallest_compaction_key,
    const InternalKey* largest_compaction_key) {
  PERF_TIMER_GUARD(new_table_iterator_nanos);

  Status s;
+  bool create_new_table_reader = false;
  TableReader* table_reader = nullptr;
  Cache::Handle* handle = nullptr;
  if (table_reader_ptr != nullptr) {
    *table_reader_ptr = nullptr;
  }
-  bool for_compaction = caller == TableReaderCaller::kCompaction;
+  size_t readahead = 0;
+  if (for_compaction) {
+#ifndef NDEBUG
+    bool use_direct_reads_for_compaction = env_options.use_direct_reads;
+    TEST_SYNC_POINT_CALLBACK("TableCache::NewIterator:for_compaction",
+                             &use_direct_reads_for_compaction);
+#endif  // !NDEBUG
+    if (ioptions_.new_table_reader_for_compaction_inputs) {
+      // get compaction_readahead_size from env_options allows us to set the
+      // value dynamically
+      readahead = env_options.compaction_readahead_size;
+      create_new_table_reader = true;
+    }
+  }
+
  auto& fd = file_meta.fd;
-  table_reader = fd.table_reader;
-  if (table_reader == nullptr) {
-    s = FindTable(env_options, icomparator, fd, &handle, prefix_extractor,
-                  options.read_tier == kBlockCacheTier /* no_io */,
-                  !for_compaction /* record_read_stats */, file_read_hist,
-                  skip_filters, level);
+  if (create_new_table_reader) {
+    std::unique_ptr<TableReader> table_reader_unique_ptr;
+    s = GetTableReader(
+        env_options, icomparator, fd, true /* sequential_mode */, readahead,
+        !for_compaction /* record stats */, nullptr, &table_reader_unique_ptr,
+        prefix_extractor, false /* skip_filters */, level,
+        true /* prefetch_index_and_filter_in_cache */, for_compaction);
    if (s.ok()) {
-      table_reader = GetTableReaderFromHandle(handle);
+      table_reader = table_reader_unique_ptr.release();
+    }
+  } else {
+    table_reader = fd.table_reader;
+    if (table_reader == nullptr) {
+      s = FindTable(env_options, icomparator, fd, &handle, prefix_extractor,
+                    options.read_tier == kBlockCacheTier /* no_io */,
+                    !for_compaction /* record read_stats */, file_read_hist,
+                    skip_filters, level);
+      if (s.ok()) {
+        table_reader = GetTableReaderFromHandle(handle);
+      }
    }
  }
  InternalIterator* result = nullptr;
@@ -207,10 +247,13 @@ InternalIterator* TableCache::NewIterator(
      result = NewEmptyInternalIterator<Slice>(arena);
    } else {
      result = table_reader->NewIterator(options, prefix_extractor, arena,
-                                         skip_filters, caller,
-                                         env_options.compaction_readahead_size);
+                                         skip_filters, for_compaction);
    }
-    if (handle != nullptr) {
+    if (create_new_table_reader) {
+      assert(handle == nullptr);
+      result->RegisterCleanup(&DeleteTableReader, table_reader,
+                              ioptions_.statistics);
+    } else if (handle != nullptr) {
      result->RegisterCleanup(&UnrefEntry, cache_, handle);
      handle = nullptr;  // prevent from releasing below
    }
@@ -279,23 +322,8 @@ Status TableCache::Get(const ReadOptions& options,
    // sequence key increases. However, to support caching snapshot
    // reads, we append the sequence number (incremented by 1 to
    // distinguish from 0) only in this case.
-    // If the snapshot is larger than the largest seqno in the file,
-    // all data should be exposed to the snapshot, so we treat it
-    // the same as there is no snapshot. The exception is that if
-    // a seq-checking callback is registered, some internal keys
-    // may still be filtered out.
-    uint64_t seq_no = 0;
-    // Maybe we can include the whole file ifsnapshot == fd.largest_seqno.
-    if (options.snapshot != nullptr &&
-        (get_context->has_callback() ||
-         static_cast_with_check<const SnapshotImpl, const Snapshot>(
-             options.snapshot)
-                 ->GetSequenceNumber() <= fd.largest_seqno)) {
-      // We should consider to use options.snapshot->GetSequenceNumber()
-      // instead of GetInternalKeySeqno(k), which will make the code
-      // easier to understand.
-      seq_no = 1 + GetInternalKeySeqno(k);
-    }
+    uint64_t seq_no =
+        options.snapshot == nullptr ? 0 : 1 + GetInternalKeySeqno(k);

    // Compute row cache key.
    row_cache_key.TrimAppend(row_cache_key.Size(), row_cache_id_.data(),
@@ -505,30 +533,4 @@ void TableCache::Evict(Cache* cache, uint64_t file_number) {
  cache->Erase(GetSliceForFileNumber(&file_number));
 }

-uint64_t TableCache::ApproximateOffsetOf(
-    const Slice& key, const FileDescriptor& fd, TableReaderCaller caller,
-    const InternalKeyComparator& internal_comparator,
-    const SliceTransform* prefix_extractor) {
-  uint64_t result = 0;
-  TableReader* table_reader = fd.table_reader;
-  Cache::Handle* table_handle = nullptr;
-  if (table_reader == nullptr) {
-    const bool for_compaction = (caller == TableReaderCaller::kCompaction);
-    Status s = FindTable(env_options_, internal_comparator, fd, &table_handle,
-                         prefix_extractor, false /* no_io */,
-                         !for_compaction /* record_read_stats */);
-    if (s.ok()) {
-      table_reader = GetTableReaderFromHandle(table_handle);
-    }
-  }
-
-  if (table_reader != nullptr) {
-    result = table_reader->ApproximateOffsetOf(key, caller);
-  }
-  if (table_handle != nullptr) {
-    ReleaseHandle(table_handle);
-  }
-
-  return result;
-}
 }  // namespace rocksdb
@@ -70,10 +70,12 @@ class TableCache {
      const ReadOptions& options, const EnvOptions& toptions,
      const InternalKeyComparator& internal_comparator,
      const FileMetaData& file_meta, RangeDelAggregator* range_del_agg,
-      const SliceTransform* prefix_extractor, TableReader** table_reader_ptr,
-      HistogramImpl* file_read_hist, TableReaderCaller caller, Arena* arena,
-      bool skip_filters, int level, const InternalKey* smallest_compaction_key,
-      const InternalKey* largest_compaction_key);
+      const SliceTransform* prefix_extractor = nullptr,
+      TableReader** table_reader_ptr = nullptr,
+      HistogramImpl* file_read_hist = nullptr, bool for_compaction = false,
+      Arena* arena = nullptr, bool skip_filters = false, int level = -1,
+      const InternalKey* smallest_compaction_key = nullptr,
+      const InternalKey* largest_compaction_key = nullptr);

  // If a seek to internal key "k" in specified file finds an entry,
  // call get_context->SaveValue() repeatedly until
@@ -153,12 +155,6 @@ class TableCache {
      const FileDescriptor& fd,
      const SliceTransform* prefix_extractor = nullptr);

-  // Returns approximated offset of a key in a file represented by fd.
-  uint64_t ApproximateOffsetOf(
-      const Slice& key, const FileDescriptor& fd, TableReaderCaller caller,
-      const InternalKeyComparator& internal_comparator,
-      const SliceTransform* prefix_extractor = nullptr);
-
  // Release the handle from a cache
  void ReleaseHandle(Cache::Handle* handle);

@@ -181,11 +177,13 @@ class TableCache {
  Status GetTableReader(const EnvOptions& env_options,
                        const InternalKeyComparator& internal_comparator,
                        const FileDescriptor& fd, bool sequential_mode,
-                        bool record_read_stats, HistogramImpl* file_read_hist,
+                        size_t readahead, bool record_read_stats,
+                        HistogramImpl* file_read_hist,
                        std::unique_ptr<TableReader>* table_reader,
                        const SliceTransform* prefix_extractor = nullptr,
                        bool skip_filters = false, int level = -1,
-                        bool prefetch_index_and_filter_in_cache = true);
+                        bool prefetch_index_and_filter_in_cache = true,
+                        bool for_compaction = false);

  const ImmutableCFOptions& ioptions_;
  const EnvOptions& env_options_;
@@ -199,8 +199,7 @@ void TransactionLogIteratorImpl::NextImpl(bool internal) {
      if (current_last_seq_ == versions_->LastSequence()) {
        current_status_ = Status::OK();
      } else {
-        const char* msg = "Create a new iterator to fetch the new tail.";
-        current_status_ = Status::TryAgain(msg);
+        current_status_ = Status::Corruption("NO MORE DATA LEFT");
      }
      return;
    }
@@ -52,8 +52,6 @@ struct FileDescriptor {
        smallest_seqno(_smallest_seqno),
        largest_seqno(_largest_seqno) {}

-  FileDescriptor(const FileDescriptor& fd) { *this=fd; }
-
  FileDescriptor& operator=(const FileDescriptor& fd) {
    table_reader = fd.table_reader;
    packed_number_and_path_id = fd.packed_number_and_path_id;
@@ -850,15 +850,14 @@ namespace {

 class LevelIterator final : public InternalIterator {
 public:
-  LevelIterator(TableCache* table_cache, const ReadOptions& read_options,
-                const EnvOptions& env_options,
-                const InternalKeyComparator& icomparator,
-                const LevelFilesBrief* flevel,
-                const SliceTransform* prefix_extractor, bool should_sample,
-                HistogramImpl* file_read_hist, TableReaderCaller caller,
-                bool skip_filters, int level, RangeDelAggregator* range_del_agg,
-                const std::vector<AtomicCompactionUnitBoundary>*
-                    compaction_boundaries = nullptr)
+  LevelIterator(
+      TableCache* table_cache, const ReadOptions& read_options,
+      const EnvOptions& env_options, const InternalKeyComparator& icomparator,
+      const LevelFilesBrief* flevel, const SliceTransform* prefix_extractor,
+      bool should_sample, HistogramImpl* file_read_hist, bool for_compaction,
+      bool skip_filters, int level, RangeDelAggregator* range_del_agg,
+      const std::vector<AtomicCompactionUnitBoundary>* compaction_boundaries =
+          nullptr)
      : table_cache_(table_cache),
        read_options_(read_options),
        env_options_(env_options),
@@ -868,7 +867,7 @@ class LevelIterator final : public InternalIterator {
        prefix_extractor_(prefix_extractor),
        file_read_hist_(file_read_hist),
        should_sample_(should_sample),
-        caller_(caller),
+        for_compaction_(for_compaction),
        skip_filters_(skip_filters),
        file_index_(flevel_->num_files),
        level_(level),
@@ -886,7 +885,7 @@ class LevelIterator final : public InternalIterator {
  void SeekToFirst() override;
  void SeekToLast() override;
  void Next() final override;
-  bool NextAndGetResult(IterateResult* result) override;
+  bool NextAndGetResult(Slice* ret_key) override;
  void Prev() override;

  bool Valid() const override { return file_iter_.Valid(); }
@@ -894,38 +893,23 @@ class LevelIterator final : public InternalIterator {
    assert(Valid());
    return file_iter_.key();
  }
-
  Slice value() const override {
    assert(Valid());
    return file_iter_.value();
  }
-
  Status status() const override {
    return file_iter_.iter() ? file_iter_.status() : Status::OK();
  }
-
-  inline bool MayBeOutOfLowerBound() override {
-    assert(Valid());
-    return may_be_out_of_lower_bound_ && file_iter_.MayBeOutOfLowerBound();
-  }
-
-  inline bool MayBeOutOfUpperBound() override {
-    assert(Valid());
-    return file_iter_.MayBeOutOfUpperBound();
-  }
-
  void SetPinnedItersMgr(PinnedIteratorsManager* pinned_iters_mgr) override {
    pinned_iters_mgr_ = pinned_iters_mgr;
    if (file_iter_.iter()) {
      file_iter_.SetPinnedItersMgr(pinned_iters_mgr);
    }
  }
-
  bool IsKeyPinned() const override {
    return pinned_iters_mgr_ && pinned_iters_mgr_->PinningEnabled() &&
           file_iter_.iter() && file_iter_.IsKeyPinned();
  }
-
  bool IsValuePinned() const override {
    return pinned_iters_mgr_ && pinned_iters_mgr_->PinningEnabled() &&
           file_iter_.iter() && file_iter_.IsValuePinned();
@@ -969,27 +953,12 @@ class LevelIterator final : public InternalIterator {
      smallest_compaction_key = (*compaction_boundaries_)[file_index_].smallest;
      largest_compaction_key = (*compaction_boundaries_)[file_index_].largest;
    }
-    CheckMayBeOutOfLowerBound();
    return table_cache_->NewIterator(
        read_options_, env_options_, icomparator_, *file_meta.file_metadata,
        range_del_agg_, prefix_extractor_,
-        nullptr /* don't need reference to table */, file_read_hist_, caller_,
-        /*arena=*/nullptr, skip_filters_, level_, smallest_compaction_key,
-        largest_compaction_key);
-  }
-
-  // Check if current file being fully within iterate_lower_bound.
-  //
-  // Note MyRocks may update iterate bounds between seek. To workaround it,
-  // we need to check and update may_be_out_of_lower_bound_ accordingly.
-  void CheckMayBeOutOfLowerBound() {
-    if (read_options_.iterate_lower_bound != nullptr &&
-        file_index_ < flevel_->num_files) {
-      may_be_out_of_lower_bound_ =
-          user_comparator_.Compare(
-              ExtractUserKey(file_smallest_key(file_index_)),
-              *read_options_.iterate_lower_bound) < 0;
-    }
+        nullptr /* don't need reference to table */,
+        file_read_hist_, for_compaction_, nullptr /* arena */, skip_filters_,
+        level_, smallest_compaction_key, largest_compaction_key);
  }

  TableCache* table_cache_;
@@ -1003,9 +972,8 @@ class LevelIterator final : public InternalIterator {

  HistogramImpl* file_read_hist_;
  bool should_sample_;
-  TableReaderCaller caller_;
+  bool for_compaction_;
  bool skip_filters_;
-  bool may_be_out_of_lower_bound_ = true;
  size_t file_index_;
  int level_;
  RangeDelAggregator* range_del_agg_;
@@ -1041,7 +1009,6 @@ void LevelIterator::Seek(const Slice& target) {
    file_iter_.Seek(target);
  }
  SkipEmptyFileForward();
-  CheckMayBeOutOfLowerBound();
 }

 void LevelIterator::SeekForPrev(const Slice& target) {
@@ -1055,7 +1022,6 @@ void LevelIterator::SeekForPrev(const Slice& target) {
    file_iter_.SeekForPrev(target);
    SkipEmptyFileBackward();
  }
-  CheckMayBeOutOfLowerBound();
 }

 void LevelIterator::SeekToFirst() {
@@ -1064,7 +1030,6 @@ void LevelIterator::SeekToFirst() {
    file_iter_.SeekToFirst();
  }
  SkipEmptyFileForward();
-  CheckMayBeOutOfLowerBound();
 }

 void LevelIterator::SeekToLast() {
@@ -1073,17 +1038,15 @@ void LevelIterator::SeekToLast() {
    file_iter_.SeekToLast();
  }
  SkipEmptyFileBackward();
-  CheckMayBeOutOfLowerBound();
 }

 void LevelIterator::Next() { NextImpl(); }

-bool LevelIterator::NextAndGetResult(IterateResult* result) {
+bool LevelIterator::NextAndGetResult(Slice* ret_key) {
  NextImpl();
  bool is_valid = Valid();
  if (is_valid) {
-    result->key = key();
-    result->may_be_out_of_upper_bound = MayBeOutOfUpperBound();
+    *ret_key = key();
  }
  return is_valid;
 }
@@ -1230,7 +1193,8 @@ Status Version::GetTableProperties(std::shared_ptr<const TableProperties>* tp,
      new RandomAccessFileReader(
          std::move(file), file_name, nullptr /* env */, nullptr /* stats */,
          0 /* hist_type */, nullptr /* file_read_hist */,
-          nullptr /* rate_limiter */, ioptions->listeners));
+          nullptr /* rate_limiter */, false /* for_compaction*/,
+          ioptions->listeners));
  s = ReadTableProperties(
      file_reader.get(), file_meta->fd.GetFileSize(),
      Footer::kInvalidTableMagicNumber /* table's magic number */, *ioptions,
@@ -1477,14 +1441,10 @@ void Version::AddIteratorsForLevel(const ReadOptions& read_options,
    for (size_t i = 0; i < storage_info_.LevelFilesBrief(0).num_files; i++) {
      const auto& file = storage_info_.LevelFilesBrief(0).files[i];
      merge_iter_builder->AddIterator(cfd_->table_cache()->NewIterator(
-          read_options, soptions, cfd_->internal_comparator(),
-          *file.file_metadata, range_del_agg,
-          mutable_cf_options_.prefix_extractor.get(), nullptr,
-          cfd_->internal_stats()->GetFileReadHist(0),
-          TableReaderCaller::kUserIterator, arena,
-          /*skip_filters=*/false, /*level=*/0,
-          /*smallest_compaction_key=*/nullptr,
-          /*largest_compaction_key=*/nullptr));
+          read_options, soptions, cfd_->internal_comparator(), *file.file_metadata,
+          range_del_agg, mutable_cf_options_.prefix_extractor.get(), nullptr,
+          cfd_->internal_stats()->GetFileReadHist(0), false, arena,
+          false /* skip_filters */, 0 /* level */));
    }
    if (should_sample) {
      // Count ones for every L0 files. This is done per iterator creation
@@ -1505,8 +1465,8 @@ void Version::AddIteratorsForLevel(const ReadOptions& read_options,
        cfd_->internal_comparator(), &storage_info_.LevelFilesBrief(level),
        mutable_cf_options_.prefix_extractor.get(), should_sample_file_read(),
        cfd_->internal_stats()->GetFileReadHist(level),
-        TableReaderCaller::kUserIterator, IsFilterSkipped(level), level,
-        range_del_agg, /*largest_compaction_key=*/nullptr));
+        false /* for_compaction */, IsFilterSkipped(level), level,
+        range_del_agg));
  }
 }

@@ -1535,14 +1495,10 @@ Status Version::OverlapWithLevelIterator(const ReadOptions& read_options,
        continue;
      }
      ScopedArenaIterator iter(cfd_->table_cache()->NewIterator(
-          read_options, env_options, cfd_->internal_comparator(),
-          *file->file_metadata, &range_del_agg,
-          mutable_cf_options_.prefix_extractor.get(), nullptr,
-          cfd_->internal_stats()->GetFileReadHist(0),
-          TableReaderCaller::kUserIterator, &arena,
-          /*skip_filters=*/false, /*level=*/0,
-          /*smallest_compaction_key=*/nullptr,
-          /*largest_compaction_key=*/nullptr));
+          read_options, env_options, cfd_->internal_comparator(), *file->file_metadata,
+          &range_del_agg, mutable_cf_options_.prefix_extractor.get(), nullptr,
+          cfd_->internal_stats()->GetFileReadHist(0), false, &arena,
+          false /* skip_filters */, 0 /* level */));
      status = OverlapWithIterator(
          ucmp, smallest_user_key, largest_user_key, iter.get(), overlap);
      if (!status.ok() || *overlap) {
@@ -1556,7 +1512,7 @@ Status Version::OverlapWithLevelIterator(const ReadOptions& read_options,
        cfd_->internal_comparator(), &storage_info_.LevelFilesBrief(level),
        mutable_cf_options_.prefix_extractor.get(), should_sample_file_read(),
        cfd_->internal_stats()->GetFileReadHist(level),
-        TableReaderCaller::kUserIterator, IsFilterSkipped(level), level,
+        false /* for_compaction */, IsFilterSkipped(level), level,
        &range_del_agg));
    status = OverlapWithIterator(
        ucmp, smallest_user_key, largest_user_key, iter.get(), overlap);
@@ -1662,17 +1618,11 @@ void Version::Get(const ReadOptions& read_options, const LookupKey& k,
  }

  PinnedIteratorsManager pinned_iters_mgr;
-  uint64_t tracing_get_id = BlockCacheTraceHelper::kReservedGetId;
-  if (vset_ && vset_->block_cache_tracer_ &&
-      vset_->block_cache_tracer_->is_tracing_enabled()) {
-    tracing_get_id = vset_->block_cache_tracer_->NextGetId();
-  }
  GetContext get_context(
      user_comparator(), merge_operator_, info_log_, db_statistics_,
      status->ok() ? GetContext::kNotFound : GetContext::kMerge, user_key,
      value, value_found, merge_context, max_covering_tombstone_seq, this->env_,
-      seq, merge_operator_ ? &pinned_iters_mgr : nullptr, callback, is_blob,
-      tracing_get_id);
+      seq, merge_operator_ ? &pinned_iters_mgr : nullptr, callback, is_blob);

  // Pin blocks that we read to hold merge operands
  if (merge_operator_) {
@@ -1790,12 +1740,7 @@ void Version::MultiGet(const ReadOptions& read_options, MultiGetRange* range,
  if (merge_operator_) {
    pinned_iters_mgr.StartPinning();
  }
-  uint64_t tracing_mget_id = BlockCacheTraceHelper::kReservedGetId;

-  if (vset_ && vset_->block_cache_tracer_ &&
-      vset_->block_cache_tracer_->is_tracing_enabled()) {
-    tracing_mget_id = vset_->block_cache_tracer_->NextGetId();
-  }
  // Even though we know the batch size won't be > MAX_BATCH_SIZE,
  // use autovector in order to avoid unnecessary construction of GetContext
  // objects, which is expensive
@@ -1807,8 +1752,7 @@ void Version::MultiGet(const ReadOptions& read_options, MultiGetRange* range,
        iter->s->ok() ? GetContext::kNotFound : GetContext::kMerge, iter->ukey,
        iter->value, nullptr, &(iter->merge_context),
        &iter->max_covering_tombstone_seq, this->env_, &iter->seq,
-        merge_operator_ ? &pinned_iters_mgr : nullptr, callback, is_blob,
-        tracing_mget_id);
+        merge_operator_ ? &pinned_iters_mgr : nullptr, callback, is_blob);
  }
  int get_ctx_index = 0;
  for (auto iter = range->begin(); iter != range->end();
@@ -3419,10 +3363,16 @@ VersionSet::VersionSet(const std::string& dbname,
      env_options_(storage_options),
      block_cache_tracer_(block_cache_tracer) {}

+void CloseTables(void* ptr, size_t) {
+  TableReader* table_reader = reinterpret_cast<TableReader*>(ptr);
+  table_reader->Close();
+}
+
 VersionSet::~VersionSet() {
  // we need to delete column_family_set_ because its destructor depends on
  // VersionSet
  Cache* table_cache = column_family_set_->get_table_cache();
+  table_cache->ApplyToAllCacheEntries(&CloseTables, false /* thread_safe */);
  column_family_set_.reset();
  for (auto& file : obsolete_files_) {
    if (file.metadata->table_reader_handle) {
@@ -4260,8 +4210,7 @@ Status VersionSet::Recover(
      return s;
    }
    manifest_file_reader.reset(
-        new SequentialFileReader(std::move(manifest_file), manifest_path,
-                                 db_options_->log_readahead_size));
+        new SequentialFileReader(std::move(manifest_file), manifest_path));
  }
  uint64_t current_manifest_file_size;
  s = env_->GetFileSize(manifest_path, &current_manifest_file_size);
@@ -4407,9 +4356,10 @@ Status VersionSet::Recover(
        ", last_sequence is %" PRIu64 ", log_number is %" PRIu64
        ",prev_log_number is %" PRIu64 ",max_column_family is %" PRIu32
        ",min_log_number_to_keep is %" PRIu64 "\n",
-        manifest_path.c_str(), manifest_file_number_, next_file_number_.load(),
-        last_sequence_.load(), log_number, prev_log_number_,
-        column_family_set_->GetMaxColumnFamily(), min_log_number_to_keep_2pc());
+        manifest_path.c_str(), manifest_file_number_,
+        next_file_number_.load(), last_sequence_.load(), log_number,
+        prev_log_number_, column_family_set_->GetMaxColumnFamily(),
+        min_log_number_to_keep_2pc());

    for (auto cfd : *column_family_set_) {
      if (cfd->IsDropped()) {
@@ -4591,8 +4541,7 @@ Status VersionSet::DumpManifest(Options& options, std::string& dscname,
    if (!s.ok()) {
      return s;
    }
-    file_reader.reset(new SequentialFileReader(
-        std::move(file), dscname, db_options_->log_readahead_size));
+    file_reader.reset(new SequentialFileReader(std::move(file), dscname));
  }

  bool have_prev_log_number = false;
@@ -4873,7 +4822,7 @@ Status VersionSet::WriteSnapshot(log::Writer* log) {
 // maintain state of where they first appear in the files.
 uint64_t VersionSet::ApproximateSize(Version* v, const Slice& start,
                                     const Slice& end, int start_level,
-                                     int end_level, TableReaderCaller caller) {
+                                     int end_level, bool for_compaction) {
  // pre-condition
  assert(v->cfd_->internal_comparator().Compare(start, end) <= 0);

@@ -4894,7 +4843,7 @@ uint64_t VersionSet::ApproximateSize(Version* v, const Slice& start,

    if (!level) {
      // level 0 data is sorted order, handle the use case explicitly
-      size += ApproximateSizeLevel0(v, files_brief, start, end, caller);
+      size += ApproximateSizeLevel0(v, files_brief, start, end, for_compaction);
      continue;
    }

@@ -4911,7 +4860,7 @@ uint64_t VersionSet::ApproximateSize(Version* v, const Slice& start,
    // inferred from the sorted order
    for (uint64_t i = idx_start; i < files_brief.num_files; i++) {
      uint64_t val;
-      val = ApproximateSize(v, files_brief.files[i], end, caller);
+      val = ApproximateSize(v, files_brief.files[i], end, for_compaction);
      if (!val) {
        // the files after this will not have the range
        break;
@@ -4922,7 +4871,7 @@ uint64_t VersionSet::ApproximateSize(Version* v, const Slice& start,
      if (i == idx_start) {
        // subtract the bytes needed to be scanned to get to the starting
        // key
-        val = ApproximateSize(v, files_brief.files[i], start, caller);
+        val = ApproximateSize(v, files_brief.files[i], start, for_compaction);
        assert(size >= val);
        size -= val;
      }
@@ -4936,15 +4885,15 @@ uint64_t VersionSet::ApproximateSizeLevel0(Version* v,
                                           const LevelFilesBrief& files_brief,
                                           const Slice& key_start,
                                           const Slice& key_end,
-                                           TableReaderCaller caller) {
+                                           bool for_compaction) {
  // level 0 files are not in sorted order, we need to iterate through
  // the list to compute the total bytes that require scanning
  uint64_t size = 0;
  for (size_t i = 0; i < files_brief.num_files; i++) {
    const uint64_t start =
-        ApproximateSize(v, files_brief.files[i], key_start, caller);
+        ApproximateSize(v, files_brief.files[i], key_start, for_compaction);
    const uint64_t end =
-        ApproximateSize(v, files_brief.files[i], key_end, caller);
+        ApproximateSize(v, files_brief.files[i], key_end, for_compaction);
    assert(end >= start);
    size += end - start;
  }
@@ -4952,8 +4901,7 @@ uint64_t VersionSet::ApproximateSizeLevel0(Version* v,
 }

 uint64_t VersionSet::ApproximateSize(Version* v, const FdWithKeyRange& f,
-                                     const Slice& key,
-                                     TableReaderCaller caller) {
+                                     const Slice& key, bool for_compaction) {
  // pre-condition
  assert(v);

@@ -4967,12 +4915,15 @@ uint64_t VersionSet::ApproximateSize(Version* v, const FdWithKeyRange& f,
  } else {
    // "key" falls in the range for this table.  Add the
    // approximate offset of "key" within the table.
-    TableCache* table_cache = v->cfd_->table_cache();
-    if (table_cache != nullptr) {
-      result = table_cache->ApproximateOffsetOf(
-          key, f.file_metadata->fd, caller, v->cfd()->internal_comparator(),
-          v->GetMutableCFOptions().prefix_extractor.get());
+    TableReader* table_reader_ptr;
+    InternalIterator* iter = v->cfd_->table_cache()->NewIterator(
+        ReadOptions(), v->env_options_, v->cfd_->internal_comparator(),
+        *f.file_metadata, nullptr /* range_del_agg */,
+        v->GetMutableCFOptions().prefix_extractor.get(), &table_reader_ptr);
+    if (table_reader_ptr != nullptr) {
+      result = table_reader_ptr->ApproximateOffsetOf(key, for_compaction);
    }
+    delete iter;
  }
  return result;
 }
@@ -5049,12 +5000,10 @@ InternalIterator* VersionSet::MakeInputIterator(
              read_options, env_options_compactions, cfd->internal_comparator(),
              *flevel->files[i].file_metadata, range_del_agg,
              c->mutable_cf_options()->prefix_extractor.get(),
-              /*table_reader_ptr=*/nullptr,
-              /*file_read_hist=*/nullptr, TableReaderCaller::kCompaction,
-              /*arena=*/nullptr,
-              /*skip_filters=*/false, /*level=*/static_cast<int>(which),
-              /*smallest_compaction_key=*/nullptr,
-              /*largest_compaction_key=*/nullptr);
+              nullptr /* table_reader_ptr */,
+              nullptr /* no per level latency histogram */,
+              true /* for_compaction */, nullptr /* arena */,
+              false /* skip_filters */, static_cast<int>(which) /* level */);
        }
      } else {
        // Create concatenating iterator for the files from this level
@@ -5062,10 +5011,10 @@ InternalIterator* VersionSet::MakeInputIterator(
            cfd->table_cache(), read_options, env_options_compactions,
            cfd->internal_comparator(), c->input_levels(which),
            c->mutable_cf_options()->prefix_extractor.get(),
-            /*should_sample=*/false,
-            /*no per level latency histogram=*/nullptr,
-            TableReaderCaller::kCompaction, /*skip_filters=*/false,
-            /*level=*/static_cast<int>(which), range_del_agg,
+            false /* should_sample */,
+            nullptr /* no per level latency histogram */,
+            true /* for_compaction */, false /* skip_filters */,
+            static_cast<int>(which) /* level */, range_del_agg,
            c->boundaries(which));
      }
    }
@@ -5709,8 +5658,7 @@ Status ReactiveVersionSet::MaybeSwitchManifest(
    std::unique_ptr<SequentialFileReader> manifest_file_reader;
    if (s.ok()) {
      manifest_file_reader.reset(
-          new SequentialFileReader(std::move(manifest_file), manifest_path,
-                                   db_options_->log_readahead_size));
+          new SequentialFileReader(std::move(manifest_file), manifest_path));
      manifest_reader->reset(new log::FragmentBufferedReader(
          nullptr, std::move(manifest_file_reader), reporter,
          true /* checksum */, 0 /* log_number */));
@@ -655,7 +655,7 @@ class Version {

  uint64_t GetSstFilesSize();

-  const MutableCFOptions& GetMutableCFOptions() { return mutable_cf_options_; }
+  MutableCFOptions GetMutableCFOptions() { return mutable_cf_options_; }

 private:
  Env* env_;
@@ -981,11 +981,10 @@ class VersionSet {
  void AddLiveFiles(std::vector<FileDescriptor>* live_list);

  // Return the approximate size of data to be scanned for range [start, end)
-  // in levels [start_level, end_level). If end_level == -1 it will search
+  // in levels [start_level, end_level). If end_level == 0 it will search
  // through all non-empty levels
  uint64_t ApproximateSize(Version* v, const Slice& start, const Slice& end,
-                           int start_level, int end_level,
-                           TableReaderCaller caller);
+                           int start_level, int end_level, bool for_compaction);

  // Return the size of the current manifest file
  uint64_t manifest_file_size() const { return manifest_file_size_; }
@@ -1036,10 +1035,10 @@ class VersionSet {
  // ApproximateSize helper
  uint64_t ApproximateSizeLevel0(Version* v, const LevelFilesBrief& files_brief,
                                 const Slice& start, const Slice& end,
-                                 TableReaderCaller caller);
+                                 bool for_compaction);

  uint64_t ApproximateSize(Version* v, const FdWithKeyRange& f,
-                           const Slice& key, TableReaderCaller caller);
+                           const Slice& key, bool for_compaction);

  // Save current contents to *log
  Status WriteSnapshot(log::Writer* log);
@@ -293,29 +293,6 @@ TEST_F(WalManagerTest, TransactionLogIteratorJustEmptyFile) {
  // Check that an empty iterator is returned
  ASSERT_TRUE(!iter->Valid());
 }
-  
-TEST_F(WalManagerTest, TransactionLogIteratorNewFileWhileScanning) {
-  Init();
-  CreateArchiveLogs(2, 100);
-  auto iter = OpenTransactionLogIter(0);
-  CreateArchiveLogs(1, 100);
-  int i = 0;
-  for (; iter->Valid(); iter->Next()) {
-    i++;
-  }
-  ASSERT_EQ(i, 200);
-  // A new log file was added after the iterator was created.
-  // TryAgain indicates a new iterator is needed to fetch the new data
-  ASSERT_TRUE(iter->status().IsTryAgain());
-  
-  iter = OpenTransactionLogIter(0);
-  i = 0;
-  for (; iter->Valid(); iter->Next()) {
-    i++;
-  }
-  ASSERT_EQ(i, 300);
-  ASSERT_TRUE(iter->status().ok());
-}

 }  // namespace rocksdb

@@ -135,105 +135,6 @@ struct BatchContentClassifier : public WriteBatch::Handler {
  }
 };

-class TimestampAssigner : public WriteBatch::Handler {
- public:
-  explicit TimestampAssigner(const Slice& ts)
-      : timestamp_(ts), timestamps_(kEmptyTimestampList) {}
-  explicit TimestampAssigner(const std::vector<Slice>& ts_list)
-      : timestamps_(ts_list) {
-    SanityCheck();
-  }
-  ~TimestampAssigner() override {}
-
-  Status PutCF(uint32_t, const Slice& key, const Slice&) override {
-    AssignTimestamp(key);
-    ++idx_;
-    return Status::OK();
-  }
-
-  Status DeleteCF(uint32_t, const Slice& key) override {
-    AssignTimestamp(key);
-    ++idx_;
-    return Status::OK();
-  }
-
-  Status SingleDeleteCF(uint32_t, const Slice& key) override {
-    AssignTimestamp(key);
-    ++idx_;
-    return Status::OK();
-  }
-
-  Status DeleteRangeCF(uint32_t, const Slice& begin_key,
-                       const Slice& end_key) override {
-    AssignTimestamp(begin_key);
-    AssignTimestamp(end_key);
-    ++idx_;
-    return Status::OK();
-  }
-
-  Status MergeCF(uint32_t, const Slice& key, const Slice&) override {
-    AssignTimestamp(key);
-    ++idx_;
-    return Status::OK();
-  }
-
-  Status PutBlobIndexCF(uint32_t, const Slice&, const Slice&) override {
-    // TODO (yanqin): support blob db in the future.
-    return Status::OK();
-  }
-
-  Status MarkBeginPrepare(bool) override {
-    // TODO (yanqin): support in the future.
-    return Status::OK();
-  }
-
-  Status MarkEndPrepare(const Slice&) override {
-    // TODO (yanqin): support in the future.
-    return Status::OK();
-  }
-
-  Status MarkCommit(const Slice&) override {
-    // TODO (yanqin): support in the future.
-    return Status::OK();
-  }
-
-  Status MarkRollback(const Slice&) override {
-    // TODO (yanqin): support in the future.
-    return Status::OK();
-  }
-
- private:
-  void SanityCheck() const {
-    assert(!timestamps_.empty());
-#ifndef NDEBUG
-    const size_t ts_sz = timestamps_[0].size();
-    for (size_t i = 1; i != timestamps_.size(); ++i) {
-      assert(ts_sz == timestamps_[i].size());
-    }
-#endif  // !NDEBUG
-  }
-
-  void AssignTimestamp(const Slice& key) {
-    assert(timestamps_.empty() || idx_ < timestamps_.size());
-    const Slice& ts = timestamps_.empty() ? timestamp_ : timestamps_[idx_];
-    size_t ts_sz = ts.size();
-    char* ptr = const_cast<char*>(key.data() + key.size() - ts_sz);
-    memcpy(ptr, ts.data(), ts_sz);
-  }
-
-  static const std::vector<Slice> kEmptyTimestampList;
-  const Slice timestamp_;
-  const std::vector<Slice>& timestamps_;
-  size_t idx_ = 0;
-
-  // No copy or move.
-  TimestampAssigner(const TimestampAssigner&) = delete;
-  TimestampAssigner(TimestampAssigner&&) = delete;
-  TimestampAssigner& operator=(const TimestampAssigner&) = delete;
-  TimestampAssigner&& operator=(TimestampAssigner&&) = delete;
-};
-const std::vector<Slice> TimestampAssigner::kEmptyTimestampList;
-
 }  // anon namespace

 struct SavePoints {
@@ -241,15 +142,7 @@ struct SavePoints {
 };

 WriteBatch::WriteBatch(size_t reserved_bytes, size_t max_bytes)
-    : content_flags_(0), max_bytes_(max_bytes), rep_(), timestamp_size_(0) {
-  rep_.reserve((reserved_bytes > WriteBatchInternal::kHeader)
-                   ? reserved_bytes
-                   : WriteBatchInternal::kHeader);
-  rep_.resize(WriteBatchInternal::kHeader);
-}
-
-WriteBatch::WriteBatch(size_t reserved_bytes, size_t max_bytes, size_t ts_sz)
-    : content_flags_(0), max_bytes_(max_bytes), rep_(), timestamp_size_(ts_sz) {
+    : content_flags_(0), max_bytes_(max_bytes), rep_() {
  rep_.reserve((reserved_bytes > WriteBatchInternal::kHeader) ?
    reserved_bytes : WriteBatchInternal::kHeader);
  rep_.resize(WriteBatchInternal::kHeader);
@@ -258,21 +151,18 @@ WriteBatch::WriteBatch(size_t reserved_bytes, size_t max_bytes, size_t ts_sz)
 WriteBatch::WriteBatch(const std::string& rep)
    : content_flags_(ContentFlags::DEFERRED),
      max_bytes_(0),
-      rep_(rep),
-      timestamp_size_(0) {}
+      rep_(rep) {}

 WriteBatch::WriteBatch(std::string&& rep)
    : content_flags_(ContentFlags::DEFERRED),
      max_bytes_(0),
-      rep_(std::move(rep)),
-      timestamp_size_(0) {}
+      rep_(std::move(rep)) {}

 WriteBatch::WriteBatch(const WriteBatch& src)
    : wal_term_point_(src.wal_term_point_),
      content_flags_(src.content_flags_.load(std::memory_order_relaxed)),
      max_bytes_(src.max_bytes_),
-      rep_(src.rep_),
-      timestamp_size_(src.timestamp_size_) {
+      rep_(src.rep_) {
  if (src.save_points_ != nullptr) {
    save_points_.reset(new SavePoints());
    save_points_->stack = src.save_points_->stack;
@@ -284,8 +174,7 @@ WriteBatch::WriteBatch(WriteBatch&& src) noexcept
      wal_term_point_(std::move(src.wal_term_point_)),
      content_flags_(src.content_flags_.load(std::memory_order_relaxed)),
      max_bytes_(src.max_bytes_),
-      rep_(std::move(src.rep_)),
-      timestamp_size_(src.timestamp_size_) {}
+      rep_(std::move(src.rep_)) {}

 WriteBatch& WriteBatch::operator=(const WriteBatch& src) {
  if (&src != this) {
@@ -754,14 +643,7 @@ Status WriteBatchInternal::Put(WriteBatch* b, uint32_t column_family_id,
    b->rep_.push_back(static_cast<char>(kTypeColumnFamilyValue));
    PutVarint32(&b->rep_, column_family_id);
  }
-  if (0 == b->timestamp_size_) {
-    PutLengthPrefixedSlice(&b->rep_, key);
-  } else {
-    PutVarint32(&b->rep_,
-                static_cast<uint32_t>(key.size() + b->timestamp_size_));
-    b->rep_.append(key.data(), key.size());
-    b->rep_.append(b->timestamp_size_, '\0');
-  }
+  PutLengthPrefixedSlice(&b->rep_, key);
  PutLengthPrefixedSlice(&b->rep_, value);
  b->content_flags_.store(
      b->content_flags_.load(std::memory_order_relaxed) | ContentFlags::HAS_PUT,
@@ -810,11 +692,7 @@ Status WriteBatchInternal::Put(WriteBatch* b, uint32_t column_family_id,
    b->rep_.push_back(static_cast<char>(kTypeColumnFamilyValue));
    PutVarint32(&b->rep_, column_family_id);
  }
-  if (0 == b->timestamp_size_) {
-    PutLengthPrefixedSliceParts(&b->rep_, key);
-  } else {
-    PutLengthPrefixedSlicePartsWithPadding(&b->rep_, key, b->timestamp_size_);
-  }
+  PutLengthPrefixedSliceParts(&b->rep_, key);
  PutLengthPrefixedSliceParts(&b->rep_, value);
  b->content_flags_.store(
      b->content_flags_.load(std::memory_order_relaxed) | ContentFlags::HAS_PUT,
@@ -1160,16 +1038,6 @@ Status WriteBatch::PopSavePoint() {
  return Status::OK();
 }

-Status WriteBatch::AssignTimestamp(const Slice& ts) {
-  TimestampAssigner ts_assigner(ts);
-  return Iterate(&ts_assigner);
-}
-
-Status WriteBatch::AssignTimestamps(const std::vector<Slice>& ts_list) {
-  TimestampAssigner ts_assigner(ts_list);
-  return Iterate(&ts_assigner);
-}
-
 class MemTableInserter : public WriteBatch::Handler {

  SequenceNumber sequence_;
@@ -8,13 +8,9 @@ def test_binary(
        test_cc,
        parallelism,
        rocksdb_arch_preprocessor_flags,
-        rocksdb_os_preprocessor_flags,
        rocksdb_compiler_flags,
        rocksdb_preprocessor_flags,
-        rocksdb_external_deps,
-        rocksdb_os_deps,
-        extra_deps,
-        extra_compiler_flags):
+        rocksdb_external_deps):
    TEST_RUNNER = native.package_name() + "/buckifier/rocks_test_runner.sh"

    ttype = "gtest" if parallelism == "parallel" else "simple"
@@ -24,11 +20,9 @@ def test_binary(
        name = test_bin,
        srcs = [test_cc],
        arch_preprocessor_flags = rocksdb_arch_preprocessor_flags,
-        os_preprocessor_flags = rocksdb_os_preprocessor_flags,
-        compiler_flags = rocksdb_compiler_flags + extra_compiler_flags,
+        compiler_flags = rocksdb_compiler_flags,
        preprocessor_flags = rocksdb_preprocessor_flags,
-        deps = [":rocksdb_test_lib"] + extra_deps,
-        os_deps = rocksdb_os_deps,
+        deps = [":rocksdb_test_lib"],
        external_deps = rocksdb_external_deps,
    )

@@ -10,13 +10,11 @@
 #include "rocksdb/env.h"

 #include <thread>
-#include "logging/env_logger.h"
 #include "memory/arena.h"
 #include "options/db_options.h"
 #include "port/port.h"
 #include "port/sys_time.h"
 #include "rocksdb/options.h"
-#include "rocksdb/utilities/object_registry.h"
 #include "util/autovector.h"

 namespace rocksdb {
@@ -24,25 +22,6 @@ namespace rocksdb {
 Env::~Env() {
 }

-Status Env::NewLogger(const std::string& fname,
-                      std::shared_ptr<Logger>* result) {
-  return NewEnvLogger(fname, this, result);
-}
-
-Status Env::LoadEnv(const std::string& value, Env** result) {
-  Env* env = *result;
-  Status s;
-#ifndef ROCKSDB_LITE
-  s = ObjectRegistry::NewInstance()->NewStaticObject<Env>(value, &env);
-#else
-  s = Status::NotSupported("Cannot load environment in LITE mode: ", value);
-#endif
-  if (s.ok()) {
-    *result = env;
-  }
-  return s;
-}
-
 std::string Env::PriorityToString(Env::Priority priority) {
  switch (priority) {
    case Env::Priority::BOTTOM:
@@ -443,20 +422,5 @@ EnvOptions::EnvOptions() {
  AssignEnvOptions(this, options);
 }

-Status NewEnvLogger(const std::string& fname, Env* env,
-                    std::shared_ptr<Logger>* result) {
-  EnvOptions options;
-  // TODO: Tune the buffer size.
-  options.writable_file_max_buffer_size = 1024 * 1024;
-  std::unique_ptr<WritableFile> writable_file;
-  const auto status = env->NewWritableFile(fname, &writable_file, options);
-  if (!status.ok()) {
-    return status;
-  }
-
-  *result = std::make_shared<EnvLogger>(std::move(writable_file), fname,
-                                        options, env);
-  return Status::OK();
-}

 }  // namespace rocksdb
@@ -11,6 +11,7 @@

 #include "env/mock_env.h"
 #include "rocksdb/env.h"
+#include "rocksdb/utilities/object_registry.h"
 #include "test_util/testharness.h"

 namespace rocksdb {
@@ -103,12 +104,13 @@ namespace {
 // ValuesIn() will skip running tests when given an empty collection.
 std::vector<Env*> GetCustomEnvs() {
  static Env* custom_env;
+  static std::unique_ptr<Env> custom_env_guard;
  static bool init = false;
  if (!init) {
    init = true;
    const char* uri = getenv("TEST_ENV_URI");
    if (uri != nullptr) {
-      Env::LoadEnv(uri, &custom_env);
+      custom_env = NewCustomObject<Env>(uri, &custom_env_guard);
    }
  }

@@ -195,26 +195,23 @@ class EncryptedWritableFile : public WritableFileWrapper {
  EncryptedWritableFile(WritableFile* f, BlockAccessCipherStream* s, size_t prefixLength)
    : WritableFileWrapper(f), file_(f), stream_(s), prefixLength_(prefixLength) { }

-  Status Append(const Slice& data) override {
+  Status Append(const Slice& data) override { 
    AlignedBuffer buf;
    Status status;
-    Slice dataToAppend(data);
+    Slice dataToAppend(data); 
    if (data.size() > 0) {
      auto offset = file_->GetFileSize(); // size including prefix
      // Encrypt in cloned buffer
      buf.Alignment(GetRequiredBufferAlignment());
      buf.AllocateNewBuffer(data.size());
-      // TODO (sagar0): Modify AlignedBuffer.Append to allow doing a memmove
-      // so that the next two lines can be replaced with buf.Append().
      memmove(buf.BufferStart(), data.data(), data.size());
-      buf.Size(data.size());
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      status = stream_->Encrypt(offset, buf.BufferStart(), data.size());
      if (!status.ok()) {
        return status;
      }
-      dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
+      dataToAppend = Slice(buf.BufferStart(), data.size());
    }
-    status = file_->Append(dataToAppend);
+    status = file_->Append(dataToAppend); 
    if (!status.ok()) {
      return status;
    }
@@ -224,19 +221,18 @@ class EncryptedWritableFile : public WritableFileWrapper {
  Status PositionedAppend(const Slice& data, uint64_t offset) override {
    AlignedBuffer buf;
    Status status;
-    Slice dataToAppend(data);
+    Slice dataToAppend(data); 
    offset += prefixLength_;
    if (data.size() > 0) {
      // Encrypt in cloned buffer
      buf.Alignment(GetRequiredBufferAlignment());
      buf.AllocateNewBuffer(data.size());
      memmove(buf.BufferStart(), data.data(), data.size());
-      buf.Size(data.size());
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      status = stream_->Encrypt(offset, buf.BufferStart(), data.size());
      if (!status.ok()) {
        return status;
      }
-      dataToAppend = Slice(buf.BufferStart(), buf.CurrentSize());
+      dataToAppend = Slice(buf.BufferStart(), data.size());
    }
    status = file_->PositionedAppend(dataToAppend, offset);
    if (!status.ok()) {
@@ -329,19 +325,18 @@ class EncryptedRandomRWFile : public RandomRWFile {
  Status Write(uint64_t offset, const Slice& data) override {
    AlignedBuffer buf;
    Status status;
-    Slice dataToWrite(data);
+    Slice dataToWrite(data); 
    offset += prefixLength_;
    if (data.size() > 0) {
      // Encrypt in cloned buffer
      buf.Alignment(GetRequiredBufferAlignment());
      buf.AllocateNewBuffer(data.size());
      memmove(buf.BufferStart(), data.data(), data.size());
-      buf.Size(data.size());
-      status = stream_->Encrypt(offset, buf.BufferStart(), buf.CurrentSize());
+      status = stream_->Encrypt(offset, buf.BufferStart(), data.size());
      if (!status.ok()) {
        return status;
      }
-      dataToWrite = Slice(buf.BufferStart(), buf.CurrentSize());
+      dataToWrite = Slice(buf.BufferStart(), data.size());
    }
    status = file_->Write(offset, dataToWrite);
    return status;
@@ -398,14 +393,13 @@ class EncryptedEnv : public EnvWrapper {
    Slice prefixSlice;
    size_t prefixLength = provider_->GetPrefixLength();
    if (prefixLength > 0) {
-      // Read prefix
+      // Read prefix 
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
      status = underlying->Read(prefixLength, &prefixSlice, prefixBuf.BufferStart());
      if (!status.ok()) {
        return status;
      }
-      prefixBuf.Size(prefixLength);
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
@@ -436,14 +430,13 @@ class EncryptedEnv : public EnvWrapper {
    Slice prefixSlice;
    size_t prefixLength = provider_->GetPrefixLength();
    if (prefixLength > 0) {
-      // Read prefix
+      // Read prefix 
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
      status = underlying->Read(0, prefixLength, &prefixSlice, prefixBuf.BufferStart());
      if (!status.ok()) {
        return status;
      }
-      prefixBuf.Size(prefixLength);
    }
    // Create cipher stream
    std::unique_ptr<BlockAccessCipherStream> stream;
@@ -474,13 +467,12 @@ class EncryptedEnv : public EnvWrapper {
    Slice prefixSlice;
    size_t prefixLength = provider_->GetPrefixLength();
    if (prefixLength > 0) {
-      // Initialize prefix
+      // Initialize prefix 
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
      provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(), prefixLength);
-      prefixBuf.Size(prefixLength);
-      prefixSlice = Slice(prefixBuf.BufferStart(), prefixBuf.CurrentSize());
-      // Write prefix
+      prefixSlice = Slice(prefixBuf.BufferStart(), prefixLength);
+      // Write prefix 
      status = underlying->Append(prefixSlice);
      if (!status.ok()) {
        return status;
@@ -521,13 +513,12 @@ class EncryptedEnv : public EnvWrapper {
    Slice prefixSlice;
    size_t prefixLength = provider_->GetPrefixLength();
    if (prefixLength > 0) {
-      // Initialize prefix
+      // Initialize prefix 
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
      provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(), prefixLength);
-      prefixBuf.Size(prefixLength);
-      prefixSlice = Slice(prefixBuf.BufferStart(), prefixBuf.CurrentSize());
-      // Write prefix
+      prefixSlice = Slice(prefixBuf.BufferStart(), prefixLength);
+      // Write prefix 
      status = underlying->Append(prefixSlice);
      if (!status.ok()) {
        return status;
@@ -563,13 +554,12 @@ class EncryptedEnv : public EnvWrapper {
    Slice prefixSlice;
    size_t prefixLength = provider_->GetPrefixLength();
    if (prefixLength > 0) {
-      // Initialize prefix
+      // Initialize prefix 
      prefixBuf.Alignment(underlying->GetRequiredBufferAlignment());
      prefixBuf.AllocateNewBuffer(prefixLength);
      provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(), prefixLength);
-      prefixBuf.Size(prefixLength);
-      prefixSlice = Slice(prefixBuf.BufferStart(), prefixBuf.CurrentSize());
-      // Write prefix
+      prefixSlice = Slice(prefixBuf.BufferStart(), prefixLength);
+      // Write prefix 
      status = underlying->Append(prefixSlice);
      if (!status.ok()) {
        return status;
@@ -619,13 +609,11 @@ class EncryptedEnv : public EnvWrapper {
        if (!status.ok()) {
          return status;
        }
-        prefixBuf.Size(prefixLength);
      } else {
-        // File is new, initialize & write prefix
+        // File is new, initialize & write prefix 
        provider_->CreateNewPrefix(fname, prefixBuf.BufferStart(), prefixLength);
-        prefixBuf.Size(prefixLength);
-        prefixSlice = Slice(prefixBuf.BufferStart(), prefixBuf.CurrentSize());
-        // Write prefix
+        prefixSlice = Slice(prefixBuf.BufferStart(), prefixLength);
+        // Write prefix 
        status = underlying->Write(0, prefixSlice);
        if (!status.ok()) {
          return status;
@@ -642,7 +630,7 @@ class EncryptedEnv : public EnvWrapper {
    return Status::OK();
  }

-  // Store in *result the attributes of the children of the specified directory.
+    // Store in *result the attributes of the children of the specified directory.
  // In case the implementation lists the directory prior to iterating the files
  // and files are concurrently deleted, the deleted files will be omitted from
  // result.
@@ -682,7 +670,8 @@ class EncryptedEnv : public EnvWrapper {
  EncryptionProvider *provider_;
 };

-// Returns an Env that encrypts data when stored on disk and decrypts data when
+
+// Returns an Env that encrypts data when stored on disk and decrypts data when 
 // read from disk.
 Env* NewEncryptedEnv(Env* base_env, EncryptionProvider* provider) {
  return new EncryptedEnv(base_env, provider);
@@ -705,14 +694,14 @@ Status BlockAccessCipherStream::Encrypt(uint64_t fileOffset, char *data, size_t
    char *block = data;
    size_t n = std::min(dataSize, blockSize - blockOffset);
    if (n != blockSize) {
-      // We're not encrypting a full block.
+      // We're not encrypting a full block. 
      // Copy data to blockBuffer
      if (!blockBuffer.get()) {
        // Allocate buffer
        blockBuffer = std::unique_ptr<char[]>(new char[blockSize]);
      }
      block = blockBuffer.get();
-      // Copy plain data to block buffer
+      // Copy plain data to block buffer 
      memmove(block + blockOffset, data, n);
    }
    auto status = EncryptBlock(blockIndex, block, (char*)scratch.data());
@@ -752,14 +741,14 @@ Status BlockAccessCipherStream::Decrypt(uint64_t fileOffset, char *data, size_t
    char *block = data;
    size_t n = std::min(dataSize, blockSize - blockOffset);
    if (n != blockSize) {
-      // We're not decrypting a full block.
+      // We're not decrypting a full block. 
      // Copy data to blockBuffer
      if (!blockBuffer.get()) {
        // Allocate buffer
        blockBuffer = std::unique_ptr<char[]>(new char[blockSize]);
      }
      block = blockBuffer.get();
-      // Copy encrypted data to block buffer
+      // Copy encrypted data to block buffer 
      memmove(block + blockOffset, data, n);
    }
    auto status = DecryptBlock(blockIndex, block, (char*)scratch.data());
@@ -818,7 +807,7 @@ Status CTRCipherStream::EncryptBlock(uint64_t blockIndex, char *data, char* scra
  memmove(scratch, iv_.data(), blockSize);
  EncodeFixed64(scratch, blockIndex + initialCounter_);

-  // Encrypt nonce+counter
+  // Encrypt nonce+counter 
  auto status = cipher_.Encrypt(scratch);
  if (!status.ok()) {
    return status;
@@ -834,13 +823,13 @@ Status CTRCipherStream::EncryptBlock(uint64_t blockIndex, char *data, char* scra
 // Decrypt a block of data at the given block index.
 // Length of data is equal to BlockSize();
 Status CTRCipherStream::DecryptBlock(uint64_t blockIndex, char *data, char* scratch) {
-  // For CTR decryption & encryption are the same
+  // For CTR decryption & encryption are the same 
  return EncryptBlock(blockIndex, data, scratch);
 }

 // GetPrefixLength returns the length of the prefix that is added to every file
 // and used for storing encryption options.
-// For optimal performance, the prefix length should be a multiple of
+// For optimal performance, the prefix length should be a multiple of 
 // the page size.
 size_t CTREncryptionProvider::GetPrefixLength() {
  return defaultPrefixLength;
@@ -855,7 +844,7 @@ static void decodeCTRParameters(const char *prefix, size_t blockSize, uint64_t &
  iv = Slice(prefix + blockSize, blockSize);
 }

-// CreateNewPrefix initialized an allocated block of prefix memory
+// CreateNewPrefix initialized an allocated block of prefix memory 
 // for a new file.
 Status CTREncryptionProvider::CreateNewPrefix(const std::string& /*fname*/,
                                              char* prefix,
@@ -884,7 +873,7 @@ Status CTREncryptionProvider::CreateNewPrefix(const std::string& /*fname*/,
  return Status::OK();
 }

-// PopulateSecretPrefixPart initializes the data into a new prefix block
+// PopulateSecretPrefixPart initializes the data into a new prefix block 
 // in plain text.
 // Returns the amount of space (starting from the start of the prefix)
 // that has been initialized.
@@ -919,7 +908,7 @@ Status CTREncryptionProvider::CreateCipherStream(
    return status;
  }

-  // Create cipher stream
+  // Create cipher stream 
  return CreateCipherStreamFromPrefix(fname, options, initialCounter, iv, prefix, result);
 }

@@ -420,7 +420,7 @@ Status HdfsEnv::NewRandomAccessFile(const std::string& fname,
 // create a new file for writing
 Status HdfsEnv::NewWritableFile(const std::string& fname,
                                std::unique_ptr<WritableFile>* result,
-                                const EnvOptions& options) {
+                                const EnvOptions& /*options*/) {
  result->reset();
  Status s;
  HdfsWritableFile* f = new HdfsWritableFile(fileSys_, fname, options);
@@ -590,11 +590,6 @@ Status HdfsEnv::UnlockFile(FileLock* /*lock*/) { return Status::OK(); }

 Status HdfsEnv::NewLogger(const std::string& fname,
                          std::shared_ptr<Logger>* result) {
-  // EnvOptions is used exclusively for its `strict_bytes_per_sync` value. That
-  // option is only intended for WAL/flush/compaction writes, so turn it off in
-  // the logger.
-  EnvOptions options;
-  options.strict_bytes_per_sync = false;
  HdfsWritableFile* f = new HdfsWritableFile(fileSys_, fname, options);
  if (f == nullptr || !f->isValid()) {
    delete f;
@@ -887,14 +887,13 @@ class PosixEnv : public Env {
    FILE* f;
    {
      IOSTATS_TIMER_GUARD(open_nanos);
-      f = fopen(fname.c_str(),
-                "w"
+      f = fopen(fname.c_str(), "w"
 #ifdef __GLIBC_PREREQ
 #if __GLIBC_PREREQ(2, 7)
-                "e"  // glibc extension to enable O_CLOEXEC
+          "e" // glibc extension to enable O_CLOEXEC
 #endif
 #endif
-      );
+          );
    }
    if (f == nullptr) {
      result->reset();
@@ -14,7 +14,6 @@
 #include <algorithm>
 #if defined(OS_LINUX)
 #include <linux/fs.h>
-#include <linux/falloc.h>
 #endif
 #include <stdio.h>
 #include <stdlib.h>
@@ -28,6 +27,7 @@
 #include <sys/syscall.h>
 #include <sys/sysmacros.h>
 #endif
+#include "logging/posix_logger.h"
 #include "monitoring/iostats_context_imp.h"
 #include "port/port.h"
 #include "rocksdb/slice.h"
@@ -804,8 +804,8 @@ Status PosixMmapFile::InvalidateCache(size_t offset, size_t length) {

 #ifdef ROCKSDB_FALLOCATE_PRESENT
 Status PosixMmapFile::Allocate(uint64_t offset, uint64_t len) {
-  assert(offset <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
-  assert(len <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
+  assert(offset <= std::numeric_limits<off_t>::max());
+  assert(len <= std::numeric_limits<off_t>::max());
  TEST_KILL_RANDOM("PosixMmapFile::Allocate:0", rocksdb_kill_odds);
  int alloc_status = 0;
  if (allow_fallocate_) {
@@ -874,7 +874,7 @@ Status PosixWritableFile::PositionedAppend(const Slice& data, uint64_t offset) {
    assert(IsSectorAligned(data.size(), GetRequiredBufferAlignment()));
    assert(IsSectorAligned(data.data(), GetRequiredBufferAlignment()));
  }
-  assert(offset <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
+  assert(offset <= std::numeric_limits<off_t>::max());
  const char* src = data.data();
  size_t nbytes = data.size();
  if (!PosixPositionedWrite(fd_, src, nbytes, static_cast<off_t>(offset))) {
@@ -1010,8 +1010,8 @@ Status PosixWritableFile::InvalidateCache(size_t offset, size_t length) {

 #ifdef ROCKSDB_FALLOCATE_PRESENT
 Status PosixWritableFile::Allocate(uint64_t offset, uint64_t len) {
-  assert(offset <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
-  assert(len <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
+  assert(offset <= std::numeric_limits<off_t>::max());
+  assert(len <= std::numeric_limits<off_t>::max());
  TEST_KILL_RANDOM("PosixWritableFile::Allocate:0", rocksdb_kill_odds);
  IOSTATS_TIMER_GUARD(allocate_nanos);
  int alloc_status = 0;
@@ -1032,8 +1032,8 @@ Status PosixWritableFile::Allocate(uint64_t offset, uint64_t len) {

 Status PosixWritableFile::RangeSync(uint64_t offset, uint64_t nbytes) {
 #ifdef ROCKSDB_RANGESYNC_PRESENT
-  assert(offset <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
-  assert(nbytes <= static_cast<uint64_t>(std::numeric_limits<off_t>::max()));
+  assert(offset <= std::numeric_limits<off_t>::max());
+  assert(nbytes <= std::numeric_limits<off_t>::max());
  if (sync_file_range_supported_) {
    int ret;
    if (strict_bytes_per_sync_) {
@@ -110,7 +110,6 @@ Status DeleteDBFile(const ImmutableDBOptions* db_options,

 bool IsWalDirSameAsDBPath(const ImmutableDBOptions* db_options) {
  bool same = false;
-  assert(!db_options->db_paths.empty());
  Status s = db_options->env->AreFilesSame(db_options->wal_dir,
                                           db_options->db_paths[0].path, &same);
  if (s.IsNotSupported()) {
@@ -59,7 +59,7 @@ struct LRUCacheOptions {
  //
  // See also
  // BlockBasedTableOptions::cache_index_and_filter_blocks_with_high_priority.
-  double high_pri_pool_ratio = 0.5;
+  double high_pri_pool_ratio = 0.0;

  // If non-nullptr will use this allocator instead of system allocator when
  // allocating memory for cache blocks. Call this method before you start using
@@ -99,7 +99,7 @@ struct LRUCacheOptions {
 // will be at least 512KB and number of shard bits will not exceed 6.
 extern std::shared_ptr<Cache> NewLRUCache(
    size_t capacity, int num_shard_bits = -1,
-    bool strict_capacity_limit = false, double high_pri_pool_ratio = 0.5,
+    bool strict_capacity_limit = false, double high_pri_pool_ratio = 0.0,
    std::shared_ptr<MemoryAllocator> memory_allocator = nullptr,
    bool use_adaptive_mutex = kDefaultToAdaptiveMutex);

@@ -250,6 +250,11 @@ class Cache {

  virtual std::string GetPrintableOptions() const { return ""; }

+  // Mark the last inserted object as being a raw data block. This will be used
+  // in tests. The default implementation does nothing.
+  virtual void TEST_mark_as_data_block(const Slice& /*key*/,
+                                       size_t /*charge*/) {}
+
  MemoryAllocator* memory_allocator() const { return memory_allocator_.get(); }

 private:
@@ -35,7 +35,6 @@ class Comparator {

  virtual ~Comparator() {}

-  static const char* Type() { return "Comparator"; }
  // Three-way comparison.  Returns value:
  //   < 0 iff "a" < "b",
  //   == 0 iff "a" == "b",
@@ -808,7 +808,7 @@ class DB {
  // stats should be included, or file stats approximation or both
  enum SizeApproximationFlags : uint8_t {
    NONE = 0,
-    INCLUDE_MEMTABLES = 1 << 0,
+    INCLUDE_MEMTABLES = 1,
    INCLUDE_FILES = 1 << 1
  };

@@ -818,24 +818,14 @@ class DB {
  // Note that the returned sizes measure file system space usage, so
  // if the user data compresses by a factor of ten, the returned
  // sizes will be one-tenth the size of the corresponding user data size.
-  virtual Status GetApproximateSizes(const SizeApproximationOptions& options,
-                                     ColumnFamilyHandle* column_family,
-                                     const Range* range, int n,
-                                     uint64_t* sizes) = 0;
-
-  // Simpler versions of the GetApproximateSizes() method above.
-  // The include_flags argumenbt must of type DB::SizeApproximationFlags
-  // and can not be NONE.
+  //
+  // If include_flags defines whether the returned size should include
+  // the recently written data in the mem-tables (if
+  // the mem-table type supports it), data serialized to disk, or both.
+  // include_flags should be of type DB::SizeApproximationFlags
  virtual void GetApproximateSizes(ColumnFamilyHandle* column_family,
                                   const Range* range, int n, uint64_t* sizes,
-                                   uint8_t include_flags = INCLUDE_FILES) {
-    SizeApproximationOptions options;
-    options.include_memtabtles =
-        (include_flags & SizeApproximationFlags::INCLUDE_MEMTABLES) != 0;
-    options.include_files =
-        (include_flags & SizeApproximationFlags::INCLUDE_FILES) != 0;
-    GetApproximateSizes(options, column_family, range, n, sizes);
-  }
+                                   uint8_t include_flags = INCLUDE_FILES) = 0;
  virtual void GetApproximateSizes(const Range* range, int n, uint64_t* sizes,
                                   uint8_t include_flags = INCLUDE_FILES) {
    GetApproximateSizes(DefaultColumnFamily(), range, n, sizes, include_flags);
@@ -1184,27 +1174,6 @@ class DB {
  virtual Status IngestExternalFiles(
      const std::vector<IngestExternalFileArg>& args) = 0;

-  // CreateColumnFamilyWithImport() will create a new column family with
-  // column_family_name and import external SST files specified in metadata into
-  // this column family.
-  // (1) External SST files can be created using SstFileWriter.
-  // (2) External SST files can be exported from a particular column family in
-  //     an existing DB.
-  // Option in import_options specifies whether the external files are copied or
-  // moved (default is copy). When option specifies copy, managing files at
-  // external_file_path is caller's responsibility. When option specifies a
-  // move, the call ensures that the specified files at external_file_path are
-  // deleted on successful return and files are not modified on any error
-  // return.
-  // On error return, column family handle returned will be nullptr.
-  // ColumnFamily will be present on successful return and will not be present
-  // on error return. ColumnFamily may be present on any crash during this call.
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& options, const std::string& column_family_name,
-      const ImportColumnFamilyOptions& import_options,
-      const ExportImportFilesMetaData& metadata,
-      ColumnFamilyHandle** handle) = 0;
-
  virtual Status VerifyChecksum() = 0;

  // AddFile() is deprecated, please use IngestExternalFile()
@@ -118,10 +118,10 @@ struct EnvOptions {
  bool fallocate_with_keep_size = true;

  // See DBOptions doc
-  size_t compaction_readahead_size = 0;
+  size_t compaction_readahead_size;

  // See DBOptions doc
-  size_t random_access_max_buffer_size = 0;
+  size_t random_access_max_buffer_size;

  // See DBOptions doc
  size_t writable_file_max_buffer_size = 1024 * 1024;
@@ -144,11 +144,6 @@ class Env {

  virtual ~Env();

-  static const char* Type() { return "Environment"; }
-
-  // Loads the environment specified by the input value into the result
-  static Status LoadEnv(const std::string& value, Env** result);
-
  // Return a default environment suitable for the current operating
  // system.  Sophisticated users may wish to provide their own Env
  // implementation instead of relying on this default environment.
@@ -400,11 +395,9 @@ class Env {
  // same directory.
  virtual Status GetTestDirectory(std::string* path) = 0;

-  // Create and returns a default logger (an instance of EnvLogger) for storing
-  // informational messages. Derived classes can overide to provide custom
-  // logger.
+  // Create and return a log file for storing informational messages.
  virtual Status NewLogger(const std::string& fname,
-                           std::shared_ptr<Logger>* result);
+                           std::shared_ptr<Logger>* result) = 0;

  // Returns the number of micro-seconds since some fixed point in time.
  // It is often used as system time such as in GenericRateLimiter
@@ -1570,10 +1563,4 @@ Status NewHdfsEnv(Env** hdfs_env, const std::string& fsname);
 // This is a factory method for TimedEnv defined in utilities/env_timed.cc.
 Env* NewTimedEnv(Env* base_env);

-// Returns an instance of logger that can be used for storing informational
-// messages.
-// This is a factory method for EnvLogger declared in logging/env_logging.h
-Status NewEnvLogger(const std::string& fname, Env* env,
-                    std::shared_ptr<Logger>* result);
-
 }  // namespace rocksdb
@@ -46,7 +46,6 @@ class Logger;
 class MergeOperator {
 public:
  virtual ~MergeOperator() {}
-  static const char* Type() { return "MergeOperator"; }

  // Gives the client a way to express the read -> modify -> write semantics
  // key:      (IN)    The key that's associated with this merge operation.
@@ -108,11 +108,4 @@ struct LiveFileMetaData : SstFileMetaData {
  int level;                       // Level at which this file resides.
  LiveFileMetaData() : column_family_name(), level(0) {}
 };
-
-// Metadata returned as output from ExportColumnFamily() and used as input to
-// CreateColumnFamiliesWithImport().
-struct ExportImportFilesMetaData {
-  std::string db_comparator_name;       // Used to safety check at import.
-  std::vector<LiveFileMetaData> files;  // Vector of file metadata.
-};
 }  // namespace rocksdb
@@ -275,8 +275,6 @@ struct ColumnFamilyOptions : public AdvancedColumnFamilyOptions {
  // this option helps reducing the cpu usage of long-running compactions. The
  // feature is disabled when max_subcompactions is greater than one.
  //
-  // NOTE: This feautre is currently incompatible with RangeDeletes.
-  //
  // Default: 0
  //
  // Dynamically changeable through SetOptions() API
@@ -762,8 +760,6 @@ struct DBOptions {
  // for this mode if using block-based table.
  //
  // Default: false
-  // This flag has no affect on the behavior of compaction and plan to delete
-  // in the future.
  bool new_table_reader_for_compaction_inputs = false;

  // If non-zero, we perform bigger reads when doing compaction. If you're
@@ -1089,13 +1085,6 @@ struct DBOptions {
  // If set to true, takes precedence over
  // ReadOptions::background_purge_on_iterator_cleanup.
  bool avoid_unnecessary_blocking_io = false;
-
-  // The number of bytes to prefetch when reading the log. This is mostly useful
-  // for reading a remotely located log, as it can save the number of
-  // round-trips. If 0, then the prefetching is disabled.
-  //
-  // Default: 0
-  size_t log_readahead_size = 0;
 };

 // Options to control the behavior of a database (passed to DB::Open)
@@ -1500,20 +1489,4 @@ struct TraceOptions {
  uint64_t filter = kTraceFilterNone;
 };

-// ImportColumnFamilyOptions is used by ImportColumnFamily()
-struct ImportColumnFamilyOptions {
-  // Can be set to true to move the files instead of copying them.
-  bool move_files = false;
-};
-
-// Options used with DB::GetApproximateSizes()
-struct SizeApproximationOptions {
-  // Defines whether the returned size should include the recently written
-  // data in the mem-tables. If set to false, include_files must be true.
-  bool include_memtabtles = false;
-  // Defines whether the returned size should include data serialized to disk.
-  // If set to false, include_memtabtles must be true.
-  bool include_files = true;
-};
-
 }  // namespace rocksdb
@@ -480,7 +480,7 @@ enum StatsLevel : uint8_t {
 class Statistics {
 public:
  virtual ~Statistics() {}
-  static const char* Type() { return "Statistics"; }
+
  virtual uint64_t getTickerCount(uint32_t tickerType) const = 0;
  virtual void histogramData(uint32_t type,
                             HistogramData* const data) const = 0;
@@ -74,7 +74,7 @@ struct BlockBasedTableOptions {
  // blocks with high priority. If set to true, depending on implementation of
  // block cache, index and filter blocks may be less likely to be evicted
  // than data blocks.
-  bool cache_index_and_filter_blocks_with_high_priority = true;
+  bool cache_index_and_filter_blocks_with_high_priority = false;

  // if cache_index_and_filter_blocks is true and the below is true, then
  // filter and index blocks are stored in the cache, but a reference is
@@ -93,32 +93,14 @@ struct BlockBasedTableOptions {
  enum IndexType : char {
    // A space efficient index block that is optimized for
    // binary-search-based index.
-    kBinarySearch = 0x00,
+    kBinarySearch,

    // The hash index, if enabled, will do the hash lookup when
    // `Options.prefix_extractor` is provided.
-    kHashSearch = 0x01,
+    kHashSearch,

    // A two-level index implementation. Both levels are binary search indexes.
-    kTwoLevelIndexSearch = 0x02,
-
-    // Like kBinarySearch, but index also contains first key of each block.
-    // This allows iterators to defer reading the block until it's actually
-    // needed. May significantly reduce read amplification of short range scans.
-    // Without it, iterator seek usually reads one block from each level-0 file
-    // and from each level, which may be expensive.
-    // Works best in combination with:
-    //  - IndexShorteningMode::kNoShortening,
-    //  - custom FlushBlockPolicy to cut blocks at some meaningful boundaries,
-    //    e.g. when prefix changes.
-    // Makes the index significantly bigger (2x or more), especially when keys
-    // are long.
-    //
-    // IO errors are not handled correctly in this mode right now: if an error
-    // happens when lazily reading a block in value(), value() returns empty
-    // slice, and you need to call Valid()/status() afterwards.
-    // TODO(kolmike): Fix it.
-    kBinarySearchWithFirstKey = 0x03,
+    kTwoLevelIndexSearch,
  };

  IndexType index_type = kBinarySearch;
@@ -9,15 +9,11 @@
 #ifndef ROCKSDB_LITE

 #include <string>
-#include <vector>
 #include "rocksdb/status.h"

 namespace rocksdb {

 class DB;
-class ColumnFamilyHandle;
-struct LiveFileMetaData;
-struct ExportImportFilesMetaData;

 class Checkpoint {
 public:
@@ -40,16 +36,6 @@ class Checkpoint {
  virtual Status CreateCheckpoint(const std::string& checkpoint_dir,
                                  uint64_t log_size_for_flush = 0);

-  // Exports all live SST files of a specified Column Family onto export_dir,
-  // returning SST files information in metadata.
-  // - SST files will be created as hard links when the directory specified
-  //   is in the same partition as the db directory, copied otherwise.
-  // - export_dir should not already exist and will be created by this API.
-  // - Always triggers a flush.
-  virtual Status ExportColumnFamily(ColumnFamilyHandle* handle,
-                                    const std::string& export_dir,
-                                    ExportImportFilesMetaData** metadata);
-
  virtual ~Checkpoint() {}
 };

@@ -40,10 +40,6 @@ Status GetAllKeyVersions(DB* db, Slice begin_key, Slice end_key,
                         size_t max_num_ikeys,
                         std::vector<KeyVersion>* key_versions);

-Status GetAllKeyVersions(DB* db, ColumnFamilyHandle* cfh, Slice begin_key,
-                         Slice end_key, size_t max_num_ikeys,
-                         std::vector<KeyVersion>* key_versions);
-
 }  // namespace rocksdb

 #endif  // ROCKSDB_LITE
@@ -31,7 +31,6 @@ class LDBCommand {
  // Command-line arguments
  static const std::string ARG_DB;
  static const std::string ARG_PATH;
-  static const std::string ARG_SECONDARY_PATH;
  static const std::string ARG_HEX;
  static const std::string ARG_KEY_HEX;
  static const std::string ARG_VALUE_HEX;
@@ -129,10 +128,6 @@ class LDBCommand {
 protected:
  LDBCommandExecuteResult exec_state_;
  std::string db_path_;
-  // If empty, open DB as primary. If non-empty, open the DB as secondary
-  // with this secondary path. When running against a database opened by
-  // another process, ldb wll leave the source directory completely intact. 
-  std::string secondary_path_;
  std::string column_family_name_;
  DB* db_;
  DBWithTTL* db_ttl_;
@@ -11,195 +11,80 @@
 #include <memory>
 #include <regex>
 #include <string>
-#include <unordered_map>
 #include <vector>
-#include "rocksdb/status.h"
+
+#include "rocksdb/env.h"

 namespace rocksdb {
-class Logger;
+
+// Creates a new T using the factory function that was registered with a pattern
+// that matches the provided "target" string according to std::regex_match.
+//
+// If no registered functions match, returns nullptr. If multiple functions
+// match, the factory function used is unspecified.
+//
+// Populates res_guard with result pointer if caller is granted ownership.
+template <typename T>
+T* NewCustomObject(const std::string& target, std::unique_ptr<T>* res_guard);
+
 // Returns a new T when called with a string. Populates the std::unique_ptr
 // argument if granting ownership to caller.
 template <typename T>
-using FactoryFunc =
-    std::function<T*(const std::string&, std::unique_ptr<T>*, std::string*)>;
+using FactoryFunc = std::function<T*(const std::string&, std::unique_ptr<T>*)>;

-class ObjectLibrary {
+// To register a factory function for a type T, initialize a Registrar<T> object
+// with static storage duration. For example:
+//
+//   static Registrar<Env> hdfs_reg("hdfs://.*", &CreateHdfsEnv);
+//
+// Then, calling NewCustomObject<Env>("hdfs://some_path", ...) will match the
+// regex provided above, so it returns the result of invoking CreateHdfsEnv.
+template <typename T>
+class Registrar {
 public:
-  // Base class for an Entry in the Registry.
-  class Entry {
-   public:
-    virtual ~Entry() {}
-    Entry(const std::string& name) : name_(std::move(name)) {}
-
-    // Checks to see if the target matches this entry
-    virtual bool matches(const std::string& target) const {
-      return name_ == target;
-    }
-    const std::string& Name() const { return name_; }
-
-   private:
-    const std::string name_;  // The name of the Entry
-  };                          // End class Entry
-
-  // An Entry containing a FactoryFunc for creating new Objects
-  template <typename T>
-  class FactoryEntry : public Entry {
-   public:
-    FactoryEntry(const std::string& name, FactoryFunc<T> f)
-        : Entry(name), pattern_(std::move(name)), factory_(std::move(f)) {}
-    ~FactoryEntry() override {}
-    bool matches(const std::string& target) const override {
-      return std::regex_match(target, pattern_);
-    }
-    // Creates a new T object.
-    T* NewFactoryObject(const std::string& target, std::unique_ptr<T>* guard,
-                        std::string* msg) const {
-      return factory_(target, guard, msg);
-    }
-
-   private:
-    std::regex pattern_;  // The pattern for this entry
-    FactoryFunc<T> factory_;
-  };  // End class FactoryEntry
- public:
-  // Finds the entry matching the input name and type
-  const Entry* FindEntry(const std::string& type,
-                         const std::string& name) const;
-  void Dump(Logger* logger) const;
-
-  // Registers the factory with the library for the pattern.
-  // If the pattern matches, the factory may be used to create a new object.
-  template <typename T>
-  const FactoryFunc<T>& Register(const std::string& pattern,
-                                 const FactoryFunc<T>& factory) {
-    std::unique_ptr<Entry> entry(new FactoryEntry<T>(pattern, factory));
-    AddEntry(T::Type(), entry);
-    return factory;
-  }
-  // Returns the default ObjectLibrary
-  static std::shared_ptr<ObjectLibrary>& Default();
-
- private:
-  // Adds the input entry to the list for the given type
-  void AddEntry(const std::string& type, std::unique_ptr<Entry>& entry);
-
-  // ** FactoryFunctions for this loader, organized by type
-  std::unordered_map<std::string, std::vector<std::unique_ptr<Entry>>> entries_;
+  explicit Registrar(std::string pattern, FactoryFunc<T> factory);
 };

-// The ObjectRegistry is used to register objects that can be created by a
-// name/pattern at run-time where the specific implementation of the object may
-// not be known in advance.
-class ObjectRegistry {
- public:
-  static std::shared_ptr<ObjectRegistry> NewInstance();
+// Implementation details follow.

-  ObjectRegistry();
+namespace internal {

-  void AddLibrary(const std::shared_ptr<ObjectLibrary>& library) {
-    libraries_.emplace_back(library);
+template <typename T>
+struct RegistryEntry {
+  std::regex pattern;
+  FactoryFunc<T> factory;
+};
+
+template <typename T>
+struct Registry {
+  static Registry* Get() {
+    static Registry<T> instance;
+    return &instance;
  }
-
-  // Creates a new T using the factory function that was registered with a
-  // pattern that matches the provided "target" string according to
-  // std::regex_match.
-  //
-  // If no registered functions match, returns nullptr. If multiple functions
-  // match, the factory function used is unspecified.
-  //
-  // Populates res_guard with result pointer if caller is granted ownership.
-  template <typename T>
-  T* NewObject(const std::string& target, std::unique_ptr<T>* guard,
-               std::string* errmsg) {
-    guard->reset();
-    const auto* basic = FindEntry(T::Type(), target);
-    if (basic != nullptr) {
-      const auto* factory =
-          static_cast<const ObjectLibrary::FactoryEntry<T>*>(basic);
-      return factory->NewFactoryObject(target, guard, errmsg);
-    } else {
-      *errmsg = std::string("Could not load ") + T::Type();
-      return nullptr;
-    }
-  }
-
-  // Creates a new unique T using the input factory functions.
-  // Returns OK if a new unique T was successfully created
-  // Returns NotFound if the type/target could not be created
-  // Returns InvalidArgument if the factory return an unguarded object
-  //                      (meaning it cannot be managed by a unique ptr)
-  template <typename T>
-  Status NewUniqueObject(const std::string& target,
-                         std::unique_ptr<T>* result) {
-    std::string errmsg;
-    T* ptr = NewObject(target, result, &errmsg);
-    if (ptr == nullptr) {
-      return Status::NotFound(errmsg, target);
-    } else if (*result) {
-      return Status::OK();
-    } else {
-      return Status::InvalidArgument(std::string("Cannot make a unique ") +
-                                         T::Type() + " from unguarded one ",
-                                     target);
-    }
-  }
-
-  // Creates a new shared T using the input factory functions.
-  // Returns OK if a new shared T was successfully created
-  // Returns NotFound if the type/target could not be created
-  // Returns InvalidArgument if the factory return an unguarded object
-  //                      (meaning it cannot be managed by a shared ptr)
-  template <typename T>
-  Status NewSharedObject(const std::string& target,
-                         std::shared_ptr<T>* result) {
-    std::string errmsg;
-    std::unique_ptr<T> guard;
-    T* ptr = NewObject(target, &guard, &errmsg);
-    if (ptr == nullptr) {
-      return Status::NotFound(errmsg, target);
-    } else if (guard) {
-      result->reset(guard.release());
-      return Status::OK();
-    } else {
-      return Status::InvalidArgument(std::string("Cannot make a shared ") +
-                                         T::Type() + " from unguarded one ",
-                                     target);
-    }
-  }
-
-  // Creates a new static T using the input factory functions.
-  // Returns OK if a new static T was successfully created
-  // Returns NotFound if the type/target could not be created
-  // Returns InvalidArgument if the factory return a guarded object
-  //                      (meaning it is managed by a unique ptr)
-  template <typename T>
-  Status NewStaticObject(const std::string& target, T** result) {
-    std::string errmsg;
-    std::unique_ptr<T> guard;
-    T* ptr = NewObject(target, &guard, &errmsg);
-    if (ptr == nullptr) {
-      return Status::NotFound(errmsg, target);
-    } else if (guard.get()) {
-      return Status::InvalidArgument(std::string("Cannot make a static ") +
-                                         T::Type() + " from a guarded one ",
-                                     target);
-    } else {
-      *result = ptr;
-      return Status::OK();
-    }
-  }
-
-  // Dump the contents of the registry to the logger
-  void Dump(Logger* logger) const;
+  std::vector<RegistryEntry<T>> entries;

 private:
-  const ObjectLibrary::Entry* FindEntry(const std::string& type,
-                                        const std::string& name) const;
-
-  // The set of libraries to search for factories for this registry.
-  // The libraries are searched in reverse order (back to front) when
-  // searching for entries.
-  std::vector<std::shared_ptr<ObjectLibrary>> libraries_;
+  Registry() = default;
 };
+
+}  // namespace internal
+
+template <typename T>
+T* NewCustomObject(const std::string& target, std::unique_ptr<T>* res_guard) {
+  res_guard->reset();
+  for (const auto& entry : internal::Registry<T>::Get()->entries) {
+    if (std::regex_match(target, entry.pattern)) {
+      return entry.factory(target, res_guard);
+    }
+  }
+  return nullptr;
+}
+
+template <typename T>
+Registrar<T>::Registrar(std::string pattern, FactoryFunc<T> factory) {
+  internal::Registry<T>::Get()->entries.emplace_back(internal::RegistryEntry<T>{
+      std::regex(std::move(pattern)), std::move(factory)});
+}
+
 }  // namespace rocksdb
 #endif  // ROCKSDB_LITE
@@ -120,16 +120,6 @@ class StackableDB : public DB {
    return db_->IngestExternalFiles(args);
  }

-  using DB::CreateColumnFamilyWithImport;
-  virtual Status CreateColumnFamilyWithImport(
-      const ColumnFamilyOptions& options, const std::string& column_family_name,
-      const ImportColumnFamilyOptions& import_options,
-      const ExportImportFilesMetaData& metadata,
-      ColumnFamilyHandle** handle) override {
-    return db_->CreateColumnFamilyWithImport(options, column_family_name,
-                                             import_options, metadata, handle);
-  }
-
  virtual Status VerifyChecksum() override { return db_->VerifyChecksum(); }

  using DB::KeyMayExist;
@@ -209,11 +199,10 @@ class StackableDB : public DB {
  }

  using DB::GetApproximateSizes;
-  virtual Status GetApproximateSizes(const SizeApproximationOptions& options,
-                                     ColumnFamilyHandle* column_family,
-                                     const Range* r, int n,
-                                     uint64_t* sizes) override {
-    return db_->GetApproximateSizes(options, column_family, r, n, sizes);
+  virtual void GetApproximateSizes(
+      ColumnFamilyHandle* column_family, const Range* r, int n, uint64_t* sizes,
+      uint8_t include_flags = INCLUDE_FILES) override {
+    return db_->GetApproximateSizes(column_family, r, n, sizes, include_flags);
  }

  using DB::GetApproximateMemTableStats;
@@ -101,11 +101,6 @@ struct TransactionDBOptions {
  // ordering rather than concurrency control.
  bool skip_concurrency_control = false;

-  // This option is only valid for write unprepared. If a write batch exceeds
-  // this threshold, then the transaction will implicitly flush the currently
-  // pending writes into the database. A value of 0 or less means no limit.
-  int64_t default_write_batch_flush_threshold = 0;
-
 private:
  // 128 entries
  size_t wp_snapshot_cache_bits = static_cast<size_t>(7);
@@ -167,11 +162,6 @@ struct TransactionOptions {
  // back/commit before new transactions start.
  // Default: false
  bool skip_concurrency_control = false;
-
-  // See TransactionDBOptions::default_write_batch_flush_threshold for
-  // description. If a negative value is specified, then the default value from
-  // TransactionDBOptions is used.
-  int64_t write_batch_flush_threshold = -1;
 };

 // The per-write optimizations that do not involve transactions. TransactionDB
@@ -5,7 +5,7 @@
 #pragma once

 #define ROCKSDB_MAJOR 6
-#define ROCKSDB_MINOR 4
+#define ROCKSDB_MINOR 3
 #define ROCKSDB_PATCH 6

 // Do not use these. We made the mistake of declaring macros starting with
@@ -28,7 +28,6 @@
 #include <atomic>
 #include <memory>
 #include <string>
-#include <vector>
 #include "rocksdb/status.h"
 #include "rocksdb/write_batch_base.h"

@@ -61,7 +60,6 @@ struct SavePoint {
 class WriteBatch : public WriteBatchBase {
 public:
  explicit WriteBatch(size_t reserved_bytes = 0, size_t max_bytes = 0);
-  explicit WriteBatch(size_t reserved_bytes, size_t max_bytes, size_t ts_sz);
  ~WriteBatch() override;

  using WriteBatchBase::Put;
@@ -313,12 +311,6 @@ class WriteBatch : public WriteBatchBase {
  // Returns trie if MarkRollback will be called during Iterate
  bool HasRollback() const;

-  // Assign timestamp to write batch
-  Status AssignTimestamp(const Slice& ts);
-
-  // Assign timestamps to write batch
-  Status AssignTimestamps(const std::vector<Slice>& ts_list);
-
  using WriteBatchBase::GetWriteBatch;
  WriteBatch* GetWriteBatch() override { return this; }

@@ -369,7 +361,6 @@ class WriteBatch : public WriteBatchBase {

 protected:
  std::string rep_;  // See comment in write_batch.cc for the format of rep_
-  const size_t timestamp_size_;

  // Intentionally copyable
 };
@@ -5902,10 +5902,8 @@ class IndexTypeJni {
       return 0x0;
     case rocksdb::BlockBasedTableOptions::IndexType::kHashSearch:
       return 0x1;
-     case rocksdb::BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch:
+    case rocksdb::BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch:
       return 0x2;
-     case rocksdb::BlockBasedTableOptions::IndexType::kBinarySearchWithFirstKey:
-       return 0x3;
     default:
       return 0x7F;  // undefined
   }
@@ -5922,9 +5920,6 @@ class IndexTypeJni {
       return rocksdb::BlockBasedTableOptions::IndexType::kHashSearch;
     case 0x2:
       return rocksdb::BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
-     case 0x3:
-       return rocksdb::BlockBasedTableOptions::IndexType::
-           kBinarySearchWithFirstKey;
     default:
       // undefined/default
       return rocksdb::BlockBasedTableOptions::IndexType::kBinarySearch;
@@ -85,7 +85,7 @@ jlong Java_org_rocksdb_BlockBasedTableConfig_newTableFactoryHandle(
      std::shared_ptr<rocksdb::Cache> *pCache =
          reinterpret_cast<std::shared_ptr<rocksdb::Cache> *>(jblock_cache_handle);
      options.block_cache = *pCache;
-    } else if (jblock_cache_size >= 0) {
+    } else if (jblock_cache_size > 0) {
      if (jblock_cache_num_shard_bits > 0) {
        options.block_cache = rocksdb::NewLRUCache(
            static_cast<size_t>(jblock_cache_size),
@@ -94,9 +94,6 @@ jlong Java_org_rocksdb_BlockBasedTableConfig_newTableFactoryHandle(
        options.block_cache = rocksdb::NewLRUCache(
            static_cast<size_t>(jblock_cache_size));
      }
-    } else {
-      options.no_block_cache = true;
-      options.block_cache = nullptr;
    }
  }
  if (jpersistent_cache_handle > 0) {
@@ -14,7 +14,7 @@ import java.util.List;
 * Taken from include/rocksdb/advanced_options.h
 */
 public interface AdvancedColumnFamilyOptionsInterface
-    <T extends AdvancedColumnFamilyOptionsInterface<T>> {
+    <T extends AdvancedColumnFamilyOptionsInterface> {

  /**
   * The minimum number of write buffers that will be merged together
@@ -12,7 +12,7 @@ package org.rocksdb;
 * and MutableCFOptions in util/cf_options.h
 */
 public interface AdvancedMutableColumnFamilyOptionsInterface
-    <T extends AdvancedMutableColumnFamilyOptionsInterface<T>> {
+    <T extends AdvancedMutableColumnFamilyOptionsInterface> {

  /**
   * The maximum number of write buffers that are built up in memory.
@@ -725,7 +725,7 @@ public class BlockBasedTableConfig extends TableFormatConfig {

  /**
   * Set the size of the cache in bytes that will be used by RocksDB.
-   * If cacheSize is negative, then cache will not be used.
+   * If cacheSize is non-positive, then cache will not be used.
   * DEFAULT: 8M
   *
   * @param blockCacheSize block cache size in bytes
@@ -6,7 +6,7 @@
 package org.rocksdb;

 public interface ColumnFamilyOptionsInterface
-    <T extends ColumnFamilyOptionsInterface<T>>
+    <T extends ColumnFamilyOptionsInterface>
        extends AdvancedColumnFamilyOptionsInterface<T> {

  /**
@@ -8,7 +8,7 @@ package org.rocksdb;
 import java.util.Collection;
 import java.util.List;

-public interface DBOptionsInterface<T extends DBOptionsInterface<T>> {
+public interface DBOptionsInterface<T extends DBOptionsInterface> {

  /**
   * Use this if your DB is very small (like under 1GB) and you don't want to
@@ -6,7 +6,7 @@
 package org.rocksdb;

 public interface MutableColumnFamilyOptionsInterface
-    <T extends MutableColumnFamilyOptionsInterface<T>>
+    <T extends MutableColumnFamilyOptionsInterface>
        extends AdvancedMutableColumnFamilyOptionsInterface<T> {

  /**
@@ -1,7 +1,7 @@
 // Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
 package org.rocksdb;

-public interface MutableDBOptionsInterface<T extends MutableDBOptionsInterface<T>> {
+public interface MutableDBOptionsInterface<T extends MutableDBOptionsInterface> {

  /**
   * Specifies the maximum number of concurrent background jobs (both flushes
@@ -16,7 +16,7 @@ import java.util.List;
 * during the creation of a {@link org.rocksdb.RocksDB} (i.e., RocksDB.open()).
 *
 * If {@link #dispose()} function is not called, then it will be GC'd
- * automatically and native resources will be released as part of the process.
+ * automaticallyand native resources will be released as part of the process.
 */
 public class Options extends RocksObject
    implements DBOptionsInterface<Options>,
@@ -155,11 +155,6 @@ std::string AutoRollLogger::ValistToString(const char* format,

 void AutoRollLogger::LogInternal(const char* format, ...) {
  mutex_.AssertHeld();
-
-  if (!logger_) {
-    return;
-  }
-
  va_list args;
  va_start(args, format);
  logger_->Logv(format, args);
@@ -168,10 +163,7 @@ void AutoRollLogger::LogInternal(const char* format, ...) {

 void AutoRollLogger::Logv(const char* format, va_list ap) {
  assert(GetStatus().ok());
-  if (!logger_) {
-    return;
-  }
-  
+
  std::shared_ptr<Logger> logger;
  {
    MutexLock l(&mutex_);
@@ -215,10 +207,6 @@ void AutoRollLogger::WriteHeaderInfo() {
 }

 void AutoRollLogger::LogHeader(const char* format, va_list args) {
-  if (!logger_) {
-    return;
-  }
-
  // header message are to be retained in memory. Since we cannot make any
  // assumptions about the data contained in va_list, we will retain them as
  // strings
@@ -41,10 +41,6 @@ class AutoRollLogger : public Logger {
  }

  size_t GetLogFileSize() const override {
-    if (!logger_) {
-      return 0;
-    }
-
    std::shared_ptr<Logger> logger;
    {
      MutexLock l(&mutex_);
@@ -22,7 +22,6 @@
 #include "rocksdb/db.h"
 #include "test_util/sync_point.h"
 #include "test_util/testharness.h"
-#include "test_util/testutil.h"

 namespace rocksdb {
 namespace {
@@ -445,7 +444,7 @@ TEST_F(AutoRollLoggerTest, LogFlushWhileRolling) {
       {"AutoRollLogger::Flush:PinnedLogger", "PosixLogger::Flush:Begin2"}});
  rocksdb::SyncPoint::GetInstance()->EnableProcessing();

-  flush_thread = port::Thread([&]() { auto_roll_logger->Flush(); });
+  flush_thread = port::Thread ([&]() { auto_roll_logger->Flush(); });
  TEST_SYNC_POINT(
      "AutoRollLoggerTest::LogFlushWhileRolling:PreRollAndPostThreadInit");
  RollLogFileBySizeTest(auto_roll_logger, options.max_log_file_size,
@@ -558,6 +557,25 @@ static std::vector<std::string> GetOldFileNames(const std::string& path) {
  return ret;
 }

+// Return the number of lines where a given pattern was found in the file
+static size_t GetLinesCount(const std::string& fname,
+                            const std::string& pattern) {
+  std::stringstream ssbuf;
+  std::string line;
+  size_t count = 0;
+
+  std::ifstream inFile(fname.c_str());
+  ssbuf << inFile.rdbuf();
+
+  while (getline(ssbuf, line)) {
+    if (line.find(pattern) != std::string::npos) {
+      count++;
+    }
+  }
+
+  return count;
+}
+
 TEST_F(AutoRollLoggerTest, LogHeaderTest) {
  static const size_t MAX_HEADERS = 10;
  static const size_t LOG_MAX_SIZE = 1024 * 5;
@@ -609,7 +627,7 @@ TEST_F(AutoRollLoggerTest, LogHeaderTest) {
      // verify that the files rolled over
      ASSERT_NE(oldfname, newfname);
      // verify that the old log contains all the header logs
-      ASSERT_EQ(test::GetLinesCount(oldfname, HEADER_STR), MAX_HEADERS);
+      ASSERT_EQ(GetLinesCount(oldfname, HEADER_STR), MAX_HEADERS);
    }
  }
 }
@@ -1,165 +0,0 @@
-//  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-//  This source code is licensed under both the GPLv2 (found in the
-//  COPYING file in the root directory) and Apache 2.0 License
-//  (found in the LICENSE.Apache file in the root directory).
-//
-// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
-//
-// Logger implementation that uses custom Env object for logging.
-
-#pragma once
-
-#include <atomic>
-#include <memory>
-#include "port/sys_time.h"
-#include <time.h>
-
-#include "monitoring/iostats_context_imp.h"
-#include "rocksdb/env.h"
-#include "rocksdb/slice.h"
-#include "test_util/sync_point.h"
-#include "util/file_reader_writer.h"
-#include "util/mutexlock.h"
-
-namespace rocksdb {
-
-class EnvLogger : public Logger {
- public:
-  EnvLogger(std::unique_ptr<WritableFile>&& writable_file,
-            const std::string& fname, const EnvOptions& options, Env* env,
-            InfoLogLevel log_level = InfoLogLevel::ERROR_LEVEL)
-      : Logger(log_level),
-        file_(std::move(writable_file), fname, options, env),
-        last_flush_micros_(0),
-        env_(env),
-        flush_pending_(false) {}
-
-  ~EnvLogger() {
-    if (!closed_) {
-      closed_ = true;
-      CloseHelper();
-    }
-  }
-
- private:
-  void FlushLocked() {
-    mutex_.AssertHeld();
-    if (flush_pending_) {
-      flush_pending_ = false;
-      file_.Flush();
-    }
-    last_flush_micros_ = env_->NowMicros();
-  }
-
-  void Flush() override {
-    TEST_SYNC_POINT("EnvLogger::Flush:Begin1");
-    TEST_SYNC_POINT("EnvLogger::Flush:Begin2");
-
-    MutexLock l(&mutex_);
-    FlushLocked();
-  }
-
-  Status CloseImpl() override { return CloseHelper(); }
-
-  Status CloseHelper() {
-    mutex_.Lock();
-    const auto close_status = file_.Close();
-    mutex_.Unlock();
-
-    if (close_status.ok()) {
-      return close_status;
-    }
-    return Status::IOError("Close of log file failed with error:" +
-                           (close_status.getState()
-                                ? std::string(close_status.getState())
-                                : std::string()));
-  }
-
-  using Logger::Logv;
-  void Logv(const char* format, va_list ap) override {
-    IOSTATS_TIMER_GUARD(logger_nanos);
-
-    const uint64_t thread_id = env_->GetThreadID();
-
-    // We try twice: the first time with a fixed-size stack allocated buffer,
-    // and the second time with a much larger dynamically allocated buffer.
-    char buffer[500];
-    for (int iter = 0; iter < 2; iter++) {
-      char* base;
-      int bufsize;
-      if (iter == 0) {
-        bufsize = sizeof(buffer);
-        base = buffer;
-      } else {
-        bufsize = 65536;
-        base = new char[bufsize];
-      }
-      char* p = base;
-      char* limit = base + bufsize;
-
-      struct timeval now_tv;
-      gettimeofday(&now_tv, nullptr);
-      const time_t seconds = now_tv.tv_sec;
-      struct tm t;
-      localtime_r(&seconds, &t);
-      p += snprintf(p, limit - p, "%04d/%02d/%02d-%02d:%02d:%02d.%06d %llx ",
-                    t.tm_year + 1900, t.tm_mon + 1, t.tm_mday, t.tm_hour,
-                    t.tm_min, t.tm_sec, static_cast<int>(now_tv.tv_usec),
-                    static_cast<long long unsigned int>(thread_id));
-
-      // Print the message
-      if (p < limit) {
-        va_list backup_ap;
-        va_copy(backup_ap, ap);
-        p += vsnprintf(p, limit - p, format, backup_ap);
-        va_end(backup_ap);
-      }
-
-      // Truncate to available space if necessary
-      if (p >= limit) {
-        if (iter == 0) {
-          continue;  // Try again with larger buffer
-        } else {
-          p = limit - 1;
-        }
-      }
-
-      // Add newline if necessary
-      if (p == base || p[-1] != '\n') {
-        *p++ = '\n';
-      }
-
-      assert(p <= limit);
-      mutex_.Lock();
-      // We will ignore any error returned by Append().
-      file_.Append(Slice(base, p - base));
-      flush_pending_ = true;
-      const uint64_t now_micros = env_->NowMicros();
-      if (now_micros - last_flush_micros_ >= flush_every_seconds_ * 1000000) {
-        FlushLocked();
-      }
-      mutex_.Unlock();
-      if (base != buffer) {
-        delete[] base;
-      }
-      break;
-    }
-  }
-
-  size_t GetLogFileSize() const override {
-    MutexLock l(&mutex_);
-    return file_.GetFileSize();
-  }
-
- private:
-  WritableFileWriter file_;
-  mutable port::Mutex mutex_;  // Mutex to protect the shared variables below.
-  const static uint64_t flush_every_seconds_ = 5;
-  std::atomic_uint_fast64_t last_flush_micros_;
-  Env* env_;
-  std::atomic<bool> flush_pending_;
-};
-
-}  // namespace rocksdb
Author	SHA1	Message	Date
Vijay Nadimpalli	efb42cfa47	Making platform 007 (gcc 7) default in build_detect_platform.sh (#5947 ) Summary: Making platform 007 (gcc 7) default in build_detect_platform.sh. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5947 Differential Revision: D18038837 Pulled By: vjnadimpalli fbshipit-source-id: 9ac2ddaa93bf328a416faec028970e039886378e	2019-10-30 10:30:21 -07:00
Adam Retter	29b8a70a2e	Fixes for building RocksJava releases on arm64v8 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5674 Differential Revision: D16870338 fbshipit-source-id: c8dac644b1479fa734b491f3a8d50151772290f7	2019-10-23 15:19:39 -07:00
sdong	d47cdbc188	Bump up version to 6.3.6	2019-10-01 17:04:43 -07:00
sdong	56cb92bec6	Revert "Merging iterator to avoid child iterator reseek for some cases (#5286 )" This reverts commit `9fad3e21eb`.	2019-10-01 16:48:35 -07:00
sdong	fbf178ec68	Bump up the version to 6.3.5	2019-09-17 17:32:10 -07:00
sdong	648b0141fa	Merging iterator to disble reseek optimization in prefix seek (#5815 ) Summary: We are seeing a bug of wrong results with merging iterator's reseek avoidence feature and prefix extractor. Disable this optimization for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5815 Test Plan: Validated the same MyRocks case was fixed; run all existing tests. Differential Revision: D17430776 fbshipit-source-id: aef664277ba0ab8a2e68331ff0db6ae682535371	2019-09-17 17:30:56 -07:00
Yanqin Jin	d3f28558e7	Bump patch version to 6.3.4	2019-09-03 13:17:54 -07:00
Yanqin Jin	17a3c25564	Fix a bug in file ingestion (#5760 ) Summary: Before this PR, when the number of column families involved in a file ingestion exceeds 2, a bug in the looping logic prevents correct file number being assigned to each ingestion job. Also skip deleting non-existing hard links during cleanup-after-failure. Test plan (devserver) ``` $COMPILE_WITH_ASAN=1 make all $./external_sst_file_test --gtest_filter=ExternalSSTFileTest/ExternalSSTFileTest.IngestFilesIntoMultipleColumnFamilies_/ $makke check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5760 Differential Revision: D17142982 Pulled By: riversand963 fbshipit-source-id: 06c1847a4e7a402647bcf28d124e70f2a0f9daf6	2019-09-03 13:05:22 -07:00
Maysam Yabandeh	4ff493bb38	Disable snapshot refresh feature when snap_refresh_nanos is 0 (#5724 ) Summary: The comments of snap_refresh_nanos advertise that the snapshot refresh feature will be disabled when the option is set to 0. This contract is however not honored in the code: https://github.com/facebook/rocksdb/pull/5278 The patch fixes that and also adds an assert to ensure that the feature is not used when the option is zero. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5724 Differential Revision: D16918185 Pulled By: maysamyabandeh fbshipit-source-id: fec167287df7d85093e087fc39c0eb243e3bbd7e	2019-08-20 13:20:32 -07:00
Levi Tamasi	0e7d927217	Update HISTORY.md for 6.3.2	2019-08-16 11:26:11 -07:00
Levi Tamasi	bb2ab26b9f	Fix regression affecting partitioned indexes/filters when cache_index_and_filter_blocks is false (#5705 ) Summary: PR https://github.com/facebook/rocksdb/issues/5298 (and subsequent related patches) unintentionally changed the semantics of cache_index_and_filter_blocks: historically, this option only affected the main index/filter block; with the changes, it affects index/filter partitions as well. This can cause performance issues when cache_index_and_filter_blocks is false since in this case, partitions are neither cached nor preloaded (i.e. they are loaded on demand upon each access). The patch reverts to the earlier behavior, that is, partitions are cached similarly to data blocks regardless of the value of the above option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5705 Test Plan: make check ./db_bench -benchmarks=fillrandom --statistics --stats_interval_seconds=1 --duration=30 --num=500000000 --bloom_bits=20 --partition_index_and_filters=true --cache_index_and_filter_blocks=false ./db_bench -benchmarks=readrandom --use_existing_db --statistics --stats_interval_seconds=1 --duration=10 --num=500000000 --bloom_bits=20 --partition_index_and_filters=true --cache_index_and_filter_blocks=false --cache_size=8000000000 Relevant statistics from the readrandom benchmark with the old code: rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.filter.bytes.evict COUNT : 0 With the new code: rocksdb.block.cache.index.miss COUNT : 2500 rocksdb.block.cache.index.hit COUNT : 42696 rocksdb.block.cache.index.add COUNT : 2500 rocksdb.block.cache.index.bytes.insert COUNT : 4050048 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 2500 rocksdb.block.cache.filter.hit COUNT : 4550493 rocksdb.block.cache.filter.add COUNT : 2500 rocksdb.block.cache.filter.bytes.insert COUNT : 10331040 rocksdb.block.cache.filter.bytes.evict COUNT : 0 Differential Revision: D16817382 Pulled By: ltamasi fbshipit-source-id: 28a516b0da1f041a03313e0b70b28cf5cf205d00	2019-08-15 12:54:34 -07:00
Levi Tamasi	4ce0542eeb	Make the 'block read count' performance counters consistent (#5484 ) Summary: The patch brings the semantics of per-block-type read performance context counters in sync with the generic block_read_count by only incrementing the counter if the block was actually read from the file. It also fixes index_block_read_count, which fell victim to the refactoring in PR https://github.com/facebook/rocksdb/issues/5298. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5484 Test Plan: Extended the unit tests. Differential Revision: D15887431 Pulled By: ltamasi fbshipit-source-id: a3889759d0ac5759d56625d692cd828d1b9207a6	2019-08-15 11:48:40 -07:00
Levi Tamasi	0d3679d0af	Revert to respecting only the read_tier read option for index blocks (#5481 ) Summary: PR https://github.com/facebook/rocksdb/issues/5298 subtly changed how read options are applied to the index block during a Get, MultiGet, or iteration. Earlier, only the read_tier option applied to the index block read; since PR https://github.com/facebook/rocksdb/issues/5298, fill_cache and verify_checksums also have an effect. This patch restores the earlier behavior to prevent surprise memory increases for clients due to the index block not being cached. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5481 Test Plan: make check Differential Revision: D15883082 Pulled By: ltamasi fbshipit-source-id: 9a065ec3a6db5a365cf6dd5e95190a20c5756356	2019-08-15 11:44:23 -07:00
Levi Tamasi	317a78197b	Bump version to 6.3.2	2019-08-15 11:25:42 -07:00
Levi Tamasi	1a523dc802	Fix the changelog for 6.3.0 on the 6.3 branch as well	2019-07-31 10:18:33 -07:00
sdong	5369e08d83	Add HISTORY.md for the bug fix.	2019-07-24 13:01:39 -07:00
sdong	92f04d6c5f	Fix regression bug of Auto rolling logger when handling failures (#5622 ) Summary: Auto roll logger fails to handle file creation error in the correct way, which may expose to seg fault condition to users. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5622 Test Plan: Add a unit test on creating file under a non-existing directory. The test fails without the fix. Differential Revision: D16460853 fbshipit-source-id: e96da4bef4f16db171ea04a11b2ec5a9448ddbde	2019-07-24 12:57:07 -07:00
Maysam Yabandeh	8e7a29ca1f	Disable refresh snapshot feature by default (#5606 ) Summary: There are concerns about the correctness of this patch. Disabling by default until the concerns are resolved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5606 Differential Revision: D16428064 Pulled By: maysamyabandeh fbshipit-source-id: a89280f0ea85796c9c9dfbfd9a8e91dad9b000b3	2019-07-22 20:09:26 -07:00
anand76	eff6e2a549	MultiGet parallel IO (#5464 ) Summary: Enhancement to MultiGet batching to read data blocks required for keys in a batch in parallel from disk. It uses Env::MultiRead() API to read multiple blocks and reduce latency. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5464 Test Plan: 1. make check 2. make asan_check 3. make asan_crash Differential Revision: D15911771 Pulled By: anand1976 fbshipit-source-id: 605036b9af0f90ca0020dc87c3a86b4da6e83394	2019-07-12 15:38:53 -07:00
anand76	b6c329e406	Fix bugs in WAL trash file handling (#5520 ) Summary: 1. Cleanup WAL trash files on open 2. Don't apply deletion rate limit if WAL dir is different from db dir Pull Request resolved: https://github.com/facebook/rocksdb/pull/5520 Test Plan: Add new unit tests and make check Differential Revision: D16096750 Pulled By: anand1976 fbshipit-source-id: 6f07858ad864b754b711db416f0389c45ede599b	2019-07-12 13:58:35 -07:00
Yanqin Jin	3e1a6dd665	Ref and unref cfd before and after calling WaitForFlushMemTables (#5513 ) Summary: This is to prevent bg flush thread from unrefing and deleting the cfd that has been dropped by a concurrent thread. Before RocksDB calls `DBImpl::WaitForFlushMemTables`, we should increase the refcount of each `ColumnFamilyData` so that its ref count will not drop to 0 even if the column family is dropped by another thread. Otherwise the bg flush thread can deref the cfd and deletes it, causing a segfault in `WaitForFlushMemtables` upon accessing `cfd`. Test plan (on devserver): ``` $make clean && COMPILE_WITH_ASAN=1 make -j32 $make check ``` All unit tests must pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5513 Differential Revision: D16062898 Pulled By: riversand963 fbshipit-source-id: 37dc511f1dc99f036d0201bbd7f0a8f5677c763d	2019-07-11 23:47:58 -07:00
Yanqin Jin	1dcbe51bdf	Add C binding for secondary instance (#5505 ) Summary: Add C binding for secondary instance as well as unit test. Test plan (on devserver) ``` $make clean && COMPILE_WITH_ASAN=1 make -j20 all $./c_test $make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5505 Differential Revision: D16000043 Pulled By: riversand963 fbshipit-source-id: 3361ef6bfdf4ce12438cee7290a0ac203b5250bd	2019-07-11 23:47:29 -07:00
haoyuhuang	6d10405754	Block cache tracer: Do not populate block cache trace record when tracing is disabled. (#5510 ) Summary: This PR makes sure that trace record is not populated when tracing is disabled. Before this PR: DB path: [/data/mysql/rocks_regression_tests/OPTIONS-myrocks-40-33-10000000/2019-06-26-13-04-41/db] readwhilewriting : 9.803 micros/op 1550408 ops/sec; 107.9 MB/s (5000000 of 5000000 found) Microseconds per read: Count: 80000000 Average: 9.8045 StdDev: 12.64 Min: 1 Median: 7.5246 Max: 25343 Percentiles: P50: 7.52 P75: 12.10 P99: 37.44 P99.9: 75.07 P99.99: 133.60 After this PR: DB path: [/data/mysql/rocks_regression_tests/OPTIONS-myrocks-40-33-10000000/2019-06-26-14-08-21/db] readwhilewriting : 8.723 micros/op 1662882 ops/sec; 115.8 MB/s (5000000 of 5000000 found) Microseconds per read: Count: 80000000 Average: 8.7236 StdDev: 12.19 Min: 1 Median: 6.7262 Max: 25229 Percentiles: P50: 6.73 P75: 10.50 P99: 31.54 P99.9: 74.81 P99.99: 132.82 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5510 Differential Revision: D16016428 Pulled By: HaoyuHuang fbshipit-source-id: 3b3d11e6accf207d18ec2545b802aa01ee65901f	2019-07-10 12:08:55 -07:00
Vaibhav Gogte	b4f8173984	Export Cache::GetCharge (#5476 ) Summary: Exporting GetCharge to cache.hh Pull Request resolved: https://github.com/facebook/rocksdb/pull/5476 Differential Revision: D15881882 Pulled By: riversand963 fbshipit-source-id: 3d99084d10059b4fcaaaba240606ed50bc23351c	2019-06-24 10:19:31 -07:00
Fosco Marotto	d9aecd909f	Update version to 6.3.0	2019-06-18 13:59:54 -07:00