Fix old comment about HyperClockCache and SecondaryCache (#11517 )

Summary: Support was added in 8.1.0 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11517 Test Plan: comments only Reviewed By: anand1976 Differential Revision: D46489929 Pulled By: pdillinger fbshipit-source-id: 4fd30078389065c9ec225bf55b6773f1641f0646
Fix build error: variable 'base_level' may be uninitialized (#11435 )
2026-11-06 19:12:36 -05:00 · 2023-06-09 08:12:26 -07:00 · 2023-05-10 09:01:51 -07:00 · 2023-05-04 16:22:16 -07:00 · 2023-05-04 16:19:18 -07:00 · 2023-05-01 11:54:14 -07:00
206 changed files with 4606 additions and 2597 deletions
@@ -1,5 +1,31 @@
 # Rocksdb Change Log
-## Unreleased
+## 8.2.1 (05/04/2023)
+* Public API comments and a Meta-internal fix to the buck build
+
+## 8.2.0 (04/24/2023)
+### Public API Changes
+* `SstFileWriter::DeleteRange()` now returns `Status::InvalidArgument` if the range's end key comes before its start key according to the user comparator. Previously the behavior was undefined.
+* Add `multi_get_for_update` to C API.
+* Remove unnecessary constructor for CompressionOptions.
+
+### Behavior changes
+* Changed default block cache size from an 8MB to 32MB LRUCache, which increases the default number of cache shards from 16 to 64. This change is intended to minimize cache mutex contention under stress conditions. See https://github.com/facebook/rocksdb/wiki/Block-Cache for more information.
+* For level compaction with `level_compaction_dynamic_level_bytes=true`, RocksDB now trivially moves levels down to fill LSM starting from bottommost level during DB open. See more in comments for option `level_compaction_dynamic_level_bytes` (#11321).
+* User-provided `ReadOptions` take effect for more reads of non-`CacheEntryRole::kDataBlock` blocks.
+* For level compaction with `level_compaction_dynamic_level_bytes=true`, RocksDB now drains unnecessary levels through background compaction automatically (#11340). This together with #11321 makes it automatic to migrate other compaction settings to level compaction with `level_compaction_dynamic_level_bytes=true`. In addition, a live DB that becomes smaller will now have unnecessary levels drained which can help to reduce read and space amp.
+* If `CompactRange()` is called with `CompactRangeOptions::bottommost_level_compaction=kForce*` to compact from L0 to L1, RocksDB now will try to do trivial move from L0 to L1 and then do an intra L1 compaction, instead of a L0 to L1 compaction with trivial move disabled (#11375)).
+
+### Bug Fixes
+* In the DB::VerifyFileChecksums API, ensure that file system reads of SST files are equal to the readahead_size in ReadOptions, if specified. Previously, each read was 2x the readahead_size.
+* In block cache tracing, fixed some cases of bad hit/miss information (and more) with MultiGet.
+
+### New Features
+* Add experimental `PerfContext` counters `iter_{next|prev|seek}_count` for db iterator, each counting the times of corresponding API being called.
+* Allow runtime changes to whether `WriteBufferManager` allows stall or not by calling `SetAllowStall()`
+* Added statistics tickers BYTES_COMPRESSED_FROM, BYTES_COMPRESSED_TO, BYTES_COMPRESSION_BYPASSED, BYTES_COMPRESSION_REJECTED, NUMBER_BLOCK_COMPRESSION_BYPASSED, and NUMBER_BLOCK_COMPRESSION_REJECTED. Disabled/deprecated histograms BYTES_COMPRESSED and BYTES_DECOMPRESSED, and ticker NUMBER_BLOCK_NOT_COMPRESSED. The new tickers offer more inight into compression ratios, rejected vs. disabled compression, etc. (#11388)
+* New statistics `rocksdb.file.read.{flush|compaction}.micros` that measure read time of block-based SST tables or blob files during flush or compaction.
+
+## 8.1.0 (03/18/2023)
 ### Behavior changes
 * Compaction output file cutting logic now considers range tombstone start keys. For example, SST partitioner now may receive ParitionRequest for range tombstone start keys.
 * If the async_io ReadOption is specified for MultiGet or NewIterator on a platform that doesn't support IO uring, the option is ignored and synchronous IO is used.
@@ -337,7 +337,7 @@ ifneq ($(MACHINE), arm64)
 # linking with jemalloc (as it won't be arm64-compatible) and remove some other options
 # set during platform detection
 DISABLE_JEMALLOC=1
-PLATFORM_CFLAGS := $(filter-out -march=native -DHAVE_SSE42 -DHAVE_AVX2, $(PLATFORM_CFLAGS))
+PLATFORM_CCFLAGS := $(filter-out -march=native -DHAVE_SSE42 -DHAVE_AVX2, $(PLATFORM_CCFLAGS))
 PLATFORM_CXXFLAGS := $(filter-out -march=native -DHAVE_SSE42 -DHAVE_AVX2, $(PLATFORM_CXXFLAGS))
 endif
 endif
@@ -539,7 +539,7 @@ endif

 ifdef USE_CLANG
 	# Used by some teams in Facebook
-	WARNING_FLAGS += -Wshift-sign-overflow
+	WARNING_FLAGS += -Wshift-sign-overflow -Wambiguous-reversed-operator
 endif

 ifeq ($(PLATFORM), OS_OPENBSD)
@@ -1,8 +1,6 @@
 ## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

 [![CircleCI Status](https://circleci.com/gh/facebook/rocksdb.svg?style=svg)](https://circleci.com/gh/facebook/rocksdb)
-[![Appveyor Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/main?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/main)
-[![PPC64le Build Status](http://140-211-168-68-openstack.osuosl.org:8080/buildStatus/icon?job=rocksdb&style=plastic)](http://140-211-168-68-openstack.osuosl.org:8080/job/rocksdb)

 RocksDB is developed and maintained by Facebook Database Engineering Team.
 It is built on earlier work on [LevelDB](https://github.com/google/leveldb) by Sanjay Ghemawat (sanjay@google.com)
@@ -354,351 +354,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
        "//folly/synchronization:distributed_mutex",
    ], headers=None, link_whole=False, extra_test_libs=False)

-cpp_library_wrapper(name="rocksdb_whole_archive_lib", srcs=[
-        "cache/cache.cc",
-        "cache/cache_entry_roles.cc",
-        "cache/cache_helpers.cc",
-        "cache/cache_key.cc",
-        "cache/cache_reservation_manager.cc",
-        "cache/charged_cache.cc",
-        "cache/clock_cache.cc",
-        "cache/compressed_secondary_cache.cc",
-        "cache/lru_cache.cc",
-        "cache/secondary_cache.cc",
-        "cache/secondary_cache_adapter.cc",
-        "cache/sharded_cache.cc",
-        "db/arena_wrapped_db_iter.cc",
-        "db/blob/blob_contents.cc",
-        "db/blob/blob_fetcher.cc",
-        "db/blob/blob_file_addition.cc",
-        "db/blob/blob_file_builder.cc",
-        "db/blob/blob_file_cache.cc",
-        "db/blob/blob_file_garbage.cc",
-        "db/blob/blob_file_meta.cc",
-        "db/blob/blob_file_reader.cc",
-        "db/blob/blob_garbage_meter.cc",
-        "db/blob/blob_log_format.cc",
-        "db/blob/blob_log_sequential_reader.cc",
-        "db/blob/blob_log_writer.cc",
-        "db/blob/blob_source.cc",
-        "db/blob/prefetch_buffer_collection.cc",
-        "db/builder.cc",
-        "db/c.cc",
-        "db/column_family.cc",
-        "db/compaction/compaction.cc",
-        "db/compaction/compaction_iterator.cc",
-        "db/compaction/compaction_job.cc",
-        "db/compaction/compaction_outputs.cc",
-        "db/compaction/compaction_picker.cc",
-        "db/compaction/compaction_picker_fifo.cc",
-        "db/compaction/compaction_picker_level.cc",
-        "db/compaction/compaction_picker_universal.cc",
-        "db/compaction/compaction_service_job.cc",
-        "db/compaction/compaction_state.cc",
-        "db/compaction/sst_partitioner.cc",
-        "db/compaction/subcompaction_state.cc",
-        "db/convenience.cc",
-        "db/db_filesnapshot.cc",
-        "db/db_impl/compacted_db_impl.cc",
-        "db/db_impl/db_impl.cc",
-        "db/db_impl/db_impl_compaction_flush.cc",
-        "db/db_impl/db_impl_debug.cc",
-        "db/db_impl/db_impl_experimental.cc",
-        "db/db_impl/db_impl_files.cc",
-        "db/db_impl/db_impl_open.cc",
-        "db/db_impl/db_impl_readonly.cc",
-        "db/db_impl/db_impl_secondary.cc",
-        "db/db_impl/db_impl_write.cc",
-        "db/db_info_dumper.cc",
-        "db/db_iter.cc",
-        "db/dbformat.cc",
-        "db/error_handler.cc",
-        "db/event_helpers.cc",
-        "db/experimental.cc",
-        "db/external_sst_file_ingestion_job.cc",
-        "db/file_indexer.cc",
-        "db/flush_job.cc",
-        "db/flush_scheduler.cc",
-        "db/forward_iterator.cc",
-        "db/import_column_family_job.cc",
-        "db/internal_stats.cc",
-        "db/log_reader.cc",
-        "db/log_writer.cc",
-        "db/logs_with_prep_tracker.cc",
-        "db/malloc_stats.cc",
-        "db/memtable.cc",
-        "db/memtable_list.cc",
-        "db/merge_helper.cc",
-        "db/merge_operator.cc",
-        "db/output_validator.cc",
-        "db/periodic_task_scheduler.cc",
-        "db/range_del_aggregator.cc",
-        "db/range_tombstone_fragmenter.cc",
-        "db/repair.cc",
-        "db/seqno_to_time_mapping.cc",
-        "db/snapshot_impl.cc",
-        "db/table_cache.cc",
-        "db/table_properties_collector.cc",
-        "db/transaction_log_impl.cc",
-        "db/trim_history_scheduler.cc",
-        "db/version_builder.cc",
-        "db/version_edit.cc",
-        "db/version_edit_handler.cc",
-        "db/version_set.cc",
-        "db/wal_edit.cc",
-        "db/wal_manager.cc",
-        "db/wide/wide_column_serialization.cc",
-        "db/wide/wide_columns.cc",
-        "db/write_batch.cc",
-        "db/write_batch_base.cc",
-        "db/write_controller.cc",
-        "db/write_stall_stats.cc",
-        "db/write_thread.cc",
-        "env/composite_env.cc",
-        "env/env.cc",
-        "env/env_chroot.cc",
-        "env/env_encryption.cc",
-        "env/env_posix.cc",
-        "env/file_system.cc",
-        "env/file_system_tracer.cc",
-        "env/fs_posix.cc",
-        "env/fs_remap.cc",
-        "env/io_posix.cc",
-        "env/mock_env.cc",
-        "env/unique_id_gen.cc",
-        "file/delete_scheduler.cc",
-        "file/file_prefetch_buffer.cc",
-        "file/file_util.cc",
-        "file/filename.cc",
-        "file/line_file_reader.cc",
-        "file/random_access_file_reader.cc",
-        "file/read_write_util.cc",
-        "file/readahead_raf.cc",
-        "file/sequence_file_reader.cc",
-        "file/sst_file_manager_impl.cc",
-        "file/writable_file_writer.cc",
-        "logging/auto_roll_logger.cc",
-        "logging/event_logger.cc",
-        "logging/log_buffer.cc",
-        "memory/arena.cc",
-        "memory/concurrent_arena.cc",
-        "memory/jemalloc_nodump_allocator.cc",
-        "memory/memkind_kmem_allocator.cc",
-        "memory/memory_allocator.cc",
-        "memtable/alloc_tracker.cc",
-        "memtable/hash_linklist_rep.cc",
-        "memtable/hash_skiplist_rep.cc",
-        "memtable/skiplistrep.cc",
-        "memtable/vectorrep.cc",
-        "memtable/write_buffer_manager.cc",
-        "monitoring/histogram.cc",
-        "monitoring/histogram_windowing.cc",
-        "monitoring/in_memory_stats_history.cc",
-        "monitoring/instrumented_mutex.cc",
-        "monitoring/iostats_context.cc",
-        "monitoring/perf_context.cc",
-        "monitoring/perf_level.cc",
-        "monitoring/persistent_stats_history.cc",
-        "monitoring/statistics.cc",
-        "monitoring/thread_status_impl.cc",
-        "monitoring/thread_status_updater.cc",
-        "monitoring/thread_status_updater_debug.cc",
-        "monitoring/thread_status_util.cc",
-        "monitoring/thread_status_util_debug.cc",
-        "options/cf_options.cc",
-        "options/configurable.cc",
-        "options/customizable.cc",
-        "options/db_options.cc",
-        "options/options.cc",
-        "options/options_helper.cc",
-        "options/options_parser.cc",
-        "port/mmap.cc",
-        "port/port_posix.cc",
-        "port/stack_trace.cc",
-        "port/win/env_default.cc",
-        "port/win/env_win.cc",
-        "port/win/io_win.cc",
-        "port/win/port_win.cc",
-        "port/win/win_logger.cc",
-        "port/win/win_thread.cc",
-        "table/adaptive/adaptive_table_factory.cc",
-        "table/block_based/binary_search_index_reader.cc",
-        "table/block_based/block.cc",
-        "table/block_based/block_based_table_builder.cc",
-        "table/block_based/block_based_table_factory.cc",
-        "table/block_based/block_based_table_iterator.cc",
-        "table/block_based/block_based_table_reader.cc",
-        "table/block_based/block_builder.cc",
-        "table/block_based/block_cache.cc",
-        "table/block_based/block_prefetcher.cc",
-        "table/block_based/block_prefix_index.cc",
-        "table/block_based/data_block_footer.cc",
-        "table/block_based/data_block_hash_index.cc",
-        "table/block_based/filter_block_reader_common.cc",
-        "table/block_based/filter_policy.cc",
-        "table/block_based/flush_block_policy.cc",
-        "table/block_based/full_filter_block.cc",
-        "table/block_based/hash_index_reader.cc",
-        "table/block_based/index_builder.cc",
-        "table/block_based/index_reader_common.cc",
-        "table/block_based/parsed_full_filter_block.cc",
-        "table/block_based/partitioned_filter_block.cc",
-        "table/block_based/partitioned_index_iterator.cc",
-        "table/block_based/partitioned_index_reader.cc",
-        "table/block_based/reader_common.cc",
-        "table/block_based/uncompression_dict_reader.cc",
-        "table/block_fetcher.cc",
-        "table/compaction_merging_iterator.cc",
-        "table/cuckoo/cuckoo_table_builder.cc",
-        "table/cuckoo/cuckoo_table_factory.cc",
-        "table/cuckoo/cuckoo_table_reader.cc",
-        "table/format.cc",
-        "table/get_context.cc",
-        "table/iterator.cc",
-        "table/merging_iterator.cc",
-        "table/meta_blocks.cc",
-        "table/persistent_cache_helper.cc",
-        "table/plain/plain_table_bloom.cc",
-        "table/plain/plain_table_builder.cc",
-        "table/plain/plain_table_factory.cc",
-        "table/plain/plain_table_index.cc",
-        "table/plain/plain_table_key_coding.cc",
-        "table/plain/plain_table_reader.cc",
-        "table/sst_file_dumper.cc",
-        "table/sst_file_reader.cc",
-        "table/sst_file_writer.cc",
-        "table/table_factory.cc",
-        "table/table_properties.cc",
-        "table/two_level_iterator.cc",
-        "table/unique_id.cc",
-        "test_util/sync_point.cc",
-        "test_util/sync_point_impl.cc",
-        "test_util/transaction_test_util.cc",
-        "tools/dump/db_dump_tool.cc",
-        "tools/io_tracer_parser_tool.cc",
-        "tools/ldb_cmd.cc",
-        "tools/ldb_tool.cc",
-        "tools/sst_dump_tool.cc",
-        "trace_replay/block_cache_tracer.cc",
-        "trace_replay/io_tracer.cc",
-        "trace_replay/trace_record.cc",
-        "trace_replay/trace_record_handler.cc",
-        "trace_replay/trace_record_result.cc",
-        "trace_replay/trace_replay.cc",
-        "util/async_file_reader.cc",
-        "util/build_version.cc",
-        "util/cleanable.cc",
-        "util/coding.cc",
-        "util/compaction_job_stats_impl.cc",
-        "util/comparator.cc",
-        "util/compression.cc",
-        "util/compression_context_cache.cc",
-        "util/concurrent_task_limiter_impl.cc",
-        "util/crc32c.cc",
-        "util/crc32c_arm64.cc",
-        "util/data_structure.cc",
-        "util/dynamic_bloom.cc",
-        "util/file_checksum_helper.cc",
-        "util/hash.cc",
-        "util/murmurhash.cc",
-        "util/random.cc",
-        "util/rate_limiter.cc",
-        "util/ribbon_config.cc",
-        "util/slice.cc",
-        "util/status.cc",
-        "util/stderr_logger.cc",
-        "util/string_util.cc",
-        "util/thread_local.cc",
-        "util/threadpool_imp.cc",
-        "util/xxhash.cc",
-        "utilities/agg_merge/agg_merge.cc",
-        "utilities/backup/backup_engine.cc",
-        "utilities/blob_db/blob_compaction_filter.cc",
-        "utilities/blob_db/blob_db.cc",
-        "utilities/blob_db/blob_db_impl.cc",
-        "utilities/blob_db/blob_db_impl_filesnapshot.cc",
-        "utilities/blob_db/blob_dump_tool.cc",
-        "utilities/blob_db/blob_file.cc",
-        "utilities/cache_dump_load.cc",
-        "utilities/cache_dump_load_impl.cc",
-        "utilities/cassandra/cassandra_compaction_filter.cc",
-        "utilities/cassandra/format.cc",
-        "utilities/cassandra/merge_operator.cc",
-        "utilities/checkpoint/checkpoint_impl.cc",
-        "utilities/compaction_filters.cc",
-        "utilities/compaction_filters/remove_emptyvalue_compactionfilter.cc",
-        "utilities/convenience/info_log_finder.cc",
-        "utilities/counted_fs.cc",
-        "utilities/debug.cc",
-        "utilities/env_mirror.cc",
-        "utilities/env_timed.cc",
-        "utilities/fault_injection_env.cc",
-        "utilities/fault_injection_fs.cc",
-        "utilities/fault_injection_secondary_cache.cc",
-        "utilities/leveldb_options/leveldb_options.cc",
-        "utilities/memory/memory_util.cc",
-        "utilities/merge_operators.cc",
-        "utilities/merge_operators/bytesxor.cc",
-        "utilities/merge_operators/max.cc",
-        "utilities/merge_operators/put.cc",
-        "utilities/merge_operators/sortlist.cc",
-        "utilities/merge_operators/string_append/stringappend.cc",
-        "utilities/merge_operators/string_append/stringappend2.cc",
-        "utilities/merge_operators/uint64add.cc",
-        "utilities/object_registry.cc",
-        "utilities/option_change_migration/option_change_migration.cc",
-        "utilities/options/options_util.cc",
-        "utilities/persistent_cache/block_cache_tier.cc",
-        "utilities/persistent_cache/block_cache_tier_file.cc",
-        "utilities/persistent_cache/block_cache_tier_metadata.cc",
-        "utilities/persistent_cache/persistent_cache_tier.cc",
-        "utilities/persistent_cache/volatile_tier_impl.cc",
-        "utilities/simulator_cache/cache_simulator.cc",
-        "utilities/simulator_cache/sim_cache.cc",
-        "utilities/table_properties_collectors/compact_on_deletion_collector.cc",
-        "utilities/trace/file_trace_reader_writer.cc",
-        "utilities/trace/replayer_impl.cc",
-        "utilities/transactions/lock/lock_manager.cc",
-        "utilities/transactions/lock/point/point_lock_manager.cc",
-        "utilities/transactions/lock/point/point_lock_tracker.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/concurrent_tree.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/keyrange.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/lock_request.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/locktree.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/manager.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/range_buffer.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/treenode.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/txnid_set.cc",
-        "utilities/transactions/lock/range/range_tree/lib/locktree/wfg.cc",
-        "utilities/transactions/lock/range/range_tree/lib/standalone_port.cc",
-        "utilities/transactions/lock/range/range_tree/lib/util/dbt.cc",
-        "utilities/transactions/lock/range/range_tree/lib/util/memarena.cc",
-        "utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc",
-        "utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc",
-        "utilities/transactions/optimistic_transaction.cc",
-        "utilities/transactions/optimistic_transaction_db_impl.cc",
-        "utilities/transactions/pessimistic_transaction.cc",
-        "utilities/transactions/pessimistic_transaction_db.cc",
-        "utilities/transactions/snapshot_checker.cc",
-        "utilities/transactions/transaction_base.cc",
-        "utilities/transactions/transaction_db_mutex_impl.cc",
-        "utilities/transactions/transaction_util.cc",
-        "utilities/transactions/write_prepared_txn.cc",
-        "utilities/transactions/write_prepared_txn_db.cc",
-        "utilities/transactions/write_unprepared_txn.cc",
-        "utilities/transactions/write_unprepared_txn_db.cc",
-        "utilities/ttl/db_ttl_impl.cc",
-        "utilities/wal_filter.cc",
-        "utilities/write_batch_with_index/write_batch_with_index.cc",
-        "utilities/write_batch_with_index/write_batch_with_index_internal.cc",
-    ], deps=[
-        "//folly/container:f14_hash",
-        "//folly/experimental/coro:blocking_wait",
-        "//folly/experimental/coro:collect",
-        "//folly/experimental/coro:coroutine",
-        "//folly/experimental/coro:task",
-        "//folly/synchronization:distributed_mutex",
-    ], headers=None, link_whole=True, extra_test_libs=False)
+cpp_library_wrapper(name="rocksdb_whole_archive_lib", srcs=[], deps=[":rocksdb_lib"], headers=None, link_whole=True, extra_test_libs=False)

 cpp_library_wrapper(name="rocksdb_test_lib", srcs=[
        "db/db_test_util.cc",
@@ -29,6 +29,9 @@ Learn more about those use cases in a Tech Talk by Ankit Gupta and Naveen Somasu
 ## Yahoo
 Yahoo is using RocksDB as a storage engine for their biggest distributed data store Sherpa. Learn more about it here: http://yahooeng.tumblr.com/post/120730204806/sherpa-scales-new-heights

+## Tencent
+[PaxosStore](https://github.com/Tencent/paxosstore) is a distributed database supporting WeChat. It uses RocksDB as its storage engine.
+
 ## Baidu
 [Apache Doris](http://doris.apache.org/master/en/) is a MPP analytical database engine released by Baidu. It [uses RocksDB](http://doris.apache.org/master/en/administrator-guide/operation/tablet-meta-tool.html) to manage its tablet's metadata.

@@ -154,16 +154,9 @@ def generate_targets(repo_path, deps_map):
    # rocksdb_whole_archive_lib
    TARGETS.add_library(
        "rocksdb_whole_archive_lib",
-        src_mk["LIB_SOURCES"] +
-        # always add range_tree, it's only excluded on ppc64, which we don't use internally
-        src_mk["RANGE_TREE_SOURCES"] + src_mk["TOOL_LIB_SOURCES"],
+        [],
        deps=[
-            "//folly/container:f14_hash",
-            "//folly/experimental/coro:blocking_wait",
-            "//folly/experimental/coro:collect",
-            "//folly/experimental/coro:coroutine",
-            "//folly/experimental/coro:task",
-            "//folly/synchronization:distributed_mutex",
+            ":rocksdb_lib",
        ],
        headers=None,
        extra_external_deps="",
@@ -63,13 +63,7 @@ if [ -z "$ROCKSDB_NO_FBCODE" -a -d /mnt/gvfs/third-party ]; then
    if [ "$LIB_MODE" == "shared" ]; then
      PIC_BUILD=1
    fi
-    if [ -n "$ROCKSDB_FBCODE_BUILD_WITH_PLATFORM010" ]; then
-      source "$PWD/build_tools/fbcode_config_platform010.sh"
-    elif [ -n "$ROCKSDB_FBCODE_BUILD_WITH_PLATFORM009" ]; then
-      source "$PWD/build_tools/fbcode_config_platform009.sh"
-    else
-      source "$PWD/build_tools/fbcode_config_platform009.sh"
-    fi
+    source "$PWD/build_tools/fbcode_config_platform010.sh"
 fi

 # Delete existing output, if it exists
@@ -1,22 +0,0 @@
-# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
-GCC_BASE=/mnt/gvfs/third-party2/gcc/1795efe5f06778c15a92c8f9a2aba5dc496d9d4d/9.x/centos7-native/3bed279
-CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/7318eaac22659b6ff2fe43918e4b69fd0772a8a7/9.0.0/platform009/651ee30
-LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/4959b39cfbe5965a37c861c4c327fa7c5c759b87/9.x/platform009/9202ce7
-GLIBC_BASE=/mnt/gvfs/third-party2/glibc/45ce3375cdc77ecb2520bbf8f0ecddd3f98efd7a/2.30/platform009/f259413
-SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/be4de3205e029101b18aa8103daa696c2bef3b19/1.1.3/platform009/7f3b187
-ZLIB_BASE=/mnt/gvfs/third-party2/zlib/3c160ac5c67e257501e24c6c1d00ad5e01d73db6/1.2.8/platform009/7f3b187
-BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/73a237ac5bc0a5f5d67b39b8d253cfebaab88684/1.0.6/platform009/7f3b187
-LZ4_BASE=/mnt/gvfs/third-party2/lz4/6ca38d3c390be2774d61a300f151464bbd632d62/1.9.1/platform009/7f3b187
-ZSTD_BASE=/mnt/gvfs/third-party2/zstd/64c58a207d2495e83abc57a500a956df09b79a7c/1.4.x/platform009/ba86d1f
-GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/824d0a8a5abb5b121afd1b35fc3896407ea50092/2.2.0/platform009/7f3b187
-JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/b62912d333ef33f9760efa6219dbe3fe6abb3b0e/master/platform009/c305944
-NUMA_BASE=/mnt/gvfs/third-party2/numa/0af65f71e23a67bf65dc91b11f95caa39325c432/2.0.11/platform009/7f3b187
-LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/02486dac347645d31dce116f44e1de3177315be2/1.4/platform009/5191652
-TBB_BASE=/mnt/gvfs/third-party2/tbb/2e0ec671e550bfca347300bf3f789d9c0fff24ad/2018_U5/platform009/7f3b187
-LIBURING_BASE=/mnt/gvfs/third-party2/liburing/70dbd9cfee63a25611417d09433a86d7711b3990/20200729/platform009/7f3b187
-KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/32b8a2407b634df3f8f948ba373fc4acc6a18296/fb/platform009/da39a3e
-BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/08634589372fa5f237bfd374e8c644a8364e78c1/2.32/platform009/ba86d1f/
-VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/6ae525939ad02e5e676855082fbbc7828dbafeac/3.15.0/platform009/7f3b187
-LUA_BASE=/mnt/gvfs/third-party2/lua/162efd9561a3d21f6869f4814011e9cf1b3ff4dc/5.3.4/platform009/a6271c4
-BENCHMARK_BASE=/mnt/gvfs/third-party2/benchmark/30bf49ad6414325e17f3425b0edcb64239427ae3/1.6.1/platform009/7f3b187
-GLOG_BASE=/mnt/gvfs/third-party2/glog/32d751bd5673375b438158717ab6a57c1cc57e3d/0.3.2_fb/platform009/10a364d
@@ -1,170 +0,0 @@
-#!/bin/sh
-# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
-#
-# Set environment variables so that we can compile rocksdb using
-# fbcode settings.  It uses the latest g++ and clang compilers and also
-# uses jemalloc
-# Environment variables that change the behavior of this script:
-# PIC_BUILD -- if true, it will only take pic versions of libraries from fbcode. libraries that don't have pic variant will not be included
-
-
-BASEDIR=`dirname $BASH_SOURCE`
-source "$BASEDIR/dependencies_platform009.sh"
-
-CFLAGS=""
-
-# libgcc
-LIBGCC_INCLUDE="$LIBGCC_BASE/include/c++/9.3.0 -I $LIBGCC_BASE/include/c++/9.3.0/backward"
-LIBGCC_LIBS=" -L $LIBGCC_BASE/lib"
-
-# glibc
-GLIBC_INCLUDE="$GLIBC_BASE/include"
-GLIBC_LIBS=" -L $GLIBC_BASE/lib"
-
-if test -z $PIC_BUILD; then
-  MAYBE_PIC=
-else
-  MAYBE_PIC=_pic
-fi
-
-if ! test $ROCKSDB_DISABLE_SNAPPY; then
-  # snappy
-  SNAPPY_INCLUDE=" -I $SNAPPY_BASE/include/"
-  SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy${MAYBE_PIC}.a"
-  CFLAGS+=" -DSNAPPY"
-fi
-
-if ! test $ROCKSDB_DISABLE_ZLIB; then
-  # location of zlib headers and libraries
-  ZLIB_INCLUDE=" -I $ZLIB_BASE/include/"
-  ZLIB_LIBS=" $ZLIB_BASE/lib/libz${MAYBE_PIC}.a"
-  CFLAGS+=" -DZLIB"
-fi
-
-if ! test $ROCKSDB_DISABLE_BZIP; then
-  # location of bzip headers and libraries
-  BZIP_INCLUDE=" -I $BZIP2_BASE/include/"
-  BZIP_LIBS=" $BZIP2_BASE/lib/libbz2${MAYBE_PIC}.a"
-  CFLAGS+=" -DBZIP2"
-fi
-
-if ! test $ROCKSDB_DISABLE_LZ4; then
-  LZ4_INCLUDE=" -I $LZ4_BASE/include/"
-  LZ4_LIBS=" $LZ4_BASE/lib/liblz4${MAYBE_PIC}.a"
-  CFLAGS+=" -DLZ4"
-fi
-
-if ! test $ROCKSDB_DISABLE_ZSTD; then
-  ZSTD_INCLUDE=" -I $ZSTD_BASE/include/"
-  ZSTD_LIBS=" $ZSTD_BASE/lib/libzstd${MAYBE_PIC}.a"
-  CFLAGS+=" -DZSTD"
-fi
-
-# location of gflags headers and libraries
-GFLAGS_INCLUDE=" -I $GFLAGS_BASE/include/"
-GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags${MAYBE_PIC}.a"
-CFLAGS+=" -DGFLAGS=gflags"
-
-BENCHMARK_INCLUDE=" -I $BENCHMARK_BASE/include/"
-BENCHMARK_LIBS=" $BENCHMARK_BASE/lib/libbenchmark${MAYBE_PIC}.a"
-
-GLOG_INCLUDE=" -I $GLOG_BASE/include/"
-GLOG_LIBS=" $GLOG_BASE/lib/libglog${MAYBE_PIC}.a"
-
-# location of jemalloc
-JEMALLOC_INCLUDE=" -I $JEMALLOC_BASE/include/"
-JEMALLOC_LIB=" $JEMALLOC_BASE/lib/libjemalloc${MAYBE_PIC}.a"
-
-# location of numa
-NUMA_INCLUDE=" -I $NUMA_BASE/include/"
-NUMA_LIB=" $NUMA_BASE/lib/libnuma${MAYBE_PIC}.a"
-CFLAGS+=" -DNUMA"
-
-# location of libunwind
-LIBUNWIND="$LIBUNWIND_BASE/lib/libunwind${MAYBE_PIC}.a"
-
-# location of TBB
-TBB_INCLUDE=" -isystem $TBB_BASE/include/"
-TBB_LIBS="$TBB_BASE/lib/libtbb${MAYBE_PIC}.a"
-CFLAGS+=" -DTBB"
-
-# location of LIBURING
-LIBURING_INCLUDE=" -isystem $LIBURING_BASE/include/"
-LIBURING_LIBS="$LIBURING_BASE/lib/liburing${MAYBE_PIC}.a"
-CFLAGS+=" -DLIBURING"
-
-test "$USE_SSE" || USE_SSE=1
-export USE_SSE
-test "$PORTABLE" || PORTABLE=1
-export PORTABLE
-
-BINUTILS="$BINUTILS_BASE/bin"
-AR="$BINUTILS/ar"
-AS="$BINUTILS/as"
-
-DEPS_INCLUDE="$SNAPPY_INCLUDE $ZLIB_INCLUDE $BZIP_INCLUDE $LZ4_INCLUDE $ZSTD_INCLUDE $GFLAGS_INCLUDE $NUMA_INCLUDE $TBB_INCLUDE $LIBURING_INCLUDE $BENCHMARK_INCLUDE $GLOG_INCLUDE"
-
-STDLIBS="-L $GCC_BASE/lib64"
-
-CLANG_BIN="$CLANG_BASE/bin"
-CLANG_LIB="$CLANG_BASE/lib"
-CLANG_SRC="$CLANG_BASE/../../src"
-
-CLANG_ANALYZER="$CLANG_BIN/clang++"
-CLANG_SCAN_BUILD="$CLANG_SRC/llvm/clang/tools/scan-build/bin/scan-build"
-
-if [ -z "$USE_CLANG" ]; then
-  # gcc
-  CC="$GCC_BASE/bin/gcc"
-  CXX="$GCC_BASE/bin/g++"
-  AR="$GCC_BASE/bin/gcc-ar"
-
-  CFLAGS+=" -B$BINUTILS"
-  CFLAGS+=" -isystem $LIBGCC_INCLUDE"
-  CFLAGS+=" -isystem $GLIBC_INCLUDE"
-  JEMALLOC=1
-else
-  # clang
-  CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
-  CC="$CLANG_BIN/clang"
-  CXX="$CLANG_BIN/clang++"
-  AR="$CLANG_BIN/llvm-ar"
-
-  KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
-
-  CFLAGS+=" -B$BINUTILS -nostdinc -nostdlib"
-  CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/9.x "
-  CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/9.x/x86_64-facebook-linux "
-  CFLAGS+=" -isystem $GLIBC_INCLUDE"
-  CFLAGS+=" -isystem $LIBGCC_INCLUDE"
-  CFLAGS+=" -isystem $CLANG_INCLUDE"
-  CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE/linux "
-  CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE "
-  CFLAGS+=" -Wno-expansion-to-defined "
-  CXXFLAGS="-nostdinc++"
-fi
-
-CFLAGS+=" $DEPS_INCLUDE"
-CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DROCKSDB_FALLOCATE_PRESENT -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DHAVE_SSE42 -DROCKSDB_IOURING_PRESENT"
-CXXFLAGS+=" $CFLAGS"
-
-EXEC_LDFLAGS=" $SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $NUMA_LIB $TBB_LIBS $LIBURING_LIBS $BENCHMARK_LIBS"
-EXEC_LDFLAGS+=" -Wl,--dynamic-linker,/usr/local/fbcode/platform009/lib/ld.so"
-EXEC_LDFLAGS+=" $LIBUNWIND"
-EXEC_LDFLAGS+=" -Wl,-rpath=/usr/local/fbcode/platform009/lib"
-EXEC_LDFLAGS+=" -Wl,-rpath=$GCC_BASE/lib64"
-# required by libtbb
-EXEC_LDFLAGS+=" -ldl"
-
-PLATFORM_LDFLAGS="$LIBGCC_LIBS $GLIBC_LIBS $STDLIBS -lgcc -lstdc++"
-PLATFORM_LDFLAGS+=" -B$BINUTILS"
-
-EXEC_LDFLAGS_SHARED="$SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $TBB_LIBS $LIBURING_LIBS $BENCHMARK_LIBS"
-
-VALGRIND_VER="$VALGRIND_BASE/bin/"
-
-# lua not supported because it's on track for deprecation, I think
-LUA_PATH=
-LUA_LIB=
-
-export CC CXX AR AS CFLAGS CXXFLAGS EXEC_LDFLAGS EXEC_LDFLAGS_SHARED VALGRIND_VER JEMALLOC_LIB JEMALLOC_INCLUDE CLANG_ANALYZER CLANG_SCAN_BUILD LUA_PATH LUA_LIB
@@ -360,7 +360,7 @@ function send_to_ods {
    echo >&2 "ERROR: Key $key doesn't have a value."
    return
  fi
-  curl --silent "https://www.intern.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build&key=$key&value=$value" \
+  curl --silent "https://www.facebook.com/intern/agent/ods_set.php?entity=rocksdb_build&key=$key&value=$value" \
    --connect-timeout 60
 }

@@ -104,46 +104,3 @@ get_lib_base valgrind   LATEST platform010
 get_lib_base lua        5.3.4  platform010

 git diff $OUTPUT
-
-
-###########################################################
-#                platform009 dependencies                 #
-###########################################################
-
-OUTPUT="$BASEDIR/dependencies_platform009.sh"
-
-rm -f "$OUTPUT"
-touch "$OUTPUT"
-
-echo "Writing dependencies to $OUTPUT"
-
-# Compilers locations
-GCC_BASE=`readlink -f $TP2_LATEST/gcc/9.x/centos7-native/*/`
-CLANG_BASE=`readlink -f $TP2_LATEST/llvm-fb/9.0.0/platform009/*/`
-
-log_header
-log_variable GCC_BASE
-log_variable CLANG_BASE
-
-# Libraries locations
-get_lib_base libgcc     9.x     platform009
-get_lib_base glibc      2.30    platform009
-get_lib_base snappy     LATEST  platform009
-get_lib_base zlib       LATEST  platform009
-get_lib_base bzip2      LATEST  platform009
-get_lib_base lz4        LATEST  platform009
-get_lib_base zstd       LATEST  platform009
-get_lib_base gflags     LATEST  platform009
-get_lib_base jemalloc   LATEST  platform009
-get_lib_base numa       LATEST  platform009
-get_lib_base libunwind  LATEST  platform009
-get_lib_base tbb        2018_U5 platform009
-get_lib_base liburing   LATEST  platform009
-get_lib_base benchmark  LATEST  platform009
-
-get_lib_base kernel-headers fb platform009
-get_lib_base binutils   LATEST centos7-native
-get_lib_base valgrind   LATEST platform009
-get_lib_base lua        5.3.4  platform009
-
-git diff $OUTPUT
@@ -95,8 +95,7 @@ Cache::Handle* CacheWithSecondaryAdapter::Promote(
  PERF_COUNTER_ADD(secondary_cache_hit_count, 1);
  RecordTick(stats, SECONDARY_CACHE_HITS);

-  // FIXME? using charge=Size() is suspicious but inherited from older
-  // implementation
+  // Note: SecondaryCache::Size() is really charge (from the CreateCallback)
  size_t charge = secondary_handle->Size();
  Handle* result = nullptr;
  // Insert into primary cache, possibly as a standalone+dummy entries.
@@ -226,7 +226,7 @@ class ShardedCache : public ShardedCacheBase {
    return SumOverShards2(&CacheShard::GetPinnedUsage);
  }
  size_t GetOccupancyCount() const override {
-    return SumOverShards2(&CacheShard::GetPinnedUsage);
+    return SumOverShards2(&CacheShard::GetOccupancyCount);
  }
  size_t GetTableAddressCount() const override {
    return SumOverShards2(&CacheShard::GetTableAddressCount);
@@ -12,7 +12,7 @@ fi
 ROOT=".."
 # Fetch right version of gcov
 if [ -d /mnt/gvfs/third-party -a -z "$CXX" ]; then
-  source $ROOT/build_tools/fbcode_config_platform009.sh
+  source $ROOT/build_tools/fbcode_config_platform010.sh
  GCOV=$GCC_BASE/bin/gcov
 else
  GCOV=$(which gcov)
@@ -259,6 +259,7 @@ Status BlobFileBuilder::CompressBlobIfNeeded(
    return Status::OK();
  }

+  // TODO: allow user CompressionOptions, including max_compressed_bytes_per_kb
  CompressionOptions opts;
  CompressionContext context(blob_compression_type_);
  constexpr uint64_t sample_for_compression = 0;
@@ -37,7 +37,7 @@ BlobFileCache::BlobFileCache(Cache* cache,
 }

 Status BlobFileCache::GetBlobFileReader(
-    uint64_t blob_file_number,
+    const ReadOptions& read_options, uint64_t blob_file_number,
    CacheHandleGuard<BlobFileReader>* blob_file_reader) {
  assert(blob_file_reader);
  assert(blob_file_reader->IsEmpty());
@@ -73,7 +73,7 @@ Status BlobFileCache::GetBlobFileReader(
  {
    assert(file_options_);
    const Status s = BlobFileReader::Create(
-        *immutable_options_, *file_options_, column_family_id_,
+        *immutable_options_, read_options, *file_options_, column_family_id_,
        blob_file_read_hist_, blob_file_number, io_tracer_, &reader);
    if (!s.ok()) {
      RecordTick(statistics, NO_FILE_ERRORS);
@@ -32,7 +32,8 @@ class BlobFileCache {
  BlobFileCache(const BlobFileCache&) = delete;
  BlobFileCache& operator=(const BlobFileCache&) = delete;

-  Status GetBlobFileReader(uint64_t blob_file_number,
+  Status GetBlobFileReader(const ReadOptions& read_options,
+                           uint64_t blob_file_number,
                           CacheHandleGuard<BlobFileReader>* blob_file_reader);

 private:
@@ -118,7 +118,9 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader) {
  // First try: reader should be opened and put in cache
  CacheHandleGuard<BlobFileReader> first;

-  ASSERT_OK(blob_file_cache.GetBlobFileReader(blob_file_number, &first));
+  const ReadOptions read_options;
+  ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
+                                              &first));
  ASSERT_NE(first.GetValue(), nullptr);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -126,7 +128,8 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader) {
  // Second try: reader should be served from cache
  CacheHandleGuard<BlobFileReader> second;

-  ASSERT_OK(blob_file_cache.GetBlobFileReader(blob_file_number, &second));
+  ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
+                                              &second));
  ASSERT_NE(second.GetValue(), nullptr);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -163,19 +166,21 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_Race) {
  CacheHandleGuard<BlobFileReader> first;
  CacheHandleGuard<BlobFileReader> second;

+  const ReadOptions read_options;
  SyncPoint::GetInstance()->SetCallBack(
      "BlobFileCache::GetBlobFileReader:DoubleCheck", [&](void* /* arg */) {
        // Disabling sync points to prevent infinite recursion
        SyncPoint::GetInstance()->DisableProcessing();
-
-        ASSERT_OK(blob_file_cache.GetBlobFileReader(blob_file_number, &second));
+        ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options,
+                                                    blob_file_number, &second));
        ASSERT_NE(second.GetValue(), nullptr);
        ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
        ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
      });
  SyncPoint::GetInstance()->EnableProcessing();

-  ASSERT_OK(blob_file_cache.GetBlobFileReader(blob_file_number, &first));
+  ASSERT_OK(blob_file_cache.GetBlobFileReader(read_options, blob_file_number,
+                                              &first));
  ASSERT_NE(first.GetValue(), nullptr);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 0);
@@ -213,8 +218,10 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_IOError) {

  CacheHandleGuard<BlobFileReader> reader;

+  const ReadOptions read_options;
  ASSERT_TRUE(
-      blob_file_cache.GetBlobFileReader(blob_file_number, &reader).IsIOError());
+      blob_file_cache.GetBlobFileReader(read_options, blob_file_number, &reader)
+          .IsIOError());
  ASSERT_EQ(reader.GetValue(), nullptr);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 1);
@@ -253,8 +260,10 @@ TEST_F(BlobFileCacheTest, GetBlobFileReader_CacheFull) {
  // strict_capacity_limit is set
  CacheHandleGuard<BlobFileReader> reader;

-  ASSERT_TRUE(blob_file_cache.GetBlobFileReader(blob_file_number, &reader)
-                  .IsMemoryLimit());
+  const ReadOptions read_options;
+  ASSERT_TRUE(
+      blob_file_cache.GetBlobFileReader(read_options, blob_file_number, &reader)
+          .IsMemoryLimit());
  ASSERT_EQ(reader.GetValue(), nullptr);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_OPENS), 1);
  ASSERT_EQ(options.statistics->getTickerCount(NO_FILE_ERRORS), 1);
@@ -26,9 +26,10 @@
 namespace ROCKSDB_NAMESPACE {

 Status BlobFileReader::Create(
-    const ImmutableOptions& immutable_options, const FileOptions& file_options,
-    uint32_t column_family_id, HistogramImpl* blob_file_read_hist,
-    uint64_t blob_file_number, const std::shared_ptr<IOTracer>& io_tracer,
+    const ImmutableOptions& immutable_options, const ReadOptions& read_options,
+    const FileOptions& file_options, uint32_t column_family_id,
+    HistogramImpl* blob_file_read_hist, uint64_t blob_file_number,
+    const std::shared_ptr<IOTracer>& io_tracer,
    std::unique_ptr<BlobFileReader>* blob_file_reader) {
  assert(blob_file_reader);
  assert(!*blob_file_reader);
@@ -52,15 +53,17 @@ Status BlobFileReader::Create(
  CompressionType compression_type = kNoCompression;

  {
-    const Status s = ReadHeader(file_reader.get(), column_family_id, statistics,
-                                &compression_type);
+    const Status s =
+        ReadHeader(file_reader.get(), read_options, column_family_id,
+                   statistics, &compression_type);
    if (!s.ok()) {
      return s;
    }
  }

  {
-    const Status s = ReadFooter(file_reader.get(), file_size, statistics);
+    const Status s =
+        ReadFooter(file_reader.get(), read_options, file_size, statistics);
    if (!s.ok()) {
      return s;
    }
@@ -134,6 +137,7 @@ Status BlobFileReader::OpenFile(
 }

 Status BlobFileReader::ReadHeader(const RandomAccessFileReader* file_reader,
+                                  const ReadOptions& read_options,
                                  uint32_t column_family_id,
                                  Statistics* statistics,
                                  CompressionType* compression_type) {
@@ -151,9 +155,10 @@ Status BlobFileReader::ReadHeader(const RandomAccessFileReader* file_reader,
    constexpr size_t read_size = BlobLogHeader::kSize;

    // TODO: rate limit reading headers from blob files.
-    const Status s = ReadFromFile(file_reader, read_offset, read_size,
-                                  statistics, &header_slice, &buf, &aligned_buf,
-                                  Env::IO_TOTAL /* rate_limiter_priority */);
+    const Status s =
+        ReadFromFile(file_reader, read_options, read_offset, read_size,
+                     statistics, &header_slice, &buf, &aligned_buf,
+                     Env::IO_TOTAL /* rate_limiter_priority */);
    if (!s.ok()) {
      return s;
    }
@@ -187,6 +192,7 @@ Status BlobFileReader::ReadHeader(const RandomAccessFileReader* file_reader,
 }

 Status BlobFileReader::ReadFooter(const RandomAccessFileReader* file_reader,
+                                  const ReadOptions& read_options,
                                  uint64_t file_size, Statistics* statistics) {
  assert(file_size >= BlobLogHeader::kSize + BlobLogFooter::kSize);
  assert(file_reader);
@@ -202,9 +208,10 @@ Status BlobFileReader::ReadFooter(const RandomAccessFileReader* file_reader,
    constexpr size_t read_size = BlobLogFooter::kSize;

    // TODO: rate limit reading footers from blob files.
-    const Status s = ReadFromFile(file_reader, read_offset, read_size,
-                                  statistics, &footer_slice, &buf, &aligned_buf,
-                                  Env::IO_TOTAL /* rate_limiter_priority */);
+    const Status s =
+        ReadFromFile(file_reader, read_options, read_offset, read_size,
+                     statistics, &footer_slice, &buf, &aligned_buf,
+                     Env::IO_TOTAL /* rate_limiter_priority */);
    if (!s.ok()) {
      return s;
    }
@@ -232,6 +239,7 @@ Status BlobFileReader::ReadFooter(const RandomAccessFileReader* file_reader,
 }

 Status BlobFileReader::ReadFromFile(const RandomAccessFileReader* file_reader,
+                                    const ReadOptions& read_options,
                                    uint64_t read_offset, size_t read_size,
                                    Statistics* statistics, Slice* slice,
                                    Buffer* buf, AlignedBuf* aligned_buf,
@@ -246,17 +254,23 @@ Status BlobFileReader::ReadFromFile(const RandomAccessFileReader* file_reader,

  Status s;

+  IOOptions io_options;
+  s = file_reader->PrepareIOOptions(read_options, io_options);
+  if (!s.ok()) {
+    return s;
+  }
+
  if (file_reader->use_direct_io()) {
    constexpr char* scratch = nullptr;

-    s = file_reader->Read(IOOptions(), read_offset, read_size, slice, scratch,
+    s = file_reader->Read(io_options, read_offset, read_size, slice, scratch,
                          aligned_buf, rate_limiter_priority);
  } else {
    buf->reset(new char[read_size]);
    constexpr AlignedBuf* aligned_scratch = nullptr;

-    s = file_reader->Read(IOOptions(), read_offset, read_size, slice,
-                          buf->get(), aligned_scratch, rate_limiter_priority);
+    s = file_reader->Read(io_options, read_offset, read_size, slice, buf->get(),
+                          aligned_scratch, rate_limiter_priority);
  }

  if (!s.ok()) {
@@ -324,8 +338,13 @@ Status BlobFileReader::GetBlob(
    Status s;
    constexpr bool for_compaction = true;

+    IOOptions io_options;
+    s = file_reader_->PrepareIOOptions(read_options, io_options);
+    if (!s.ok()) {
+      return s;
+    }
    prefetched = prefetch_buffer->TryReadFromCache(
-        IOOptions(), file_reader_.get(), record_offset,
+        io_options, file_reader_.get(), record_offset,
        static_cast<size_t>(record_size), &record_slice, &s,
        read_options.rate_limiter_priority, for_compaction);
    if (!s.ok()) {
@@ -338,10 +357,10 @@ Status BlobFileReader::GetBlob(
    PERF_COUNTER_ADD(blob_read_count, 1);
    PERF_COUNTER_ADD(blob_read_byte, record_size);
    PERF_TIMER_GUARD(blob_read_time);
-    const Status s = ReadFromFile(file_reader_.get(), record_offset,
-                                  static_cast<size_t>(record_size), statistics_,
-                                  &record_slice, &buf, &aligned_buf,
-                                  read_options.rate_limiter_priority);
+    const Status s = ReadFromFile(
+        file_reader_.get(), read_options, record_offset,
+        static_cast<size_t>(record_size), statistics_, &record_slice, &buf,
+        &aligned_buf, read_options.rate_limiter_priority);
    if (!s.ok()) {
      return s;
    }
@@ -29,6 +29,7 @@ class Statistics;
 class BlobFileReader {
 public:
  static Status Create(const ImmutableOptions& immutable_options,
+                       const ReadOptions& read_options,
                       const FileOptions& file_options,
                       uint32_t column_family_id,
                       HistogramImpl* blob_file_read_hist,
@@ -74,15 +75,18 @@ class BlobFileReader {
                         std::unique_ptr<RandomAccessFileReader>* file_reader);

  static Status ReadHeader(const RandomAccessFileReader* file_reader,
+                           const ReadOptions& read_options,
                           uint32_t column_family_id, Statistics* statistics,
                           CompressionType* compression_type);

  static Status ReadFooter(const RandomAccessFileReader* file_reader,
-                           uint64_t file_size, Statistics* statistics);
+                           const ReadOptions& read_options, uint64_t file_size,
+                           Statistics* statistics);

  using Buffer = std::unique_ptr<char[]>;

  static Status ReadFromFile(const RandomAccessFileReader* file_reader,
+                             const ReadOptions& read_options,
                             uint64_t read_offset, size_t read_size,
                             Statistics* statistics, Slice* slice, Buffer* buf,
                             AlignedBuf* aligned_buf,
@@ -172,12 +172,12 @@ TEST_F(BlobFileReaderTest, CreateReaderAndGetBlob) {

  std::unique_ptr<BlobFileReader> reader;

+  ReadOptions read_options;
  ASSERT_OK(BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader));
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader));

  // Make sure the blob can be retrieved with and without checksum verification
-  ReadOptions read_options;
  read_options.verify_checksums = false;

  constexpr FilePrefetchBuffer* prefetch_buffer = nullptr;
@@ -479,11 +479,11 @@ TEST_F(BlobFileReaderTest, Malformed) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
-  ASSERT_TRUE(BlobFileReader::Create(immutable_options, FileOptions(),
-                                     column_family_id, blob_file_read_hist,
-                                     blob_file_number, nullptr /*IOTracer*/,
-                                     &reader)
+  const ReadOptions read_options;
+  ASSERT_TRUE(BlobFileReader::Create(immutable_options, read_options,
+                                     FileOptions(), column_family_id,
+                                     blob_file_read_hist, blob_file_number,
+                                     nullptr /*IOTracer*/, &reader)
                  .IsCorruption());
 }

@@ -513,11 +513,11 @@ TEST_F(BlobFileReaderTest, TTL) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
-  ASSERT_TRUE(BlobFileReader::Create(immutable_options, FileOptions(),
-                                     column_family_id, blob_file_read_hist,
-                                     blob_file_number, nullptr /*IOTracer*/,
-                                     &reader)
+  const ReadOptions read_options;
+  ASSERT_TRUE(BlobFileReader::Create(immutable_options, read_options,
+                                     FileOptions(), column_family_id,
+                                     blob_file_read_hist, blob_file_number,
+                                     nullptr /*IOTracer*/, &reader)
                  .IsCorruption());
 }

@@ -552,11 +552,11 @@ TEST_F(BlobFileReaderTest, ExpirationRangeInHeader) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
-  ASSERT_TRUE(BlobFileReader::Create(immutable_options, FileOptions(),
-                                     column_family_id, blob_file_read_hist,
-                                     blob_file_number, nullptr /*IOTracer*/,
-                                     &reader)
+  const ReadOptions read_options;
+  ASSERT_TRUE(BlobFileReader::Create(immutable_options, read_options,
+                                     FileOptions(), column_family_id,
+                                     blob_file_read_hist, blob_file_number,
+                                     nullptr /*IOTracer*/, &reader)
                  .IsCorruption());
 }

@@ -591,11 +591,11 @@ TEST_F(BlobFileReaderTest, ExpirationRangeInFooter) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
-  ASSERT_TRUE(BlobFileReader::Create(immutable_options, FileOptions(),
-                                     column_family_id, blob_file_read_hist,
-                                     blob_file_number, nullptr /*IOTracer*/,
-                                     &reader)
+  const ReadOptions read_options;
+  ASSERT_TRUE(BlobFileReader::Create(immutable_options, read_options,
+                                     FileOptions(), column_family_id,
+                                     blob_file_read_hist, blob_file_number,
+                                     nullptr /*IOTracer*/, &reader)
                  .IsCorruption());
 }

@@ -629,9 +629,9 @@ TEST_F(BlobFileReaderTest, IncorrectColumnFamily) {
  std::unique_ptr<BlobFileReader> reader;

  constexpr uint32_t incorrect_column_family_id = 2;
-
-  ASSERT_TRUE(BlobFileReader::Create(immutable_options, FileOptions(),
-                                     incorrect_column_family_id,
+  const ReadOptions read_options;
+  ASSERT_TRUE(BlobFileReader::Create(immutable_options, read_options,
+                                     FileOptions(), incorrect_column_family_id,
                                     blob_file_read_hist, blob_file_number,
                                     nullptr /*IOTracer*/, &reader)
                  .IsCorruption());
@@ -664,10 +664,10 @@ TEST_F(BlobFileReaderTest, BlobCRCError) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
+  const ReadOptions read_options;
  ASSERT_OK(BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader));
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader));

  SyncPoint::GetInstance()->SetCallBack(
      "BlobFileReader::VerifyBlob:CheckBlobCRC", [](void* arg) {
@@ -728,13 +728,12 @@ TEST_F(BlobFileReaderTest, Compression) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
+  ReadOptions read_options;
  ASSERT_OK(BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader));
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader));

  // Make sure the blob can be retrieved with and without checksum verification
-  ReadOptions read_options;
  read_options.verify_checksums = false;

  constexpr FilePrefetchBuffer* prefetch_buffer = nullptr;
@@ -803,10 +802,10 @@ TEST_F(BlobFileReaderTest, UncompressionError) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
+  const ReadOptions read_options;
  ASSERT_OK(BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader));
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader));

  SyncPoint::GetInstance()->SetCallBack(
      "BlobFileReader::UncompressBlobIfNeeded:TamperWithResult", [](void* arg) {
@@ -895,10 +894,10 @@ TEST_P(BlobFileReaderIOErrorTest, IOError) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
+  const ReadOptions read_options;
  const Status s = BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader);
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader);

  const bool fail_during_create =
      (sync_point_ != "BlobFileReader::GetBlob:ReadFromFile");
@@ -983,10 +982,10 @@ TEST_P(BlobFileReaderDecodingErrorTest, DecodingError) {
  constexpr HistogramImpl* blob_file_read_hist = nullptr;

  std::unique_ptr<BlobFileReader> reader;
-
+  const ReadOptions read_options;
  const Status s = BlobFileReader::Create(
-      immutable_options, FileOptions(), column_family_id, blob_file_read_hist,
-      blob_file_number, nullptr /*IOTracer*/, &reader);
+      immutable_options, read_options, FileOptions(), column_family_id,
+      blob_file_read_hist, blob_file_number, nullptr /*IOTracer*/, &reader);

  const bool fail_during_create =
      sync_point_ != "BlobFileReader::GetBlob:TamperWithResult";
@@ -209,7 +209,8 @@ Status BlobSource::GetBlob(const ReadOptions& read_options,

  {
    CacheHandleGuard<BlobFileReader> blob_file_reader;
-    s = blob_file_cache_->GetBlobFileReader(file_number, &blob_file_reader);
+    s = blob_file_cache_->GetBlobFileReader(read_options, file_number,
+                                            &blob_file_reader);
    if (!s.ok()) {
      return s;
    }
@@ -372,8 +373,8 @@ void BlobSource::MultiGetBlobFromOneFile(const ReadOptions& read_options,
    }

    CacheHandleGuard<BlobFileReader> blob_file_reader;
-    Status s =
-        blob_file_cache_->GetBlobFileReader(file_number, &blob_file_reader);
+    Status s = blob_file_cache_->GetBlobFileReader(read_options, file_number,
+                                                   &blob_file_reader);
    if (!s.ok()) {
      for (size_t i = 0; i < _blob_reqs.size(); ++i) {
        BlobReadRequest* const req = _blob_reqs[i].first;
@@ -95,9 +95,9 @@ class BlobSource {
                               uint64_t* bytes_read);

  inline Status GetBlobFileReader(
-      uint64_t blob_file_number,
+      const ReadOptions& read_options, uint64_t blob_file_number,
      CacheHandleGuard<BlobFileReader>* blob_file_reader) {
-    return blob_file_cache_->GetBlobFileReader(blob_file_number,
+    return blob_file_cache_->GetBlobFileReader(read_options, blob_file_number,
                                               blob_file_reader);
  }

@@ -517,7 +517,8 @@ TEST_F(BlobSourceTest, GetCompressedBlobs) {
                  compression, blob_offsets, blob_sizes);

    CacheHandleGuard<BlobFileReader> blob_file_reader;
-    ASSERT_OK(blob_source.GetBlobFileReader(file_number, &blob_file_reader));
+    ASSERT_OK(blob_source.GetBlobFileReader(read_options, file_number,
+                                            &blob_file_reader));
    ASSERT_NE(blob_file_reader.GetValue(), nullptr);

    const uint64_t file_size = blob_file_reader.GetValue()->GetFileSize();
@@ -1139,12 +1140,13 @@ TEST_F(BlobSecondaryCacheTest, GetBlobsFromSecondaryCache) {
                         blob_file_cache.get());

  CacheHandleGuard<BlobFileReader> file_reader;
-  ASSERT_OK(blob_source.GetBlobFileReader(file_number, &file_reader));
+  ReadOptions read_options;
+  ASSERT_OK(
+      blob_source.GetBlobFileReader(read_options, file_number, &file_reader));
  ASSERT_NE(file_reader.GetValue(), nullptr);
  const uint64_t file_size = file_reader.GetValue()->GetFileSize();
  ASSERT_EQ(file_reader.GetValue()->GetCompressionType(), kNoCompression);

-  ReadOptions read_options;
  read_options.verify_checksums = true;

  auto blob_cache = options_.blob_cache;
@@ -56,8 +56,8 @@ TableBuilder* NewTableBuilder(const TableBuilderOptions& tboptions,
 Status BuildTable(
    const std::string& dbname, VersionSet* versions,
    const ImmutableDBOptions& db_options, const TableBuilderOptions& tboptions,
-    const FileOptions& file_options, TableCache* table_cache,
-    InternalIterator* iter,
+    const FileOptions& file_options, const ReadOptions& read_options,
+    TableCache* table_cache, InternalIterator* iter,
    std::vector<std::unique_ptr<FragmentedRangeTombstoneIterator>>
        range_del_iters,
    FileMetaData* meta, std::vector<BlobFileAddition>* blob_file_additions,
@@ -255,8 +255,8 @@ Status BuildTable(
            SizeApproximationOptions approx_opts;
            approx_opts.files_size_error_margin = 0.1;
            meta->compensated_range_deletion_size += versions->ApproximateSize(
-                approx_opts, version, kv.first.Encode(), tombstone_end.Encode(),
-                0 /* start_level */, -1 /* end_level */,
+                approx_opts, read_options, version, kv.first.Encode(),
+                tombstone_end.Encode(), 0 /* start_level */, -1 /* end_level */,
                TableReaderCaller::kFlush);
          }
          last_tombstone_start_user_key = range_del_it->start_key();
@@ -369,7 +369,6 @@ Status BuildTable(
      // here because this is a special case after we finish the table building.
      // No matter whether use_direct_io_for_flush_and_compaction is true,
      // the goal is to cache it here for further user reads.
-      ReadOptions read_options;
      std::unique_ptr<InternalIterator> it(table_cache->NewIterator(
          read_options, file_options, tboptions.internal_comparator, *meta,
          nullptr /* range_del_agg */, mutable_cf_options.prefix_extractor,
@@ -53,8 +53,8 @@ TableBuilder* NewTableBuilder(const TableBuilderOptions& tboptions,
 extern Status BuildTable(
    const std::string& dbname, VersionSet* versions,
    const ImmutableDBOptions& db_options, const TableBuilderOptions& tboptions,
-    const FileOptions& file_options, TableCache* table_cache,
-    InternalIterator* iter,
+    const FileOptions& file_options, const ReadOptions& read_options,
+    TableCache* table_cache, InternalIterator* iter,
    std::vector<std::unique_ptr<FragmentedRangeTombstoneIterator>>
        range_del_iters,
    FileMetaData* meta, std::vector<BlobFileAddition>* blob_file_additions,
@@ -4064,6 +4064,8 @@ uint64_t rocksdb_perfcontext_metric(rocksdb_perfcontext_t* context,
      return rep->blob_decompress_time;
    case rocksdb_internal_range_del_reseek_count:
      return rep->internal_range_del_reseek_count;
+    case rocksdb_block_read_cpu_time:
+      return rep->block_read_cpu_time;
    default:
      break;
  }
@@ -4679,7 +4681,7 @@ rocksdb_cache_t* rocksdb_cache_create_lru_with_strict_capacity_limit(
 }

 rocksdb_cache_t* rocksdb_cache_create_lru_opts(
-    rocksdb_lru_cache_options_t* opt) {
+    const rocksdb_lru_cache_options_t* opt) {
  rocksdb_cache_t* c = new rocksdb_cache_t;
  c->rep = NewLRUCache(opt->rep);
  return c;
@@ -4726,7 +4728,7 @@ rocksdb_cache_t* rocksdb_cache_create_hyper_clock(
 }

 rocksdb_cache_t* rocksdb_cache_create_hyper_clock_opts(
-    rocksdb_hyper_clock_cache_options_t* opts) {
+    const rocksdb_hyper_clock_cache_options_t* opts) {
  rocksdb_cache_t* c = new rocksdb_cache_t;
  c->rep = opts->rep.MakeSharedCache();
  return c;
@@ -4742,18 +4744,26 @@ void rocksdb_cache_set_capacity(rocksdb_cache_t* cache, size_t capacity) {
  cache->rep->SetCapacity(capacity);
 }

-size_t rocksdb_cache_get_capacity(rocksdb_cache_t* cache) {
+size_t rocksdb_cache_get_capacity(const rocksdb_cache_t* cache) {
  return cache->rep->GetCapacity();
 }

-size_t rocksdb_cache_get_usage(rocksdb_cache_t* cache) {
+size_t rocksdb_cache_get_usage(const rocksdb_cache_t* cache) {
  return cache->rep->GetUsage();
 }

-size_t rocksdb_cache_get_pinned_usage(rocksdb_cache_t* cache) {
+size_t rocksdb_cache_get_pinned_usage(const rocksdb_cache_t* cache) {
  return cache->rep->GetPinnedUsage();
 }

+size_t rocksdb_cache_get_table_address_count(const rocksdb_cache_t* cache) {
+  return cache->rep->GetTableAddressCount();
+}
+
+size_t rocksdb_cache_get_occupancy_count(const rocksdb_cache_t* cache) {
+  return cache->rep->GetOccupancyCount();
+}
+
 rocksdb_dbpath_t* rocksdb_dbpath_create(const char* path,
                                        uint64_t target_size) {
  rocksdb_dbpath_t* result = new rocksdb_dbpath_t;
@@ -5853,6 +5863,35 @@ void rocksdb_transaction_multi_get(rocksdb_transaction_t* txn,
  }
 }

+void rocksdb_transaction_multi_get_for_update(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    size_t num_keys, const char* const* keys_list,
+    const size_t* keys_list_sizes, char** values_list,
+    size_t* values_list_sizes, char** errs) {
+  std::vector<Slice> keys(num_keys);
+  for (size_t i = 0; i < num_keys; i++) {
+    keys[i] = Slice(keys_list[i], keys_list_sizes[i]);
+  }
+  std::vector<std::string> values(num_keys);
+  std::vector<Status> statuses =
+      txn->rep->MultiGetForUpdate(options->rep, keys, &values);
+  for (size_t i = 0; i < num_keys; i++) {
+    if (statuses[i].ok()) {
+      values_list[i] = CopyString(values[i]);
+      values_list_sizes[i] = values[i].size();
+      errs[i] = nullptr;
+    } else {
+      values_list[i] = nullptr;
+      values_list_sizes[i] = 0;
+      if (!statuses[i].IsNotFound()) {
+        errs[i] = strdup(statuses[i].ToString().c_str());
+      } else {
+        errs[i] = nullptr;
+      }
+    }
+  }
+}
+
 void rocksdb_transaction_multi_get_cf(
    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
    const rocksdb_column_family_handle_t* const* column_families,
@@ -5885,6 +5924,38 @@ void rocksdb_transaction_multi_get_cf(
  }
 }

+void rocksdb_transaction_multi_get_for_update_cf(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    const rocksdb_column_family_handle_t* const* column_families,
+    size_t num_keys, const char* const* keys_list,
+    const size_t* keys_list_sizes, char** values_list,
+    size_t* values_list_sizes, char** errs) {
+  std::vector<Slice> keys(num_keys);
+  std::vector<ColumnFamilyHandle*> cfs(num_keys);
+  for (size_t i = 0; i < num_keys; i++) {
+    keys[i] = Slice(keys_list[i], keys_list_sizes[i]);
+    cfs[i] = column_families[i]->rep;
+  }
+  std::vector<std::string> values(num_keys);
+  std::vector<Status> statuses =
+      txn->rep->MultiGetForUpdate(options->rep, cfs, keys, &values);
+  for (size_t i = 0; i < num_keys; i++) {
+    if (statuses[i].ok()) {
+      values_list[i] = CopyString(values[i]);
+      values_list_sizes[i] = values[i].size();
+      errs[i] = nullptr;
+    } else {
+      values_list[i] = nullptr;
+      values_list_sizes[i] = 0;
+      if (!statuses[i].IsNotFound()) {
+        errs[i] = strdup(statuses[i].ToString().c_str());
+      } else {
+        errs[i] = nullptr;
+      }
+    }
+  }
+}
+
 // Read a key outside a transaction
 char* rocksdb_transactiondb_get(rocksdb_transactiondb_t* txn_db,
                                const rocksdb_readoptions_t* options,
@@ -488,6 +488,19 @@ static void CheckTxnPinGetCF(rocksdb_transaction_t* txn,
  rocksdb_pinnableslice_destroy(p);
 }

+static void CheckTxnGetForUpdate(rocksdb_transaction_t* txn,
+                                 const rocksdb_readoptions_t* options,
+                                 const char* key, const char* expected) {
+  char* err = NULL;
+  size_t val_len;
+  char* val;
+  val = rocksdb_transaction_get_for_update(txn, options, key, strlen(key),
+                                           &val_len, true, &err);
+  CheckNoError(err);
+  CheckEqual(expected, val, val_len);
+  Free(&val);
+}
+
 static void CheckTxnDBGet(rocksdb_transactiondb_t* txn_db,
                          const rocksdb_readoptions_t* options, const char* key,
                          const char* expected) {
@@ -515,6 +528,20 @@ static void CheckTxnDBGetCF(rocksdb_transactiondb_t* txn_db,
  Free(&val);
 }

+static void CheckTxnGetForUpdateCF(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key,
+    const char* expected) {
+  char* err = NULL;
+  size_t val_len;
+  char* val;
+  val = rocksdb_transaction_get_for_update_cf(
+      txn, options, column_family, key, strlen(key), &val_len, true, &err);
+  CheckNoError(err);
+  CheckEqual(expected, val, val_len);
+  Free(&val);
+}
+
 static void CheckTxnDBPinGet(rocksdb_transactiondb_t* txn_db,
                             const rocksdb_readoptions_t* options,
                             const char* key, const char* expected) {
@@ -3204,6 +3231,120 @@ int main(int argc, char** argv) {
    rocksdb_transactiondb_options_destroy(txn_db_options);
  }

+  StartPhase("transactions_multi_get_for_update");
+  {
+    // open a TransactionDB
+    txn_db_options = rocksdb_transactiondb_options_create();
+    rocksdb_transactiondb_options_set_transaction_lock_timeout(txn_db_options,
+                                                               0);
+    txn_options = rocksdb_transaction_options_create();
+    rocksdb_options_set_create_if_missing(options, 1);
+    txn_db = rocksdb_transactiondb_open(options, txn_db_options, dbname, &err);
+    CheckNoError(err);
+
+    rocksdb_transactiondb_put(txn_db, woptions, "foo", 3, "hey", 3, &err);
+    CheckNoError(err);
+    rocksdb_transactiondb_put(txn_db, woptions, "bar", 3, "hello", 5, &err);
+    CheckNoError(err);
+
+    // begin transactions
+    txn = rocksdb_transaction_begin(txn_db, woptions, txn_options, NULL);
+    rocksdb_transaction_t* txn2 =
+        rocksdb_transaction_begin(txn_db, woptions, txn_options, NULL);
+
+    // multi get
+    {
+      const char* keys[2] = {"foo", "bar"};
+      const size_t keys_sizes[2] = {3, 3};
+      char* vals[2];
+      size_t vals_sizes[2];
+      char* errs[2];
+      const char* expected[2] = {"hey", "hello"};
+      rocksdb_transaction_multi_get_for_update(
+          txn, roptions, 2, keys, keys_sizes, vals, vals_sizes, errs);
+      CheckMultiGetValues(2, vals, vals_sizes, errs, expected);
+    }
+
+    char* conflict_err = NULL;
+    size_t val_len;
+    rocksdb_transaction_get_for_update(txn2, roptions, "foo", 3, &val_len, true,
+                                       &conflict_err);
+    // get-for-update conflict
+    CheckCondition(conflict_err != NULL);
+    Free(&conflict_err);
+
+    // commit
+    rocksdb_transaction_commit(txn, &err);
+    CheckNoError(err);
+
+    // should work after first tx is commited
+    CheckTxnGetForUpdate(txn2, roptions, "foo", "hey");
+
+    // commit the second one
+    rocksdb_transaction_commit(txn2, &err);
+    CheckNoError(err);
+
+    // destroy txns
+    rocksdb_transaction_destroy(txn);
+    rocksdb_transaction_destroy(txn2);
+
+    // same for column families
+
+    rocksdb_column_family_handle_t* cfh;
+    cfh = rocksdb_transactiondb_create_column_family(txn_db, options,
+                                                     "txn_db_cf", &err);
+    CheckNoError(err);
+
+    rocksdb_transactiondb_put_cf(txn_db, woptions, cfh, "cf_foo", 6, "cf_hello",
+                                 8, &err);
+    CheckNoError(err);
+    rocksdb_transactiondb_put_cf(txn_db, woptions, cfh, "cf_bar", 6, "cf_hey",
+                                 6, &err);
+    CheckNoError(err);
+
+    txn = rocksdb_transaction_begin(txn_db, woptions, txn_options, NULL);
+    txn2 = rocksdb_transaction_begin(txn_db, woptions, txn_options, NULL);
+
+    {
+      const rocksdb_column_family_handle_t* get_handles[2] = {cfh, cfh};
+      const char* keys[2] = {"cf_foo", "cf_bar"};
+      const size_t keys_sizes[2] = {6, 6};
+      char* vals[2];
+      size_t vals_sizes[2];
+      char* errs[2];
+      const char* expected[2] = {"cf_hello", "cf_hey"};
+      rocksdb_transaction_multi_get_for_update_cf(txn, roptions, get_handles, 2,
+                                                  keys, keys_sizes, vals,
+                                                  vals_sizes, errs);
+      CheckMultiGetValues(2, vals, vals_sizes, errs, expected);
+    }
+
+    char* conflict_err_cf = NULL;
+    size_t val_len_cf;
+    rocksdb_transaction_get_for_update_cf(txn2, roptions, cfh, "cf_foo", 6,
+                                          &val_len_cf, true, &conflict_err_cf);
+    CheckCondition(conflict_err_cf != NULL);
+    Free(&conflict_err_cf);
+
+    rocksdb_transaction_commit(txn, &err);
+    CheckNoError(err);
+
+    CheckTxnGetForUpdateCF(txn2, roptions, cfh, "cf_foo", "cf_hello");
+
+    rocksdb_transaction_commit(txn2, &err);
+    CheckNoError(err);
+
+    // close and destroy
+    rocksdb_column_family_handle_destroy(cfh);
+    rocksdb_transaction_destroy(txn);
+    rocksdb_transaction_destroy(txn2);
+    rocksdb_transactiondb_close(txn_db);
+    rocksdb_destroy_db(options, dbname, &err);
+    CheckNoError(err);
+    rocksdb_transaction_options_destroy(txn_options);
+    rocksdb_transactiondb_options_destroy(txn_db_options);
+  }
+
  StartPhase("optimistic_transactions");
  {
    rocksdb_options_t* db_options = rocksdb_options_create();
@@ -1141,6 +1141,7 @@ Status ColumnFamilyData::RangesOverlapWithMemtables(
  *overlap = false;
  // Create an InternalIterator over all unflushed memtables
  Arena arena;
+  // TODO: plumb Env::IOActivity
  ReadOptions read_opts;
  read_opts.total_order_seek = true;
  MergeIteratorBuilder merge_iter_builder(&internal_comparator_, &arena);
@@ -120,7 +120,9 @@ TEST_F(CompactFilesTest, L0ConflictsFiles) {
 TEST_F(CompactFilesTest, MultipleLevel) {
  Options options;
  options.create_if_missing = true;
-  options.level_compaction_dynamic_level_bytes = true;
+  // Otherwise background compaction can happen to
+  // drain unnecessary level
+  options.level_compaction_dynamic_level_bytes = false;
  options.num_levels = 6;
  // Add listener
  FlushedFileCollector* collector = new FlushedFileCollector();
@@ -181,7 +183,6 @@ TEST_F(CompactFilesTest, MultipleLevel) {
  for (int invalid_output_level = 0; invalid_output_level < 5;
       invalid_output_level++) {
    s = db->CompactFiles(CompactionOptions(), files, invalid_output_level);
-    std::cout << s.ToString() << std::endl;
    ASSERT_TRUE(s.IsInvalidArgument());
  }

@@ -546,13 +546,16 @@ struct PerKeyPlacementContext {
  const Slice value;
  const SequenceNumber seq_num;

-  bool output_to_penultimate_level;
+  bool& output_to_penultimate_level;

  PerKeyPlacementContext(int _level, Slice _key, Slice _value,
-                         SequenceNumber _seq_num)
-      : level(_level), key(_key), value(_value), seq_num(_seq_num) {
-    output_to_penultimate_level = false;
-  }
+                         SequenceNumber _seq_num,
+                         bool& _output_to_penultimate_level)
+      : level(_level),
+        key(_key),
+        value(_value),
+        seq_num(_seq_num),
+        output_to_penultimate_level(_output_to_penultimate_level) {}
 };
 #endif /* !NDEBUG */

@@ -1201,17 +1201,7 @@ void CompactionIterator::GarbageCollectBlobIfNeeded() {

 void CompactionIterator::DecideOutputLevel() {
  assert(compaction_->SupportsPerKeyPlacement());
-#ifndef NDEBUG
-  // Could be overridden by unittest
-  PerKeyPlacementContext context(level_, ikey_.user_key, value_,
-                                 ikey_.sequence);
-  TEST_SYNC_POINT_CALLBACK("CompactionIterator::PrepareOutput.context",
-                           &context);
-  output_to_penultimate_level_ = context.output_to_penultimate_level;
-#else
  output_to_penultimate_level_ = false;
-#endif  // NDEBUG
-
  // if the key is newer than the cutoff sequence or within the earliest
  // snapshot, it should output to the penultimate level.
  if (ikey_.sequence > preclude_last_level_min_seqno_ ||
@@ -1219,6 +1209,17 @@ void CompactionIterator::DecideOutputLevel() {
    output_to_penultimate_level_ = true;
  }

+#ifndef NDEBUG
+  // Could be overridden by unittest
+  PerKeyPlacementContext context(level_, ikey_.user_key, value_, ikey_.sequence,
+                                 output_to_penultimate_level_);
+  TEST_SYNC_POINT_CALLBACK("CompactionIterator::PrepareOutput.context",
+                           &context);
+  if (ikey_.sequence > earliest_snapshot_) {
+    output_to_penultimate_level_ = true;
+  }
+#endif  // NDEBUG
+
  if (output_to_penultimate_level_) {
    // If it's decided to output to the penultimate level, but unsafe to do so,
    // still output to the last level. For example, moving the data from a lower
@@ -1411,6 +1412,7 @@ std::unique_ptr<BlobFetcher> CompactionIterator::CreateBlobFetcherIfNeeded(
  }

  ReadOptions read_options;
+  read_options.io_activity = Env::IOActivity::kCompaction;
  read_options.fill_cache = false;

  return std::unique_ptr<BlobFetcher>(new BlobFetcher(version, read_options));
@@ -192,8 +192,8 @@ CompactionJob::CompactionJob(
  assert(log_buffer_ != nullptr);

  const auto* cfd = compact_->compaction->column_family_data();
-  ThreadStatusUtil::SetColumnFamily(cfd, cfd->ioptions()->env,
-                                    db_options_.enable_thread_tracking);
+  ThreadStatusUtil::SetEnableTracking(db_options_.enable_thread_tracking);
+  ThreadStatusUtil::SetColumnFamily(cfd);
  ThreadStatusUtil::SetThreadOperation(ThreadStatus::OP_COMPACTION);
  ReportStartedCompaction(compaction);
 }
@@ -204,10 +204,6 @@ CompactionJob::~CompactionJob() {
 }

 void CompactionJob::ReportStartedCompaction(Compaction* compaction) {
-  const auto* cfd = compact_->compaction->column_family_data();
-  ThreadStatusUtil::SetColumnFamily(cfd, cfd->ioptions()->env,
-                                    db_options_.enable_thread_tracking);
-
  ThreadStatusUtil::SetThreadOperationProperty(ThreadStatus::COMPACTION_JOB_ID,
                                               job_id_);

@@ -291,12 +287,14 @@ void CompactionJob::Prepare() {
               c->immutable_options()->preclude_last_level_data_seconds);

  if (preserve_time_duration > 0) {
+    const ReadOptions read_options(Env::IOActivity::kCompaction);
    // setup seqno_time_mapping_
    seqno_time_mapping_.SetMaxTimeDuration(preserve_time_duration);
    for (const auto& each_level : *c->inputs()) {
      for (const auto& fmd : each_level.files) {
        std::shared_ptr<const TableProperties> tp;
-        Status s = cfd->current()->GetTableProperties(&tp, fmd, nullptr);
+        Status s =
+            cfd->current()->GetTableProperties(read_options, &tp, fmd, nullptr);
        if (s.ok()) {
          seqno_time_mapping_.Add(tp->seqno_to_time_mapping)
              .PermitUncheckedError();
@@ -472,7 +470,7 @@ void CompactionJob::GenSubcompactionBoundaries() {
  // overlap with N-1 other ranges. Since we requested a relatively large number
  // (128) of ranges from each input files, even N range overlapping would
  // cause relatively small inaccuracy.
-
+  const ReadOptions read_options(Env::IOActivity::kCompaction);
  auto* c = compact_->compaction;
  if (c->max_subcompactions() <= 1 &&
      !(c->immutable_options()->compaction_pri == kRoundRobin &&
@@ -506,7 +504,7 @@ void CompactionJob::GenSubcompactionBoundaries() {
        FileMetaData* f = flevel->files[i].file_metadata;
        std::vector<TableReader::Anchor> my_anchors;
        Status s = cfd->table_cache()->ApproximateKeyAnchors(
-            ReadOptions(), icomp, *f, my_anchors);
+            read_options, icomp, *f, my_anchors);
        if (!s.ok() || my_anchors.empty()) {
          my_anchors.emplace_back(f->largest.user_key(), f->fd.GetFileSize());
        }
@@ -722,11 +720,12 @@ Status CompactionJob::Run() {
        // use_direct_io_for_flush_and_compaction is true, we will regard this
        // verification as user reads since the goal is to cache it here for
        // further user reads
-        ReadOptions read_options;
+        const ReadOptions verify_table_read_options(
+            Env::IOActivity::kCompaction);
        InternalIterator* iter = cfd->table_cache()->NewIterator(
-            read_options, file_options_, cfd->internal_comparator(),
-            files_output[file_idx]->meta, /*range_del_agg=*/nullptr,
-            prefix_extractor,
+            verify_table_read_options, file_options_,
+            cfd->internal_comparator(), files_output[file_idx]->meta,
+            /*range_del_agg=*/nullptr, prefix_extractor,
            /*table_reader_ptr=*/nullptr,
            cfd->internal_stats()->GetFileReadHist(
                compact_->compaction->output_level()),
@@ -1032,7 +1031,6 @@ void CompactionJob::NotifyOnSubcompactionCompleted(
 void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
  assert(sub_compact);
  assert(sub_compact->compaction);
-
  if (db_options_.compaction_service) {
    CompactionServiceJobStatus comp_status =
        ProcessKeyValueCompactionWithCompactionService(sub_compact);
@@ -1083,6 +1081,7 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) {
  read_options.verify_checksums = true;
  read_options.fill_cache = false;
  read_options.rate_limiter_priority = GetRateLimiterPriority();
+  read_options.io_activity = Env::IOActivity::kCompaction;
  // Compaction iterators shouldn't be confined to a single prefix.
  // Compactions use Seek() for
  // (a) concurrent compactions,
@@ -1640,6 +1639,7 @@ Status CompactionJob::InstallCompactionResults(

  db_mutex_->AssertHeld();

+  const ReadOptions read_options(Env::IOActivity::kCompaction);
  auto* compaction = compact_->compaction;
  assert(compaction);

@@ -1717,8 +1717,8 @@ Status CompactionJob::InstallCompactionResults(
  }

  return versions_->LogAndApply(compaction->column_family_data(),
-                                mutable_cf_options, edit, db_mutex_,
-                                db_directory_);
+                                mutable_cf_options, read_options, edit,
+                                db_mutex_, db_directory_);
 }

 void CompactionJob::RecordCompactionIOStats() {
@@ -389,9 +389,9 @@ class CompactionJobTestBase : public testing::Test {
        0);

    mutex_.Lock();
-    EXPECT_OK(
-        versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
-                               mutable_cf_options_, &edit, &mutex_, nullptr));
+    EXPECT_OK(versions_->LogAndApply(
+        versions_->GetColumnFamilySet()->GetDefault(), mutable_cf_options_,
+        read_options_, &edit, &mutex_, nullptr));
    mutex_.Unlock();
  }

@@ -727,6 +727,7 @@ class CompactionJobTestBase : public testing::Test {
  ColumnFamilyOptions cf_options_;
  MutableCFOptions mutable_cf_options_;
  MutableDBOptions mutable_db_options_;
+  const ReadOptions read_options_;
  std::shared_ptr<Cache> table_cache_;
  WriteController write_controller_;
  WriteBufferManager write_buffer_manager_;
@@ -2440,4 +2441,3 @@ int main(int argc, char** argv) {
  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
-
@@ -574,6 +574,7 @@ Status CompactionOutputs::AddRangeDels(
  auto it = range_del_agg_->NewIterator(lower_bound, upper_bound);
  Slice last_tombstone_start_user_key{};
  bool reached_lower_bound = false;
+  const ReadOptions read_options(Env::IOActivity::kCompaction);
  for (it->SeekToFirst(); it->Valid(); it->Next()) {
    auto tombstone = it->Tombstone();
    auto kv = tombstone.Serialize();
@@ -713,7 +714,7 @@ Status CompactionOutputs::AddRangeDels(
          approx_opts.files_size_error_margin = 0.1;
          auto approximate_covered_size =
              compaction_->input_version()->version_set()->ApproximateSize(
-                  approx_opts, compaction_->input_version(),
+                  approx_opts, read_options, compaction_->input_version(),
                  tombstone_start.Encode(), tombstone_end.Encode(),
                  compaction_->output_level() + 1 /* start_level */,
                  -1 /* end_level */, kCompaction);
@@ -83,7 +83,7 @@ class LevelCompactionBuilder {

  Compaction* GetCompaction();

-  // For the specfied level, pick a file that we want to compact.
+  // From `start_level_`, pick files to compact to `output_level_`.
  // Returns false if there is no file to compact.
  // If it returns true, inputs->files.size() will be exactly one for
  // all compaction priorities except round-robin. For round-robin,
@@ -107,8 +107,9 @@ class LevelCompactionBuilder {
  bool PickIntraL0Compaction();

  // Return true if TrivialMove is extended. `start_index` is the index of
-  // the intiial file picked, which should already be in `start_level_inputs_`.
-  bool TryExtendNonL0TrivialMove(int start_index);
+  // the initial file picked, which should already be in `start_level_inputs_`.
+  bool TryExtendNonL0TrivialMove(int start_index,
+                                 bool only_expand_right = false);

  // Picks a file from level_files to compact.
  // level_files is a vector of (level, file metadata) in ascending order of
@@ -355,7 +356,8 @@ void LevelCompactionBuilder::SetupOtherFilesWithRoundRobinExpansion() {
  vstorage_->GetOverlappingInputs(output_level_, &smallest, &largest,
                                  &output_level_inputs.files);
  if (output_level_inputs.empty()) {
-    if (TryExtendNonL0TrivialMove((int)start_index)) {
+    if (TryExtendNonL0TrivialMove((int)start_index,
+                                  true /* only_expand_right */)) {
      return;
    }
  }
@@ -501,6 +503,16 @@ Compaction* LevelCompactionBuilder::PickCompaction() {
 }

 Compaction* LevelCompactionBuilder::GetCompaction() {
+  // TryPickL0TrivialMove() does not apply to the case when compacting L0 to an
+  // empty output level. So L0 files is picked in PickFileToCompact() by
+  // compaction score. We may still be able to do trivial move when this file
+  // does not overlap with other L0s. This happens when
+  // compaction_inputs_[0].size() == 1 since SetupOtherL0FilesIfNeeded() did not
+  // pull in more L0s.
+  assert(!compaction_inputs_.empty());
+  bool l0_files_might_overlap =
+      start_level_ == 0 && !is_l0_trivial_move_ &&
+      (compaction_inputs_.size() > 1 || compaction_inputs_[0].size() > 1);
  auto c = new Compaction(
      vstorage_, ioptions_, mutable_cf_options_, mutable_db_options_,
      std::move(compaction_inputs_), output_level_,
@@ -515,8 +527,7 @@ Compaction* LevelCompactionBuilder::GetCompaction() {
      Temperature::kUnknown,
      /* max_subcompactions */ 0, std::move(grandparents_), is_manual_,
      /* trim_ts */ "", start_level_score_, false /* deletion_compaction */,
-      /* l0_files_might_overlap */ start_level_ == 0 && !is_l0_trivial_move_,
-      compaction_reason_);
+      l0_files_might_overlap, compaction_reason_);

  // If it's level 0 compaction, make sure we don't execute any other level 0
  // compactions in parallel
@@ -653,7 +664,8 @@ bool LevelCompactionBuilder::TryPickL0TrivialMove() {
  return false;
 }

-bool LevelCompactionBuilder::TryExtendNonL0TrivialMove(int start_index) {
+bool LevelCompactionBuilder::TryExtendNonL0TrivialMove(int start_index,
+                                                       bool only_expand_right) {
  if (start_level_inputs_.size() == 1 &&
      (ioptions_.db_paths.empty() || ioptions_.db_paths.size() == 1) &&
      (mutable_cf_options_.compression_per_level.empty())) {
@@ -670,6 +682,7 @@ bool LevelCompactionBuilder::TryExtendNonL0TrivialMove(int start_index) {
    size_t total_size = initial_file->fd.GetFileSize();
    CompactionInputFiles output_level_inputs;
    output_level_inputs.level = output_level_;
+    // Expand towards right
    for (int i = start_index + 1;
         i < static_cast<int>(level_files.size()) &&
         start_level_inputs_.size() < kMaxMultiTrivialMove;
@@ -702,6 +715,37 @@ bool LevelCompactionBuilder::TryExtendNonL0TrivialMove(int start_index) {
      }
      start_level_inputs_.files.push_back(next_file);
    }
+    // Expand towards left
+    if (!only_expand_right) {
+      for (int i = start_index - 1;
+           i >= 0 && start_level_inputs_.size() < kMaxMultiTrivialMove; i--) {
+        FileMetaData* next_file = level_files[i];
+        if (next_file->being_compacted) {
+          break;
+        }
+        vstorage_->GetOverlappingInputs(output_level_, &(next_file->smallest),
+                                        &(initial_file->largest),
+                                        &output_level_inputs.files);
+        if (!output_level_inputs.empty()) {
+          break;
+        }
+        if (i > 0 && compaction_picker_->icmp()
+                             ->user_comparator()
+                             ->CompareWithoutTimestamp(
+                                 next_file->smallest.user_key(),
+                                 level_files[i - 1]->largest.user_key()) == 0) {
+          // Not a clean up after adding the next file. Skip.
+          break;
+        }
+        total_size += next_file->fd.GetFileSize();
+        if (total_size > mutable_cf_options_.max_compaction_bytes) {
+          break;
+        }
+        // keep `files` sorted in increasing order by key range
+        start_level_inputs_.files.insert(start_level_inputs_.files.begin(),
+                                         next_file);
+      }
+    }
    return start_level_inputs_.size() > 1;
  }
  return false;
@@ -785,7 +829,10 @@ bool LevelCompactionBuilder::PickFileToCompact() {
    vstorage_->GetOverlappingInputs(output_level_, &smallest, &largest,
                                    &output_level_inputs.files);
    if (output_level_inputs.empty()) {
-      if (TryExtendNonL0TrivialMove(index)) {
+      if (start_level_ > 0 &&
+          TryExtendNonL0TrivialMove(index,
+                                    ioptions_.compaction_pri ==
+                                        kRoundRobin /* only_expand_right */)) {
        break;
      }
    } else {
@@ -2520,6 +2520,61 @@ TEST_F(CompactionPickerTest, L0TrivialMoveWholeL0) {
  ASSERT_TRUE(compaction->IsTrivialMove());
 }

+TEST_F(CompactionPickerTest, NonL0TrivialMoveExtendBothDirection) {
+  mutable_cf_options_.max_bytes_for_level_base = 5000;
+  mutable_cf_options_.level0_file_num_compaction_trigger = 4;
+  mutable_cf_options_.max_compaction_bytes = 10000000u;
+  ioptions_.level_compaction_dynamic_level_bytes = false;
+  NewVersionStorage(6, kCompactionStyleLevel);
+
+  Add(1, 1U, "300", "350", 3000U, 0, 710, 800, 3000U);
+  Add(1, 2U, "600", "651", 3001U, 0, 610, 700, 3001U);
+  Add(1, 3U, "700", "750", 3000U, 0, 500, 550, 3000U);
+  Add(2, 4U, "800", "850", 4000U, 0, 150, 200, 4000U);
+
+  UpdateVersionStorageInfo();
+  // File #2 should be picked first, and expand both directions to include
+  // files #1 and #3.
+  std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
+      cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
+      &log_buffer_));
+  ASSERT_TRUE(compaction.get() != nullptr);
+  ASSERT_EQ(1, compaction->num_input_levels());
+  ASSERT_EQ(3, compaction->num_input_files(0));
+  ASSERT_EQ(1, compaction->input(0, 0)->fd.GetNumber());
+  ASSERT_EQ(2, compaction->input(0, 1)->fd.GetNumber());
+  ASSERT_EQ(3, compaction->input(0, 2)->fd.GetNumber());
+  ASSERT_TRUE(compaction->IsTrivialMove());
+}
+
+TEST_F(CompactionPickerTest, L0TrivialMoveToEmptyLevel) {
+  mutable_cf_options_.max_bytes_for_level_base = 5000;
+  mutable_cf_options_.level0_file_num_compaction_trigger = 4;
+  mutable_cf_options_.max_compaction_bytes = 10000000u;
+  ioptions_.level_compaction_dynamic_level_bytes = false;
+  NewVersionStorage(6, kCompactionStyleLevel);
+
+  // File 2 will be picked first, which by itself is trivial movable.
+  // There was a bug before where compaction also picks file 3 and 4,
+  // (and then file 1 since it overlaps with the key range),
+  // which makes the compaction not trivial movable.
+  Add(0, 1U, "450", "599", 3000U, 0, 710, 800, 3000U);
+  Add(0, 2U, "600", "651", 3001U, 0, 610, 700, 3001U);
+  Add(0, 3U, "300", "350", 3000U, 0, 500, 550, 3000U);
+  Add(0, 4U, "500", "550", 2999U, 0, 300, 350, 2999U);
+
+  UpdateVersionStorageInfo();
+
+  std::unique_ptr<Compaction> compaction(level_compaction_picker.PickCompaction(
+      cf_name_, mutable_cf_options_, mutable_db_options_, vstorage_.get(),
+      &log_buffer_));
+  ASSERT_TRUE(compaction.get() != nullptr);
+  ASSERT_EQ(1, compaction->num_input_levels());
+  ASSERT_EQ(1, compaction->num_input_files(0));
+  ASSERT_EQ(2, compaction->input(0, 0)->fd.GetNumber());
+  ASSERT_TRUE(compaction->IsTrivialMove());
+}
+
 TEST_F(CompactionPickerTest, IsTrivialMoveOffSstPartitioned) {
  mutable_cf_options_.max_bytes_for_level_base = 10000u;
  mutable_cf_options_.max_compaction_bytes = 10001u;
@@ -1111,6 +1111,7 @@ TEST_P(TieredCompactionTest, RangeBasedTieredStorageLevel) {
  options.num_levels = kNumLevels;
  options.statistics = CreateDBStatistics();
  options.max_subcompactions = 10;
+  options.preclude_last_level_data_seconds = 10000;
  DestroyAndReopen(options);
  auto cmp = options.comparator;

@@ -33,7 +33,9 @@ Status DeleteFilesInRanges(DB* db, ColumnFamilyHandle* column_family,
 Status VerifySstFileChecksum(const Options& options,
                             const EnvOptions& env_options,
                             const std::string& file_path) {
-  return VerifySstFileChecksum(options, env_options, ReadOptions(), file_path);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  return VerifySstFileChecksum(options, env_options, read_options, file_path);
 }
 Status VerifySstFileChecksum(const Options& options,
                             const EnvOptions& env_options,
@@ -56,8 +58,9 @@ Status VerifySstFileChecksum(const Options& options,
  std::unique_ptr<RandomAccessFileReader> file_reader(
      new RandomAccessFileReader(
          std::move(file), file_path, ioptions.clock, nullptr /* io_tracer */,
-          nullptr /* stats */, 0 /* hist_type */, nullptr /* file_read_hist */,
-          ioptions.rate_limiter.get()));
+          ioptions.stats /* stats */,
+          Histograms::SST_READ_MICROS /* hist_type */,
+          nullptr /* file_read_hist */, ioptions.rate_limiter.get()));
  const bool kImmortal = true;
  auto reader_options = TableReaderOptions(
      ioptions, options.prefix_extractor, env_options, internal_comparator,
@@ -76,4 +79,3 @@ Status VerifySstFileChecksum(const Options& options,
 }

 }  // namespace ROCKSDB_NAMESPACE
-
@@ -762,9 +762,11 @@ TEST_F(CorruptionTest, RangeDeletionCorrupted) {
      fs->GetFileSize(filename, file_opts.io_options, &file_size, nullptr));

  BlockHandle range_del_handle;
-  ASSERT_OK(FindMetaBlockInFile(
-      file_reader.get(), file_size, kBlockBasedTableMagicNumber,
-      ImmutableOptions(options_), kRangeDelBlockName, &range_del_handle));
+  const ReadOptions read_options;
+  ASSERT_OK(FindMetaBlockInFile(file_reader.get(), file_size,
+                                kBlockBasedTableMagicNumber,
+                                ImmutableOptions(options_), read_options,
+                                kRangeDelBlockName, &range_del_handle));

  ASSERT_OK(TryReopen());
  ASSERT_OK(test::CorruptFile(env_.get(), filename,
@@ -1666,4 +1668,3 @@ int main(int argc, char** argv) {
  RegisterCustomObjects(argc, argv);
  return RUN_ALL_TESTS();
 }
-
@@ -4494,6 +4494,63 @@ TEST_F(DBBasicTest, VerifyFileChecksums) {
  ASSERT_TRUE(db_->VerifyFileChecksums(ReadOptions()).IsInvalidArgument());
 }

+TEST_F(DBBasicTest, VerifyFileChecksumsReadahead) {
+  Options options = GetDefaultOptions();
+  options.create_if_missing = true;
+  options.env = env_;
+  options.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
+  DestroyAndReopen(options);
+
+  Random rnd(301);
+  int alignment = 256 * 1024;
+  for (int i = 0; i < 16; ++i) {
+    ASSERT_OK(Put("key" + std::to_string(i), rnd.RandomString(alignment)));
+  }
+  ASSERT_OK(Flush());
+
+  std::vector<std::string> filenames;
+  int sst_cnt = 0;
+  std::string sst_name;
+  uint64_t sst_size;
+  uint64_t number;
+  FileType type;
+  ASSERT_OK(env_->GetChildren(dbname_, &filenames));
+  for (auto name : filenames) {
+    if (ParseFileName(name, &number, &type)) {
+      if (type == kTableFile) {
+        sst_cnt++;
+        sst_name = name;
+      }
+    }
+  }
+  ASSERT_EQ(sst_cnt, 1);
+  ASSERT_OK(env_->GetFileSize(dbname_ + '/' + sst_name, &sst_size));
+
+  bool last_read = false;
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "GenerateOneFileChecksum::Chunk:0", [&](void* /*arg*/) {
+        if (env_->random_read_bytes_counter_.load() == sst_size) {
+          EXPECT_FALSE(last_read);
+          last_read = true;
+        } else {
+          ASSERT_EQ(env_->random_read_bytes_counter_.load() & (alignment - 1),
+                    0);
+        }
+      });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+  env_->count_random_reads_ = true;
+  env_->random_read_bytes_counter_ = 0;
+  env_->random_read_counter_.Reset();
+
+  ReadOptions ro;
+  ro.readahead_size = alignment;
+  ASSERT_OK(db_->VerifyFileChecksums(ro));
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+  ASSERT_TRUE(last_read);
+  ASSERT_EQ(env_->random_read_counter_.Read(),
+            (sst_size + alignment - 1) / (alignment));
+}
+
 // TODO: re-enable after we provide finer-grained control for WAL tracking to
 // meet the needs of different use cases, durability levels and recovery modes.
 TEST_F(DBBasicTest, DISABLED_ManualWalSync) {
@@ -2437,9 +2437,11 @@ TEST_F(DBBloomFilterTest, PrefixScan) {
 }

 TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
+  const int kNumKeysPerFlush = 1000;
+
  Options options = CurrentOptions();
-  options.write_buffer_size = 64 * 1024;
-  options.arena_block_size = 4 * 1024;
+  options.memtable_factory.reset(
+      test::NewSpecialSkipListFactory(kNumKeysPerFlush));
  options.target_file_size_base = 64 * 1024;
  options.level0_file_num_compaction_trigger = 2;
  options.level0_slowdown_writes_trigger = 2;
@@ -2475,8 +2477,13 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
  int num_inserted = 0;
  for (int key : keys) {
    ASSERT_OK(Put(1, Key(key), "val"));
-    if (++num_inserted % 1000 == 0) {
-      ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable());
+    num_inserted++;
+    // The write after each `kNumKeysPerFlush` keys triggers a flush. Always
+    // wait for that flush and any follow-on compactions for deterministic LSM
+    // shape.
+    if (num_inserted > kNumKeysPerFlush &&
+        num_inserted % kNumKeysPerFlush == 1) {
+      ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[1]));
      ASSERT_OK(dbfull()->TEST_WaitForCompact());
    }
  }
@@ -742,7 +742,7 @@ TEST_F(DBTestCompactionFilter, CompactionFilterContextCfId) {
  ASSERT_TRUE(filter->compaction_filter_created());
 }

-// Compaction filters aplies to all records, regardless snapshots.
+// Compaction filters applies to all records, regardless snapshots.
 TEST_F(DBTestCompactionFilter, CompactionFilterIgnoreSnapshot) {
  std::string five = std::to_string(5);
  Options options = CurrentOptions();
@@ -136,11 +136,12 @@ class DBCompactionTestWithParam

 class DBCompactionTestWithBottommostParam
    : public DBTestBase,
-      public testing::WithParamInterface<BottommostLevelCompaction> {
+      public testing::WithParamInterface<
+          std::tuple<BottommostLevelCompaction, bool>> {
 public:
  DBCompactionTestWithBottommostParam()
      : DBTestBase("db_compaction_test", /*env_do_fsync=*/true) {
-    bottommost_level_compaction_ = GetParam();
+    bottommost_level_compaction_ = std::get<0>(GetParam());
  }

  BottommostLevelCompaction bottommost_level_compaction_;
@@ -3613,7 +3614,10 @@ TEST_F(DBCompactionTest, CancelCompactionWaitingOnConflict) {
  Random rnd(301);
  for (int i = 0; i < kNumSortedRuns; ++i) {
    int key_idx = 0;
-    GenerateNewFile(&rnd, &key_idx, true /* nowait */);
+    // We hold the compaction from happening, so when generating the last SST
+    // file, we cannot wait. Otherwise, we'll hit a deadlock.
+    GenerateNewFile(&rnd, &key_idx,
+                    (i == kNumSortedRuns - 1) ? true : false /* nowait */);
  }
  auto_compaction_sleeping_task.WaitUntilSleeping();

@@ -5506,8 +5510,9 @@ TEST_F(DBCompactionTest, CompactionLimiter) {

  for (int n = 0; n < options.level0_file_num_compaction_trigger; n++) {
    for (unsigned int cf = 0; cf < cf_count; cf++) {
+      // All L0s should overlap with each other
      for (int i = 0; i < kNumKeysPerFile; i++) {
-        ASSERT_OK(Put(cf, Key(keyIndex++), ""));
+        ASSERT_OK(Put(cf, Key(i), ""));
      }
      // put extra key to trigger flush
      ASSERT_OK(Put(cf, "", ""));
@@ -7335,10 +7340,63 @@ TEST_P(DBCompactionTestL0FilesMisorderCorruptionWithParam,
  Destroy(options_);
 }

+TEST_F(DBCompactionTest, SingleLevelUniveresal) {
+  // Tests that manual compaction works with single level universal compaction.
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleUniversal;
+  options.disable_auto_compactions = true;
+  options.num_levels = 1;
+  DestroyAndReopen(options);
+
+  Random rnd(31);
+  for (int i = 0; i < 10; ++i) {
+    for (int j = 0; j < 50; ++j) {
+      ASSERT_OK(Put(Key(i * 100 + j), rnd.RandomString(50)));
+    }
+    ASSERT_OK(Flush());
+  }
+  ASSERT_EQ(NumTableFilesAtLevel(0), 10);
+  ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
+  ASSERT_EQ(NumTableFilesAtLevel(0), 1);
+}
+
+TEST_F(DBCompactionTest, SingleOverlappingNonL0BottommostManualCompaction) {
+  // Tests that manual compact will rewrite bottommost level
+  // when there is only a single non-L0 level that overlaps with
+  // manual compaction range.
+  constexpr int kSstNum = 10;
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.num_levels = 7;
+  for (auto b : {BottommostLevelCompaction::kForce,
+                 BottommostLevelCompaction::kForceOptimized}) {
+    DestroyAndReopen(options);
+
+    // Generate some sst files on level 0 with sequence keys (no overlap)
+    for (int i = 0; i < kSstNum; i++) {
+      for (int j = 1; j < UCHAR_MAX; j++) {
+        auto key = std::string(kSstNum, '\0');
+        key[kSstNum - i] += static_cast<char>(j);
+        ASSERT_OK(Put(key, std::string(i % 1000, 'A')));
+      }
+      ASSERT_OK(Flush());
+    }
+    MoveFilesToLevel(4);
+    ASSERT_EQ(NumTableFilesAtLevel(4), kSstNum);
+    CompactRangeOptions cro;
+    cro.bottommost_level_compaction = b;
+    ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+    ASSERT_EQ(NumTableFilesAtLevel(4), 1);
+  }
+}
+
 TEST_P(DBCompactionTestWithBottommostParam, SequenceKeysManualCompaction) {
  constexpr int kSstNum = 10;
  Options options = CurrentOptions();
  options.disable_auto_compactions = true;
+  options.num_levels = 7;
+  const bool dynamic_level = std::get<1>(GetParam());
+  options.level_compaction_dynamic_level_bytes = dynamic_level;
  DestroyAndReopen(options);

  // Generate some sst files on level 0 with sequence keys (no overlap)
@@ -7356,25 +7414,42 @@ TEST_P(DBCompactionTestWithBottommostParam, SequenceKeysManualCompaction) {

  auto cro = CompactRangeOptions();
  cro.bottommost_level_compaction = bottommost_level_compaction_;
+  bool trivial_moved = false;
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "DBImpl::BackgroundCompaction:TrivialMove",
+      [&](void* /*arg*/) { trivial_moved = true; });
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+  // All bottommost_level_compaction options should allow l0 -> l1 trivial move.
  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+  ASSERT_TRUE(trivial_moved);
  if (bottommost_level_compaction_ == BottommostLevelCompaction::kForce ||
      bottommost_level_compaction_ ==
          BottommostLevelCompaction::kForceOptimized) {
-    // Real compaction to compact all sst files from level 0 to 1 file on level
-    // 1
-    ASSERT_EQ("0,1", FilesPerLevel(0));
+    // bottommost level should go through intra-level compaction
+    // and has only 1 file
+    if (dynamic_level) {
+      ASSERT_EQ("0,0,0,0,0,0,1", FilesPerLevel(0));
+    } else {
+      ASSERT_EQ("0,1", FilesPerLevel(0));
+    }
  } else {
-    // Just trivial move from level 0 -> 1
-    ASSERT_EQ("0," + std::to_string(kSstNum), FilesPerLevel(0));
+    // Just trivial move from level 0 -> 1/base
+    if (dynamic_level) {
+      ASSERT_EQ("0,0,0,0,0,0," + std::to_string(kSstNum), FilesPerLevel(0));
+    } else {
+      ASSERT_EQ("0," + std::to_string(kSstNum), FilesPerLevel(0));
+    }
  }
 }

 INSTANTIATE_TEST_CASE_P(
    DBCompactionTestWithBottommostParam, DBCompactionTestWithBottommostParam,
-    ::testing::Values(BottommostLevelCompaction::kSkip,
-                      BottommostLevelCompaction::kIfHaveCompactionFilter,
-                      BottommostLevelCompaction::kForce,
-                      BottommostLevelCompaction::kForceOptimized));
+    ::testing::Combine(
+        ::testing::Values(BottommostLevelCompaction::kSkip,
+                          BottommostLevelCompaction::kIfHaveCompactionFilter,
+                          BottommostLevelCompaction::kForce,
+                          BottommostLevelCompaction::kForceOptimized),
+        ::testing::Bool()));

 TEST_F(DBCompactionTest, UpdateLevelSubCompactionTest) {
  Options options = CurrentOptions();
@@ -9108,6 +9183,235 @@ TEST_F(DBCompactionTest, BottommostFileCompactionAllowIngestBehind) {
  // ASSERT_OK(dbfull()->TEST_WaitForCompact(true /* wait_unscheduled */));
 }

+TEST_F(DBCompactionTest, TurnOnLevelCompactionDynamicLevelBytes) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleLevel;
+  options.allow_ingest_behind = false;
+  options.level_compaction_dynamic_level_bytes = false;
+  options.num_levels = 6;
+  options.compression = kNoCompression;
+  options.max_bytes_for_level_base = 1 << 20;
+  options.max_bytes_for_level_multiplier = 10;
+  DestroyAndReopen(options);
+
+  // put files in L0, L1 and L2
+  WriteOptions write_opts;
+  ASSERT_OK(db_->Put(write_opts, Key(1), "val1"));
+  Random rnd(33);
+  // Fill L2 with size larger than max_bytes_for_level_base,
+  // so the level above it won't be drained.
+  for (int i = 2; i <= (1 << 10); ++i) {
+    ASSERT_OK(db_->Put(write_opts, Key(i), rnd.RandomString(2 << 10)));
+  }
+  ASSERT_OK(Flush());
+  MoveFilesToLevel(2);
+  ASSERT_OK(db_->Put(write_opts, Key(2), "val2"));
+  ASSERT_OK(Flush());
+  MoveFilesToLevel(2);
+  ASSERT_OK(db_->Put(write_opts, Key(1), "new_val1"));
+  ASSERT_OK(Flush());
+  MoveFilesToLevel(1);
+  ASSERT_OK(db_->Put(write_opts, Key(3), "val3"));
+  ASSERT_OK(Flush());
+  ASSERT_EQ("1,1,2", FilesPerLevel());
+  auto verify_db = [&]() {
+    ASSERT_EQ(Get(Key(1)), "new_val1");
+    ASSERT_EQ(Get(Key(2)), "val2");
+    ASSERT_EQ(Get(Key(3)), "val3");
+  };
+  verify_db();
+
+  options.level_compaction_dynamic_level_bytes = true;
+  Reopen(options);
+  // except for L0, files should be pushed down as much as possible
+  ASSERT_EQ("1,0,0,0,1,2", FilesPerLevel());
+  verify_db();
+
+  // turning the options on and off should be safe
+  options.level_compaction_dynamic_level_bytes = false;
+  Reopen(options);
+  MoveFilesToLevel(1);
+  ASSERT_EQ("0,1,0,0,1,2", FilesPerLevel());
+  verify_db();
+
+  // newly flushed file is also pushed down
+  options.level_compaction_dynamic_level_bytes = true;
+  Reopen(options);
+  // Files in L1 should be trivially moved down during DB opening.
+  // The file should be moved to L3, and then may be drained and compacted to
+  // L4. So we just check L1 and L2 here.
+  ASSERT_EQ(0, NumTableFilesAtLevel(1));
+  ASSERT_EQ(0, NumTableFilesAtLevel(2));
+  verify_db();
+}
+
+TEST_F(DBCompactionTest, TurnOnLevelCompactionDynamicLevelBytesUCToLC) {
+  // Basic test for migrating from UC to LC.
+  // DB has non-empty L1 that should be pushed down to last level (L49).
+  Options options = CurrentOptions();
+  options.compaction_style = CompactionStyle::kCompactionStyleUniversal;
+  options.allow_ingest_behind = false;
+  options.level_compaction_dynamic_level_bytes = false;
+  options.num_levels = 50;
+  CreateAndReopenWithCF({"pikachu"}, options);
+
+  Random rnd(33);
+  for (int f = 0; f < 10; ++f) {
+    ASSERT_OK(Put(1, Key(f), rnd.RandomString(1000)));
+    ASSERT_OK(Flush(1));
+  }
+  CompactRangeOptions compact_options;
+  compact_options.change_level = true;
+  compact_options.target_level = 1;
+  ASSERT_OK(db_->CompactRange(compact_options, handles_[1], nullptr, nullptr));
+  ASSERT_EQ("0,1", FilesPerLevel(1));
+
+  options.compaction_style = CompactionStyle::kCompactionStyleLevel;
+  options.level_compaction_dynamic_level_bytes = true;
+  ReopenWithColumnFamilies({"default", "pikachu"}, options);
+  std::string expected_lsm = "";
+  for (int i = 0; i < 49; ++i) {
+    expected_lsm += "0,";
+  }
+  expected_lsm += "1";
+  ASSERT_EQ(expected_lsm, FilesPerLevel(1));
+
+  // Tests that entries for trial move in MANIFEST should be valid
+  ReopenWithColumnFamilies({"default", "pikachu"}, options);
+  ASSERT_EQ(expected_lsm, FilesPerLevel(1));
+}
+
+TEST_F(DBCompactionTest, DrainUnnecessaryLevelsAfterMultiplierChanged) {
+  // When the level size multiplier increases such that fewer levels become
+  // necessary, unnecessary levels should to be drained.
+  const int kBaseLevelBytes = 256 << 10;  // 256KB
+  const int kFileBytes = 64 << 10;        // 64KB
+  const int kInitMultiplier = 2, kChangedMultiplier = 10;
+  const int kNumFiles = 32;
+  const int kNumLevels = 5;
+  const int kValueBytes = 1 << 10;  // 1KB
+
+  Options options = CurrentOptions();
+  options.compression = kNoCompression;
+  options.level_compaction_dynamic_level_bytes = true;
+  options.max_bytes_for_level_base = kBaseLevelBytes;
+  options.max_bytes_for_level_multiplier = kInitMultiplier;
+  options.num_levels = kNumLevels;
+  Reopen(options);
+
+  // Initially we setup the LSM to look roughly as follows:
+  //
+  // L0: empty
+  // L1: 256KB
+  // ...
+  // L4: 1MB
+  Random rnd(301);
+  for (int file = 0; file < kNumFiles; ++file) {
+    for (int i = 0; i < kFileBytes / kValueBytes; ++i) {
+      ASSERT_OK(Put(Key(file * kFileBytes / kValueBytes + i),
+                    rnd.RandomString(kValueBytes)));
+    }
+    ASSERT_OK(Flush());
+  }
+
+  int init_num_nonempty = 0;
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  for (int level = 1; level < kNumLevels; ++level) {
+    if (NumTableFilesAtLevel(level) > 0) {
+      ++init_num_nonempty;
+    }
+  }
+
+  // After increasing the multiplier and running compaction fewer levels are
+  // needed to hold all the data. Unnecessary levels should be drained.
+  ASSERT_OK(db_->SetOptions({{"max_bytes_for_level_multiplier",
+                              std::to_string(kChangedMultiplier)}}));
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  int final_num_nonempty = 0;
+  for (int level = 1; level < kNumLevels; ++level) {
+    if (NumTableFilesAtLevel(level) > 0) {
+      ++final_num_nonempty;
+    }
+  }
+  ASSERT_GT(init_num_nonempty, final_num_nonempty);
+}
+
+TEST_F(DBCompactionTest, DrainUnnecessaryLevelsAfterDBBecomesSmall) {
+  // When the DB size is smaller, e.g., large chunk of data deleted by
+  // DeleteRange(), unnecessary levels should to be drained.
+  const int kBaseLevelBytes = 256 << 10;  // 256KB
+  const int kFileBytes = 64 << 10;        // 64KB
+  const int kMultiplier = 2;
+  const int kNumFiles = 32;
+  const int kNumLevels = 5;
+  const int kValueBytes = 1 << 10;  // 1KB
+  const int kDeleteFileNum = 8;
+
+  Options options = CurrentOptions();
+  options.compression = kNoCompression;
+  options.level_compaction_dynamic_level_bytes = true;
+  options.max_bytes_for_level_base = kBaseLevelBytes;
+  options.max_bytes_for_level_multiplier = kMultiplier;
+  options.num_levels = kNumLevels;
+  Reopen(options);
+
+  // Initially we setup the LSM to look roughly as follows:
+  //
+  // L0: empty
+  // L1: 256KB
+  // ...
+  // L4: 1MB
+  Random rnd(301);
+  for (int file = 0; file < kNumFiles; ++file) {
+    for (int i = 0; i < kFileBytes / kValueBytes; ++i) {
+      ASSERT_OK(Put(Key(file * kFileBytes / kValueBytes + i),
+                    rnd.RandomString(kValueBytes)));
+    }
+    ASSERT_OK(Flush());
+    if (file == kDeleteFileNum) {
+      // Ensure the DeleteRange() call below only delete data from last level
+      ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
+      ASSERT_EQ(NumTableFilesAtLevel(kNumLevels - 1), kDeleteFileNum + 1);
+    }
+  }
+
+  int init_num_nonempty = 0;
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  for (int level = 1; level < kNumLevels; ++level) {
+    if (NumTableFilesAtLevel(level) > 0) {
+      ++init_num_nonempty;
+    }
+  }
+
+  // Disable auto compaction CompactRange() below
+  ASSERT_OK(dbfull()->SetOptions({{"disable_auto_compactions", "true"}}));
+  // Delete keys within first (kDeleteFileNum + 1) files' key ranges.
+  // This should reduce DB size enough such that there is now
+  // an unneeded level.
+  std::string begin = Key(0);
+  std::string end = Key(kDeleteFileNum * kFileBytes / kValueBytes);
+  ASSERT_OK(
+      db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), begin, end));
+  Slice begin_slice = begin;
+  Slice end_slice = end;
+  ASSERT_OK(db_->CompactRange(CompactRangeOptions(), &begin_slice, &end_slice));
+  int after_delete_range_nonempty = 0;
+  for (int level = 1; level < kNumLevels; ++level) {
+    if (NumTableFilesAtLevel(level) > 0) {
+      ++after_delete_range_nonempty;
+    }
+  }
+  ASSERT_OK(dbfull()->SetOptions({{"disable_auto_compactions", "false"}}));
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  int final_num_nonempty = 0;
+  for (int level = 1; level < kNumLevels; ++level) {
+    if (NumTableFilesAtLevel(level) > 0) {
+      ++final_num_nonempty;
+    }
+  }
+  ASSERT_GE(init_num_nonempty, after_delete_range_nonempty);
+  ASSERT_GT(after_delete_range_nonempty, final_num_nonempty);
+}

 }  // namespace ROCKSDB_NAMESPACE

@@ -46,6 +46,11 @@ Status CompactedDBImpl::Get(const ReadOptions& options, ColumnFamilyHandle*,
 Status CompactedDBImpl::Get(const ReadOptions& options, ColumnFamilyHandle*,
                            const Slice& key, PinnableSlice* value,
                            std::string* timestamp) {
+  if (options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call Get with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
  assert(user_comparator_);
  if (options.timestamp) {
    const Status s = FailIfTsMismatchCf(
@@ -327,6 +327,9 @@ Status DBImpl::Resume() {
 //    means a new super version wouldn't have been installed
 Status DBImpl::ResumeImpl(DBRecoverContext context) {
  mutex_.AssertHeld();
+
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  WaitForBackgroundWork();

  Status s;
@@ -368,7 +371,7 @@ Status DBImpl::ResumeImpl(DBRecoverContext context) {
      assert(cfh);
      ColumnFamilyData* cfd = cfh->cfd();
      const MutableCFOptions& cf_opts = *cfd->GetLatestMutableCFOptions();
-      s = versions_->LogAndApply(cfd, cf_opts, &edit, &mutex_,
+      s = versions_->LogAndApply(cfd, cf_opts, read_options, &edit, &mutex_,
                                 directories_.GetDbDir());
      if (!s.ok()) {
        io_s = versions_->io_status();
@@ -1147,6 +1150,8 @@ FSDirectory* DBImpl::GetDataDir(ColumnFamilyData* cfd, size_t path_id) const {
 Status DBImpl::SetOptions(
    ColumnFamilyHandle* column_family,
    const std::unordered_map<std::string, std::string>& options_map) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  auto* cfd =
      static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
  if (options_map.empty()) {
@@ -1168,8 +1173,8 @@ Status DBImpl::SetOptions(
      new_options = *cfd->GetLatestMutableCFOptions();
      // Append new version to recompute compaction score.
      VersionEdit dummy_edit;
-      s = versions_->LogAndApply(cfd, new_options, &dummy_edit, &mutex_,
-                                 directories_.GetDbDir());
+      s = versions_->LogAndApply(cfd, new_options, read_options, &dummy_edit,
+                                 &mutex_, directories_.GetDbDir());
      // Trigger possible flush/compactions. This has to be before we persist
      // options to file, otherwise there will be a deadlock with writer
      // thread.
@@ -1507,7 +1512,9 @@ Status DBImpl::SyncWAL() {
  }
  if (status.ok() && synced_wals.IsWalAddition()) {
    InstrumentedMutexLock l(&mutex_);
-    status = ApplyWALToManifest(&synced_wals);
+    // TODO: plumb Env::IOActivity
+    const ReadOptions read_options;
+    status = ApplyWALToManifest(read_options, &synced_wals);
  }

  TEST_SYNC_POINT("DBImpl::SyncWAL:BeforeMarkLogsSynced:2");
@@ -1515,11 +1522,13 @@ Status DBImpl::SyncWAL() {
  return status;
 }

-Status DBImpl::ApplyWALToManifest(VersionEdit* synced_wals) {
+Status DBImpl::ApplyWALToManifest(const ReadOptions& read_options,
+                                  VersionEdit* synced_wals) {
  // not empty, write to MANIFEST.
  mutex_.AssertHeld();
+
  Status status = versions_->LogAndApplyToDefaultColumnFamily(
-      synced_wals, &mutex_, directories_.GetDbDir());
+      read_options, synced_wals, &mutex_, directories_.GetDbDir());
  if (!status.ok() && versions_->io_status().IsIOError()) {
    status = error_handler_.SetBGError(versions_->io_status(),
                                       BackgroundErrorReason::kManifestWrite);
@@ -1936,6 +1945,12 @@ Status DBImpl::GetEntity(const ReadOptions& read_options,
        "Cannot call GetEntity without a PinnableWideColumns object");
  }

+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call GetEntity with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
+
  columns->Reset();

  GetImplOptions get_impl_options;
@@ -1981,6 +1996,12 @@ Status DBImpl::GetImpl(const ReadOptions& read_options, const Slice& key,

  assert(get_impl_options.column_family);

+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call Get with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
+
  if (read_options.timestamp) {
    const Status s = FailIfTsMismatchCf(get_impl_options.column_family,
                                        *(read_options.timestamp),
@@ -2931,6 +2952,11 @@ Status DBImpl::MultiGetImpl(
    autovector<KeyContext*, MultiGetContext::MAX_BATCH_SIZE>* sorted_keys,
    SuperVersion* super_version, SequenceNumber snapshot,
    ReadCallback* callback) {
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call MultiGet with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
  PERF_CPU_TIMER_GUARD(get_cpu_nanos, immutable_db_options_.clock);
  StopWatch sw(immutable_db_options_.clock, stats_, DB_MULTIGET);

@@ -3129,6 +3155,8 @@ Status DBImpl::CreateColumnFamilies(
 Status DBImpl::CreateColumnFamilyImpl(const ColumnFamilyOptions& cf_options,
                                      const std::string& column_family_name,
                                      ColumnFamilyHandle** handle) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  Status s;
  *handle = nullptr;

@@ -3169,9 +3197,9 @@ Status DBImpl::CreateColumnFamilyImpl(const ColumnFamilyOptions& cf_options,
      write_thread_.EnterUnbatched(&w, &mutex_);
      // LogAndApply will both write the creation in MANIFEST and create
      // ColumnFamilyData object
-      s = versions_->LogAndApply(nullptr, MutableCFOptions(cf_options), &edit,
-                                 &mutex_, directories_.GetDbDir(), false,
-                                 &cf_options);
+      s = versions_->LogAndApply(nullptr, MutableCFOptions(cf_options),
+                                 read_options, &edit, &mutex_,
+                                 directories_.GetDbDir(), false, &cf_options);
      write_thread_.ExitUnbatched(&w);
    }
    if (s.ok()) {
@@ -3250,6 +3278,8 @@ Status DBImpl::DropColumnFamilies(
 }

 Status DBImpl::DropColumnFamilyImpl(ColumnFamilyHandle* column_family) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
  auto cfd = cfh->cfd();
  if (cfd->GetID() == 0) {
@@ -3272,8 +3302,9 @@ Status DBImpl::DropColumnFamilyImpl(ColumnFamilyHandle* column_family) {
      // we drop column family from a single write thread
      WriteThread::Writer w;
      write_thread_.EnterUnbatched(&w, &mutex_);
-      s = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(), &edit,
-                                 &mutex_, directories_.GetDbDir());
+      s = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
+                                 read_options, &edit, &mutex_,
+                                 directories_.GetDbDir());
      write_thread_.ExitUnbatched(&w);
    }
    if (s.ok()) {
@@ -3324,10 +3355,13 @@ bool DBImpl::KeyMayExist(const ReadOptions& read_options,
                         std::string* value, std::string* timestamp,
                         bool* value_found) {
  assert(value != nullptr);
+  assert(read_options.io_activity == Env::IOActivity::kUnknown);
+
  if (value_found != nullptr) {
    // falsify later if key-may-exist but can't fetch value
    *value_found = true;
  }
+  // TODO: plumb Env::IOActivity
  ReadOptions roptions = read_options;
  roptions.read_tier = kBlockCacheTier;  // read from block cache only
  PinnableSlice pinnable_val;
@@ -3356,6 +3390,11 @@ Iterator* DBImpl::NewIterator(const ReadOptions& read_options,
    return NewErrorIterator(Status::NotSupported(
        "ReadTier::kPersistedData is not yet supported in iterators."));
  }
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return NewErrorIterator(Status::InvalidArgument(
+        "Cannot call NewIterator with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`"));
+  }

  assert(column_family);

@@ -3491,6 +3530,11 @@ Status DBImpl::NewIterators(
    return Status::NotSupported(
        "ReadTier::kPersistedData is not yet supported in iterators.");
  }
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call NewIterators with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }

  if (read_options.timestamp) {
    for (auto* cf : column_families) {
@@ -3808,7 +3852,9 @@ Status DBImpl::GetPropertiesOfAllTables(ColumnFamilyHandle* column_family,
  version->Ref();
  mutex_.Unlock();

-  auto s = version->GetPropertiesOfAllTables(props);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  auto s = version->GetPropertiesOfAllTables(read_options, props);

  // Decrement the ref count
  mutex_.Lock();
@@ -3830,7 +3876,9 @@ Status DBImpl::GetPropertiesOfTablesInRange(ColumnFamilyHandle* column_family,
  version->Ref();
  mutex_.Unlock();

-  auto s = version->GetPropertiesOfTablesInRange(range, n, props);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  auto s = version->GetPropertiesOfTablesInRange(read_options, range, n, props);

  // Decrement the ref count
  mutex_.Lock();
@@ -4163,6 +4211,8 @@ Status DBImpl::GetApproximateSizes(const SizeApproximationOptions& options,
  SuperVersion* sv = GetAndRefSuperVersion(cfd);
  v = sv->current;

+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  for (int i = 0; i < n; i++) {
    Slice start = range[i].start;
    Slice limit = range[i].limit;
@@ -4184,7 +4234,7 @@ Status DBImpl::GetApproximateSizes(const SizeApproximationOptions& options,
    sizes[i] = 0;
    if (options.include_files) {
      sizes[i] += versions_->ApproximateSize(
-          options, v, k1.Encode(), k2.Encode(), /*start_level=*/0,
+          options, read_options, v, k1.Encode(), k2.Encode(), /*start_level=*/0,
          /*end_level=*/-1, TableReaderCaller::kUserApproximateSize);
    }
    if (options.include_memtables) {
@@ -4232,6 +4282,8 @@ Status DBImpl::GetUpdatesSince(
 }

 Status DBImpl::DeleteFile(std::string name) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  uint64_t number;
  FileType type;
  WalFileType log_type;
@@ -4311,7 +4363,8 @@ Status DBImpl::DeleteFile(std::string name) {
    edit.SetColumnFamily(cfd->GetID());
    edit.DeleteFile(level, number);
    status = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
-                                    &edit, &mutex_, directories_.GetDbDir());
+                                    read_options, &edit, &mutex_,
+                                    directories_.GetDbDir());
    if (status.ok()) {
      InstallSuperVersionAndScheduleWork(cfd,
                                         &job_context.superversion_contexts[0],
@@ -4333,6 +4386,8 @@ Status DBImpl::DeleteFile(std::string name) {
 Status DBImpl::DeleteFilesInRanges(ColumnFamilyHandle* column_family,
                                   const RangePtr* ranges, size_t n,
                                   bool include_end) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  Status status = Status::OK();
  auto cfh = static_cast_with_check<ColumnFamilyHandleImpl>(column_family);
  ColumnFamilyData* cfd = cfh->cfd();
@@ -4398,7 +4453,8 @@ Status DBImpl::DeleteFilesInRanges(ColumnFamilyHandle* column_family,
    }
    input_version->Ref();
    status = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
-                                    &edit, &mutex_, directories_.GetDbDir());
+                                    read_options, &edit, &mutex_,
+                                    directories_.GetDbDir());
    if (status.ok()) {
      InstallSuperVersionAndScheduleWork(cfd,
                                         &job_context.superversion_contexts[0],
@@ -5034,6 +5090,7 @@ Status DBImpl::GetLatestSequenceForKey(
  MergeContext merge_context;
  SequenceNumber max_covering_tombstone_seq = 0;

+  // TODO: plumb Env::IOActivity
  ReadOptions read_options;
  SequenceNumber current_seq = versions_->LastSequence();

@@ -5189,6 +5246,8 @@ Status DBImpl::IngestExternalFile(

 Status DBImpl::IngestExternalFiles(
    const std::vector<IngestExternalFileArg>& args) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  if (args.empty()) {
    return Status::InvalidArgument("ingestion arg list is empty");
  }
@@ -5406,9 +5465,9 @@ Status DBImpl::IngestExternalFiles(
        }
        assert(0 == num_entries);
      }
-      status =
-          versions_->LogAndApply(cfds_to_commit, mutable_cf_options_list,
-                                 edit_lists, &mutex_, directories_.GetDbDir());
+      status = versions_->LogAndApply(cfds_to_commit, mutable_cf_options_list,
+                                      read_options, edit_lists, &mutex_,
+                                      directories_.GetDbDir());
      // It is safe to update VersionSet last seqno here after LogAndApply since
      // LogAndApply persists last sequence number from VersionEdits,
      // which are from file's largest seqno and not from VersionSet.
@@ -5509,6 +5568,8 @@ Status DBImpl::CreateColumnFamilyWithImport(
    const ExportImportFilesMetaData& metadata, ColumnFamilyHandle** handle) {
  assert(handle != nullptr);
  assert(*handle == nullptr);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  std::string cf_comparator_name = options.comparator->Name();
  if (cf_comparator_name != metadata.db_comparator_name) {
    return Status::InvalidArgument("Comparator name mismatch");
@@ -5550,8 +5611,9 @@ Status DBImpl::CreateColumnFamilyWithImport(
      // file, we have to make sure the file number will never being reused.
      next_file_number = versions_->FetchAddFileNumber(metadata.files.size());
      auto cf_options = cfd->GetLatestMutableCFOptions();
-      status = versions_->LogAndApply(cfd, *cf_options, &dummy_edit, &mutex_,
-                                      directories_.GetDbDir());
+      status =
+          versions_->LogAndApply(cfd, *cf_options, read_options, &dummy_edit,
+                                 &mutex_, directories_.GetDbDir());
      if (status.ok()) {
        InstallSuperVersionAndScheduleWork(cfd, &dummy_sv_ctx, *cf_options);
      }
@@ -5587,8 +5649,9 @@ Status DBImpl::CreateColumnFamilyWithImport(
      // Install job edit [Mutex will be unlocked here]
      if (status.ok()) {
        auto cf_options = cfd->GetLatestMutableCFOptions();
-        status = versions_->LogAndApply(cfd, *cf_options, import_job.edit(),
-                                        &mutex_, directories_.GetDbDir());
+        status = versions_->LogAndApply(cfd, *cf_options, read_options,
+                                        import_job.edit(), &mutex_,
+                                        directories_.GetDbDir());
        if (status.ok()) {
          InstallSuperVersionAndScheduleWork(cfd, &sv_context, *cf_options);
        }
@@ -5648,6 +5711,12 @@ Status DBImpl::VerifyChecksumInternal(const ReadOptions& read_options,

  Status s;

+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    s = Status::InvalidArgument(
+        "Cannot verify file checksum with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+    return s;
+  }
  if (use_file_checksum) {
    FileChecksumGenFactory* const file_checksum_gen_factory =
        immutable_db_options_.file_checksum_gen_factory.get();
@@ -5761,6 +5830,12 @@ Status DBImpl::VerifyFullFileChecksum(const std::string& file_checksum_expected,
                                      const std::string& func_name_expected,
                                      const std::string& fname,
                                      const ReadOptions& read_options) {
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call VerifyChecksum with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
+
  Status s;
  if (file_checksum_expected == kUnknownFileChecksum) {
    return s;
@@ -5893,6 +5968,8 @@ Status DBImpl::ReserveFileNumbersBeforeIngestion(
    ColumnFamilyData* cfd, uint64_t num,
    std::unique_ptr<std::list<uint64_t>::iterator>& pending_output_elem,
    uint64_t* next_file_number) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  Status s;
  SuperVersionContext dummy_sv_ctx(true /* create_superversion */);
  assert(nullptr != next_file_number);
@@ -5910,8 +5987,8 @@ Status DBImpl::ReserveFileNumbersBeforeIngestion(
  // reuse the file number that has already assigned to the internal file,
  // and this will overwrite the external file. To protect the external
  // file, we have to make sure the file number will never being reused.
-  s = versions_->LogAndApply(cfd, *cf_options, &dummy_edit, &mutex_,
-                             directories_.GetDbDir());
+  s = versions_->LogAndApply(cfd, *cf_options, read_options, &dummy_edit,
+                             &mutex_, directories_.GetDbDir());
  if (s.ok()) {
    InstallSuperVersionAndScheduleWork(cfd, &dummy_sv_ctx, *cf_options);
  }
@@ -734,13 +734,17 @@ class DBImpl : public DB {
  // max_file_num_to_ignore allows bottom level compaction to filter out newly
  // compacted SST files. Setting max_file_num_to_ignore to kMaxUint64 will
  // disable the filtering
+  // If `final_output_level` is not nullptr, it is set to manual compaction's
+  // output level if returned status is OK, and it may or may not be set to
+  // manual compaction's output level if returned status is not OK.
  Status RunManualCompaction(ColumnFamilyData* cfd, int input_level,
                             int output_level,
                             const CompactRangeOptions& compact_range_options,
                             const Slice* begin, const Slice* end,
                             bool exclusive, bool disallow_trivial_move,
                             uint64_t max_file_num_to_ignore,
-                             const std::string& trim_ts);
+                             const std::string& trim_ts,
+                             int* final_output_level = nullptr);

  // Return an internal iterator over the current state of the database.
  // The keys of this iterator are internal keys (see format.h).
@@ -2123,7 +2127,7 @@ class DBImpl : public DB {

  // helper function to call after some of the logs_ were synced
  void MarkLogsSynced(uint64_t up_to, bool synced_dir, VersionEdit* edit);
-  Status ApplyWALToManifest(VersionEdit* edit);
+  Status ApplyWALToManifest(const ReadOptions& read_options, VersionEdit* edit);
  // WALs with log number up to up_to are not synced successfully.
  void MarkLogsNotSynced(uint64_t up_to);

@@ -229,7 +229,9 @@ Status DBImpl::FlushMemTableToOutputFile(
    log_io_s = SyncClosedLogs(job_context, &synced_wals);
    mutex_.Lock();
    if (log_io_s.ok() && synced_wals.IsWalAddition()) {
-      log_io_s = status_to_io_status(ApplyWALToManifest(&synced_wals));
+      const ReadOptions read_options(Env::IOActivity::kFlush);
+      log_io_s =
+          status_to_io_status(ApplyWALToManifest(read_options, &synced_wals));
      TEST_SYNC_POINT_CALLBACK("DBImpl::FlushMemTableToOutputFile:CommitWal:1",
                               nullptr);
    }
@@ -492,7 +494,9 @@ Status DBImpl::AtomicFlushMemTablesToOutputFiles(
    log_io_s = SyncClosedLogs(job_context, &synced_wals);
    mutex_.Lock();
    if (log_io_s.ok() && synced_wals.IsWalAddition()) {
-      log_io_s = status_to_io_status(ApplyWALToManifest(&synced_wals));
+      const ReadOptions read_options(Env::IOActivity::kFlush);
+      log_io_s =
+          status_to_io_status(ApplyWALToManifest(read_options, &synced_wals));
    }

    if (!log_io_s.ok() && !log_io_s.IsShutdownInProgress() &&
@@ -956,6 +960,9 @@ Status DBImpl::IncreaseFullHistoryTsLowImpl(ColumnFamilyData* cfd,
  VersionEdit edit;
  edit.SetColumnFamily(cfd->GetID());
  edit.SetFullHistoryTsLow(ts_low);
+
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  TEST_SYNC_POINT_CALLBACK("DBImpl::IncreaseFullHistoryTsLowImpl:BeforeEdit",
                           &edit);

@@ -969,7 +976,8 @@ Status DBImpl::IncreaseFullHistoryTsLowImpl(ColumnFamilyData* cfd,
  }

  Status s = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
-                                    &edit, &mutex_, directories_.GetDbDir());
+                                    read_options, &edit, &mutex_,
+                                    directories_.GetDbDir());
  if (!s.ok()) {
    return s;
  }
@@ -1054,8 +1062,8 @@ Status DBImpl::CompactRangeInternal(const CompactRangeOptions& options,
    }
    s = RunManualCompaction(cfd, ColumnFamilyData::kCompactAllLevels,
                            final_output_level, options, begin, end, exclusive,
-                            false, std::numeric_limits<uint64_t>::max(),
-                            trim_ts);
+                            false /* disable_trivial_move */,
+                            std::numeric_limits<uint64_t>::max(), trim_ts);
  } else {
    int first_overlapped_level = kInvalidLevel;
    int max_overlapped_level = kInvalidLevel;
@@ -1080,6 +1088,7 @@ Status DBImpl::CompactRangeInternal(const CompactRangeOptions& options,

      ReadOptions ro;
      ro.total_order_seek = true;
+      ro.io_activity = Env::IOActivity::kCompaction;
      bool overlap;
      for (int level = 0;
           level < current_version->storage_info()->num_non_empty_levels();
@@ -1142,74 +1151,83 @@ Status DBImpl::CompactRangeInternal(const CompactRangeOptions& options,
      CleanupSuperVersion(super_version);
    }
    if (s.ok() && first_overlapped_level != kInvalidLevel) {
-      // max_file_num_to_ignore can be used to filter out newly created SST
-      // files, useful for bottom level compaction in a manual compaction
-      uint64_t max_file_num_to_ignore = std::numeric_limits<uint64_t>::max();
-      uint64_t next_file_number = versions_->current_next_file_number();
-      final_output_level = max_overlapped_level;
-      int output_level;
-      for (int level = first_overlapped_level; level <= max_overlapped_level;
-           level++) {
-        bool disallow_trivial_move = false;
-        // in case the compaction is universal or if we're compacting the
-        // bottom-most level, the output level will be the same as input one.
-        // level 0 can never be the bottommost level (i.e. if all files are in
-        // level 0, we will compact to level 1)
-        if (cfd->ioptions()->compaction_style == kCompactionStyleUniversal ||
-            cfd->ioptions()->compaction_style == kCompactionStyleFIFO) {
-          output_level = level;
-        } else if (level == max_overlapped_level && level > 0) {
-          if (options.bottommost_level_compaction ==
-              BottommostLevelCompaction::kSkip) {
-            // Skip bottommost level compaction
-            continue;
-          } else if (options.bottommost_level_compaction ==
-                         BottommostLevelCompaction::kIfHaveCompactionFilter &&
-                     cfd->ioptions()->compaction_filter == nullptr &&
-                     cfd->ioptions()->compaction_filter_factory == nullptr) {
-            // Skip bottommost level compaction since we don't have a compaction
-            // filter
-            continue;
-          }
-          output_level = level;
-          // update max_file_num_to_ignore only for bottom level compaction
-          // because data in newly compacted files in middle levels may still
-          // need to be pushed down
-          max_file_num_to_ignore = next_file_number;
-        } else {
+      if (cfd->ioptions()->compaction_style == kCompactionStyleUniversal ||
+          cfd->ioptions()->compaction_style == kCompactionStyleFIFO) {
+        assert(first_overlapped_level == 0);
+        s = RunManualCompaction(
+            cfd, first_overlapped_level, first_overlapped_level, options, begin,
+            end, exclusive, true /* disallow_trivial_move */,
+            std::numeric_limits<uint64_t>::max() /* max_file_num_to_ignore */,
+            trim_ts);
+        final_output_level = max_overlapped_level;
+      } else {
+        assert(cfd->ioptions()->compaction_style == kCompactionStyleLevel);
+        uint64_t next_file_number = versions_->current_next_file_number();
+        // Start compaction from `first_overlapped_level`, one level down at a
+        // time, until output level >= max_overlapped_level.
+        // When max_overlapped_level == 0, we will still compact from L0 -> L1
+        // (or LBase), and followed by a bottommost level intra-level compaction
+        // at L1 (or LBase), if applicable.
+        int level = first_overlapped_level;
+        final_output_level = level;
+        int output_level = 0, base_level = 0;
+        while (level < max_overlapped_level || level == 0) {
          output_level = level + 1;
-          if (cfd->ioptions()->compaction_style == kCompactionStyleLevel &&
-              cfd->ioptions()->level_compaction_dynamic_level_bytes &&
+          if (cfd->ioptions()->level_compaction_dynamic_level_bytes &&
              level == 0) {
            output_level = ColumnFamilyData::kCompactToBaseLevel;
          }
-          // if it's a BottommostLevel compaction and `kForce*` compaction is
-          // set, disallow trivial move
-          if (level == max_overlapped_level &&
-              (options.bottommost_level_compaction ==
-                   BottommostLevelCompaction::kForce ||
-               options.bottommost_level_compaction ==
-                   BottommostLevelCompaction::kForceOptimized)) {
-            disallow_trivial_move = true;
+          // Use max value for `max_file_num_to_ignore` to always compact
+          // files down.
+          s = RunManualCompaction(
+              cfd, level, output_level, options, begin, end, exclusive,
+              !trim_ts.empty() /* disallow_trivial_move */,
+              std::numeric_limits<uint64_t>::max() /* max_file_num_to_ignore */,
+              trim_ts,
+              output_level == ColumnFamilyData::kCompactToBaseLevel
+                  ? &base_level
+                  : nullptr);
+          if (!s.ok()) {
+            break;
+          }
+          if (output_level == ColumnFamilyData::kCompactToBaseLevel) {
+            assert(base_level > 0);
+            level = base_level;
+          } else {
+            ++level;
+          }
+          final_output_level = level;
+          TEST_SYNC_POINT("DBImpl::RunManualCompaction()::1");
+          TEST_SYNC_POINT("DBImpl::RunManualCompaction()::2");
+        }
+        if (s.ok()) {
+          assert(final_output_level > 0);
+          // bottommost level intra-level compaction
+          // TODO(cbi): this preserves earlier behavior where if
+          //  max_overlapped_level = 0 and bottommost_level_compaction is
+          //  kIfHaveCompactionFilter, we only do a L0 -> LBase compaction
+          //  and do not do intra-LBase compaction even when user configures
+          //  compaction filter. We may want to still do a LBase -> LBase
+          //  compaction in case there is some file in LBase that did not go
+          //  through L0 -> LBase compaction, and hence did not go through
+          //  compaction filter.
+          if ((options.bottommost_level_compaction ==
+                   BottommostLevelCompaction::kIfHaveCompactionFilter &&
+               max_overlapped_level != 0 &&
+               (cfd->ioptions()->compaction_filter != nullptr ||
+                cfd->ioptions()->compaction_filter_factory != nullptr)) ||
+              options.bottommost_level_compaction ==
+                  BottommostLevelCompaction::kForceOptimized ||
+              options.bottommost_level_compaction ==
+                  BottommostLevelCompaction::kForce) {
+            // Use `next_file_number` as `max_file_num_to_ignore` to avoid
+            // rewriting newly compacted files when it is kForceOptimized.
+            s = RunManualCompaction(
+                cfd, final_output_level, final_output_level, options, begin,
+                end, exclusive, !trim_ts.empty() /* disallow_trivial_move */,
+                next_file_number /* max_file_num_to_ignore */, trim_ts);
          }
        }
-        // trim_ts need real compaction to remove latest record
-        if (!trim_ts.empty()) {
-          disallow_trivial_move = true;
-        }
-        s = RunManualCompaction(cfd, level, output_level, options, begin, end,
-                                exclusive, disallow_trivial_move,
-                                max_file_num_to_ignore, trim_ts);
-        if (!s.ok()) {
-          break;
-        }
-        if (output_level == ColumnFamilyData::kCompactToBaseLevel) {
-          final_output_level = cfd->NumberLevels() - 1;
-        } else if (output_level > final_output_level) {
-          final_output_level = output_level;
-        }
-        TEST_SYNC_POINT("DBImpl::RunManualCompaction()::1");
-        TEST_SYNC_POINT("DBImpl::RunManualCompaction()::2");
      }
    }
  }
@@ -1630,6 +1648,8 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
    return Status::InvalidArgument("Target level exceeds number of levels");
  }

+  const ReadOptions read_options(Env::IOActivity::kCompaction);
+
  SuperVersionContext sv_context(/* create_superversion */ true);

  InstrumentedMutexLock guard_lock(&mutex_);
@@ -1744,8 +1764,9 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
                    "[%s] Apply version edit:\n%s", cfd->GetName().c_str(),
                    edit.DebugString().data());

-    Status status = versions_->LogAndApply(cfd, mutable_cf_options, &edit,
-                                           &mutex_, directories_.GetDbDir());
+    Status status =
+        versions_->LogAndApply(cfd, mutable_cf_options, read_options, &edit,
+                               &mutex_, directories_.GetDbDir());

    cfd->compaction_picker()->UnregisterCompaction(c.get());
    c.reset();
@@ -1853,7 +1874,8 @@ Status DBImpl::RunManualCompaction(
    ColumnFamilyData* cfd, int input_level, int output_level,
    const CompactRangeOptions& compact_range_options, const Slice* begin,
    const Slice* end, bool exclusive, bool disallow_trivial_move,
-    uint64_t max_file_num_to_ignore, const std::string& trim_ts) {
+    uint64_t max_file_num_to_ignore, const std::string& trim_ts,
+    int* final_output_level) {
  assert(input_level == ColumnFamilyData::kCompactAllLevels ||
         input_level >= 0);

@@ -2004,6 +2026,15 @@ Status DBImpl::RunManualCompaction(
    } else if (!scheduled) {
      if (compaction == nullptr) {
        manual.done = true;
+        if (final_output_level) {
+          // No compaction needed or there is a conflicting compaction.
+          // Still set `final_output_level` to the level where we would
+          // have compacted to.
+          *final_output_level = output_level;
+          if (output_level == ColumnFamilyData::kCompactToBaseLevel) {
+            *final_output_level = cfd->current()->storage_info()->base_level();
+          }
+        }
        bg_cv_.SignalAll();
        continue;
      }
@@ -2037,6 +2068,9 @@ Status DBImpl::RunManualCompaction(
      }
      scheduled = true;
      TEST_SYNC_POINT("DBImpl::RunManualCompaction:Scheduled");
+      if (final_output_level) {
+        *final_output_level = compaction->output_level();
+      }
    }
  }

@@ -3167,6 +3201,8 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
  mutex_.AssertHeld();
  TEST_SYNC_POINT("DBImpl::BackgroundCompaction:Start");

+  const ReadOptions read_options(Env::IOActivity::kCompaction);
+
  bool is_manual = (manual_compaction != nullptr);
  std::unique_ptr<Compaction> c;
  if (prepicked_compaction != nullptr &&
@@ -3377,9 +3413,9 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
    for (const auto& f : *c->inputs(0)) {
      c->edit()->DeleteFile(c->level(), f->fd.GetNumber());
    }
-    status = versions_->LogAndApply(c->column_family_data(),
-                                    *c->mutable_cf_options(), c->edit(),
-                                    &mutex_, directories_.GetDbDir());
+    status = versions_->LogAndApply(
+        c->column_family_data(), *c->mutable_cf_options(), read_options,
+        c->edit(), &mutex_, directories_.GetDbDir());
    io_s = versions_->io_status();
    InstallSuperVersionAndScheduleWork(c->column_family_data(),
                                       &job_context->superversion_contexts[0],
@@ -3396,9 +3432,7 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
                             c->column_family_data());
    // Instrument for event update
    // TODO(yhchiang): add op details for showing trivial-move.
-    ThreadStatusUtil::SetColumnFamily(
-        c->column_family_data(), c->column_family_data()->ioptions()->env,
-        immutable_db_options_.enable_thread_tracking);
+    ThreadStatusUtil::SetColumnFamily(c->column_family_data());
    ThreadStatusUtil::SetThreadOperation(ThreadStatus::OP_COMPACTION);

    compaction_job_stats.num_input_files = c->num_input_files(0);
@@ -3444,9 +3478,9 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
            vstorage->GetNextCompactCursor(start_level, c->num_input_files(0)));
      }
    }
-    status = versions_->LogAndApply(c->column_family_data(),
-                                    *c->mutable_cf_options(), c->edit(),
-                                    &mutex_, directories_.GetDbDir());
+    status = versions_->LogAndApply(
+        c->column_family_data(), *c->mutable_cf_options(), read_options,
+        c->edit(), &mutex_, directories_.GetDbDir());
    io_s = versions_->io_status();
    // Use latest MutableCFOptions
    InstallSuperVersionAndScheduleWork(c->column_family_data(),
@@ -3760,6 +3794,8 @@ void DBImpl::BuildCompactionJobInfo(
  compaction_job_info->table_properties = c->GetOutputTableProperties();
  compaction_job_info->compaction_reason = c->compaction_reason();
  compaction_job_info->compression = c->output_compression();
+
+  const ReadOptions read_options(Env::IOActivity::kCompaction);
  for (size_t i = 0; i < c->num_input_levels(); ++i) {
    for (const auto fmd : *c->inputs(i)) {
      const FileDescriptor& desc = fmd->fd;
@@ -3771,7 +3807,7 @@ void DBImpl::BuildCompactionJobInfo(
          static_cast<int>(i), file_number, fmd->oldest_blob_file_number});
      if (compaction_job_info->table_properties.count(fn) == 0) {
        std::shared_ptr<const TableProperties> tp;
-        auto s = current->GetTableProperties(&tp, fmd, &fn);
+        auto s = current->GetTableProperties(read_options, &tp, fmd, &fn);
        if (s.ok()) {
          compaction_job_info->table_properties[fn] = tp;
        }
@@ -61,7 +61,8 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
                   "PromoteL0 FAILED. Invalid target level %d\n", target_level);
    return Status::InvalidArgument("Invalid target level");
  }
-
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  Status status;
  VersionEdit edit;
  JobContext job_context(next_job_id_.fetch_add(1), true);
@@ -141,7 +142,8 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
    }

    status = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
-                                    &edit, &mutex_, directories_.GetDbDir());
+                                    read_options, &edit, &mutex_,
+                                    directories_.GetDbDir());
    if (status.ok()) {
      InstallSuperVersionAndScheduleWork(cfd,
                                         &job_context.superversion_contexts[0],
@@ -19,6 +19,7 @@
 #include "file/writable_file_writer.h"
 #include "logging/logging.h"
 #include "monitoring/persistent_stats_history.h"
+#include "monitoring/thread_status_util.h"
 #include "options/options_helper.h"
 #include "rocksdb/table.h"
 #include "rocksdb/wal_filter.h"
@@ -533,6 +534,103 @@ Status DBImpl::Recover(
  if (!s.ok()) {
    return s;
  }
+  if (s.ok() && !read_only) {
+    for (auto cfd : *versions_->GetColumnFamilySet()) {
+      // Try to trivially move files down the LSM tree to start from bottommost
+      // level when level_compaction_dynamic_level_bytes is enabled. This should
+      // only be useful when user is migrating to turning on this option.
+      // If a user is migrating from Level Compaction with a smaller level
+      // multiplier or from Universal Compaction, there may be too many
+      // non-empty levels and the trivial moves here are not sufficed for
+      // migration. Additional compactions are needed to drain unnecessary
+      // levels.
+      //
+      // Note that this step moves files down LSM without consulting
+      // SSTPartitioner. Further compactions are still needed if
+      // the user wants to partition SST files.
+      // Note that files moved in this step may not respect the compression
+      // option in target level.
+      if (cfd->ioptions()->compaction_style ==
+              CompactionStyle::kCompactionStyleLevel &&
+          cfd->ioptions()->level_compaction_dynamic_level_bytes &&
+          !cfd->GetLatestMutableCFOptions()->disable_auto_compactions) {
+        int to_level = cfd->ioptions()->num_levels - 1;
+        // last level is reserved
+        // allow_ingest_behind does not support Level Compaction,
+        // and per_key_placement can have infinite compaction loop for Level
+        // Compaction. Adjust to_level here just to be safe.
+        if (cfd->ioptions()->allow_ingest_behind ||
+            cfd->ioptions()->preclude_last_level_data_seconds > 0) {
+          to_level -= 1;
+        }
+        // Whether this column family has a level trivially moved
+        bool moved = false;
+        // Fill the LSM starting from to_level and going up one level at a time.
+        // Some loop invariants (when last level is not reserved):
+        // - levels in (from_level, to_level] are empty, and
+        // - levels in (to_level, last_level] are non-empty.
+        for (int from_level = to_level; from_level >= 0; --from_level) {
+          const std::vector<FileMetaData*>& level_files =
+              cfd->current()->storage_info()->LevelFiles(from_level);
+          if (level_files.empty() || from_level == 0) {
+            continue;
+          }
+          assert(from_level <= to_level);
+          // Trivial move files from `from_level` to `to_level`
+          if (from_level < to_level) {
+            if (!moved) {
+              // lsm_state will look like "[1,2,3,4,5,6,0]" for an LSM with
+              // 7 levels
+              std::string lsm_state = "[";
+              for (int i = 0; i < cfd->ioptions()->num_levels; ++i) {
+                lsm_state += std::to_string(
+                    cfd->current()->storage_info()->NumLevelFiles(i));
+                if (i < cfd->ioptions()->num_levels - 1) {
+                  lsm_state += ",";
+                }
+              }
+              lsm_state += "]";
+              ROCKS_LOG_WARN(immutable_db_options_.info_log,
+                             "[%s] Trivially move files down the LSM when open "
+                             "with level_compaction_dynamic_level_bytes=true,"
+                             " lsm_state: %s (Files are moved only if DB "
+                             "Recovery is successful).",
+                             cfd->GetName().c_str(), lsm_state.c_str());
+              moved = true;
+            }
+            ROCKS_LOG_WARN(
+                immutable_db_options_.info_log,
+                "[%s] Moving %zu files from from_level-%d to from_level-%d",
+                cfd->GetName().c_str(), level_files.size(), from_level,
+                to_level);
+            VersionEdit edit;
+            edit.SetColumnFamily(cfd->GetID());
+            for (const FileMetaData* f : level_files) {
+              edit.DeleteFile(from_level, f->fd.GetNumber());
+              edit.AddFile(to_level, f->fd.GetNumber(), f->fd.GetPathId(),
+                           f->fd.GetFileSize(), f->smallest, f->largest,
+                           f->fd.smallest_seqno, f->fd.largest_seqno,
+                           f->marked_for_compaction,
+                           f->temperature,  // this can be different from
+                                            // `last_level_temperature`
+                           f->oldest_blob_file_number, f->oldest_ancester_time,
+                           f->file_creation_time, f->epoch_number,
+                           f->file_checksum, f->file_checksum_func_name,
+                           f->unique_id, f->compensated_range_deletion_size);
+              ROCKS_LOG_WARN(immutable_db_options_.info_log,
+                             "[%s] Moving #%" PRIu64
+                             " from from_level-%d to from_level-%d %" PRIu64
+                             " bytes\n",
+                             cfd->GetName().c_str(), f->fd.GetNumber(),
+                             from_level, to_level, f->fd.GetFileSize());
+            }
+            recovery_ctx->UpdateVersionEdits(cfd, edit);
+          }
+          --to_level;
+        }
+      }
+    }
+  }
  s = SetupDBId(read_only, recovery_ctx);
  ROCKS_LOG_INFO(immutable_db_options_.info_log, "DB ID: %s\n", db_id_.c_str());
  if (s.ok() && !read_only) {
@@ -826,8 +924,9 @@ Status DBImpl::InitPersistStatsColumnFamily() {
 Status DBImpl::LogAndApplyForRecovery(const RecoveryContext& recovery_ctx) {
  mutex_.AssertHeld();
  assert(versions_->descriptor_log_ == nullptr);
+  const ReadOptions read_options(Env::IOActivity::kDBOpen);
  Status s = versions_->LogAndApply(
-      recovery_ctx.cfds_, recovery_ctx.mutable_cf_opts_,
+      recovery_ctx.cfds_, recovery_ctx.mutable_cf_opts_, read_options,
      recovery_ctx.edit_lists_, &mutex_, directories_.GetDbDir());
  if (s.ok() && !(recovery_ctx.files_to_delete_.empty())) {
    mutex_.Unlock();
@@ -1480,6 +1579,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
  meta.fd = FileDescriptor(versions_->NewFileNumber(), 0, 0);
  ReadOptions ro;
  ro.total_order_seek = true;
+  ro.io_activity = Env::IOActivity::kDBOpen;
  Arena arena;
  Status s;
  TableProperties table_properties;
@@ -1538,10 +1638,11 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
      SeqnoToTimeMapping empty_seqno_time_mapping;
      Version* version = cfd->current();
      version->Ref();
+      const ReadOptions read_option(Env::IOActivity::kDBOpen);
      s = BuildTable(
          dbname_, versions_.get(), immutable_db_options_, tboptions,
-          file_options_for_compaction_, cfd->table_cache(), iter.get(),
-          std::move(range_del_iters), &meta, &blob_file_additions,
+          file_options_for_compaction_, read_option, cfd->table_cache(),
+          iter.get(), std::move(range_del_iters), &meta, &blob_file_additions,
          snapshot_seqs, earliest_write_conflict_snapshot, kMaxSequenceNumber,
          snapshot_checker, paranoid_file_checks, cfd->internal_stats(), &io_s,
          io_tracer_, BlobFileCreationReason::kRecovery,
@@ -1642,8 +1743,12 @@ Status DB::Open(const DBOptions& db_options, const std::string& dbname,
                std::vector<ColumnFamilyHandle*>* handles, DB** dbptr) {
  const bool kSeqPerBatch = true;
  const bool kBatchPerTxn = true;
-  return DBImpl::Open(db_options, dbname, column_families, handles, dbptr,
-                      !kSeqPerBatch, kBatchPerTxn);
+  ThreadStatusUtil::SetEnableTracking(db_options.enable_thread_tracking);
+  ThreadStatusUtil::SetThreadOperation(ThreadStatus::OperationType::OP_DBOPEN);
+  Status s = DBImpl::Open(db_options, dbname, column_families, handles, dbptr,
+                          !kSeqPerBatch, kBatchPerTxn);
+  ThreadStatusUtil::ResetThreadStatus();
+  return s;
 }

 // TODO: Implement the trimming in flush code path.
@@ -1828,7 +1933,9 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,

  // Handles create_if_missing, error_if_exists
  uint64_t recovered_seq(kMaxSequenceNumber);
-  s = impl->Recover(column_families, false, false, false, &recovered_seq,
+  s = impl->Recover(column_families, false /* read_only */,
+                    false /* error_if_wal_file_exists */,
+                    false /* error_if_data_exists_in_wals */, &recovered_seq,
                    &recovery_ctx);
  if (s.ok()) {
    uint64_t new_log_number = impl->versions_->NewFileNumber();
@@ -40,6 +40,11 @@ Status DBImplReadOnly::Get(const ReadOptions& read_options,
                           ColumnFamilyHandle* column_family, const Slice& key,
                           PinnableSlice* pinnable_val,
                           std::string* timestamp) {
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call Get with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
  assert(pinnable_val != nullptr);
  // TODO: stopwatch DB_GET needed?, perf timer needed?
  PERF_TIMER_GUARD(get_snapshot_time);
@@ -112,6 +117,11 @@ Status DBImplReadOnly::Get(const ReadOptions& read_options,

 Iterator* DBImplReadOnly::NewIterator(const ReadOptions& read_options,
                                      ColumnFamilyHandle* column_family) {
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return NewErrorIterator(Status::InvalidArgument(
+        "Cannot call NewIterator with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`"));
+  }
  assert(column_family);
  if (read_options.timestamp) {
    const Status s = FailIfTsMismatchCf(
@@ -345,6 +345,11 @@ Status DBImplSecondary::GetImpl(const ReadOptions& read_options,
                                ColumnFamilyHandle* column_family,
                                const Slice& key, PinnableSlice* pinnable_val,
                                std::string* timestamp) {
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call Get with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
  assert(pinnable_val != nullptr);
  PERF_CPU_TIMER_GUARD(get_cpu_nanos, immutable_db_options_.clock);
  StopWatch sw(immutable_db_options_.clock, stats_, DB_GET);
@@ -445,6 +450,11 @@ Iterator* DBImplSecondary::NewIterator(const ReadOptions& read_options,
    return NewErrorIterator(Status::NotSupported(
        "ReadTier::kPersistedData is not yet supported in iterators."));
  }
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return NewErrorIterator(Status::InvalidArgument(
+        "Cannot call NewIterator with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`"));
+  }

  assert(column_family);
  if (read_options.timestamp) {
@@ -511,6 +521,11 @@ Status DBImplSecondary::NewIterators(
    return Status::NotSupported(
        "ReadTier::kPersistedData is not yet supported in iterators.");
  }
+  if (read_options.io_activity != Env::IOActivity::kUnknown) {
+    return Status::InvalidArgument(
+        "Cannot call NewIterators with `ReadOptions::io_activity` != "
+        "`Env::IOActivity::kUnknown`");
+  }
  ReadCallback* read_callback = nullptr;  // No read callback provided.
  if (iterators == nullptr) {
    return Status::InvalidArgument("iterators not allowed to be nullptr");
@@ -606,7 +606,9 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
    log_write_mutex_.Unlock();
    if (status.ok() && synced_wals.IsWalAddition()) {
      InstrumentedMutexLock l(&mutex_);
-      status = ApplyWALToManifest(&synced_wals);
+      // TODO: plumb Env::IOActivity
+      const ReadOptions read_options;
+      status = ApplyWALToManifest(read_options, &synced_wals);
    }

    // Requesting sync with two_write_queues_ is expected to be very rare. We
@@ -767,7 +769,9 @@ Status DBImpl::PipelinedWriteImpl(const WriteOptions& write_options,
    }
    if (w.status.ok() && synced_wals.IsWalAddition()) {
      InstrumentedMutexLock l(&mutex_);
-      w.status = ApplyWALToManifest(&synced_wals);
+      // TODO: plumb Env::IOActivity
+      const ReadOptions read_options;
+      w.status = ApplyWALToManifest(read_options, &synced_wals);
    }
    write_thread_.ExitAsBatchGroupLeader(wal_write_group, w.status);
  }
@@ -1805,7 +1809,7 @@ Status DBImpl::DelayWrite(uint64_t num_bytes, WriteThread& write_thread,
  bool delayed = false;
  {
    StopWatch sw(immutable_db_options_.clock, stats_, WRITE_STALL,
-                 &time_delayed);
+                 Histograms::HISTOGRAM_ENUM_MAX, &time_delayed);
    // To avoid parallel timed delays (bad throttling), only support them
    // on the primary write queue.
    uint64_t delay;
@@ -2086,6 +2090,8 @@ void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
 // two_write_queues_ is true (This is to simplify the reasoning.)
 Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
  mutex_.AssertHeld();
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  log::Writer* new_log = nullptr;
  MemTable* new_mem = nullptr;
  IOStatus io_s;
@@ -2237,8 +2243,8 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {

      VersionEdit wal_deletion;
      wal_deletion.DeleteWalsBefore(min_wal_number_to_keep);
-      s = versions_->LogAndApplyToDefaultColumnFamily(&wal_deletion, &mutex_,
-                                                      directories_.GetDbDir());
+      s = versions_->LogAndApplyToDefaultColumnFamily(
+          read_options, &wal_deletion, &mutex_, directories_.GetDbDir());
      if (!s.ok() && versions_->io_status().IsIOError()) {
        s = error_handler_.SetBGError(versions_->io_status(),
                                      BackgroundErrorReason::kManifestWrite);
@@ -131,6 +131,7 @@ void DBIter::Next() {
  assert(valid_);
  assert(status_.ok());

+  PERF_COUNTER_ADD(iter_next_count, 1);
  PERF_CPU_TIMER_GUARD(iter_next_cpu_nanos, clock_);
  // Release temporarily pinned blocks from last operation
  ReleaseTempPinnedData();
@@ -195,6 +196,7 @@ bool DBIter::SetBlobValueIfNeeded(const Slice& user_key,

  // TODO: consider moving ReadOptions from ArenaWrappedDBIter to DBIter to
  // avoid having to copy options back and forth.
+  // TODO: plumb Env::IOActivity
  ReadOptions read_options;
  read_options.read_tier = read_tier_;
  read_options.fill_cache = fill_cache_;
@@ -637,6 +639,7 @@ void DBIter::Prev() {
  assert(valid_);
  assert(status_.ok());

+  PERF_COUNTER_ADD(iter_prev_count, 1);
  PERF_CPU_TIMER_GUARD(iter_prev_cpu_nanos, clock_);
  ReleaseTempPinnedData();
  ResetBlobValue();
@@ -1444,6 +1447,7 @@ void DBIter::SetSavedKeyToSeekForPrevTarget(const Slice& target) {
 }

 void DBIter::Seek(const Slice& target) {
+  PERF_COUNTER_ADD(iter_seek_count, 1);
  PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
  StopWatch sw(clock_, statistics_, DB_SEEK);

@@ -1517,6 +1521,7 @@ void DBIter::Seek(const Slice& target) {
 }

 void DBIter::SeekForPrev(const Slice& target) {
+  PERF_COUNTER_ADD(iter_seek_count, 1);
  PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
  StopWatch sw(clock_, statistics_, DB_SEEK);

@@ -1590,6 +1595,7 @@ void DBIter::SeekToFirst() {
    Seek(*iterate_lower_bound_);
    return;
  }
+  PERF_COUNTER_ADD(iter_seek_count, 1);
  PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
  // Don't use iter_::Seek() if we set a prefix extractor
  // because prefix seek will be used.
@@ -1652,6 +1658,7 @@ void DBIter::SeekToLast() {
    return;
  }

+  PERF_COUNTER_ADD(iter_seek_count, 1);
  PERF_CPU_TIMER_GUARD(iter_seek_cpu_nanos, clock_);
  // Don't use iter_::Seek() if we set a prefix extractor
  // because prefix seek will be used.
@@ -30,12 +30,63 @@ class DummyReadCallback : public ReadCallback {
  void SetSnapshot(SequenceNumber seq) { max_visible_seq_ = seq; }
 };

+class DBIteratorBaseTest : public DBTestBase {
+ public:
+  DBIteratorBaseTest()
+      : DBTestBase("db_iterator_test", /*env_do_fsync=*/true) {}
+};
+
+TEST_F(DBIteratorBaseTest, APICallsWithPerfContext) {
+  // Set up the DB
+  Options options = CurrentOptions();
+  DestroyAndReopen(options);
+  Random rnd(301);
+  for (int i = 1; i <= 3; i++) {
+    ASSERT_OK(Put(std::to_string(i), std::to_string(i)));
+  }
+
+  // Setup iterator and PerfContext
+  Iterator* iter = db_->NewIterator(ReadOptions());
+  std::string key_str = std::to_string(2);
+  Slice key(key_str);
+  SetPerfLevel(kEnableCount);
+  get_perf_context()->Reset();
+
+  // Initial PerfContext counters
+  ASSERT_EQ(0, get_perf_context()->iter_seek_count);
+  ASSERT_EQ(0, get_perf_context()->iter_next_count);
+  ASSERT_EQ(0, get_perf_context()->iter_prev_count);
+
+  // Test Seek-related API calls PerfContext counter
+  iter->Seek(key);
+  iter->SeekToFirst();
+  iter->SeekToLast();
+  iter->SeekForPrev(key);
+  ASSERT_EQ(4, get_perf_context()->iter_seek_count);
+  ASSERT_EQ(0, get_perf_context()->iter_next_count);
+  ASSERT_EQ(0, get_perf_context()->iter_prev_count);
+
+  // Test Next() calls PerfContext counter
+  iter->Next();
+  ASSERT_EQ(4, get_perf_context()->iter_seek_count);
+  ASSERT_EQ(1, get_perf_context()->iter_next_count);
+  ASSERT_EQ(0, get_perf_context()->iter_prev_count);
+
+  // Test Prev() calls PerfContext counter
+  iter->Prev();
+  ASSERT_EQ(4, get_perf_context()->iter_seek_count);
+  ASSERT_EQ(1, get_perf_context()->iter_next_count);
+  ASSERT_EQ(1, get_perf_context()->iter_prev_count);
+
+  delete iter;
+}
+
 // Test param:
 //   bool: whether to pass read_callback to NewIterator().
-class DBIteratorTest : public DBTestBase,
+class DBIteratorTest : public DBIteratorBaseTest,
                       public testing::WithParamInterface<bool> {
 public:
-  DBIteratorTest() : DBTestBase("db_iterator_test", /*env_do_fsync=*/true) {}
+  DBIteratorTest() {}

  Iterator* NewIterator(const ReadOptions& read_options,
                        ColumnFamilyHandle* column_family = nullptr) {
@@ -358,6 +358,48 @@ TEST_F(DBMergeOperatorTest, MergeOperatorFailsWithMustMerge) {
  }
 }

+TEST_F(DBMergeOperatorTest, DataBlockBinaryAndHash) {
+  // Basic test to check that merge operator works with data block index type
+  // DataBlockBinaryAndHash.
+  Options options;
+  options.create_if_missing = true;
+  options.merge_operator.reset(new TestPutOperator());
+  options.env = env_;
+  BlockBasedTableOptions table_options;
+  table_options.block_restart_interval = 16;
+  table_options.data_block_index_type =
+      BlockBasedTableOptions::DataBlockIndexType::kDataBlockBinaryAndHash;
+  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
+  DestroyAndReopen(options);
+
+  const int kNumKeys = 100;
+  for (int i = 0; i < kNumKeys; ++i) {
+    ASSERT_OK(db_->Merge(WriteOptions(), Key(i), std::to_string(i)));
+  }
+  ASSERT_OK(Flush());
+  std::string value;
+  for (int i = 0; i < kNumKeys; ++i) {
+    ASSERT_OK(db_->Get(ReadOptions(), Key(i), &value));
+    ASSERT_EQ(std::to_string(i), value);
+  }
+
+  std::vector<const Snapshot*> snapshots;
+  for (int i = 0; i < kNumKeys; ++i) {
+    ASSERT_OK(db_->Delete(WriteOptions(), Key(i)));
+    for (int j = 0; j < 3; ++j) {
+      ASSERT_OK(db_->Merge(WriteOptions(), Key(i), std::to_string(i * 3 + j)));
+      snapshots.push_back(db_->GetSnapshot());
+    }
+  }
+  ASSERT_OK(Flush());
+  for (int i = 0; i < kNumKeys; ++i) {
+    ASSERT_OK(db_->Get(ReadOptions(), Key(i), &value));
+    ASSERT_EQ(std::to_string(i * 3 + 2), value);
+  }
+  for (auto snapshot : snapshots) {
+    db_->ReleaseSnapshot(snapshot);
+  }
+}

 class MergeOperatorPinningTest : public DBMergeOperatorTest,
                                 public testing::WithParamInterface<bool> {
@@ -2109,25 +2109,26 @@ TEST_F(DBPropertiesTest, GetMapPropertyBlockCacheEntryStats) {

 TEST_F(DBPropertiesTest, WriteStallStatsSanityCheck) {
  for (uint32_t i = 0; i < static_cast<uint32_t>(WriteStallCause::kNone); ++i) {
-    std::string str = kWriteStallCauseToHyphenString[i];
+    WriteStallCause cause = static_cast<WriteStallCause>(i);
+    const std::string& str = WriteStallCauseToHyphenString(cause);
    ASSERT_TRUE(!str.empty())
        << "Please ensure mapping from `WriteStallCause` to "
-           "`kWriteStallCauseToHyphenString` is complete";
-    WriteStallCause cause = static_cast<WriteStallCause>(i);
+           "`WriteStallCauseToHyphenString` is complete";
    if (cause == WriteStallCause::kCFScopeWriteStallCauseEnumMax ||
        cause == WriteStallCause::kDBScopeWriteStallCauseEnumMax) {
-      ASSERT_EQ(str, kInvalidWriteStallCauseHyphenString)
-          << "Please ensure order in `kWriteStallCauseToHyphenString` is "
+      ASSERT_EQ(str, InvalidWriteStallHyphenString())
+          << "Please ensure order in `WriteStallCauseToHyphenString` is "
             "consistent with `WriteStallCause`";
    }
  }

  for (uint32_t i = 0; i < static_cast<uint32_t>(WriteStallCondition::kNormal);
       ++i) {
-    std::string str = kWriteStallConditionToHyphenString[i];
+    WriteStallCondition condition = static_cast<WriteStallCondition>(i);
+    const std::string& str = WriteStallConditionToHyphenString(condition);
    ASSERT_TRUE(!str.empty())
        << "Please ensure mapping from `WriteStallCondition` to "
-           "`kWriteStallConditionToHyphenString` is complete";
+           "`WriteStallConditionToHyphenString` is complete";
  }

  for (uint32_t i = 0; i < static_cast<uint32_t>(WriteStallCause::kNone); ++i) {
@@ -2332,8 +2333,9 @@ TEST_F(DBPropertiesTest, TableMetaIndexKeys) {

    // Read metaindex
    BlockContents bc;
-    ASSERT_OK(ReadMetaIndexBlockInFile(r.get(), file_size, 0U,
-                                       ImmutableOptions(options), &bc));
+    const ReadOptions read_options;
+    ASSERT_OK(ReadMetaIndexBlockInFile(
+        r.get(), file_size, 0U, ImmutableOptions(options), read_options, &bc));
    Block metaindex_block(std::move(bc));
    std::unique_ptr<InternalIterator> meta_iter;
    meta_iter.reset(metaindex_block.NewMetaIterator());
@@ -235,8 +235,9 @@ TEST_P(DBRateLimiterOnReadTest, VerifyChecksum) {
  ASSERT_EQ(0, options_.rate_limiter->GetTotalRequests(Env::IO_USER));

  ASSERT_OK(db_->VerifyChecksum(GetReadOptions()));
-  // The files are tiny so there should have just been one read per file.
-  int expected = kNumFiles;
+  // There are 3 reads per file: ReadMetaIndexBlock,
+  // VerifyChecksumInMetaBlocks, VerifyChecksumInBlocks
+  int expected = kNumFiles * 3;
  ASSERT_EQ(expected, options_.rate_limiter->GetTotalRequests(Env::IO_USER));
 }

@@ -810,9 +810,10 @@ TEST_F(DBSSTTest, RateLimitedWALDelete) {
  // We created 4 sst files in L0
  ASSERT_EQ("4", FilesPerLevel(0));

-  // Compaction will move the 4 files in L0 to trash and create 1 L1 file
+  // Compaction will move the 4 files in L0 to trash and create 1 L1 file.
+  // Use kForceOptimized to not rewrite the new L1 file.
  CompactRangeOptions cro;
-  cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  cro.bottommost_level_compaction = BottommostLevelCompaction::kForceOptimized;
  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
  ASSERT_OK(dbfull()->TEST_WaitForCompact(true));
  ASSERT_EQ("0,1", FilesPerLevel(0));
@@ -20,76 +20,115 @@ class DBStatisticsTest : public DBTestBase {
 };

 TEST_F(DBStatisticsTest, CompressionStatsTest) {
-  CompressionType type;
+  for (CompressionType type : GetSupportedCompressions()) {
+    if (type == kNoCompression) {
+      continue;
+    }
+    if (type == kBZip2Compression) {
+      // Weird behavior in this test
+      continue;
+    }
+    SCOPED_TRACE("Compression type: " + std::to_string(type));

-  if (Snappy_Supported()) {
-    type = kSnappyCompression;
-    fprintf(stderr, "using snappy\n");
-  } else if (Zlib_Supported()) {
-    type = kZlibCompression;
-    fprintf(stderr, "using zlib\n");
-  } else if (BZip2_Supported()) {
-    type = kBZip2Compression;
-    fprintf(stderr, "using bzip2\n");
-  } else if (LZ4_Supported()) {
-    type = kLZ4Compression;
-    fprintf(stderr, "using lz4\n");
-  } else if (XPRESS_Supported()) {
-    type = kXpressCompression;
-    fprintf(stderr, "using xpress\n");
-  } else if (ZSTD_Supported()) {
-    type = kZSTD;
-    fprintf(stderr, "using ZSTD\n");
-  } else {
-    fprintf(stderr, "skipping test, compression disabled\n");
-    return;
+    Options options = CurrentOptions();
+    options.compression = type;
+    options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
+    options.statistics->set_stats_level(StatsLevel::kExceptTimeForMutex);
+    BlockBasedTableOptions bbto;
+    bbto.enable_index_compression = false;
+    options.table_factory.reset(NewBlockBasedTableFactory(bbto));
+    DestroyAndReopen(options);
+
+    auto PopStat = [&](Tickers t) -> uint64_t {
+      return options.statistics->getAndResetTickerCount(t);
+    };
+
+    int kNumKeysWritten = 100;
+    double compress_to = 0.5;
+    // About three KVs per block
+    int len = static_cast<int>(BlockBasedTableOptions().block_size / 3);
+    int uncomp_est = kNumKeysWritten * (len + 20);
+
+    Random rnd(301);
+    std::string buf;
+
+    // Check that compressions occur and are counted when compression is turned
+    // on
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      ASSERT_OK(
+          Put(Key(i), test::CompressibleString(&rnd, compress_to, len, &buf)));
+    }
+    ASSERT_OK(Flush());
+    EXPECT_EQ(34, PopStat(NUMBER_BLOCK_COMPRESSED));
+    EXPECT_NEAR2(uncomp_est, PopStat(BYTES_COMPRESSED_FROM), uncomp_est / 10);
+    EXPECT_NEAR2(uncomp_est * compress_to, PopStat(BYTES_COMPRESSED_TO),
+                 uncomp_est / 10);
+
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_DECOMPRESSED));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_FROM));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_TO));
+
+    // And decompressions
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      auto r = Get(Key(i));
+    }
+    EXPECT_EQ(34, PopStat(NUMBER_BLOCK_DECOMPRESSED));
+    EXPECT_NEAR2(uncomp_est, PopStat(BYTES_DECOMPRESSED_TO), uncomp_est / 10);
+    EXPECT_NEAR2(uncomp_est * compress_to, PopStat(BYTES_DECOMPRESSED_FROM),
+                 uncomp_est / 10);
+
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSION_BYPASSED));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSION_REJECTED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSION_BYPASSED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSION_REJECTED));
+
+    // Check when compression is rejected.
+    DestroyAndReopen(options);
+
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      ASSERT_OK(Put(Key(i), rnd.RandomBinaryString(len)));
+    }
+    ASSERT_OK(Flush());
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      auto r = Get(Key(i));
+    }
+    EXPECT_EQ(34, PopStat(NUMBER_BLOCK_COMPRESSION_REJECTED));
+    EXPECT_NEAR2(uncomp_est, PopStat(BYTES_COMPRESSION_REJECTED),
+                 uncomp_est / 10);
+
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSION_BYPASSED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_DECOMPRESSED));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSED_FROM));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSED_TO));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSION_BYPASSED));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_FROM));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_TO));
+
+    // Check when compression is disabled.
+    options.compression = kNoCompression;
+    DestroyAndReopen(options);
+
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      ASSERT_OK(Put(Key(i), rnd.RandomBinaryString(len)));
+    }
+    ASSERT_OK(Flush());
+    for (int i = 0; i < kNumKeysWritten; ++i) {
+      auto r = Get(Key(i));
+    }
+    EXPECT_EQ(34, PopStat(NUMBER_BLOCK_COMPRESSION_BYPASSED));
+    EXPECT_NEAR2(uncomp_est, PopStat(BYTES_COMPRESSION_BYPASSED),
+                 uncomp_est / 10);
+
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_COMPRESSION_REJECTED));
+    EXPECT_EQ(0, PopStat(NUMBER_BLOCK_DECOMPRESSED));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSED_FROM));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSED_TO));
+    EXPECT_EQ(0, PopStat(BYTES_COMPRESSION_REJECTED));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_FROM));
+    EXPECT_EQ(0, PopStat(BYTES_DECOMPRESSED_TO));
  }
-
-  Options options = CurrentOptions();
-  options.compression = type;
-  options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
-  options.statistics->set_stats_level(StatsLevel::kExceptTimeForMutex);
-  DestroyAndReopen(options);
-
-  int kNumKeysWritten = 100000;
-
-  // Check that compressions occur and are counted when compression is turned on
-  Random rnd(301);
-  for (int i = 0; i < kNumKeysWritten; ++i) {
-    // compressible string
-    ASSERT_OK(Put(Key(i), rnd.RandomString(128) + std::string(128, 'a')));
-  }
-  ASSERT_OK(Flush());
-  ASSERT_GT(options.statistics->getTickerCount(NUMBER_BLOCK_COMPRESSED), 0);
-
-  for (int i = 0; i < kNumKeysWritten; ++i) {
-    auto r = Get(Key(i));
-  }
-  ASSERT_GT(options.statistics->getTickerCount(NUMBER_BLOCK_DECOMPRESSED), 0);
-
-  options.compression = kNoCompression;
-  DestroyAndReopen(options);
-  uint64_t currentCompressions =
-      options.statistics->getTickerCount(NUMBER_BLOCK_COMPRESSED);
-  uint64_t currentDecompressions =
-      options.statistics->getTickerCount(NUMBER_BLOCK_DECOMPRESSED);
-
-  // Check that compressions do not occur when turned off
-  for (int i = 0; i < kNumKeysWritten; ++i) {
-    // compressible string
-    ASSERT_OK(Put(Key(i), rnd.RandomString(128) + std::string(128, 'a')));
-  }
-  ASSERT_OK(Flush());
-  ASSERT_EQ(options.statistics->getTickerCount(NUMBER_BLOCK_COMPRESSED) -
-                currentCompressions,
-            0);
-
-  for (int i = 0; i < kNumKeysWritten; ++i) {
-    auto r = Get(Key(i));
-  }
-  ASSERT_EQ(options.statistics->getTickerCount(NUMBER_BLOCK_DECOMPRESSED) -
-                currentDecompressions,
-            0);
 }

 TEST_F(DBStatisticsTest, MutexWaitStatsDisabledByDefault) {
@@ -353,7 +353,7 @@ TEST_F(DBTest, MixedSlowdownOptionsInQueue) {
          for (int i = 0; i < 2; ++i) {
            threads.emplace_back(write_no_slowdown_func);
          }
-          // Sleep for 2s to allow the threads to insert themselves into the
+          // Sleep for 3s to allow the threads to insert themselves into the
          // write queue
          env_->SleepForMicroseconds(3000000ULL);
        }
@@ -424,7 +424,7 @@ TEST_F(DBTest, MixedSlowdownOptionsStop) {
          for (int i = 0; i < 2; ++i) {
            threads.emplace_back(write_no_slowdown_func);
          }
-          // Sleep for 2s to allow the threads to insert themselves into the
+          // Sleep for 3s to allow the threads to insert themselves into the
          // write queue
          env_->SleepForMicroseconds(3000000ULL);
        }
@@ -3918,6 +3918,94 @@ TEST_F(DBBasicTestWithTimestamp, IterSeekToLastWithIterateUpperbound) {
  ASSERT_FALSE(iter->Valid());
  ASSERT_OK(iter->status());
 }
+
+TEST_F(DBBasicTestWithTimestamp, TimestampFilterTableReadOnGet) {
+  Options options = CurrentOptions();
+  options.env = env_;
+  options.create_if_missing = true;
+  options.statistics = ROCKSDB_NAMESPACE::CreateDBStatistics();
+  const size_t kTimestampSize = Timestamp(0, 0).size();
+  TestComparator test_cmp(kTimestampSize);
+  options.comparator = &test_cmp;
+  BlockBasedTableOptions bbto;
+  bbto.block_size = 100;
+  options.table_factory.reset(NewBlockBasedTableFactory(bbto));
+  DestroyAndReopen(options);
+
+  // Put
+  // Create two SST files
+  // file1: key => [1, 3], timestamp => [10, 20]
+  // file2, key => [2, 4], timestamp => [30, 40]
+  {
+    WriteOptions write_opts;
+    std::string write_ts = Timestamp(10, 0);
+    ASSERT_OK(db_->Put(write_opts, Key1(1), write_ts, "value1"));
+    write_ts = Timestamp(20, 0);
+    ASSERT_OK(db_->Put(write_opts, Key1(3), write_ts, "value3"));
+    ASSERT_OK(Flush());
+
+    write_ts = Timestamp(30, 0);
+    ASSERT_OK(db_->Put(write_opts, Key1(2), write_ts, "value2"));
+    write_ts = Timestamp(40, 0);
+    ASSERT_OK(db_->Put(write_opts, Key1(4), write_ts, "value4"));
+    ASSERT_OK(Flush());
+  }
+
+  // Get with timestamp
+  {
+    auto prev_checked_events = options.statistics->getTickerCount(
+        Tickers::TIMESTAMP_FILTER_TABLE_CHECKED);
+    auto prev_filtered_events = options.statistics->getTickerCount(
+        Tickers::TIMESTAMP_FILTER_TABLE_FILTERED);
+
+    // key=3 (ts=20) does not exist at timestamp=1
+    std::string read_ts_str = Timestamp(1, 0);
+    Slice read_ts_slice = Slice(read_ts_str);
+    ReadOptions read_opts;
+    read_opts.timestamp = &read_ts_slice;
+    std::string value_from_get = "";
+    std::string timestamp_from_get = "";
+    auto status =
+        db_->Get(read_opts, Key1(3), &value_from_get, &timestamp_from_get);
+    ASSERT_TRUE(status.IsNotFound());
+    ASSERT_EQ(value_from_get, std::string(""));
+    ASSERT_EQ(timestamp_from_get, std::string(""));
+
+    // key=3 is in the key ranges for both files, so both files will be queried.
+    // The table read was skipped because the timestamp is out of the table
+    // range, i.e.., 1 < [10,20], [30,40].
+    // The tickers increase by 2 due to 2 files.
+    ASSERT_EQ(prev_checked_events + 2,
+              options.statistics->getTickerCount(
+                  Tickers::TIMESTAMP_FILTER_TABLE_CHECKED));
+    ASSERT_EQ(prev_filtered_events + 2,
+              options.statistics->getTickerCount(
+                  Tickers::TIMESTAMP_FILTER_TABLE_FILTERED));
+
+    // key=3 (ts=20) exists at timestamp = 25
+    read_ts_str = Timestamp(25, 0);
+    read_ts_slice = Slice(read_ts_str);
+    read_opts.timestamp = &read_ts_slice;
+    ASSERT_OK(
+        db_->Get(read_opts, Key1(3), &value_from_get, &timestamp_from_get));
+    ASSERT_EQ("value3", value_from_get);
+    ASSERT_EQ(Timestamp(20, 0), timestamp_from_get);
+
+    // file1 was not skipped, because the timestamp is in range, [10,20] < 25.
+    // file2 was skipped, because the timestamp is not in range, 25 < [30,40].
+    // So the checked ticker increase by 2 due to 2 files;
+    // filtered ticker increase by 1 because file2 was skipped
+    ASSERT_EQ(prev_checked_events + 4,
+              options.statistics->getTickerCount(
+                  Tickers::TIMESTAMP_FILTER_TABLE_CHECKED));
+    ASSERT_EQ(prev_filtered_events + 3,
+              options.statistics->getTickerCount(
+                  Tickers::TIMESTAMP_FILTER_TABLE_FILTERED));
+  }
+
+  Close();
+}
+
 }  // namespace ROCKSDB_NAMESPACE

 int main(int argc, char** argv) {
@@ -846,6 +846,73 @@ TEST_P(DBWriteBufferManagerTest, StopSwitchingMemTablesOnceFlushing) {
  delete shared_wbm_db;
 }

+TEST_F(DBWriteBufferManagerTest, RuntimeChangeableAllowStall) {
+  constexpr int kBigValue = 10000;
+
+  Options options = CurrentOptions();
+  options.write_buffer_manager.reset(
+      new WriteBufferManager(1, nullptr /* cache */, true /* allow_stall */));
+  DestroyAndReopen(options);
+
+  // Pause flush thread so that
+  // (a) the only way to exist write stall below is to change the `allow_stall`
+  // (b) the write stall is "stable" without being interfered by flushes so that
+  // we can check it without flakiness
+  std::unique_ptr<test::SleepingBackgroundTask> sleeping_task(
+      new test::SleepingBackgroundTask());
+  env_->SetBackgroundThreads(1, Env::HIGH);
+  env_->Schedule(&test::SleepingBackgroundTask::DoSleepTask,
+                 sleeping_task.get(), Env::Priority::HIGH);
+  sleeping_task->WaitUntilSleeping();
+
+  // Test 1: test setting `allow_stall` from true to false
+  //
+  // Assert existence of a write stall
+  WriteOptions wo_no_slowdown;
+  wo_no_slowdown.no_slowdown = true;
+  Status s = Put(Key(0), DummyString(kBigValue), wo_no_slowdown);
+  ASSERT_TRUE(s.IsIncomplete());
+  ASSERT_TRUE(s.ToString().find("Write stall") != std::string::npos);
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
+      {{"WBMStallInterface::BlockDB",
+        "DBWriteBufferManagerTest::RuntimeChangeableThreadSafeParameters::"
+        "ChangeParameter"}});
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
+
+  // Test `SetAllowStall()`
+  port::Thread thread1([&] { ASSERT_OK(Put(Key(0), DummyString(kBigValue))); });
+  port::Thread thread2([&] {
+    TEST_SYNC_POINT(
+        "DBWriteBufferManagerTest::RuntimeChangeableThreadSafeParameters::"
+        "ChangeParameter");
+    options.write_buffer_manager->SetAllowStall(false);
+  });
+
+  // Verify `allow_stall` is successfully set to false in thread2.
+  // Othwerwise, thread1's write will be stalled and this test will hang
+  // forever.
+  thread1.join();
+  thread2.join();
+
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
+
+  // Test 2: test setting `allow_stall` from false to true
+  //
+  // Assert no write stall
+  ASSERT_OK(Put(Key(0), DummyString(kBigValue), wo_no_slowdown));
+
+  // Test `SetAllowStall()`
+  options.write_buffer_manager->SetAllowStall(true);
+
+  // Verify `allow_stall` is successfully set to true.
+  // Otherwise the following write will not be stalled and therefore succeed.
+  s = Put(Key(0), DummyString(kBigValue), wo_no_slowdown);
+  ASSERT_TRUE(s.IsIncomplete());
+  ASSERT_TRUE(s.ToString().find("Write stall") != std::string::npos);
+  sleeping_task->WakeUp();
+}

 INSTANTIATE_TEST_CASE_P(DBWriteBufferManagerTest, DBWriteBufferManagerTest,
                        testing::Bool());
@@ -636,6 +636,11 @@ TEST_P(DBWriteTest, LockWALInEffect) {
  ASSERT_TRUE(dbfull()->WALBufferIsEmpty());
  ASSERT_OK(db_->UnlockWAL());

+  // The above `TEST_SwitchWAL()` triggered a flush. That flush needs to finish
+  // before we make the filesystem inactive, otherwise the flush might hit an
+  // unrecoverable error (e.g., failed MANIFEST update).
+  ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(nullptr));
+
  // Fail the WAL flush if applicable
  fault_fs->SetFilesystemActive(false);
  Status s = Put("key2", "value");
@@ -38,6 +38,8 @@ Status UpdateManifestForFilesState(
    const DBOptions& db_opts, const std::string& db_name,
    const std::vector<ColumnFamilyDescriptor>& column_families,
    const UpdateManifestForFilesStateOptions& opts) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  OfflineManifestWriter w(db_opts, db_name);
  Status s = w.Recover(column_families);

@@ -114,7 +116,7 @@ Status UpdateManifestForFilesState(
      std::unique_ptr<FSDirectory> db_dir;
      s = fs->NewDirectory(db_name, IOOptions(), &db_dir, nullptr);
      if (s.ok()) {
-        s = w.LogAndApply(cfd, &edit, db_dir.get());
+        s = w.LogAndApply(read_options, cfd, &edit, db_dir.get());
      }
      if (s.ok()) {
        ++cfs_updated;
@@ -102,7 +102,8 @@ class ExternalSSTFileBasicTest
      // all point operators, even though sst_file_writer.DeleteRange
      // must be called before other sst_file_writer methods. This is
      // because point writes take precedence over range deletions
-      // in the same ingested sst.
+      // in the same ingested sst. This precedence is part of
+      // `SstFileWriter::DeleteRange()`'s API contract.
      std::string start_key = Key(range_deletions[i].first);
      std::string end_key = Key(range_deletions[i].second);
      s = sst_file_writer.DeleteRange(start_key, end_key);
@@ -1418,6 +1419,7 @@ TEST_P(ExternalSSTFileBasicTest, IngestionWithRangeDeletions) {
  ASSERT_EQ(4, NumTableFilesAtLevel(0));
  ASSERT_EQ(1, NumTableFilesAtLevel(kNumLevels - 2));
  ASSERT_EQ(2, NumTableFilesAtLevel(options.num_levels - 1));
+  VerifyDBFromMap(true_data);
 }

 TEST_F(ExternalSSTFileBasicTest, AdjacentRangeDeletionTombstones) {
@@ -1462,6 +1464,89 @@ TEST_F(ExternalSSTFileBasicTest, AdjacentRangeDeletionTombstones) {
  DestroyAndRecreateExternalSSTFilesDir();
 }

+TEST_F(ExternalSSTFileBasicTest, UnorderedRangeDeletions) {
+  int kNumLevels = 7;
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.num_levels = kNumLevels;
+  Reopen(options);
+
+  std::map<std::string, std::string> true_data;
+  int file_id = 1;
+
+  // prevent range deletions from being dropped due to becoming obsolete.
+  const Snapshot* snapshot = db_->GetSnapshot();
+
+  // Range del [0, 50) in memtable
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(0),
+                             Key(50)));
+
+  // Out of order range del overlaps memtable, so flush is required before file
+  // is ingested into L0
+  ASSERT_OK(GenerateAndAddExternalFile(
+      options, {60, 90}, {ValueType::kTypeValue, ValueType::kTypeValue},
+      {{65, 70}, {45, 50}}, file_id++, true /* write_global_seqno */,
+      true /* verify_checksums_before_ingest */, &true_data));
+  ASSERT_EQ(2, true_data.size());
+  ASSERT_EQ(2, NumTableFilesAtLevel(0));
+  ASSERT_EQ(0, NumTableFilesAtLevel(kNumLevels - 1));
+  VerifyDBFromMap(true_data);
+
+  // Compact to L6
+  MoveFilesToLevel(kNumLevels - 1);
+  ASSERT_EQ(0, NumTableFilesAtLevel(0));
+  ASSERT_EQ(1, NumTableFilesAtLevel(kNumLevels - 1));
+  VerifyDBFromMap(true_data);
+
+  // Ingest a file containing out of order range dels that cover nothing
+  ASSERT_OK(GenerateAndAddExternalFile(
+      options, {151, 175}, {ValueType::kTypeValue, ValueType::kTypeValue},
+      {{160, 200}, {120, 180}}, file_id++, true /* write_global_seqno */,
+      true /* verify_checksums_before_ingest */, &true_data));
+  ASSERT_EQ(4, true_data.size());
+  ASSERT_EQ(0, NumTableFilesAtLevel(0));
+  ASSERT_EQ(2, NumTableFilesAtLevel(kNumLevels - 1));
+  VerifyDBFromMap(true_data);
+
+  // Ingest a file containing out of order range dels that cover keys in L6
+  ASSERT_OK(GenerateAndAddExternalFile(
+      options, {}, {}, {{190, 200}, {170, 180}, {55, 65}}, file_id++,
+      true /* write_global_seqno */, true /* verify_checksums_before_ingest */,
+      &true_data));
+  ASSERT_EQ(2, true_data.size());
+  ASSERT_EQ(1, NumTableFilesAtLevel(kNumLevels - 2));
+  ASSERT_EQ(2, NumTableFilesAtLevel(kNumLevels - 1));
+  VerifyDBFromMap(true_data);
+
+  db_->ReleaseSnapshot(snapshot);
+}
+
+TEST_F(ExternalSSTFileBasicTest, RangeDeletionEndComesBeforeStart) {
+  Options options = CurrentOptions();
+  SstFileWriter sst_file_writer(EnvOptions(), options);
+
+  // "file.sst"
+  // Verify attempt to delete 300 => 200 fails.
+  // Then, verify attempt to delete 300 => 300 succeeds but writes nothing.
+  // Afterwards, verify attempt to delete 300 => 400 works normally.
+  std::string file = sst_files_dir_ + "file.sst";
+  ASSERT_OK(sst_file_writer.Open(file));
+  ASSERT_TRUE(
+      sst_file_writer.DeleteRange(Key(300), Key(200)).IsInvalidArgument());
+  ASSERT_OK(sst_file_writer.DeleteRange(Key(300), Key(300)));
+  ASSERT_OK(sst_file_writer.DeleteRange(Key(300), Key(400)));
+  ExternalSstFileInfo file_info;
+  Status s = sst_file_writer.Finish(&file_info);
+  ASSERT_OK(s) << s.ToString();
+  ASSERT_EQ(file_info.file_path, file);
+  ASSERT_EQ(file_info.num_entries, 0);
+  ASSERT_EQ(file_info.smallest_key, "");
+  ASSERT_EQ(file_info.largest_key, "");
+  ASSERT_EQ(file_info.num_range_del_entries, 1);
+  ASSERT_EQ(file_info.smallest_range_del_key, Key(300));
+  ASSERT_EQ(file_info.largest_range_del_key, Key(400));
+}
+
 TEST_P(ExternalSSTFileBasicTest, IngestFileWithBadBlockChecksum) {
  bool change_checksum_called = false;
  const auto& change_checksum = [&](void* arg) {
@@ -692,6 +692,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
    // If customized readahead size is needed, we can pass a user option
    // all the way to here. Right now we just rely on the default readahead
    // to keep things simple.
+    // TODO: plumb Env::IOActivity
    ReadOptions ro;
    ro.readahead_size = ingestion_options_.verify_checksums_readahead_size;
    status = table_reader->VerifyChecksum(
@@ -745,6 +746,7 @@ Status ExternalSstFileIngestionJob::GetIngestedFileInfo(
  file_to_ingest->num_range_deletions = props->num_range_deletions;

  ParsedInternalKey key;
+  // TODO: plumb Env::IOActivity
  ReadOptions ro;
  std::unique_ptr<InternalIterator> iter(table_reader->NewIterator(
      ro, sv->mutable_cf_options.prefix_extractor.get(), /*arena=*/nullptr,
@@ -855,6 +857,7 @@ Status ExternalSstFileIngestionJob::AssignLevelAndSeqnoForIngestedFile(

  bool overlap_with_db = false;
  Arena arena;
+  // TODO: plumb Env::IOActivity
  ReadOptions ro;
  ro.total_order_seek = true;
  int target_level = 0;
@@ -1088,4 +1091,3 @@ Status ExternalSstFileIngestionJob::SyncIngestedFile(TWritableFile* file) {
 }

 }  // namespace ROCKSDB_NAMESPACE
-
@@ -141,11 +141,12 @@ FlushJob::FlushJob(
 FlushJob::~FlushJob() { ThreadStatusUtil::ResetThreadStatus(); }

 void FlushJob::ReportStartedFlush() {
-  ThreadStatusUtil::SetColumnFamily(cfd_, cfd_->ioptions()->env,
-                                    db_options_.enable_thread_tracking);
+  ThreadStatusUtil::SetEnableTracking(db_options_.enable_thread_tracking);
+  ThreadStatusUtil::SetColumnFamily(cfd_);
  ThreadStatusUtil::SetThreadOperation(ThreadStatus::OP_FLUSH);
  ThreadStatusUtil::SetThreadOperationProperty(ThreadStatus::COMPACTION_JOB_ID,
                                               job_context_->job_id);
+
  IOSTATS_RESET(bytes_written);
 }

@@ -379,6 +380,7 @@ Status FlushJob::MemPurge() {
  // Create two iterators, one for the memtable data (contains
  // info from puts + deletes), and one for the memtable
  // Range Tombstones (from DeleteRanges).
+  // TODO: plumb Env::IOActivity
  ReadOptions ro;
  ro.total_order_seek = true;
  Arena arena;
@@ -669,6 +671,7 @@ bool FlushJob::MemPurgeDecider(double threshold) {
  // Cochran formula for determining sample size.
  // 95% confidence interval, 7% precision.
  //    n0 = (1.96*1.96)*0.25/(0.07*0.07) = 196.0
+  // TODO: plumb Env::IOActivity
  double n0 = 196.0;
  ReadOptions ro;
  ro.total_order_seek = true;
@@ -841,6 +844,7 @@ Status FlushJob::WriteLevel0Table() {
        range_del_iters;
    ReadOptions ro;
    ro.total_order_seek = true;
+    ro.io_activity = Env::IOActivity::kFlush;
    Arena arena;
    uint64_t total_num_entries = 0, total_num_deletes = 0;
    uint64_t total_data_size = 0;
@@ -930,17 +934,19 @@ Status FlushJob::WriteLevel0Table() {
          meta_.fd.GetNumber());
      const SequenceNumber job_snapshot_seq =
          job_context_->GetJobSnapshotSequence();
-      s = BuildTable(
-          dbname_, versions_, db_options_, tboptions, file_options_,
-          cfd_->table_cache(), iter.get(), std::move(range_del_iters), &meta_,
-          &blob_file_additions, existing_snapshots_,
-          earliest_write_conflict_snapshot_, job_snapshot_seq,
-          snapshot_checker_, mutable_cf_options_.paranoid_file_checks,
-          cfd_->internal_stats(), &io_s, io_tracer_,
-          BlobFileCreationReason::kFlush, seqno_to_time_mapping_, event_logger_,
-          job_context_->job_id, io_priority, &table_properties_, write_hint,
-          full_history_ts_low, blob_callback_, base_, &num_input_entries,
-          &memtable_payload_bytes, &memtable_garbage_bytes);
+      const ReadOptions read_options(Env::IOActivity::kFlush);
+      s = BuildTable(dbname_, versions_, db_options_, tboptions, file_options_,
+                     read_options, cfd_->table_cache(), iter.get(),
+                     std::move(range_del_iters), &meta_, &blob_file_additions,
+                     existing_snapshots_, earliest_write_conflict_snapshot_,
+                     job_snapshot_seq, snapshot_checker_,
+                     mutable_cf_options_.paranoid_file_checks,
+                     cfd_->internal_stats(), &io_s, io_tracer_,
+                     BlobFileCreationReason::kFlush, seqno_to_time_mapping_,
+                     event_logger_, job_context_->job_id, io_priority,
+                     &table_properties_, write_hint, full_history_ts_low,
+                     blob_callback_, base_, &num_input_entries,
+                     &memtable_payload_bytes, &memtable_garbage_bytes);
      // TODO: Cleanup io_status in BuildTable and table builders
      assert(!s.ok() || io_s.ok());
      io_s.PermitUncheckedError();
@@ -44,16 +44,19 @@ Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,

  auto num_files = files_to_import_.size();
  if (num_files == 0) {
-    return Status::InvalidArgument("The list of files is empty");
+    status = Status::InvalidArgument("The list of files is empty");
+    return status;
  }

  for (const auto& f : files_to_import_) {
    if (f.num_entries == 0) {
-      return Status::InvalidArgument("File contain no entries");
+      status = Status::InvalidArgument("File contain no entries");
+      return status;
    }

    if (!f.smallest_internal_key.Valid() || !f.largest_internal_key.Valid()) {
-      return Status::Corruption("File has corrupted keys");
+      status = Status::Corruption("File has corrupted keys");
+      return status;
    }
  }

@@ -272,6 +275,7 @@ Status ImportColumnFamilyJob::GetIngestedFileInfo(
  // in file_meta.
  if (file_meta.smallest.empty()) {
    assert(file_meta.largest.empty());
+    // TODO: plumb Env::IOActivity
    ReadOptions ro;
    std::unique_ptr<InternalIterator> iter(table_reader->NewIterator(
        ro, sv->mutable_cf_options.prefix_extractor.get(), /*arena=*/nullptr,
@@ -299,15 +303,24 @@ Status ImportColumnFamilyJob::GetIngestedFileInfo(
          return Status::Corruption("Corrupted key in external file. ",
                                    pik_status.getState());
        }
-        RangeTombstone tombstone(key, range_del_iter->value());
-        InternalKey start_key = tombstone.SerializeKey();
+        RangeTombstone first_tombstone(key, range_del_iter->value());
+        InternalKey start_key = first_tombstone.SerializeKey();
        const InternalKeyComparator* icmp = &cfd_->internal_comparator();
        if (!bound_set ||
            icmp->Compare(start_key, file_to_import->smallest_internal_key) <
                0) {
          file_to_import->smallest_internal_key = start_key;
        }
-        InternalKey end_key = tombstone.SerializeEndKey();
+
+        range_del_iter->SeekToLast();
+        pik_status = ParseInternalKey(range_del_iter->key(), &key,
+                                      db_options_.allow_data_in_errors);
+        if (!pik_status.ok()) {
+          return Status::Corruption("Corrupted key in external file. ",
+                                    pik_status.getState());
+        }
+        RangeTombstone last_tombstone(key, range_del_iter->value());
+        InternalKey end_key = last_tombstone.SerializeEndKey();
        if (!bound_set ||
            icmp->Compare(end_key, file_to_import->largest_internal_key) > 0) {
          file_to_import->largest_internal_key = end_key;
@@ -338,4 +351,3 @@ Status ImportColumnFamilyJob::GetIngestedFileInfo(
  return status;
 }
 }  // namespace ROCKSDB_NAMESPACE
-
@@ -294,6 +294,8 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithRangeTombstone) {
  ASSERT_OK(sfw_cf1.Put("K1", "V1"));
  ASSERT_OK(sfw_cf1.Put("K2", "V2"));
  ASSERT_OK(sfw_cf1.DeleteRange("K3", "K4"));
+  ASSERT_OK(sfw_cf1.DeleteRange("K7", "K9"));
+
  ASSERT_OK(sfw_cf1.Finish());

  // Import sst file corresponding to cf1 onto a new cf and verify
@@ -319,7 +321,7 @@ TEST_F(ImportColumnFamilyTest, ImportSSTFileWriterFilesWithRangeTombstone) {
  ASSERT_TRUE(file_meta != nullptr);
  InternalKey largest;
  largest.DecodeFrom(file_meta->largest);
-  ASSERT_EQ(largest.user_key(), "K4");
+  ASSERT_EQ(largest.user_key(), "K9");

  std::string value;
  ASSERT_OK(db_->Get(ReadOptions(), import_cfh_, "K1", &value));
@@ -628,22 +630,22 @@ TEST_F(ImportColumnFamilyTest, ImportColumnFamilyNegativeTest) {
  {
    // Create column family with existing cf name.
    ExportImportFilesMetaData metadata;
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "koko",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("Column family already exists"));
+    metadata.db_comparator_name = options.comparator->Name();
+    Status s = db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "koko",
+                                                 ImportColumnFamilyOptions(),
+                                                 metadata, &import_cfh_);
+    ASSERT_TRUE(std::strstr(s.getState(), "Column family already exists"));
    ASSERT_EQ(import_cfh_, nullptr);
  }

  {
    // Import with no files specified.
    ExportImportFilesMetaData metadata;
-
-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("The list of files is empty"));
+    metadata.db_comparator_name = options.comparator->Name();
+    Status s = db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
+                                                 ImportColumnFamilyOptions(),
+                                                 metadata, &import_cfh_);
+    ASSERT_TRUE(std::strstr(s.getState(), "The list of files is empty"));
    ASSERT_EQ(import_cfh_, nullptr);
  }

@@ -693,10 +695,10 @@ TEST_F(ImportColumnFamilyTest, ImportColumnFamilyNegativeTest) {
        LiveFileMetaDataInit(file1_sst_name, sst_files_dir_, 1, 10, 19));
    metadata.db_comparator_name = mismatch_options.comparator->Name();

-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "coco",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::InvalidArgument("Comparator name mismatch"));
+    Status s = db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "coco",
+                                                 ImportColumnFamilyOptions(),
+                                                 metadata, &import_cfh_);
+    ASSERT_TRUE(std::strstr(s.getState(), "Comparator name mismatch"));
    ASSERT_EQ(import_cfh_, nullptr);
  }

@@ -718,10 +720,10 @@ TEST_F(ImportColumnFamilyTest, ImportColumnFamilyNegativeTest) {
        LiveFileMetaDataInit(file3_sst_name, sst_files_dir_, 1, 10, 19));
    metadata.db_comparator_name = options.comparator->Name();

-    ASSERT_EQ(db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
-                                                ImportColumnFamilyOptions(),
-                                                metadata, &import_cfh_),
-              Status::IOError("No such file or directory"));
+    Status s = db_->CreateColumnFamilyWithImport(ColumnFamilyOptions(), "yoyo",
+                                                 ImportColumnFamilyOptions(),
+                                                 metadata, &import_cfh_);
+    ASSERT_TRUE(std::strstr(s.getState(), "No such file or directory"));
    ASSERT_EQ(import_cfh_, nullptr);

    // Test successful import after a failure with the same CF name. Ensures
@@ -1148,7 +1148,9 @@ bool InternalStats::HandleSsTables(std::string* value, Slice /*suffix*/) {
 bool InternalStats::HandleAggregatedTableProperties(std::string* value,
                                                    Slice /*suffix*/) {
  std::shared_ptr<const TableProperties> tp;
-  auto s = cfd_->current()->GetAggregatedTableProperties(&tp);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  auto s = cfd_->current()->GetAggregatedTableProperties(read_options, &tp);
  if (!s.ok()) {
    return false;
  }
@@ -1168,7 +1170,9 @@ static std::map<std::string, std::string> MapUint64ValuesToString(
 bool InternalStats::HandleAggregatedTablePropertiesMap(
    std::map<std::string, std::string>* values, Slice /*suffix*/) {
  std::shared_ptr<const TableProperties> tp;
-  auto s = cfd_->current()->GetAggregatedTableProperties(&tp);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  auto s = cfd_->current()->GetAggregatedTableProperties(read_options, &tp);
  if (!s.ok()) {
    return false;
  }
@@ -1184,8 +1188,10 @@ bool InternalStats::HandleAggregatedTablePropertiesAtLevel(std::string* values,
    return false;
  }
  std::shared_ptr<const TableProperties> tp;
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  auto s = cfd_->current()->GetAggregatedTableProperties(
-      &tp, static_cast<int>(level));
+      read_options, &tp, static_cast<int>(level));
  if (!s.ok()) {
    return false;
  }
@@ -1201,8 +1207,10 @@ bool InternalStats::HandleAggregatedTablePropertiesAtLevelMap(
    return false;
  }
  std::shared_ptr<const TableProperties> tp;
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  auto s = cfd_->current()->GetAggregatedTableProperties(
-      &tp, static_cast<int>(level));
+      read_options, &tp, static_cast<int>(level));
  if (!s.ok()) {
    return false;
  }
@@ -1397,7 +1405,11 @@ bool InternalStats::HandleEstimatePendingCompactionBytes(uint64_t* value,
 bool InternalStats::HandleEstimateTableReadersMem(uint64_t* value,
                                                  DBImpl* /*db*/,
                                                  Version* version) {
-  *value = (version == nullptr) ? 0 : version->GetMemoryUsageByTableReaders();
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+  *value = (version == nullptr)
+               ? 0
+               : version->GetMemoryUsageByTableReaders(read_options);
  return true;
 }

@@ -1448,9 +1460,10 @@ bool InternalStats::HandleEstimateOldestKeyTime(uint64_t* value, DBImpl* /*db*/,
          ->compaction_options_fifo.allow_compaction) {
    return false;
  }
-
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  TablePropertiesCollection collection;
-  auto s = cfd_->current()->GetPropertiesOfAllTables(&collection);
+  auto s = cfd_->current()->GetPropertiesOfAllTables(read_options, &collection);
  if (!s.ok()) {
    return false;
  }
@@ -599,6 +599,7 @@ void MemTable::ConstructFragmentedRangeTombstones() {
  assert(!IsFragmentedRangeTombstonesConstructed(false));
  // There should be no concurrent Construction
  if (!is_range_del_table_empty_.load(std::memory_order_relaxed)) {
+    // TODO: plumb Env::IOActivity
    auto* unfragmented_iter =
        new MemTableIterator(*this, ReadOptions(), nullptr /* arena */,
                             true /* use_range_del_table */);
@@ -467,6 +467,8 @@ Status MemTableList::TryInstallMemtableFlushResults(
      ThreadStatus::STAGE_MEMTABLE_INSTALL_FLUSH_RESULTS);
  mu->AssertHeld();

+  const ReadOptions read_options(Env::IOActivity::kFlush);
+
  // Flush was successful
  // Record the status on the memtable object. Either this call or a call by a
  // concurrent flush thread will read the status and write it to manifest.
@@ -578,8 +580,8 @@ Status MemTableList::TryInstallMemtableFlushResults(
      };
      if (write_edits) {
        // this can release and reacquire the mutex.
-        s = vset->LogAndApply(cfd, mutable_cf_options, edit_list, mu,
-                              db_directory, /*new_descriptor_log=*/false,
+        s = vset->LogAndApply(cfd, mutable_cf_options, read_options, edit_list,
+                              mu, db_directory, /*new_descriptor_log=*/false,
                              /*column_family_options=*/nullptr,
                              manifest_write_cb);
      } else {
@@ -798,6 +800,8 @@ Status InstallMemtableAtomicFlushResults(
      ThreadStatus::STAGE_MEMTABLE_INSTALL_FLUSH_RESULTS);
  mu->AssertHeld();

+  const ReadOptions read_options(Env::IOActivity::kFlush);
+
  size_t num = mems_list.size();
  assert(cfds.size() == num);
  if (imm_lists != nullptr) {
@@ -875,8 +879,8 @@ Status InstallMemtableAtomicFlushResults(
  }

  // this can release and reacquire the mutex.
-  s = vset->LogAndApply(cfds, mutable_cf_options_list, edit_lists, mu,
-                        db_directory);
+  s = vset->LogAndApply(cfds, mutable_cf_options_list, read_options, edit_lists,
+                        mu, db_directory);

  for (size_t k = 0; k != cfds.size(); ++k) {
    auto* imm = (imm_lists == nullptr) ? cfds[k]->imm() : imm_lists->at(k);
@@ -187,7 +187,8 @@ TEST_F(PerfContextTest, StopWatchOverhead) {
  uint64_t elapsed = 0;
  std::vector<uint64_t> timings(kTotalIterations);

-  StopWatch timer(SystemClock::Default().get(), nullptr, 0, &elapsed);
+  StopWatch timer(SystemClock::Default().get(), nullptr, 0,
+                  Histograms::HISTOGRAM_ENUM_MAX, &elapsed);
  for (auto& timing : timings) {
    timing = elapsed;
  }
@@ -329,21 +329,23 @@ class TestPlainTableFactory : public PlainTableFactory {
      std::unique_ptr<TableReader>* table,
      bool /*prefetch_index_and_filter_in_cache*/) const override {
    std::unique_ptr<TableProperties> props;
+    const ReadOptions read_options;
    auto s = ReadTableProperties(file.get(), file_size, kPlainTableMagicNumber,
-                                 table_reader_options.ioptions, &props);
+                                 table_reader_options.ioptions, read_options,
+                                 &props);
    EXPECT_TRUE(s.ok());

    if (store_index_in_file_) {
      BlockHandle bloom_block_handle;
      s = FindMetaBlockInFile(file.get(), file_size, kPlainTableMagicNumber,
-                              table_reader_options.ioptions,
+                              table_reader_options.ioptions, read_options,
                              BloomBlockBuilder::kBloomBlock,
                              &bloom_block_handle);
      EXPECT_TRUE(s.ok());

      BlockHandle index_block_handle;
      s = FindMetaBlockInFile(file.get(), file_size, kPlainTableMagicNumber,
-                              table_reader_options.ioptions,
+                              table_reader_options.ioptions, read_options,
                              PlainTableIndexBuilder::kPlainTableIndexBlock,
                              &index_block_handle);
      EXPECT_TRUE(s.ok());
@@ -1344,4 +1346,3 @@ int main(int argc, char** argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
 }
-
@@ -145,6 +145,8 @@ class Repairer {
  // Adds a column family to the VersionSet with cf_options_ and updates
  // manifest.
  Status AddColumnFamily(const std::string& cf_name, uint32_t cf_id) {
+    // TODO: plumb Env::IOActivity;
+    const ReadOptions read_options;
    const auto* cf_opts = GetColumnFamilyOptions(cf_name);
    if (cf_opts == nullptr) {
      return Status::Corruption("Encountered unknown column family with name=" +
@@ -166,8 +168,9 @@ class Repairer {
    Status status = env_->GetFileSystem()->NewDirectory(dbname_, IOOptions(),
                                                        &db_dir, nullptr);
    if (status.ok()) {
-      status = vset_.LogAndApply(cfd, mut_cf_opts, &edit, &mutex_, db_dir.get(),
-                                 false /* new_descriptor_log */, cf_opts);
+      status = vset_.LogAndApply(cfd, mut_cf_opts, read_options, &edit, &mutex_,
+                                 db_dir.get(), false /* new_descriptor_log */,
+                                 cf_opts);
    }
    mutex_.Unlock();
    return status;
@@ -357,6 +360,9 @@ class Repairer {
      }
    };

+    // TODO: plumb Env::IOActivity
+    const ReadOptions read_options;
+
    // Open the log file
    std::string logname = LogFileName(wal_dir, log);
    const auto& fs = env_->GetFileSystem();
@@ -422,6 +428,7 @@ class Repairer {

      FileMetaData meta;
      meta.fd = FileDescriptor(next_file_number_++, 0, 0);
+      // TODO: plumb Env::IOActivity
      ReadOptions ro;
      ro.total_order_seek = true;
      Arena arena;
@@ -456,7 +463,7 @@ class Repairer {
      SeqnoToTimeMapping empty_seqno_time_mapping;
      status = BuildTable(
          dbname_, /* versions */ nullptr, immutable_db_options_, tboptions,
-          file_options_, table_cache_.get(), iter.get(),
+          file_options_, read_options, table_cache_.get(), iter.get(),
          std::move(range_del_iters), &meta, nullptr /* blob_file_additions */,
          {}, kMaxSequenceNumber, kMaxSequenceNumber, snapshot_checker,
          false /* paranoid_file_checks*/, nullptr /* internal_stats */, &io_s,
@@ -509,8 +516,10 @@ class Repairer {
                                file_size);
    std::shared_ptr<const TableProperties> props;
    if (status.ok()) {
-      status = table_cache_->GetTableProperties(file_options_, icmp_, t->meta,
-                                                &props);
+      // TODO: plumb Env::IOActivity
+      const ReadOptions read_options;
+      status = table_cache_->GetTableProperties(file_options_, read_options,
+                                                icmp_, t->meta, &props);
    }
    if (status.ok()) {
      auto s =
@@ -556,6 +565,7 @@ class Repairer {
      }
    }
    if (status.ok()) {
+      // TODO: plumb Env::IOActivity
      ReadOptions ropts;
      ropts.total_order_seek = true;
      InternalIterator* iter = table_cache_->NewIterator(
@@ -603,6 +613,7 @@ class Repairer {
      // an SST file is a full sorted run. This probably needs the extra logic
      // from compaction_job.cc around call to UpdateBoundariesForRange (to
      // handle range tombstones extendingg beyond range of other entries).
+      // TODO: plumb Env::IOActivity
      ReadOptions ropts;
      std::unique_ptr<FragmentedRangeTombstoneIterator> r_iter;
      status = table_cache_->GetRangeTombstoneIterator(
@@ -625,6 +636,8 @@ class Repairer {
  }

  Status AddTables() {
+    // TODO: plumb Env::IOActivity;
+    const ReadOptions read_options;
    std::unordered_map<uint32_t, std::vector<const TableInfo*>> cf_id_to_tables;
    SequenceNumber max_sequence = 0;
    for (size_t i = 0; i < tables_.size(); i++) {
@@ -706,8 +719,8 @@ class Repairer {
        s = env_->GetFileSystem()->NewDirectory(dbname_, IOOptions(), &db_dir,
                                                nullptr);
        if (s.ok()) {
-          s = vset_.LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(), &edit,
-                                &mutex_, db_dir.get(),
+          s = vset_.LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
+                                read_options, &edit, &mutex_, db_dir.get(),
                                false /* new_descriptor_log */);
        }
        mutex_.Unlock();
@@ -809,4 +822,3 @@ Status RepairDB(const std::string& dbname, const Options& options) {
 }

 }  // namespace ROCKSDB_NAMESPACE
-
@@ -560,7 +560,7 @@ Status TableCache::MultiGetFilter(
 }

 Status TableCache::GetTableProperties(
-    const FileOptions& file_options,
+    const FileOptions& file_options, const ReadOptions& read_options,
    const InternalKeyComparator& internal_comparator,
    const FileMetaData& file_meta,
    std::shared_ptr<const TableProperties>* properties,
@@ -574,7 +574,7 @@ Status TableCache::GetTableProperties(
  }

  TypedHandle* table_handle = nullptr;
-  Status s = FindTable(ReadOptions(), file_options, internal_comparator,
+  Status s = FindTable(read_options, file_options, internal_comparator,
                       file_meta, &table_handle, prefix_extractor, no_io);
  if (!s.ok()) {
    return s;
@@ -608,7 +608,7 @@ Status TableCache::ApproximateKeyAnchors(
 }

 size_t TableCache::GetMemoryUsageByTableReader(
-    const FileOptions& file_options,
+    const FileOptions& file_options, const ReadOptions& read_options,
    const InternalKeyComparator& internal_comparator,
    const FileMetaData& file_meta,
    const std::shared_ptr<const SliceTransform>& prefix_extractor) {
@@ -619,7 +619,7 @@ size_t TableCache::GetMemoryUsageByTableReader(
  }

  TypedHandle* table_handle = nullptr;
-  Status s = FindTable(ReadOptions(), file_options, internal_comparator,
+  Status s = FindTable(read_options, file_options, internal_comparator,
                       file_meta, &table_handle, prefix_extractor, true);
  if (!s.ok()) {
    return 0;
@@ -636,7 +636,8 @@ void TableCache::Evict(Cache* cache, uint64_t file_number) {
 }

 uint64_t TableCache::ApproximateOffsetOf(
-    const Slice& key, const FileMetaData& file_meta, TableReaderCaller caller,
+    const ReadOptions& read_options, const Slice& key,
+    const FileMetaData& file_meta, TableReaderCaller caller,
    const InternalKeyComparator& internal_comparator,
    const std::shared_ptr<const SliceTransform>& prefix_extractor) {
  uint64_t result = 0;
@@ -645,7 +646,7 @@ uint64_t TableCache::ApproximateOffsetOf(
  if (table_reader == nullptr) {
    const bool for_compaction = (caller == TableReaderCaller::kCompaction);
    Status s =
-        FindTable(ReadOptions(), file_options_, internal_comparator, file_meta,
+        FindTable(read_options, file_options_, internal_comparator, file_meta,
                  &table_handle, prefix_extractor, false /* no_io */,
                  !for_compaction /* record_read_stats */);
    if (s.ok()) {
@@ -654,7 +655,7 @@ uint64_t TableCache::ApproximateOffsetOf(
  }

  if (table_reader != nullptr) {
-    result = table_reader->ApproximateOffsetOf(key, caller);
+    result = table_reader->ApproximateOffsetOf(read_options, key, caller);
  }
  if (table_handle != nullptr) {
    cache_.Release(table_handle);
@@ -664,8 +665,9 @@ uint64_t TableCache::ApproximateOffsetOf(
 }

 uint64_t TableCache::ApproximateSize(
-    const Slice& start, const Slice& end, const FileMetaData& file_meta,
-    TableReaderCaller caller, const InternalKeyComparator& internal_comparator,
+    const ReadOptions& read_options, const Slice& start, const Slice& end,
+    const FileMetaData& file_meta, TableReaderCaller caller,
+    const InternalKeyComparator& internal_comparator,
    const std::shared_ptr<const SliceTransform>& prefix_extractor) {
  uint64_t result = 0;
  TableReader* table_reader = file_meta.fd.table_reader;
@@ -673,7 +675,7 @@ uint64_t TableCache::ApproximateSize(
  if (table_reader == nullptr) {
    const bool for_compaction = (caller == TableReaderCaller::kCompaction);
    Status s =
-        FindTable(ReadOptions(), file_options_, internal_comparator, file_meta,
+        FindTable(read_options, file_options_, internal_comparator, file_meta,
                  &table_handle, prefix_extractor, false /* no_io */,
                  !for_compaction /* record_read_stats */);
    if (s.ok()) {
@@ -682,7 +684,7 @@ uint64_t TableCache::ApproximateSize(
  }

  if (table_reader != nullptr) {
-    result = table_reader->ApproximateSize(start, end, caller);
+    result = table_reader->ApproximateSize(read_options, start, end, caller);
  }
  if (table_handle != nullptr) {
    cache_.Release(table_handle);
@@ -179,7 +179,7 @@ class TableCache {
  //            return Status::Incomplete() if table is not present in cache and
  //            we set `no_io` to be true.
  Status GetTableProperties(
-      const FileOptions& toptions,
+      const FileOptions& toptions, const ReadOptions& read_options,
      const InternalKeyComparator& internal_comparator,
      const FileMetaData& file_meta,
      std::shared_ptr<const TableProperties>* properties,
@@ -194,22 +194,23 @@ class TableCache {
  // Return total memory usage of the table reader of the file.
  // 0 if table reader of the file is not loaded.
  size_t GetMemoryUsageByTableReader(
-      const FileOptions& toptions,
+      const FileOptions& toptions, const ReadOptions& read_options,
      const InternalKeyComparator& internal_comparator,
      const FileMetaData& file_meta,
      const std::shared_ptr<const SliceTransform>& prefix_extractor = nullptr);

  // Returns approximated offset of a key in a file represented by fd.
  uint64_t ApproximateOffsetOf(
-      const Slice& key, const FileMetaData& file_meta, TableReaderCaller caller,
+      const ReadOptions& read_options, const Slice& key,
+      const FileMetaData& file_meta, TableReaderCaller caller,
      const InternalKeyComparator& internal_comparator,
      const std::shared_ptr<const SliceTransform>& prefix_extractor = nullptr);

  // Returns approximated data size between start and end keys in a file
  // represented by fd (the start key must not be greater than the end key).
  uint64_t ApproximateSize(
-      const Slice& start, const Slice& end, const FileMetaData& file_meta,
-      TableReaderCaller caller,
+      const ReadOptions& read_options, const Slice& start, const Slice& end,
+      const FileMetaData& file_meta, TableReaderCaller caller,
      const InternalKeyComparator& internal_comparator,
      const std::shared_ptr<const SliceTransform>& prefix_extractor = nullptr);

@@ -292,8 +292,9 @@ void TestCustomizedTablePropertiesCollector(
      new RandomAccessFileReader(std::move(source), "test"));

  std::unique_ptr<TableProperties> props;
+  const ReadOptions read_options;
  Status s = ReadTableProperties(fake_file_reader.get(), fwf->contents().size(),
-                                 magic_number, ioptions, &props);
+                                 magic_number, ioptions, read_options, &props);
  ASSERT_OK(s);

  auto user_collected = props->user_collected_properties;
@@ -429,8 +430,10 @@ void TestInternalKeyPropertiesCollector(
        new RandomAccessFileReader(std::move(source), "test"));

    std::unique_ptr<TableProperties> props;
-    Status s = ReadTableProperties(reader.get(), fwf->contents().size(),
-                                   magic_number, ioptions, &props);
+    const ReadOptions read_options;
+    Status s =
+        ReadTableProperties(reader.get(), fwf->contents().size(), magic_number,
+                            ioptions, read_options, &props);
    ASSERT_OK(s);

    auto user_collected = props->user_collected_properties;
@@ -1257,7 +1257,7 @@ class VersionBuilder::Rep {
      InternalStats* internal_stats, int max_threads,
      bool prefetch_index_and_filter_in_cache, bool is_initial_load,
      const std::shared_ptr<const SliceTransform>& prefix_extractor,
-      size_t max_file_size_for_l0_meta_pin) {
+      size_t max_file_size_for_l0_meta_pin, const ReadOptions& read_options) {
    assert(table_cache_ != nullptr);

    size_t table_cache_capacity =
@@ -1324,7 +1324,7 @@ class VersionBuilder::Rep {
        int level = files_meta[file_idx].second;
        TableCache::TypedHandle* handle = nullptr;
        statuses[file_idx] = table_cache_->FindTable(
-            ReadOptions(), file_options_,
+            read_options, file_options_,
            *(base_vstorage_->InternalComparator()), *file_meta, &handle,
            prefix_extractor, false /*no_io */, true /* record_read_stats */,
            internal_stats->GetFileReadHist(level), false, level,
@@ -1384,10 +1384,11 @@ Status VersionBuilder::LoadTableHandlers(
    InternalStats* internal_stats, int max_threads,
    bool prefetch_index_and_filter_in_cache, bool is_initial_load,
    const std::shared_ptr<const SliceTransform>& prefix_extractor,
-    size_t max_file_size_for_l0_meta_pin) {
-  return rep_->LoadTableHandlers(
-      internal_stats, max_threads, prefetch_index_and_filter_in_cache,
-      is_initial_load, prefix_extractor, max_file_size_for_l0_meta_pin);
+    size_t max_file_size_for_l0_meta_pin, const ReadOptions& read_options) {
+  return rep_->LoadTableHandlers(internal_stats, max_threads,
+                                 prefetch_index_and_filter_in_cache,
+                                 is_initial_load, prefix_extractor,
+                                 max_file_size_for_l0_meta_pin, read_options);
 }

 uint64_t VersionBuilder::GetMinOldestBlobFileNumber() const {
@@ -48,7 +48,7 @@ class VersionBuilder {
      InternalStats* internal_stats, int max_threads,
      bool prefetch_index_and_filter_in_cache, bool is_initial_load,
      const std::shared_ptr<const SliceTransform>& prefix_extractor,
-      size_t max_file_size_for_l0_meta_pin);
+      size_t max_file_size_for_l0_meta_pin, const ReadOptions& read_options);
  uint64_t GetMinOldestBlobFileNumber() const;

 private:
@@ -155,8 +155,9 @@ VersionEditHandler::VersionEditHandler(
    bool read_only, std::vector<ColumnFamilyDescriptor> column_families,
    VersionSet* version_set, bool track_missing_files,
    bool no_error_if_files_missing, const std::shared_ptr<IOTracer>& io_tracer,
-    bool skip_load_table_files, EpochNumberRequirement epoch_number_requirement)
-    : VersionEditHandlerBase(),
+    const ReadOptions& read_options, bool skip_load_table_files,
+    EpochNumberRequirement epoch_number_requirement)
+    : VersionEditHandlerBase(read_options),
      read_only_(read_only),
      column_families_(std::move(column_families)),
      version_set_(version_set),
@@ -480,7 +481,8 @@ void VersionEditHandler::CheckIterationResult(const log::Reader& reader,

 ColumnFamilyData* VersionEditHandler::CreateCfAndInit(
    const ColumnFamilyOptions& cf_options, const VersionEdit& edit) {
-  ColumnFamilyData* cfd = version_set_->CreateColumnFamily(cf_options, &edit);
+  ColumnFamilyData* cfd =
+      version_set_->CreateColumnFamily(cf_options, read_options_, &edit);
  assert(cfd != nullptr);
  cfd->set_initialized();
  assert(builders_.find(edit.column_family_) == builders_.end());
@@ -537,7 +539,7 @@ Status VersionEditHandler::MaybeCreateVersion(const VersionEdit& /*edit*/,
    if (s.ok()) {
      // Install new version
      v->PrepareAppend(
-          *cfd->GetLatestMutableCFOptions(),
+          *cfd->GetLatestMutableCFOptions(), read_options_,
          !(version_set_->db_options_->skip_stats_update_on_db_open));
      version_set_->AppendVersion(cfd, v);
    } else {
@@ -569,7 +571,8 @@ Status VersionEditHandler::LoadTables(ColumnFamilyData* cfd,
      version_set_->db_options_->max_file_opening_threads,
      prefetch_index_and_filter_in_cache, is_initial_load,
      cfd->GetLatestMutableCFOptions()->prefix_extractor,
-      MaxFileSizeForL0MetaPin(*cfd->GetLatestMutableCFOptions()));
+      MaxFileSizeForL0MetaPin(*cfd->GetLatestMutableCFOptions()),
+      read_options_);
  if ((s.IsPathNotFound() || s.IsCorruption()) && no_error_if_files_missing_) {
    s = Status::OK();
  }
@@ -647,11 +650,12 @@ Status VersionEditHandler::ExtractInfoFromVersionEdit(ColumnFamilyData* cfd,
 VersionEditHandlerPointInTime::VersionEditHandlerPointInTime(
    bool read_only, std::vector<ColumnFamilyDescriptor> column_families,
    VersionSet* version_set, const std::shared_ptr<IOTracer>& io_tracer,
+    const ReadOptions& read_options,
    EpochNumberRequirement epoch_number_requirement)
    : VersionEditHandler(read_only, column_families, version_set,
                         /*track_missing_files=*/true,
                         /*no_error_if_files_missing=*/true, io_tracer,
-                         epoch_number_requirement) {}
+                         read_options, epoch_number_requirement) {}

 VersionEditHandlerPointInTime::~VersionEditHandlerPointInTime() {
  for (const auto& elem : versions_) {
@@ -816,7 +820,8 @@ Status VersionEditHandlerPointInTime::MaybeCreateVersion(
        cfd->internal_stats(),
        version_set_->db_options_->max_file_opening_threads, false, true,
        cfd->GetLatestMutableCFOptions()->prefix_extractor,
-        MaxFileSizeForL0MetaPin(*cfd->GetLatestMutableCFOptions()));
+        MaxFileSizeForL0MetaPin(*cfd->GetLatestMutableCFOptions()),
+        read_options_);
    if (!s.ok()) {
      delete version;
      if (s.IsCorruption()) {
@@ -827,7 +832,7 @@ Status VersionEditHandlerPointInTime::MaybeCreateVersion(
    s = builder->SaveTo(version->storage_info());
    if (s.ok()) {
      version->PrepareAppend(
-          *cfd->GetLatestMutableCFOptions(),
+          *cfd->GetLatestMutableCFOptions(), read_options_,
          !version_set_->db_options_->skip_stats_update_on_db_open);
      auto v_iter = versions_.find(cfd->GetID());
      if (v_iter != versions_.end()) {
@@ -847,7 +852,8 @@ Status VersionEditHandlerPointInTime::VerifyFile(ColumnFamilyData* cfd,
                                                 const std::string& fpath,
                                                 int level,
                                                 const FileMetaData& fmeta) {
-  return version_set_->VerifyFileMetadata(cfd, fpath, level, fmeta);
+  return version_set_->VerifyFileMetadata(read_options_, cfd, fpath, level,
+                                          fmeta);
 }

 Status VersionEditHandlerPointInTime::VerifyBlobFile(
@@ -856,7 +862,9 @@ Status VersionEditHandlerPointInTime::VerifyBlobFile(
  BlobSource* blob_source = cfd->blob_source();
  assert(blob_source);
  CacheHandleGuard<BlobFileReader> blob_file_reader;
-  Status s = blob_source->GetBlobFileReader(blob_file_num, &blob_file_reader);
+
+  Status s = blob_source->GetBlobFileReader(read_options_, blob_file_num,
+                                            &blob_file_reader);
  if (!s.ok()) {
    return s;
  }
@@ -19,8 +19,9 @@ struct FileMetaData;

 class VersionEditHandlerBase {
 public:
-  explicit VersionEditHandlerBase()
-      : max_manifest_read_size_(std::numeric_limits<uint64_t>::max()) {}
+  explicit VersionEditHandlerBase(const ReadOptions& read_options)
+      : read_options_(read_options),
+        max_manifest_read_size_(std::numeric_limits<uint64_t>::max()) {}

  virtual ~VersionEditHandlerBase() {}

@@ -31,8 +32,9 @@ class VersionEditHandlerBase {
  AtomicGroupReadBuffer& GetReadBuffer() { return read_buffer_; }

 protected:
-  explicit VersionEditHandlerBase(uint64_t max_read_size)
-      : max_manifest_read_size_(max_read_size) {}
+  explicit VersionEditHandlerBase(const ReadOptions& read_options,
+                                  uint64_t max_read_size)
+      : read_options_(read_options), max_manifest_read_size_(max_read_size) {}
  virtual Status Initialize() { return Status::OK(); }

  virtual Status ApplyVersionEdit(VersionEdit& edit,
@@ -45,6 +47,8 @@ class VersionEditHandlerBase {

  Status status_;

+  const ReadOptions& read_options_;
+
 private:
  AtomicGroupReadBuffer read_buffer_;
  const uint64_t max_manifest_read_size_;
@@ -52,7 +56,8 @@ class VersionEditHandlerBase {

 class ListColumnFamiliesHandler : public VersionEditHandlerBase {
 public:
-  ListColumnFamiliesHandler() : VersionEditHandlerBase() {}
+  explicit ListColumnFamiliesHandler(const ReadOptions& read_options)
+      : VersionEditHandlerBase(read_options) {}

  ~ListColumnFamiliesHandler() override {}

@@ -72,9 +77,9 @@ class ListColumnFamiliesHandler : public VersionEditHandlerBase {

 class FileChecksumRetriever : public VersionEditHandlerBase {
 public:
-  FileChecksumRetriever(uint64_t max_read_size,
+  FileChecksumRetriever(const ReadOptions& read_options, uint64_t max_read_size,
                        FileChecksumList& file_checksum_list)
-      : VersionEditHandlerBase(max_read_size),
+      : VersionEditHandlerBase(read_options, max_read_size),
        file_checksum_list_(file_checksum_list) {}

  ~FileChecksumRetriever() override {}
@@ -111,12 +116,13 @@ class VersionEditHandler : public VersionEditHandlerBase {
      VersionSet* version_set, bool track_missing_files,
      bool no_error_if_files_missing,
      const std::shared_ptr<IOTracer>& io_tracer,
+      const ReadOptions& read_options,
      EpochNumberRequirement epoch_number_requirement =
          EpochNumberRequirement::kMustPresent)
-      : VersionEditHandler(read_only, column_families, version_set,
-                           track_missing_files, no_error_if_files_missing,
-                           io_tracer, /*skip_load_table_files=*/false,
-                           epoch_number_requirement) {}
+      : VersionEditHandler(
+            read_only, column_families, version_set, track_missing_files,
+            no_error_if_files_missing, io_tracer, read_options,
+            /*skip_load_table_files=*/false, epoch_number_requirement) {}

  ~VersionEditHandler() override {}

@@ -137,7 +143,8 @@ class VersionEditHandler : public VersionEditHandlerBase {
      bool read_only, std::vector<ColumnFamilyDescriptor> column_families,
      VersionSet* version_set, bool track_missing_files,
      bool no_error_if_files_missing,
-      const std::shared_ptr<IOTracer>& io_tracer, bool skip_load_table_files,
+      const std::shared_ptr<IOTracer>& io_tracer,
+      const ReadOptions& read_options, bool skip_load_table_files,
      EpochNumberRequirement epoch_number_requirement =
          EpochNumberRequirement::kMustPresent);

@@ -212,6 +219,7 @@ class VersionEditHandlerPointInTime : public VersionEditHandler {
  VersionEditHandlerPointInTime(
      bool read_only, std::vector<ColumnFamilyDescriptor> column_families,
      VersionSet* version_set, const std::shared_ptr<IOTracer>& io_tracer,
+      const ReadOptions& read_options,
      EpochNumberRequirement epoch_number_requirement =
          EpochNumberRequirement::kMustPresent);
  ~VersionEditHandlerPointInTime() override;
@@ -238,10 +246,11 @@ class ManifestTailer : public VersionEditHandlerPointInTime {
  explicit ManifestTailer(std::vector<ColumnFamilyDescriptor> column_families,
                          VersionSet* version_set,
                          const std::shared_ptr<IOTracer>& io_tracer,
+                          const ReadOptions& read_options,
                          EpochNumberRequirement epoch_number_requirement =
                              EpochNumberRequirement::kMustPresent)
      : VersionEditHandlerPointInTime(/*read_only=*/false, column_families,
-                                      version_set, io_tracer,
+                                      version_set, io_tracer, read_options,
                                      epoch_number_requirement),
        mode_(Mode::kRecovery) {}

@@ -281,12 +290,13 @@ class DumpManifestHandler : public VersionEditHandler {
 public:
  DumpManifestHandler(std::vector<ColumnFamilyDescriptor> column_families,
                      VersionSet* version_set,
-                      const std::shared_ptr<IOTracer>& io_tracer, bool verbose,
-                      bool hex, bool json)
+                      const std::shared_ptr<IOTracer>& io_tracer,
+                      const ReadOptions& read_options, bool verbose, bool hex,
+                      bool json)
      : VersionEditHandler(
            /*read_only=*/true, column_families, version_set,
            /*track_missing_files=*/false,
-            /*no_error_if_files_missing=*/false, io_tracer,
+            /*no_error_if_files_missing=*/false, io_tracer, read_options,
            /*skip_load_table_files=*/true),
        verbose_(verbose),
        hex_(hex),
@@ -1527,13 +1527,14 @@ void LevelIterator::InitFileIterator(size_t new_file_index) {
 }
 }  // anonymous namespace

-Status Version::GetTableProperties(std::shared_ptr<const TableProperties>* tp,
+Status Version::GetTableProperties(const ReadOptions& read_options,
+                                   std::shared_ptr<const TableProperties>* tp,
                                   const FileMetaData* file_meta,
                                   const std::string* fname) const {
  auto table_cache = cfd_->table_cache();
  auto ioptions = cfd_->ioptions();
  Status s = table_cache->GetTableProperties(
-      file_options_, cfd_->internal_comparator(), *file_meta, tp,
+      file_options_, read_options, cfd_->internal_comparator(), *file_meta, tp,
      mutable_cf_options_.prefix_extractor, true /* no io */);
  if (s.ok()) {
    return s;
@@ -1565,14 +1566,16 @@ Status Version::GetTableProperties(std::shared_ptr<const TableProperties>* tp,
  // the magic number check in the footer.
  std::unique_ptr<RandomAccessFileReader> file_reader(
      new RandomAccessFileReader(
-          std::move(file), file_name, nullptr /* env */, io_tracer_,
-          nullptr /* stats */, 0 /* hist_type */, nullptr /* file_read_hist */,
-          nullptr /* rate_limiter */, ioptions->listeners));
+          std::move(file), file_name, ioptions->clock /* clock */, io_tracer_,
+          ioptions->stats /* stats */,
+          Histograms::SST_READ_MICROS /* hist_type */,
+          nullptr /* file_read_hist */, nullptr /* rate_limiter */,
+          ioptions->listeners));
  std::unique_ptr<TableProperties> props;
  s = ReadTableProperties(
      file_reader.get(), file_meta->fd.GetFileSize(),
      Footer::kNullTableMagicNumber /* table's magic number */, *ioptions,
-      &props);
+      read_options, &props);
  if (!s.ok()) {
    return s;
  }
@@ -1581,10 +1584,11 @@ Status Version::GetTableProperties(std::shared_ptr<const TableProperties>* tp,
  return s;
 }

-Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props) {
+Status Version::GetPropertiesOfAllTables(const ReadOptions& read_options,
+                                         TablePropertiesCollection* props) {
  Status s;
  for (int level = 0; level < storage_info_.num_levels_; level++) {
-    s = GetPropertiesOfAllTables(props, level);
+    s = GetPropertiesOfAllTables(read_options, props, level);
    if (!s.ok()) {
      return s;
    }
@@ -1602,6 +1606,8 @@ Status Version::TablesRangeTombstoneSummary(int max_entries_to_print,

  std::stringstream ss;

+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  for (int level = 0; level < storage_info_.num_levels_; level++) {
    for (const auto& file_meta : storage_info_.files_[level]) {
      auto fname =
@@ -1614,7 +1620,7 @@ Status Version::TablesRangeTombstoneSummary(int max_entries_to_print,
      std::unique_ptr<FragmentedRangeTombstoneIterator> tombstone_iter;

      Status s = table_cache->GetRangeTombstoneIterator(
-          ReadOptions(), cfd_->internal_comparator(), *file_meta,
+          read_options, cfd_->internal_comparator(), *file_meta,
          &tombstone_iter);
      if (!s.ok()) {
        return s;
@@ -1648,7 +1654,8 @@ Status Version::TablesRangeTombstoneSummary(int max_entries_to_print,
  return Status::OK();
 }

-Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props,
+Status Version::GetPropertiesOfAllTables(const ReadOptions& read_options,
+                                         TablePropertiesCollection* props,
                                         int level) {
  for (const auto& file_meta : storage_info_.files_[level]) {
    auto fname =
@@ -1657,7 +1664,8 @@ Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props,
    // 1. If the table is already present in table cache, load table
    // properties from there.
    std::shared_ptr<const TableProperties> table_properties;
-    Status s = GetTableProperties(&table_properties, file_meta, &fname);
+    Status s =
+        GetTableProperties(read_options, &table_properties, file_meta, &fname);
    if (s.ok()) {
      props->insert({fname, table_properties});
    } else {
@@ -1669,7 +1677,8 @@ Status Version::GetPropertiesOfAllTables(TablePropertiesCollection* props,
 }

 Status Version::GetPropertiesOfTablesInRange(
-    const Range* range, std::size_t n, TablePropertiesCollection* props) const {
+    const ReadOptions& read_options, const Range* range, std::size_t n,
+    TablePropertiesCollection* props) const {
  for (int level = 0; level < storage_info_.num_non_empty_levels(); level++) {
    for (decltype(n) i = 0; i < n; i++) {
      // Convert user_key into a corresponding internal key.
@@ -1686,7 +1695,8 @@ Status Version::GetPropertiesOfTablesInRange(
          // 1. If the table is already present in table cache, load table
          // properties from there.
          std::shared_ptr<const TableProperties> table_properties;
-          Status s = GetTableProperties(&table_properties, file_meta, &fname);
+          Status s = GetTableProperties(read_options, &table_properties,
+                                        file_meta, &fname);
          if (s.ok()) {
            props->insert({fname, table_properties});
          } else {
@@ -1701,13 +1711,14 @@ Status Version::GetPropertiesOfTablesInRange(
 }

 Status Version::GetAggregatedTableProperties(
-    std::shared_ptr<const TableProperties>* tp, int level) {
+    const ReadOptions& read_options, std::shared_ptr<const TableProperties>* tp,
+    int level) {
  TablePropertiesCollection props;
  Status s;
  if (level < 0) {
-    s = GetPropertiesOfAllTables(&props);
+    s = GetPropertiesOfAllTables(read_options, &props);
  } else {
-    s = GetPropertiesOfAllTables(&props, level);
+    s = GetPropertiesOfAllTables(read_options, &props, level);
  }
  if (!s.ok()) {
    return s;
@@ -1721,12 +1732,12 @@ Status Version::GetAggregatedTableProperties(
  return Status::OK();
 }

-size_t Version::GetMemoryUsageByTableReaders() {
+size_t Version::GetMemoryUsageByTableReaders(const ReadOptions& read_options) {
  size_t total_usage = 0;
  for (auto& file_level : storage_info_.level_files_brief_) {
    for (size_t i = 0; i < file_level.num_files; i++) {
      total_usage += cfd_->table_cache()->GetMemoryUsageByTableReader(
-          file_options_, cfd_->internal_comparator(),
+          file_options_, read_options, cfd_->internal_comparator(),
          *file_level.files[i].file_metadata,
          mutable_cf_options_.prefix_extractor);
    }
@@ -2050,6 +2061,7 @@ VersionStorageInfo::VersionStorageInfo(
      compaction_style_(compaction_style),
      files_(new std::vector<FileMetaData*>[num_levels_]),
      base_level_(num_levels_ == 1 ? -1 : 1),
+      lowest_unnecessary_level_(-1),
      level_multiplier_(0.0),
      files_by_compaction_pri_(num_levels_),
      level0_non_overlapping_(false),
@@ -2983,24 +2995,26 @@ void VersionStorageInfo::PrepareForVersionAppend(
 }

 void Version::PrepareAppend(const MutableCFOptions& mutable_cf_options,
+                            const ReadOptions& read_options,
                            bool update_stats) {
  TEST_SYNC_POINT_CALLBACK(
      "Version::PrepareAppend:forced_check",
      reinterpret_cast<void*>(&storage_info_.force_consistency_checks_));

  if (update_stats) {
-    UpdateAccumulatedStats();
+    UpdateAccumulatedStats(read_options);
  }

  storage_info_.PrepareForVersionAppend(*cfd_->ioptions(), mutable_cf_options);
 }

-bool Version::MaybeInitializeFileMetaData(FileMetaData* file_meta) {
+bool Version::MaybeInitializeFileMetaData(const ReadOptions& read_options,
+                                          FileMetaData* file_meta) {
  if (file_meta->init_stats_from_file || file_meta->compensated_file_size > 0) {
    return false;
  }
  std::shared_ptr<const TableProperties> tp;
-  Status s = GetTableProperties(&tp, file_meta);
+  Status s = GetTableProperties(read_options, &tp, file_meta);
  file_meta->init_stats_from_file = true;
  if (!s.ok()) {
    ROCKS_LOG_ERROR(vset_->db_options_->info_log,
@@ -3045,7 +3059,7 @@ void VersionStorageInfo::RemoveCurrentStats(FileMetaData* file_meta) {
  }
 }

-void Version::UpdateAccumulatedStats() {
+void Version::UpdateAccumulatedStats(const ReadOptions& read_options) {
  // maximum number of table properties loaded from files.
  const int kMaxInitCount = 20;
  int init_count = 0;
@@ -3063,7 +3077,7 @@ void Version::UpdateAccumulatedStats() {
       level < storage_info_.num_levels_ && init_count < kMaxInitCount;
       ++level) {
    for (auto* file_meta : storage_info_.files_[level]) {
-      if (MaybeInitializeFileMetaData(file_meta)) {
+      if (MaybeInitializeFileMetaData(read_options, file_meta)) {
        // each FileMeta will be initialized only once.
        storage_info_.UpdateAccumulatedStats(file_meta);
        // when option "max_open_files" is -1, all the file metadata has
@@ -3088,7 +3102,8 @@ void Version::UpdateAccumulatedStats() {
       storage_info_.accumulated_raw_value_size_ == 0 && level >= 0; --level) {
    for (int i = static_cast<int>(storage_info_.files_[level].size()) - 1;
         storage_info_.accumulated_raw_value_size_ == 0 && i >= 0; --i) {
-      if (MaybeInitializeFileMetaData(storage_info_.files_[level][i])) {
+      if (MaybeInitializeFileMetaData(read_options,
+                                      storage_info_.files_[level][i])) {
        storage_info_.UpdateAccumulatedStats(storage_info_.files_[level][i]);
      }
    }
@@ -3262,7 +3277,7 @@ void VersionStorageInfo::ComputeCompactionScore(
  // the level's target size, and 1.0 is the threshold for triggering
  // compaction. Higher score means higher prioritization.
  // Now we keep the compaction triggering condition, but consider more
-  // factors for priorization, while still keeping the 1.0 threshold.
+  // factors for prioritization, while still keeping the 1.0 threshold.
  // In order to provide flexibility for reducing score while still
  // maintaining it to be over 1.0, we scale the original score by 10x
  // if it is larger than 1.0.
@@ -3295,7 +3310,7 @@ void VersionStorageInfo::ComputeCompactionScore(
        // compaction score for the whole DB. Adding other levels as if
        // they are L0 files.
        for (int i = 1; i < num_levels(); i++) {
-          // Its possible that a subset of the files in a level may be in a
+          // It's possible that a subset of the files in a level may be in a
          // compaction, due to delete triggered compaction or trivial move.
          // In that case, the below check may not catch a level being
          // compacted as it only checks the first file. The worst that can
@@ -3344,7 +3359,7 @@ void VersionStorageInfo::ComputeCompactionScore(
              // When calculating estimated_compaction_needed_bytes, we assume
              // L0 is qualified as pending compactions. We will need to make
              // sure that it qualifies for compaction.
-              // It might be guafanteed by logic below anyway, but we are
+              // It might be guaranteed by logic below anyway, but we are
              // explicit here to make sure we don't stop writes with no
              // compaction scheduled.
              score = std::max(score, 1.01);
@@ -3373,7 +3388,7 @@ void VersionStorageInfo::ComputeCompactionScore(
          }
        }
      }
-    } else {
+    } else {  // level > 0
      // Compute the ratio of current size to size limit.
      uint64_t level_bytes_no_compacting = 0;
      uint64_t level_total_bytes = 0;
@@ -3383,21 +3398,36 @@ void VersionStorageInfo::ComputeCompactionScore(
          level_bytes_no_compacting += f->compensated_file_size;
        }
      }
-      if (!immutable_options.level_compaction_dynamic_level_bytes ||
-          level_bytes_no_compacting < MaxBytesForLevel(level)) {
+      if (!immutable_options.level_compaction_dynamic_level_bytes) {
        score = static_cast<double>(level_bytes_no_compacting) /
                MaxBytesForLevel(level);
      } else {
-        // If there are a large mount of data being compacted down to the
-        // current level soon, we would de-prioritize compaction from
-        // a level where the incoming data would be a large ratio. We do
-        // it by dividing level size not by target level size, but
-        // the target size and the incoming compaction bytes.
-        score = static_cast<double>(level_bytes_no_compacting) /
-                (MaxBytesForLevel(level) + total_downcompact_bytes) *
-                kScoreScale;
+        if (level_bytes_no_compacting < MaxBytesForLevel(level)) {
+          score = static_cast<double>(level_bytes_no_compacting) /
+                  MaxBytesForLevel(level);
+        } else {
+          // If there are a large mount of data being compacted down to the
+          // current level soon, we would de-prioritize compaction from
+          // a level where the incoming data would be a large ratio. We do
+          // it by dividing level size not by target level size, but
+          // the target size and the incoming compaction bytes.
+          score = static_cast<double>(level_bytes_no_compacting) /
+                  (MaxBytesForLevel(level) + total_downcompact_bytes) *
+                  kScoreScale;
+        }
+        // Drain unnecessary levels, but with lower priority compared to
+        // when L0 is eligible. Only non-empty levels can be unnecessary.
+        // If there is no unnecessary levels, lowest_unnecessary_level_ = -1.
+        if (level_bytes_no_compacting > 0 &&
+            level <= lowest_unnecessary_level_) {
+          score = std::max(
+              score, kScoreScale *
+                         (1.001 + 0.001 * (lowest_unnecessary_level_ - level)));
+        }
      }
-      if (level_total_bytes > MaxBytesForLevel(level)) {
+      if (level <= lowest_unnecessary_level_) {
+        total_downcompact_bytes += level_total_bytes;
+      } else if (level_total_bytes > MaxBytesForLevel(level)) {
        total_downcompact_bytes +=
            static_cast<double>(level_total_bytes - MaxBytesForLevel(level));
      }
@@ -4470,6 +4500,7 @@ void VersionStorageInfo::CalculateBaseBytes(const ImmutableOptions& ioptions,
      }
    }
  } else {
+    assert(ioptions.compaction_style == kCompactionStyleLevel);
    uint64_t max_level_size = 0;

    int first_non_empty_level = -1;
@@ -4494,11 +4525,13 @@ void VersionStorageInfo::CalculateBaseBytes(const ImmutableOptions& ioptions,
      level_max_bytes_[i] = std::numeric_limits<uint64_t>::max();
    }

+    lowest_unnecessary_level_ = -1;
    if (max_level_size == 0) {
      // No data for L1 and up. L0 compacts to last level directly.
      // No compaction from L1+ needs to be scheduled.
      base_level_ = num_levels_ - 1;
    } else {
+      assert(first_non_empty_level >= 1);
      uint64_t base_bytes_max = options.max_bytes_for_level_base;
      uint64_t base_bytes_min = static_cast<uint64_t>(
          base_bytes_max / options.max_bytes_for_level_multiplier);
@@ -4509,20 +4542,41 @@ void VersionStorageInfo::CalculateBaseBytes(const ImmutableOptions& ioptions,
        // Round up after dividing
        cur_level_size = static_cast<uint64_t>(
            cur_level_size / options.max_bytes_for_level_multiplier);
+        if (lowest_unnecessary_level_ == -1 &&
+            cur_level_size <= base_bytes_min &&
+            (ioptions.preclude_last_level_data_seconds == 0 ||
+             i < num_levels_ - 2)) {
+          // When per_key_placement is enabled, the penultimate level is
+          // necessary.
+          lowest_unnecessary_level_ = i;
+        }
      }

      // Calculate base level and its size.
      uint64_t base_level_size;
      if (cur_level_size <= base_bytes_min) {
+        // If per_key_placement is not enabled,
+        // either there is only one non-empty level after level 0,
+        // which can less than base_bytes_min AND necessary,
+        // or there is some unnecessary level.
+        assert(first_non_empty_level == num_levels_ - 1 ||
+               ioptions.preclude_last_level_data_seconds > 0 ||
+               lowest_unnecessary_level_ != -1);
        // Case 1. If we make target size of last level to be max_level_size,
        // target size of the first non-empty level would be smaller than
        // base_bytes_min. We set it be base_bytes_min.
        base_level_size = base_bytes_min + 1U;
        base_level_ = first_non_empty_level;
-        ROCKS_LOG_INFO(ioptions.logger,
-                       "More existing levels in DB than needed. "
-                       "max_bytes_for_level_multiplier may not be guaranteed.");
+        if (base_level_ < num_levels_ - 1) {
+          ROCKS_LOG_INFO(
+              ioptions.logger,
+              "More existing levels in DB than needed: all non-zero "
+              "levels <= level %d are unnecessary.  "
+              "max_bytes_for_level_multiplier may not be guaranteed.",
+              lowest_unnecessary_level_);
+        }
      } else {
+        assert(lowest_unnecessary_level_ == -1);
        // Find base level (where L0 data is compacted to).
        base_level_ = first_non_empty_level;
        while (base_level_ > 1 && cur_level_size > base_bytes_max) {
@@ -4931,7 +4985,8 @@ void VersionSet::AppendVersion(ColumnFamilyData* column_family_data,
 Status VersionSet::ProcessManifestWrites(
    std::deque<ManifestWriter>& writers, InstrumentedMutex* mu,
    FSDirectory* dir_contains_current_file, bool new_descriptor_log,
-    const ColumnFamilyOptions* new_cf_options) {
+    const ColumnFamilyOptions* new_cf_options,
+    const ReadOptions& read_options) {
  mu->AssertHeld();
  assert(!writers.empty());
  ManifestWriter& first_writer = writers.front();
@@ -5162,7 +5217,7 @@ Status VersionSet::ProcessManifestWrites(
            true /* prefetch_index_and_filter_in_cache */,
            false /* is_initial_load */,
            mutable_cf_options_ptrs[i]->prefix_extractor,
-            MaxFileSizeForL0MetaPin(*mutable_cf_options_ptrs[i]));
+            MaxFileSizeForL0MetaPin(*mutable_cf_options_ptrs[i]), read_options);
        if (!s.ok()) {
          if (db_options_->paranoid_checks) {
            break;
@@ -5207,7 +5262,8 @@ Status VersionSet::ProcessManifestWrites(
        constexpr bool update_stats = true;

        for (int i = 0; i < static_cast<int>(versions.size()); ++i) {
-          versions[i]->PrepareAppend(*mutable_cf_options_ptrs[i], update_stats);
+          versions[i]->PrepareAppend(*mutable_cf_options_ptrs[i], read_options,
+                                     update_stats);
        }
      }

@@ -5319,7 +5375,8 @@ Status VersionSet::ProcessManifestWrites(
      assert(batch_edits.size() == 1);
      assert(new_cf_options != nullptr);
      assert(max_last_sequence == descriptor_last_sequence_);
-      CreateColumnFamily(*new_cf_options, first_writer.edit_list.front());
+      CreateColumnFamily(*new_cf_options, read_options,
+                         first_writer.edit_list.front());
    } else if (first_writer.edit_list.front()->is_column_family_drop_) {
      assert(batch_edits.size() == 1);
      assert(max_last_sequence == descriptor_last_sequence_);
@@ -5488,6 +5545,7 @@ void VersionSet::WakeUpWaitingManifestWriters() {
 Status VersionSet::LogAndApply(
    const autovector<ColumnFamilyData*>& column_family_datas,
    const autovector<const MutableCFOptions*>& mutable_cf_options_list,
+    const ReadOptions& read_options,
    const autovector<autovector<VersionEdit*>>& edit_lists,
    InstrumentedMutex* mu, FSDirectory* dir_contains_current_file,
    bool new_descriptor_log, const ColumnFamilyOptions* new_cf_options,
@@ -5565,7 +5623,8 @@ Status VersionSet::LogAndApply(
    return Status::ColumnFamilyDropped();
  }
  return ProcessManifestWrites(writers, mu, dir_contains_current_file,
-                               new_descriptor_log, new_cf_options);
+                               new_descriptor_log, new_cf_options,
+                               read_options);
 }

 void VersionSet::LogAndApplyCFHelper(VersionEdit* edit,
@@ -5649,6 +5708,7 @@ Status VersionSet::GetCurrentManifestPath(const std::string& dbname,
 Status VersionSet::Recover(
    const std::vector<ColumnFamilyDescriptor>& column_families, bool read_only,
    std::string* db_id, bool no_error_if_files_missing) {
+  const ReadOptions read_options(Env::IOActivity::kDBOpen);
  // Read "CURRENT" file, which contains a pointer to the current manifest
  // file
  std::string manifest_path;
@@ -5685,7 +5745,7 @@ Status VersionSet::Recover(
    VersionEditHandler handler(
        read_only, column_families, const_cast<VersionSet*>(this),
        /*track_missing_files=*/false, no_error_if_files_missing, io_tracer_,
-        EpochNumberRequirement::kMightMissing);
+        read_options, EpochNumberRequirement::kMightMissing);
    handler.Iterate(reader, &log_read_status);
    s = handler.status();
    if (s.ok()) {
@@ -5833,6 +5893,7 @@ Status VersionSet::TryRecoverFromOneManifest(
    const std::string& manifest_path,
    const std::vector<ColumnFamilyDescriptor>& column_families, bool read_only,
    std::string* db_id, bool* has_missing_table_file) {
+  const ReadOptions read_options(Env::IOActivity::kDBOpen);
  ROCKS_LOG_INFO(db_options_->info_log, "Trying to recover from manifest: %s\n",
                 manifest_path.c_str());
  std::unique_ptr<SequentialFileReader> manifest_file_reader;
@@ -5857,7 +5918,7 @@ Status VersionSet::TryRecoverFromOneManifest(
                     /*checksum=*/true, /*log_num=*/0);
  VersionEditHandlerPointInTime handler_pit(
      read_only, column_families, const_cast<VersionSet*>(this), io_tracer_,
-      EpochNumberRequirement::kMightMissing);
+      read_options, EpochNumberRequirement::kMightMissing);

  handler_pit.Iterate(reader, &s);

@@ -5900,6 +5961,8 @@ Status VersionSet::ListColumnFamilies(std::vector<std::string>* column_families,
 Status VersionSet::ListColumnFamiliesFromManifest(
    const std::string& manifest_path, FileSystem* fs,
    std::vector<std::string>* column_families) {
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
  std::unique_ptr<SequentialFileReader> file_reader;
  Status s;
  {
@@ -5919,7 +5982,7 @@ Status VersionSet::ListColumnFamiliesFromManifest(
  log::Reader reader(nullptr, std::move(file_reader), &reporter,
                     true /* checksum */, 0 /* log_number */);

-  ListColumnFamiliesHandler handler;
+  ListColumnFamiliesHandler handler(read_options);
  handler.Iterate(reader, &s);

  assert(column_families);
@@ -5942,6 +6005,9 @@ Status VersionSet::ReduceNumberOfLevels(const std::string& dbname,
        "Number of levels needs to be bigger than 1");
  }

+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+
  ImmutableDBOptions db_options(*options);
  ColumnFamilyOptions cf_options(*options);
  std::shared_ptr<Cache> tc(NewLRUCache(options->max_open_files - 10,
@@ -6029,8 +6095,8 @@ Status VersionSet::ReduceNumberOfLevels(const std::string& dbname,
  InstrumentedMutex dummy_mutex;
  InstrumentedMutexLock l(&dummy_mutex);
  return versions.LogAndApply(versions.GetColumnFamilySet()->GetDefault(),
-                              mutable_cf_options, &ve, &dummy_mutex, nullptr,
-                              true);
+                              mutable_cf_options, read_options, &ve,
+                              &dummy_mutex, nullptr, true);
 }

 // Get the checksum information including the checksum and checksum function
@@ -6103,6 +6169,9 @@ Status VersionSet::GetLiveFilesChecksumInfo(FileChecksumList* checksum_list) {
 Status VersionSet::DumpManifest(Options& options, std::string& dscname,
                                bool verbose, bool hex, bool json) {
  assert(options.env);
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options;
+
  std::vector<std::string> column_families;
  Status s = ListColumnFamiliesFromManifest(
      dscname, options.env->GetFileSystem().get(), &column_families);
@@ -6129,7 +6198,8 @@ Status VersionSet::DumpManifest(Options& options, std::string& dscname,
    cf_descs.emplace_back(cf, options);
  }

-  DumpManifestHandler handler(cf_descs, this, io_tracer_, verbose, hex, json);
+  DumpManifestHandler handler(cf_descs, this, io_tracer_, read_options, verbose,
+                              hex, json);
  {
    VersionSet::LogReporter reporter;
    reporter.status = &s;
@@ -6332,6 +6402,7 @@ Status VersionSet::WriteCurrentStateToManifest(
 // we avoid doing binary search for the keys b and c twice and instead somehow
 // maintain state of where they first appear in the files.
 uint64_t VersionSet::ApproximateSize(const SizeApproximationOptions& options,
+                                     const ReadOptions& read_options,
                                     Version* v, const Slice& start,
                                     const Slice& end, int start_level,
                                     int end_level, TableReaderCaller caller) {
@@ -6411,8 +6482,8 @@ uint64_t VersionSet::ApproximateSize(const SizeApproximationOptions& options,
    for (int i = idx_start + 1; i < idx_end; ++i) {
      uint64_t file_size = files_brief.files[i].fd.GetFileSize();
      // The entire file falls into the range, so we can just take its size.
-      assert(file_size ==
-             ApproximateSize(v, files_brief.files[i], start, end, caller));
+      assert(file_size == ApproximateSize(read_options, v, files_brief.files[i],
+                                          start, end, caller));
      total_full_size += file_size;
    }

@@ -6447,21 +6518,24 @@ uint64_t VersionSet::ApproximateSize(const SizeApproximationOptions& options,
    // Estimate for all the first files (might also be last files), at each
    // level
    for (const auto file_ptr : first_files) {
-      total_full_size += ApproximateSize(v, *file_ptr, start, end, caller);
+      total_full_size +=
+          ApproximateSize(read_options, v, *file_ptr, start, end, caller);
    }

    // Estimate for all the last files, at each level
    for (const auto file_ptr : last_files) {
      // We could use ApproximateSize here, but calling ApproximateOffsetOf
      // directly is just more efficient.
-      total_full_size += ApproximateOffsetOf(v, *file_ptr, end, caller);
+      total_full_size +=
+          ApproximateOffsetOf(read_options, v, *file_ptr, end, caller);
    }
  }

  return total_full_size;
 }

-uint64_t VersionSet::ApproximateOffsetOf(Version* v, const FdWithKeyRange& f,
+uint64_t VersionSet::ApproximateOffsetOf(const ReadOptions& read_options,
+                                         Version* v, const FdWithKeyRange& f,
                                         const Slice& key,
                                         TableReaderCaller caller) {
  // pre-condition
@@ -6481,14 +6555,15 @@ uint64_t VersionSet::ApproximateOffsetOf(Version* v, const FdWithKeyRange& f,
    TableCache* table_cache = v->cfd_->table_cache();
    if (table_cache != nullptr) {
      result = table_cache->ApproximateOffsetOf(
-          key, *f.file_metadata, caller, icmp,
+          read_options, key, *f.file_metadata, caller, icmp,
          v->GetMutableCFOptions().prefix_extractor);
    }
  }
  return result;
 }

-uint64_t VersionSet::ApproximateSize(Version* v, const FdWithKeyRange& f,
+uint64_t VersionSet::ApproximateSize(const ReadOptions& read_options,
+                                     Version* v, const FdWithKeyRange& f,
                                     const Slice& start, const Slice& end,
                                     TableReaderCaller caller) {
  // pre-condition
@@ -6504,13 +6579,14 @@ uint64_t VersionSet::ApproximateSize(Version* v, const FdWithKeyRange& f,

  if (icmp.Compare(f.smallest_key, start) >= 0) {
    // Start of the range is before the file start - approximate by end offset
-    return ApproximateOffsetOf(v, f, end, caller);
+    return ApproximateOffsetOf(read_options, v, f, end, caller);
  }

  if (icmp.Compare(f.largest_key, end) < 0) {
    // End of the range is after the file end - approximate by subtracting
    // start offset from the file size
-    uint64_t start_offset = ApproximateOffsetOf(v, f, start, caller);
+    uint64_t start_offset =
+        ApproximateOffsetOf(read_options, v, f, start, caller);
    assert(f.fd.GetFileSize() >= start_offset);
    return f.fd.GetFileSize() - start_offset;
  }
@@ -6521,7 +6597,7 @@ uint64_t VersionSet::ApproximateSize(Version* v, const FdWithKeyRange& f,
    return 0;
  }
  return table_cache->ApproximateSize(
-      start, end, *f.file_metadata, caller, icmp,
+      read_options, start, end, *f.file_metadata, caller, icmp,
      v->GetMutableCFOptions().prefix_extractor);
 }

@@ -6812,7 +6888,8 @@ void VersionSet::GetObsoleteFiles(std::vector<ObsoleteFileInfo>* files,
 }

 ColumnFamilyData* VersionSet::CreateColumnFamily(
-    const ColumnFamilyOptions& cf_options, const VersionEdit* edit) {
+    const ColumnFamilyOptions& cf_options, const ReadOptions& read_options,
+    const VersionEdit* edit) {
  assert(edit->is_column_family_add_);

  MutableCFOptions dummy_cf_options;
@@ -6831,7 +6908,8 @@ ColumnFamilyData* VersionSet::CreateColumnFamily(

  constexpr bool update_stats = false;

-  v->PrepareAppend(*new_cfd->GetLatestMutableCFOptions(), update_stats);
+  v->PrepareAppend(*new_cfd->GetLatestMutableCFOptions(), read_options,
+                   update_stats);

  AppendVersion(new_cfd, v);
  // GetLatestMutableCFOptions() is safe here without mutex since the
@@ -6896,7 +6974,8 @@ uint64_t VersionSet::GetTotalBlobFileSize(Version* dummy_versions) {
  return all_versions_blob_file_size;
 }

-Status VersionSet::VerifyFileMetadata(ColumnFamilyData* cfd,
+Status VersionSet::VerifyFileMetadata(const ReadOptions& read_options,
+                                      ColumnFamilyData* cfd,
                                      const std::string& fpath, int level,
                                      const FileMetaData& meta) {
  uint64_t fsize = 0;
@@ -6929,7 +7008,7 @@ Status VersionSet::VerifyFileMetadata(ColumnFamilyData* cfd,
    TableCache::TypedHandle* handle = nullptr;
    FileMetaData meta_copy = meta;
    status = table_cache->FindTable(
-        ReadOptions(), file_opts, *icmp, meta_copy, &handle, pe,
+        read_options, file_opts, *icmp, meta_copy, &handle, pe,
        /*no_io=*/false, /*record_read_stats=*/true,
        internal_stats->GetFileReadHist(level), false, level,
        /*prefetch_index_and_filter_in_cache*/ false, max_sz_for_l0_meta_pin,
@@ -6973,9 +7052,9 @@ Status ReactiveVersionSet::Recover(
  log::Reader* reader = manifest_reader->get();
  assert(reader);

-  manifest_tailer_.reset(
-      new ManifestTailer(column_families, const_cast<ReactiveVersionSet*>(this),
-                         io_tracer_, EpochNumberRequirement::kMightMissing));
+  manifest_tailer_.reset(new ManifestTailer(
+      column_families, const_cast<ReactiveVersionSet*>(this), io_tracer_,
+      read_options_, EpochNumberRequirement::kMightMissing));

  manifest_tailer_->Iterate(*reader, manifest_reader_status->get());

@@ -650,6 +650,12 @@ class VersionStorageInfo {
  // be empty. -1 if it is not level-compaction so it's not applicable.
  int base_level_;

+  // Applies to level compaction when
+  // `level_compaction_dynamic_level_bytes=true`. All non-empty levels <=
+  // lowest_unnecessary_level_ are not needed and will be drained automatically.
+  // -1 if there is no unnecessary level,
+  int lowest_unnecessary_level_;
+
  double level_multiplier_;

  // A list for the same set of files that are stored in files_,
@@ -907,7 +913,7 @@ class Version {
  // populates derived data structures. Call without mutex held. It needs to be
  // called before appending the version to the version set.
  void PrepareAppend(const MutableCFOptions& mutable_cf_options,
-                     bool update_stats);
+                     const ReadOptions& read_options, bool update_stats);

  // Reference count management (so Versions do not disappear out from
  // under live iterators)
@@ -937,7 +943,8 @@ class Version {
  // specified in "file_meta".  If the file name of "file_meta" is
  // known ahead, passing it by a non-null "fname" can save a
  // file-name conversion.
-  Status GetTableProperties(std::shared_ptr<const TableProperties>* tp,
+  Status GetTableProperties(const ReadOptions& read_options,
+                            std::shared_ptr<const TableProperties>* tp,
                            const FileMetaData* file_meta,
                            const std::string* fname = nullptr) const;

@@ -945,9 +952,12 @@ class Version {
  // On success, *props will be populated with all SSTables' table properties.
  // The keys of `props` are the sst file name, the values of `props` are the
  // tables' properties, represented as std::shared_ptr.
-  Status GetPropertiesOfAllTables(TablePropertiesCollection* props);
-  Status GetPropertiesOfAllTables(TablePropertiesCollection* props, int level);
-  Status GetPropertiesOfTablesInRange(const Range* range, std::size_t n,
+  Status GetPropertiesOfAllTables(const ReadOptions& read_options,
+                                  TablePropertiesCollection* props);
+  Status GetPropertiesOfAllTables(const ReadOptions& read_options,
+                                  TablePropertiesCollection* props, int level);
+  Status GetPropertiesOfTablesInRange(const ReadOptions& read_options,
+                                      const Range* range, std::size_t n,
                                      TablePropertiesCollection* props) const;

  // Print summary of range delete tombstones in SST files into out_str,
@@ -959,13 +969,14 @@ class Version {
  // On success, "tp" will contains the aggregated table property among
  // the table properties of all sst files in this version.
  Status GetAggregatedTableProperties(
+      const ReadOptions& read_options,
      std::shared_ptr<const TableProperties>* tp, int level = -1);

  uint64_t GetEstimatedActiveKeys() {
    return storage_info_.GetEstimatedActiveKeys();
  }

-  size_t GetMemoryUsageByTableReaders();
+  size_t GetMemoryUsageByTableReaders(const ReadOptions& read_options);

  ColumnFamilyData* cfd() const { return cfd_; }

@@ -1018,11 +1029,12 @@ class Version {
  // The helper function of UpdateAccumulatedStats, which may fill the missing
  // fields of file_meta from its associated TableProperties.
  // Returns true if it does initialize FileMetaData.
-  bool MaybeInitializeFileMetaData(FileMetaData* file_meta);
+  bool MaybeInitializeFileMetaData(const ReadOptions& read_options,
+                                   FileMetaData* file_meta);

  // Update the accumulated stats associated with the current version.
  // This accumulated stats will be used in compaction.
-  void UpdateAccumulatedStats();
+  void UpdateAccumulatedStats(const ReadOptions& read_options);

  DECLARE_SYNC_AND_ASYNC(
      /* ret_type */ Status, /* func_name */ MultiGetFromSST,
@@ -1130,13 +1142,13 @@ class VersionSet {
  virtual ~VersionSet();

  Status LogAndApplyToDefaultColumnFamily(
-      VersionEdit* edit, InstrumentedMutex* mu,
+      const ReadOptions& read_options, VersionEdit* edit, InstrumentedMutex* mu,
      FSDirectory* dir_contains_current_file, bool new_descriptor_log = false,
      const ColumnFamilyOptions* column_family_options = nullptr) {
    ColumnFamilyData* default_cf = GetColumnFamilySet()->GetDefault();
    const MutableCFOptions* cf_options =
        default_cf->GetLatestMutableCFOptions();
-    return LogAndApply(default_cf, *cf_options, edit, mu,
+    return LogAndApply(default_cf, *cf_options, read_options, edit, mu,
                       dir_contains_current_file, new_descriptor_log,
                       column_family_options);
  }
@@ -1149,9 +1161,9 @@ class VersionSet {
  // REQUIRES: no other thread concurrently calls LogAndApply()
  Status LogAndApply(
      ColumnFamilyData* column_family_data,
-      const MutableCFOptions& mutable_cf_options, VersionEdit* edit,
-      InstrumentedMutex* mu, FSDirectory* dir_contains_current_file,
-      bool new_descriptor_log = false,
+      const MutableCFOptions& mutable_cf_options,
+      const ReadOptions& read_options, VersionEdit* edit, InstrumentedMutex* mu,
+      FSDirectory* dir_contains_current_file, bool new_descriptor_log = false,
      const ColumnFamilyOptions* column_family_options = nullptr) {
    autovector<ColumnFamilyData*> cfds;
    cfds.emplace_back(column_family_data);
@@ -1161,8 +1173,8 @@ class VersionSet {
    autovector<VersionEdit*> edit_list;
    edit_list.emplace_back(edit);
    edit_lists.emplace_back(edit_list);
-    return LogAndApply(cfds, mutable_cf_options_list, edit_lists, mu,
-                       dir_contains_current_file, new_descriptor_log,
+    return LogAndApply(cfds, mutable_cf_options_list, read_options, edit_lists,
+                       mu, dir_contains_current_file, new_descriptor_log,
                       column_family_options);
  }
  // The batch version. If edit_list.size() > 1, caller must ensure that
@@ -1170,6 +1182,7 @@ class VersionSet {
  Status LogAndApply(
      ColumnFamilyData* column_family_data,
      const MutableCFOptions& mutable_cf_options,
+      const ReadOptions& read_options,
      const autovector<VersionEdit*>& edit_list, InstrumentedMutex* mu,
      FSDirectory* dir_contains_current_file, bool new_descriptor_log = false,
      const ColumnFamilyOptions* column_family_options = nullptr,
@@ -1180,8 +1193,8 @@ class VersionSet {
    mutable_cf_options_list.emplace_back(&mutable_cf_options);
    autovector<autovector<VersionEdit*>> edit_lists;
    edit_lists.emplace_back(edit_list);
-    return LogAndApply(cfds, mutable_cf_options_list, edit_lists, mu,
-                       dir_contains_current_file, new_descriptor_log,
+    return LogAndApply(cfds, mutable_cf_options_list, read_options, edit_lists,
+                       mu, dir_contains_current_file, new_descriptor_log,
                       column_family_options, {manifest_wcb});
  }

@@ -1191,6 +1204,7 @@ class VersionSet {
  virtual Status LogAndApply(
      const autovector<ColumnFamilyData*>& cfds,
      const autovector<const MutableCFOptions*>& mutable_cf_options_list,
+      const ReadOptions& read_options,
      const autovector<autovector<VersionEdit*>>& edit_lists,
      InstrumentedMutex* mu, FSDirectory* dir_contains_current_file,
      bool new_descriptor_log = false,
@@ -1204,7 +1218,7 @@ class VersionSet {
                                       uint64_t* manifest_file_number);
  void WakeUpWaitingManifestWriters();

-  // Recover the last saved descriptor from persistent storage.
+  // Recover the last saved descriptor (MANIFEST) from persistent storage.
  // If read_only == true, Recover() will not complain if some column families
  // are not opened
  Status Recover(const std::vector<ColumnFamilyDescriptor>& column_families,
@@ -1421,7 +1435,8 @@ class VersionSet {
  // Return the approximate size of data to be scanned for range [start, end)
  // in levels [start_level, end_level). If end_level == -1 it will search
  // through all non-empty levels
-  uint64_t ApproximateSize(const SizeApproximationOptions& options, Version* v,
+  uint64_t ApproximateSize(const SizeApproximationOptions& options,
+                           const ReadOptions& read_options, Version* v,
                           const Slice& start, const Slice& end,
                           int start_level, int end_level,
                           TableReaderCaller caller);
@@ -1481,7 +1496,8 @@ class VersionSet {
        new Version(cfd, this, file_options_, mutable_cf_options, io_tracer_);

    constexpr bool update_stats = false;
-    version->PrepareAppend(mutable_cf_options, update_stats);
+    const ReadOptions read_options;
+    version->PrepareAppend(mutable_cf_options, read_options, update_stats);
    AppendVersion(cfd, version);
  }

@@ -1510,14 +1526,15 @@ class VersionSet {
  void Reset();

  // Returns approximated offset of a key in a file for a given version.
-  uint64_t ApproximateOffsetOf(Version* v, const FdWithKeyRange& f,
-                               const Slice& key, TableReaderCaller caller);
+  uint64_t ApproximateOffsetOf(const ReadOptions& read_options, Version* v,
+                               const FdWithKeyRange& f, const Slice& key,
+                               TableReaderCaller caller);

  // Returns approximated data size between start and end keys in a file
  // for a given version.
-  uint64_t ApproximateSize(Version* v, const FdWithKeyRange& f,
-                           const Slice& start, const Slice& end,
-                           TableReaderCaller caller);
+  uint64_t ApproximateSize(const ReadOptions& read_options, Version* v,
+                           const FdWithKeyRange& f, const Slice& start,
+                           const Slice& end, TableReaderCaller caller);

  struct MutableCFState {
    uint64_t log_number;
@@ -1536,9 +1553,11 @@ class VersionSet {
  void AppendVersion(ColumnFamilyData* column_family_data, Version* v);

  ColumnFamilyData* CreateColumnFamily(const ColumnFamilyOptions& cf_options,
+                                       const ReadOptions& read_options,
                                       const VersionEdit* edit);

-  Status VerifyFileMetadata(ColumnFamilyData* cfd, const std::string& fpath,
+  Status VerifyFileMetadata(const ReadOptions& read_options,
+                            ColumnFamilyData* cfd, const std::string& fpath,
                            int level, const FileMetaData& meta);

  // Protected by DB mutex.
@@ -1614,7 +1633,8 @@ class VersionSet {
                               InstrumentedMutex* mu,
                               FSDirectory* dir_contains_current_file,
                               bool new_descriptor_log,
-                               const ColumnFamilyOptions* new_cf_options);
+                               const ColumnFamilyOptions* new_cf_options,
+                               const ReadOptions& read_options);

  void LogAndApplyCFHelper(VersionEdit* edit,
                           SequenceNumber* max_last_sequence);
@@ -1666,13 +1686,15 @@ class ReactiveVersionSet : public VersionSet {

 private:
  std::unique_ptr<ManifestTailer> manifest_tailer_;
-
+  // TODO: plumb Env::IOActivity
+  const ReadOptions read_options_;
  using VersionSet::LogAndApply;
  using VersionSet::Recover;

  Status LogAndApply(
      const autovector<ColumnFamilyData*>& /*cfds*/,
      const autovector<const MutableCFOptions*>& /*mutable_cf_options_list*/,
+      const ReadOptions& /* read_options */,
      const autovector<autovector<VersionEdit*>>& /*edit_lists*/,
      InstrumentedMutex* /*mu*/, FSDirectory* /*dir_contains_current_file*/,
      bool /*new_descriptor_log*/, const ColumnFamilyOptions* /*new_cf_option*/,
@@ -454,6 +454,37 @@ TEST_F(VersionStorageInfoTest, MaxBytesForLevelDynamicWithLargeL0_3) {
  ASSERT_EQ(4, vstorage_.CompactionScoreLevel(2));
 }

+TEST_F(VersionStorageInfoTest, DrainUnnecessaryLevel) {
+  ioptions_.level_compaction_dynamic_level_bytes = true;
+  mutable_cf_options_.max_bytes_for_level_base = 1000;
+  mutable_cf_options_.max_bytes_for_level_multiplier = 10;
+
+  // Create a few unnecessary levels.
+  // See if score is calculated correctly.
+  Add(5, 1U, "1", "2", 2000U);  // target size 1010000
+  Add(4, 2U, "1", "2", 200U);   // target size 101000
+  // Unnecessary levels
+  Add(3, 3U, "1", "2", 100U);  // target size 10100
+  // Level 2: target size 1010
+  Add(1, 4U, "1", "2",
+      10U);  // target size 1000 = max(base_bytes_min + 1, base_bytes_max)
+
+  UpdateVersionStorageInfo();
+
+  ASSERT_EQ(1, vstorage_.base_level());
+  ASSERT_EQ(1000, vstorage_.MaxBytesForLevel(1));
+  ASSERT_EQ(10100, vstorage_.MaxBytesForLevel(3));
+  vstorage_.ComputeCompactionScore(ioptions_, mutable_cf_options_);
+
+  // Tests that levels 1 and 3 are eligible for compaction.
+  // Levels 1 and 3 are much smaller than target size,
+  // so size does not contribute to a high compaction score.
+  ASSERT_EQ(1, vstorage_.CompactionScoreLevel(0));
+  ASSERT_GT(vstorage_.CompactionScore(0), 10);
+  ASSERT_EQ(3, vstorage_.CompactionScoreLevel(1));
+  ASSERT_GT(vstorage_.CompactionScore(1), 10);
+}
+
 TEST_F(VersionStorageInfoTest, EstimateLiveDataSize) {
  // Test whether the overlaps are detected as expected
  Add(1, 1U, "4", "7", 1U);  // Perfect overlap with last level
@@ -1276,9 +1307,9 @@ class VersionSetTestBase {

  Status LogAndApplyToDefaultCF(VersionEdit& edit) {
    mutex_.Lock();
-    Status s =
-        versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
-                               mutable_cf_options_, &edit, &mutex_, nullptr);
+    Status s = versions_->LogAndApply(
+        versions_->GetColumnFamilySet()->GetDefault(), mutable_cf_options_,
+        read_options_, &edit, &mutex_, nullptr);
    mutex_.Unlock();
    return s;
  }
@@ -1290,9 +1321,9 @@ class VersionSetTestBase {
      vedits.push_back(e.get());
    }
    mutex_.Lock();
-    Status s =
-        versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
-                               mutable_cf_options_, vedits, &mutex_, nullptr);
+    Status s = versions_->LogAndApply(
+        versions_->GetColumnFamilySet()->GetDefault(), mutable_cf_options_,
+        read_options_, vedits, &mutex_, nullptr);
    mutex_.Unlock();
    return s;
  }
@@ -1304,7 +1335,7 @@ class VersionSetTestBase {
    VersionEdit dummy;
    ASSERT_OK(versions_->LogAndApply(
        versions_->GetColumnFamilySet()->GetDefault(), mutable_cf_options_,
-        &dummy, &mutex_, db_directory, new_descriptor_log));
+        read_options_, &dummy, &mutex_, db_directory, new_descriptor_log));
    mutex_.Unlock();
  }

@@ -1319,7 +1350,8 @@ class VersionSetTestBase {
    Status s;
    mutex_.Lock();
    s = versions_->LogAndApply(/*column_family_data=*/nullptr,
-                               MutableCFOptions(cf_options), &new_cf, &mutex_,
+                               MutableCFOptions(cf_options), read_options_,
+                               &new_cf, &mutex_,
                               /*db_directory=*/nullptr,
                               /*new_descriptor_log=*/false, &cf_options);
    mutex_.Unlock();
@@ -1341,6 +1373,7 @@ class VersionSetTestBase {
  ColumnFamilyOptions cf_options_;
  ImmutableOptions immutable_options_;
  MutableCFOptions mutable_cf_options_;
+  const ReadOptions read_options_;
  std::shared_ptr<Cache> table_cache_;
  WriteController write_controller_;
  WriteBufferManager write_buffer_manager_;
@@ -1364,6 +1397,8 @@ class VersionSetTest : public VersionSetTestBase, public testing::Test {
 TEST_F(VersionSetTest, SameColumnFamilyGroupCommit) {
  NewDB();
  const int kGroupSize = 5;
+  const ReadOptions read_options;
+
  autovector<VersionEdit> edits;
  for (int i = 0; i != kGroupSize; ++i) {
    edits.emplace_back(VersionEdit());
@@ -1390,8 +1425,8 @@ TEST_F(VersionSetTest, SameColumnFamilyGroupCommit) {
      });
  SyncPoint::GetInstance()->EnableProcessing();
  mutex_.Lock();
-  Status s = versions_->LogAndApply(cfds, all_mutable_cf_options, edit_lists,
-                                    &mutex_, nullptr);
+  Status s = versions_->LogAndApply(cfds, all_mutable_cf_options, read_options,
+                                    edit_lists, &mutex_, nullptr);
  mutex_.Unlock();
  EXPECT_OK(s);
  EXPECT_EQ(kGroupSize - 1, count);
@@ -1591,9 +1626,9 @@ TEST_F(VersionSetTest, ObsoleteBlobFile) {
  edit.AddBlobFileGarbage(blob_file_number, total_blob_count, total_blob_bytes);

  mutex_.Lock();
-  Status s =
-      versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),
-                             mutable_cf_options_, &edit, &mutex_, nullptr);
+  Status s = versions_->LogAndApply(
+      versions_->GetColumnFamilySet()->GetDefault(), mutable_cf_options_,
+      read_options_, &edit, &mutex_, nullptr);
  mutex_.Unlock();

  ASSERT_OK(s);
@@ -2211,7 +2246,7 @@ class VersionSetWithTimestampTest : public VersionSetTest {
    Status s;
    mutex_.Lock();
    s = versions_->LogAndApply(cfd_, *(cfd_->GetLatestMutableCFOptions()),
-                               edits_, &mutex_, nullptr);
+                               read_options_, edits_, &mutex_, nullptr);
    mutex_.Unlock();
    ASSERT_OK(s);
    VerifyFullHistoryTsLow(*std::max_element(ts_lbs.begin(), ts_lbs.end()));
@@ -2221,6 +2256,9 @@ class VersionSetWithTimestampTest : public VersionSetTest {
  ColumnFamilyData* cfd_{nullptr};
  // edits_ must contain and own pointers to heap-alloc VersionEdit objects.
  autovector<VersionEdit*> edits_;
+
+ private:
+  const ReadOptions read_options_;
 };

 const std::string VersionSetWithTimestampTest::kNewCfName("new_cf");
@@ -2649,6 +2687,8 @@ class VersionSetTestDropOneCF : public VersionSetTestBase,
 //  Repeat the test for i = 1, 2, 3 to simulate dropping the first, middle and
 //  last column family in an atomic group.
 TEST_P(VersionSetTestDropOneCF, HandleDroppedColumnFamilyInAtomicGroup) {
+  const ReadOptions read_options;
+
  std::vector<ColumnFamilyDescriptor> column_families;
  SequenceNumber last_seqno;
  std::unique_ptr<log::Writer> log_writer;
@@ -2678,7 +2718,7 @@ TEST_P(VersionSetTestDropOneCF, HandleDroppedColumnFamilyInAtomicGroup) {
  mutex_.Lock();
  s = versions_->LogAndApply(cfd_to_drop,
                             *cfd_to_drop->GetLatestMutableCFOptions(),
-                             &drop_cf_edit, &mutex_, nullptr);
+                             read_options, &drop_cf_edit, &mutex_, nullptr);
  mutex_.Unlock();
  ASSERT_OK(s);

@@ -2727,8 +2767,8 @@ TEST_P(VersionSetTestDropOneCF, HandleDroppedColumnFamilyInAtomicGroup) {
      });
  SyncPoint::GetInstance()->EnableProcessing();
  mutex_.Lock();
-  s = versions_->LogAndApply(cfds, mutable_cf_options_list, edit_lists, &mutex_,
-                             nullptr);
+  s = versions_->LogAndApply(cfds, mutable_cf_options_list, read_options,
+                             edit_lists, &mutex_, nullptr);
  mutex_.Unlock();
  ASSERT_OK(s);
  ASSERT_EQ(1, called);
@@ -33,14 +33,15 @@ class OfflineManifestWriter {
                             /*no_error_if_files_missing*/ true);
  }

-  Status LogAndApply(ColumnFamilyData* cfd, VersionEdit* edit,
+  Status LogAndApply(const ReadOptions& read_options, ColumnFamilyData* cfd,
+                     VersionEdit* edit,
                     FSDirectory* dir_contains_current_file) {
    // Use `mutex` to imitate a locked DB mutex when calling `LogAndApply()`.
    InstrumentedMutex mutex;
    mutex.Lock();
-    Status s = versions_.LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),
-                                     edit, &mutex, dir_contains_current_file,
-                                     false /* new_descriptor_log */);
+    Status s = versions_.LogAndApply(
+        cfd, *cfd->GetLatestMutableCFOptions(), read_options, edit, &mutex,
+        dir_contains_current_file, false /* new_descriptor_log */);
    mutex.Unlock();
    return s;
  }
@@ -2036,6 +2036,7 @@ class MemTableInserter : public WriteBatch::Handler {
        // key not found in memtable. Do sst get, update, add
        SnapshotImpl read_from_snapshot;
        read_from_snapshot.number_ = sequence_;
+        // TODO: plumb Env::IOActivity
        ReadOptions ropts;
        // it's going to be overwritten for sure, so no point caching data block
        // containing the old version
@@ -2480,6 +2481,7 @@ class MemTableInserter : public WriteBatch::Handler {
      // operations in the same batch.
      SnapshotImpl read_from_snapshot;
      read_from_snapshot.number_ = sequence_;
+      // TODO: plumb Env::IOActivity
      ReadOptions read_options;
      read_options.snapshot = &read_from_snapshot;

@@ -6,26 +6,46 @@
 #include "db/write_stall_stats.h"

 namespace ROCKSDB_NAMESPACE {
-const std::string kInvalidWriteStallCauseHyphenString = "invalid";
+const std::string& InvalidWriteStallHyphenString() {
+  static const std::string kInvalidWriteStallHyphenString = "invalid";
+  return kInvalidWriteStallHyphenString;
+}

-const std::array<std::string, static_cast<uint32_t>(WriteStallCause::kNone)>
-    kWriteStallCauseToHyphenString{{
-        "memtable-limit",
-        "l0-file-count-limit",
-        "pending-compaction-bytes",
-        // WriteStallCause::kCFScopeWriteStallCauseEnumMax
-        kInvalidWriteStallCauseHyphenString,
-        "write-buffer-manager-limit",
-        // WriteStallCause::kDBScopeWriteStallCauseEnumMax
-        kInvalidWriteStallCauseHyphenString,
-    }};
+const std::string& WriteStallCauseToHyphenString(WriteStallCause cause) {
+  static const std::string kMemtableLimit = "memtable-limit";
+  static const std::string kL0FileCountLimit = "l0-file-count-limit";
+  static const std::string kPendingCompactionBytes = "pending-compaction-bytes";
+  static const std::string kWriteBufferManagerLimit =
+      "write-buffer-manager-limit";
+  switch (cause) {
+    case WriteStallCause::kMemtableLimit:
+      return kMemtableLimit;
+    case WriteStallCause::kL0FileCountLimit:
+      return kL0FileCountLimit;
+    case WriteStallCause::kPendingCompactionBytes:
+      return kPendingCompactionBytes;
+    case WriteStallCause::kWriteBufferManagerLimit:
+      return kWriteBufferManagerLimit;
+    default:
+      break;
+  }
+  return InvalidWriteStallHyphenString();
+}

-const std::array<std::string,
-                 static_cast<uint32_t>(WriteStallCondition::kNormal)>
-    kWriteStallConditionToHyphenString{{
-        "delays",
-        "stops",
-    }};
+const std::string& WriteStallConditionToHyphenString(
+    WriteStallCondition condition) {
+  static const std::string kDelayed = "delays";
+  static const std::string kStopped = "stops";
+  switch (condition) {
+    case WriteStallCondition::kDelayed:
+      return kDelayed;
+    case WriteStallCondition::kStopped:
+      return kStopped;
+    default:
+      break;
+  }
+  return InvalidWriteStallHyphenString();
+}

 InternalStats::InternalCFStatsType InternalCFStat(
    WriteStallCause cause, WriteStallCondition condition) {
@@ -139,14 +159,14 @@ std::string WriteStallStatsMapKeys::CauseConditionCount(

  std::string cause_name;
  if (isCFScopeWriteStallCause(cause) || isDBScopeWriteStallCause(cause)) {
-    cause_name = kWriteStallCauseToHyphenString[static_cast<uint32_t>(cause)];
+    cause_name = WriteStallCauseToHyphenString(cause);
  } else {
    assert(false);
    return "";
  }

  const std::string& condition_name =
-      kWriteStallConditionToHyphenString[static_cast<uint32_t>(condition)];
+      WriteStallConditionToHyphenString(condition);

  cause_condition_count_name.reserve(cause_name.size() + 1 +
                                     condition_name.size());
@@ -11,15 +11,12 @@
 #include "rocksdb/types.h"

 namespace ROCKSDB_NAMESPACE {
-extern const std::string kInvalidWriteStallCauseHyphenString;
+extern const std::string& InvalidWriteStallHyphenString();

-extern const std::array<std::string,
-                        static_cast<uint32_t>(WriteStallCause::kNone)>
-    kWriteStallCauseToHyphenString;
+extern const std::string& WriteStallCauseToHyphenString(WriteStallCause cause);

-extern const std::array<std::string,
-                        static_cast<uint32_t>(WriteStallCondition::kNormal)>
-    kWriteStallConditionToHyphenString;
+extern const std::string& WriteStallConditionToHyphenString(
+    WriteStallCondition condition);

 // REQUIRES:
 // cause` is CF-scope `WriteStallCause`, see `WriteStallCause` for more
@@ -308,34 +308,10 @@ class BatchedOpsStressTest : public StressTest {
      }
    }

-    // Compare columns ignoring the last character of column values
-    auto compare = [](const WideColumns& lhs, const WideColumns& rhs) {
-      if (lhs.size() != rhs.size()) {
-        return false;
-      }
-
-      for (size_t i = 0; i < lhs.size(); ++i) {
-        if (lhs[i].name() != rhs[i].name()) {
-          return false;
-        }
-
-        if (lhs[i].value().size() != rhs[i].value().size()) {
-          return false;
-        }
-
-        if (lhs[i].value().difference_offset(rhs[i].value()) <
-            lhs[i].value().size() - 1) {
-          return false;
-        }
-      }
-
-      return true;
-    };
-
    for (size_t i = 0; i < num_keys; ++i) {
      const WideColumns& columns = results[i].columns();

-      if (!compare(results[0].columns(), columns)) {
+      if (!CompareColumns(results[0].columns(), columns)) {
        fprintf(stderr,
                "GetEntity error: inconsistent entities for key %s: %s, %s\n",
                StringToHex(key_suffix).c_str(),
@@ -372,6 +348,99 @@ class BatchedOpsStressTest : public StressTest {
    }
  }

+  void TestMultiGetEntity(ThreadState* thread, const ReadOptions& read_opts,
+                          const std::vector<int>& rand_column_families,
+                          const std::vector<int64_t>& rand_keys) override {
+    assert(thread);
+
+    assert(!rand_column_families.empty());
+    assert(rand_column_families[0] >= 0);
+    assert(rand_column_families[0] < static_cast<int>(column_families_.size()));
+
+    ColumnFamilyHandle* const cfh = column_families_[rand_column_families[0]];
+    assert(cfh);
+
+    assert(!rand_keys.empty());
+
+    ManagedSnapshot snapshot_guard(db_);
+
+    ReadOptions read_opts_copy(read_opts);
+    read_opts_copy.snapshot = snapshot_guard.snapshot();
+
+    const size_t num_keys = rand_keys.size();
+
+    for (size_t i = 0; i < num_keys; ++i) {
+      const std::string key_suffix = Key(rand_keys[i]);
+
+      constexpr size_t num_prefixes = 10;
+
+      std::array<std::string, num_prefixes> keys;
+      std::array<Slice, num_prefixes> key_slices;
+      std::array<PinnableWideColumns, num_prefixes> results;
+      std::array<Status, num_prefixes> statuses;
+
+      for (size_t j = 0; j < num_prefixes; ++j) {
+        keys[j] = std::to_string(j) + key_suffix;
+        key_slices[j] = keys[j];
+      }
+
+      db_->MultiGetEntity(read_opts_copy, cfh, num_prefixes, key_slices.data(),
+                          results.data(), statuses.data());
+
+      for (size_t j = 0; j < num_prefixes; ++j) {
+        const Status& s = statuses[j];
+
+        if (!s.ok() && !s.IsNotFound()) {
+          fprintf(stderr, "MultiGetEntity error: %s\n", s.ToString().c_str());
+          thread->stats.AddErrors(1);
+        } else if (s.IsNotFound()) {
+          thread->stats.AddGets(1, 0);
+        } else {
+          thread->stats.AddGets(1, 1);
+        }
+
+        const WideColumns& cmp_columns = results[0].columns();
+        const WideColumns& columns = results[j].columns();
+
+        if (!CompareColumns(cmp_columns, columns)) {
+          fprintf(stderr,
+                  "MultiGetEntity error: inconsistent entities for key %s: %s, "
+                  "%s\n",
+                  StringToHex(key_suffix).c_str(),
+                  WideColumnsToHex(cmp_columns).c_str(),
+                  WideColumnsToHex(columns).c_str());
+        }
+
+        if (!columns.empty()) {
+          // The last character of each column value should be 'j' as a decimal
+          // digit
+          const char expected = static_cast<char>('0' + j);
+
+          for (const auto& column : columns) {
+            const Slice& value = column.value();
+
+            if (value.empty() || value[value.size() - 1] != expected) {
+              fprintf(stderr,
+                      "MultiGetEntity error: incorrect column value for key "
+                      "%s, entity %s, column value %s, expected %c\n",
+                      StringToHex(key_suffix).c_str(),
+                      WideColumnsToHex(columns).c_str(),
+                      value.ToString(/* hex */ true).c_str(), expected);
+            }
+          }
+
+          if (!VerifyWideColumns(columns)) {
+            fprintf(stderr,
+                    "MultiGetEntity error: inconsistent columns for key %s, "
+                    "entity %s\n",
+                    StringToHex(key_suffix).c_str(),
+                    WideColumnsToHex(columns).c_str());
+          }
+        }
+      }
+    }
+  }
+
  // Given a key, this does prefix scans for "0"+P, "1"+P, ..., "9"+P
  // in the same snapshot where P is the first FLAGS_prefix_size - 1 bytes
  // of the key. Each of these 10 scans returns a series of values;
@@ -493,6 +562,30 @@ class BatchedOpsStressTest : public StressTest {
  void VerifyDb(ThreadState* /* thread */) const override {}

  void ContinuouslyVerifyDb(ThreadState* /* thread */) const override {}
+
+  // Compare columns ignoring the last character of column values
+  bool CompareColumns(const WideColumns& lhs, const WideColumns& rhs) {
+    if (lhs.size() != rhs.size()) {
+      return false;
+    }
+
+    for (size_t i = 0; i < lhs.size(); ++i) {
+      if (lhs[i].name() != rhs[i].name()) {
+        return false;
+      }
+
+      if (lhs[i].value().size() != rhs[i].value().size()) {
+        return false;
+      }
+
+      if (lhs[i].value().difference_offset(rhs[i].value()) <
+          lhs[i].value().size() - 1) {
+        return false;
+      }
+    }
+
+    return true;
+  }
 };

 StressTest* CreateBatchedOpsStressTest() { return new BatchedOpsStressTest(); }
@@ -391,6 +391,129 @@ class CfConsistencyStressTest : public StressTest {
    }
  }

+  void TestMultiGetEntity(ThreadState* thread, const ReadOptions& read_opts,
+                          const std::vector<int>& rand_column_families,
+                          const std::vector<int64_t>& rand_keys) override {
+    assert(thread);
+    assert(thread->shared);
+    assert(!rand_column_families.empty());
+    assert(!rand_keys.empty());
+
+    ManagedSnapshot snapshot_guard(db_);
+
+    ReadOptions read_opts_copy = read_opts;
+    read_opts_copy.snapshot = snapshot_guard.snapshot();
+
+    const size_t num_cfs = rand_column_families.size();
+
+    std::vector<ColumnFamilyHandle*> cfhs;
+    cfhs.reserve(num_cfs);
+
+    for (size_t j = 0; j < num_cfs; ++j) {
+      assert(rand_column_families[j] >= 0);
+      assert(rand_column_families[j] <
+             static_cast<int>(column_families_.size()));
+
+      ColumnFamilyHandle* const cfh = column_families_[rand_column_families[j]];
+      assert(cfh);
+
+      cfhs.emplace_back(cfh);
+    }
+
+    const size_t num_keys = rand_keys.size();
+
+    for (size_t i = 0; i < num_keys; ++i) {
+      const std::string key = Key(rand_keys[i]);
+
+      std::vector<Slice> key_slices(num_cfs, key);
+      std::vector<PinnableWideColumns> results(num_cfs);
+      std::vector<Status> statuses(num_cfs);
+
+      db_->MultiGetEntity(read_opts_copy, num_cfs, cfhs.data(),
+                          key_slices.data(), results.data(), statuses.data());
+
+      bool is_consistent = true;
+
+      for (size_t j = 0; j < num_cfs; ++j) {
+        const Status& s = statuses[j];
+        const Status& cmp_s = statuses[0];
+        const WideColumns& columns = results[j].columns();
+        const WideColumns& cmp_columns = results[0].columns();
+
+        if (!s.ok() && !s.IsNotFound()) {
+          fprintf(stderr, "TestMultiGetEntity error: %s\n",
+                  s.ToString().c_str());
+          thread->stats.AddErrors(1);
+          break;
+        }
+
+        assert(cmp_s.ok() || cmp_s.IsNotFound());
+
+        if (s.IsNotFound()) {
+          if (cmp_s.ok()) {
+            fprintf(
+                stderr,
+                "MultiGetEntity returns different results for key %s: CF %s "
+                "returns entity %s, CF %s returns not found\n",
+                StringToHex(key).c_str(), column_family_names_[0].c_str(),
+                WideColumnsToHex(cmp_columns).c_str(),
+                column_family_names_[j].c_str());
+            is_consistent = false;
+            break;
+          }
+
+          continue;
+        }
+
+        assert(s.ok());
+        if (cmp_s.IsNotFound()) {
+          fprintf(stderr,
+                  "MultiGetEntity returns different results for key %s: CF %s "
+                  "returns not found, CF %s returns entity %s\n",
+                  StringToHex(key).c_str(), column_family_names_[0].c_str(),
+                  column_family_names_[j].c_str(),
+                  WideColumnsToHex(columns).c_str());
+          is_consistent = false;
+          break;
+        }
+
+        if (columns != cmp_columns) {
+          fprintf(stderr,
+                  "MultiGetEntity returns different results for key %s: CF %s "
+                  "returns entity %s, CF %s returns entity %s\n",
+                  StringToHex(key).c_str(), column_family_names_[0].c_str(),
+                  WideColumnsToHex(cmp_columns).c_str(),
+                  column_family_names_[j].c_str(),
+                  WideColumnsToHex(columns).c_str());
+          is_consistent = false;
+          break;
+        }
+
+        if (!VerifyWideColumns(columns)) {
+          fprintf(stderr,
+                  "MultiGetEntity error: inconsistent columns for key %s, "
+                  "entity %s\n",
+                  StringToHex(key).c_str(), WideColumnsToHex(columns).c_str());
+          is_consistent = false;
+          break;
+        }
+      }
+
+      if (!is_consistent) {
+        fprintf(stderr,
+                "TestMultiGetEntity error: results are not consistent\n");
+        thread->stats.AddErrors(1);
+        // Fail fast to preserve the DB state.
+        thread->shared->SetVerificationFailure();
+        break;
+      } else if (statuses[0].ok()) {
+        thread->stats.AddGets(1, 1);
+      } else if (statuses[0].IsNotFound()) {
+        thread->stats.AddGets(1, 0);
+      }
+    }
+  }
+
  Status TestPrefixScan(ThreadState* thread, const ReadOptions& readoptions,
                        const std::vector<int>& rand_column_families,
                        const std::vector<int64_t>& rand_keys) override {
@@ -214,6 +214,7 @@ DECLARE_uint64(snapshot_hold_ops);
 DECLARE_bool(long_running_snapshots);
 DECLARE_bool(use_multiget);
 DECLARE_bool(use_get_entity);
+DECLARE_bool(use_multi_get_entity);
 DECLARE_int32(readpercent);
 DECLARE_int32(prefixpercent);
 DECLARE_int32(writepercent);
@@ -310,6 +311,8 @@ DECLARE_int32(create_timestamped_snapshot_one_in);

 DECLARE_bool(allow_data_in_errors);

+DECLARE_bool(enable_thread_tracking);
+
 // Tiered storage
 DECLARE_bool(enable_tiered_storage);  // set last_level_temperature
 DECLARE_int64(preclude_last_level_data_seconds);
@@ -55,8 +55,7 @@ void ThreadBody(void* v) {
    }
  }
 }
-
-bool RunStressTest(SharedState* shared) {
+bool RunStressTestImpl(SharedState* shared) {
  SystemClock* clock = db_stress_env->GetSystemClock().get();
  StressTest* stress = shared->GetStressTest();

@@ -207,5 +206,11 @@ bool RunStressTest(SharedState* shared) {
  }
  return true;
 }
+bool RunStressTest(SharedState* shared) {
+  ThreadStatusUtil::RegisterThread(db_stress_env, ThreadStatus::USER);
+  bool result = RunStressTestImpl(shared);
+  ThreadStatusUtil::UnregisterThread();
+  return result;
+}
 }  // namespace ROCKSDB_NAMESPACE
 #endif  // GFLAGS
@@ -10,8 +10,30 @@
 #ifdef GFLAGS
 #pragma once
 #include "db_stress_tool/db_stress_common.h"
+#include "monitoring/thread_status_util.h"

 namespace ROCKSDB_NAMESPACE {
+class DbStressRandomAccessFileWrapper : public FSRandomAccessFileOwnerWrapper {
+ public:
+  explicit DbStressRandomAccessFileWrapper(
+      std::unique_ptr<FSRandomAccessFile>&& target)
+      : FSRandomAccessFileOwnerWrapper(std::move(target)) {}
+
+  IOStatus Read(uint64_t offset, size_t n, const IOOptions& options,
+                Slice* result, char* scratch,
+                IODebugContext* dbg) const override {
+#ifndef NDEBUG
+    const ThreadStatus::OperationType thread_op =
+        ThreadStatusUtil::GetThreadOperation();
+    Env::IOActivity io_activity =
+        ThreadStatusUtil::TEST_GetExpectedIOActivity(thread_op);
+    assert(io_activity == Env::IOActivity::kUnknown ||
+           io_activity == options.io_activity);
+#endif
+    return target()->Read(offset, n, options, result, scratch, dbg);
+  }
+};
+
 class DbStressFSWrapper : public FileSystemWrapper {
 public:
  explicit DbStressFSWrapper(const std::shared_ptr<FileSystem>& t)
@@ -19,6 +41,18 @@ class DbStressFSWrapper : public FileSystemWrapper {
  static const char* kClassName() { return "DbStressFS"; }
  const char* Name() const override { return kClassName(); }

+  IOStatus NewRandomAccessFile(const std::string& f,
+                               const FileOptions& file_opts,
+                               std::unique_ptr<FSRandomAccessFile>* r,
+                               IODebugContext* dbg) override {
+    std::unique_ptr<FSRandomAccessFile> file;
+    IOStatus s = target()->NewRandomAccessFile(f, file_opts, &file, dbg);
+    if (s.ok()) {
+      r->reset(new DbStressRandomAccessFileWrapper(std::move(file)));
+    }
+    return s;
+  }
+
  IOStatus DeleteFile(const std::string& f, const IOOptions& opts,
                      IODebugContext* dbg) override {
    // We determine whether it is a manifest file by searching a strong,
@@ -749,6 +749,9 @@ DEFINE_bool(use_multiget, false,

 DEFINE_bool(use_get_entity, false, "If set, use the GetEntity API for reads");

+DEFINE_bool(use_multi_get_entity, false,
+            "If set, use the MultiGetEntity API for reads");
+
 static bool ValidateInt32Percent(const char* flagname, int32_t value) {
  if (value < 0 || value > 100) {
    fprintf(stderr, "Invalid value for --%s: %d, 0<= pct <=100 \n", flagname,
@@ -1048,6 +1051,11 @@ DEFINE_bool(allow_data_in_errors,
            ROCKSDB_NAMESPACE::Options().allow_data_in_errors,
            "If true, allow logging data, e.g. key, value in LOG files.");

+DEFINE_bool(enable_thread_tracking,
+            ROCKSDB_NAMESPACE::Options().enable_thread_tracking,
+            "If true, the status of the threads involved in this DB will be "
+            "tracked and available via GetThreadList() API.");
+
 DEFINE_int32(verify_iterator_with_expected_state_one_in, 0,
             "If non-zero, when TestIterate() is to be called, there is a "
             "1/verify_iterator_with_expected_state_one_in "
@@ -985,7 +985,22 @@ void StressTest::OperateDb(ThreadState* thread) {
      if (prob_op >= 0 && prob_op < static_cast<int>(FLAGS_readpercent)) {
        assert(0 <= prob_op);
        // OPERATION read
-        if (FLAGS_use_get_entity) {
+        if (FLAGS_use_multi_get_entity) {
+          constexpr uint64_t max_batch_size = 64;
+          const uint64_t batch_size = std::min(
+              static_cast<uint64_t>(thread->rand.Uniform(max_batch_size)) + 1,
+              ops_per_open - i);
+          assert(batch_size >= 1);
+          assert(batch_size <= max_batch_size);
+          assert(i + batch_size <= ops_per_open);
+
+          rand_keys = GenerateNKeys(thread, static_cast<int>(batch_size), i);
+
+          TestMultiGetEntity(thread, read_opts, rand_column_families,
+                             rand_keys);
+
+          i += batch_size - 1;
+        } else if (FLAGS_use_get_entity) {
          TestGetEntity(thread, read_opts, rand_column_families, rand_keys);
        } else if (FLAGS_use_multiget) {
          // Leave room for one more iteration of the loop with a single key
@@ -1820,19 +1835,11 @@ Status StressTest::TestCheckpoint(ThreadState* thread,
  Options tmp_opts(options_);
  tmp_opts.listeners.clear();
  tmp_opts.env = db_stress_env;
+  // Avoid delayed deletion so whole directory can be deleted
+  tmp_opts.sst_file_manager.reset();

  DestroyDB(checkpoint_dir, tmp_opts);

-  if (db_stress_env->FileExists(checkpoint_dir).ok()) {
-    // If the directory might still exist, try to delete the files one by one.
-    // Likely a trash file is still there.
-    Status my_s = DestroyDir(db_stress_env, checkpoint_dir);
-    if (!my_s.ok()) {
-      fprintf(stderr, "Fail to destory directory before checkpoint: %s",
-              my_s.ToString().c_str());
-    }
-  }
-
  Checkpoint* checkpoint = nullptr;
  Status s = Checkpoint::Create(db_, &checkpoint);
  if (s.ok()) {
@@ -2387,6 +2394,8 @@ void StressTest::PrintEnv() const {
          FLAGS_use_multiget ? "true" : "false");
  fprintf(stdout, "Use GetEntity             : %s\n",
          FLAGS_use_get_entity ? "true" : "false");
+  fprintf(stdout, "Use MultiGetEntity        : %s\n",
+          FLAGS_use_multi_get_entity ? "true" : "false");

  const char* memtablerep = "";
  switch (FLAGS_rep_factory) {
@@ -3198,6 +3207,8 @@ void InitializeOptionsFromFlags(
  }

  options.allow_data_in_errors = FLAGS_allow_data_in_errors;
+
+  options.enable_thread_tracking = FLAGS_enable_thread_tracking;
 }

 void InitializeOptionsGeneral(
@@ -98,6 +98,11 @@ class StressTest {
                             const std::vector<int>& rand_column_families,
                             const std::vector<int64_t>& rand_keys) = 0;

+  virtual void TestMultiGetEntity(ThreadState* thread,
+                                  const ReadOptions& read_opts,
+                                  const std::vector<int>& rand_column_families,
+                                  const std::vector<int64_t>& rand_keys) = 0;
+
  virtual Status TestPrefixScan(ThreadState* thread,
                                const ReadOptions& read_opts,
                                const std::vector<int>& rand_column_families,
@@ -99,13 +99,6 @@ int db_stress_tool(int argc, char** argv) {

  env_wrapper_guard = std::make_shared<CompositeEnvWrapper>(
      raw_env, std::make_shared<DbStressFSWrapper>(raw_env->GetFileSystem()));
-  if (!env_opts && !FLAGS_use_io_uring) {
-    // If using the default Env (Posix), wrap DbStressEnvWrapper with the
-    // legacy EnvWrapper. This is a workaround to prevent MultiGet and scans
-    // from failing when IO uring is disabled. The EnvWrapper
-    // has a default implementation of ReadAsync that redirects to Read.
-    env_wrapper_guard = std::make_shared<EnvWrapper>(env_wrapper_guard);
-  }
  db_stress_env = env_wrapper_guard.get();

  FLAGS_rep_factory = StringToRepFactory(FLAGS_memtablerep.c_str());
@@ -393,6 +393,12 @@ void MultiOpsTxnsStressTest::TestGetEntity(
    const std::vector<int>& /* rand_column_families */,
    const std::vector<int64_t>& /* rand_keys */) {}

+// Wide columns are currently not supported by transactions.
+void MultiOpsTxnsStressTest::TestMultiGetEntity(
+    ThreadState* /* thread */, const ReadOptions& /* read_opts */,
+    const std::vector<int>& /* rand_column_families */,
+    const std::vector<int64_t>& /* rand_keys */) {}
+
 Status MultiOpsTxnsStressTest::TestPrefixScan(
    ThreadState* thread, const ReadOptions& read_opts,
    const std::vector<int>& rand_column_families,
@@ -1214,7 +1220,8 @@ void MultiOpsTxnsStressTest::VerifyDb(ThreadState* thread) const {
 // which can be called before TransactionDB::Open() returns to caller.
 // Therefore, at that time, db_ and txn_db_  may still be nullptr.
 // Caller has to make sure that the race condition does not happen.
-void MultiOpsTxnsStressTest::VerifyPkSkFast(int job_id) {
+void MultiOpsTxnsStressTest::VerifyPkSkFast(const ReadOptions& read_options,
+                                            int job_id) {
  DB* const db = db_aptr_.load(std::memory_order_acquire);
  if (db == nullptr) {
    return;
@@ -1243,6 +1250,7 @@ void MultiOpsTxnsStressTest::VerifyPkSkFast(int job_id) {
  ReadOptions ropts;
  ropts.snapshot = snapshot;
  ropts.total_order_seek = true;
+  ropts.io_activity = read_options.io_activity;

  std::unique_ptr<Iterator> it(db_->NewIterator(ropts));
  for (it->Seek(start_key); it->Valid(); it->Next()) {