Explicitly mark destructors with override (#13212)

Summary:
I saw these compiler warnings while preparing for the 9.10 release:

```cpp
'~CompactOnDeletionCollectorFactory' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]

'~CompactForTieringCollectorFactory' overrides a destructor but is not marked 'override' [-Werror,-Wsuggest-destructor-override]
```

This code is from a while ago so I assume that this CI check has been failing for quite some time. We should still clean this up to avoid confusion in the future.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13212

Test Plan: Existing CI checks should pass, and we should not see this CI check failure the next time we try to make a release/patch.

Reviewed By: jaykorean

Differential Revision: D67287794

Pulled By: archang19

fbshipit-source-id: a11230a919c0b7ef21a7219bf05f567d3d44b2d1

BUCK

@@ -83,6 +83,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
         "db/log_writer.cc",
         "db/logs_with_prep_tracker.cc",
         "db/malloc_stats.cc",
+        "db/manifest_ops.cc",
         "db/memtable.cc",
         "db/memtable_list.cc",
         "db/merge_helper.cc",
@@ -367,6 +368,11 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
 
 cpp_library_wrapper(name="rocksdb_whole_archive_lib", srcs=[], deps=[":rocksdb_lib"], headers=[], link_whole=True, extra_test_libs=False)
 
+cpp_library_wrapper(name="rocksdb_with_faiss_lib", srcs=["utilities/secondary_index/faiss_ivf_index.cc"], deps=[
+        "//faiss:faiss",
+        ":rocksdb_lib",
+    ], headers=[], link_whole=False, extra_test_libs=False)
+
 cpp_library_wrapper(name="rocksdb_test_lib", srcs=[
         "db/db_test_util.cc",
         "db/db_with_timestamp_test_util.cc",
@@ -381,6 +387,20 @@ cpp_library_wrapper(name="rocksdb_test_lib", srcs=[
         "utilities/cassandra/test_utils.cc",
     ], deps=[":rocksdb_lib"], headers=[], link_whole=False, extra_test_libs=True)
 
+cpp_library_wrapper(name="rocksdb_with_faiss_test_lib", srcs=[
+        "db/db_test_util.cc",
+        "db/db_with_timestamp_test_util.cc",
+        "table/mock_table.cc",
+        "test_util/mock_time_env.cc",
+        "test_util/secondary_cache_test_util.cc",
+        "test_util/testharness.cc",
+        "test_util/testutil.cc",
+        "tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
+        "tools/trace_analyzer_tool.cc",
+        "utilities/agg_merge/test_agg_merge.cc",
+        "utilities/cassandra/test_utils.cc",
+    ], deps=[":rocksdb_with_faiss_lib"], headers=[], link_whole=False, extra_test_libs=True)
+
 cpp_library_wrapper(name="rocksdb_tools_lib", srcs=[
         "test_util/testutil.cc",
         "tools/block_cache_analyzer/block_cache_trace_analyzer.cc",
@@ -5077,6 +5097,12 @@ cpp_unittest_wrapper(name="external_sst_file_test",
             extra_compiler_flags=[])
 
 
+cpp_unittest_wrapper(name="faiss_ivf_index_test",
+            srcs=["utilities/secondary_index/faiss_ivf_index_test.cc"],
+            deps=[":rocksdb_with_faiss_test_lib"],
+            extra_compiler_flags=[])
+
+
 cpp_unittest_wrapper(name="fault_injection_test",
             srcs=["db/fault_injection_test.cc"],
             deps=[":rocksdb_test_lib"],

CMakeLists.txt

@@ -716,6 +716,7 @@ set(SOURCES
         db/log_reader.cc
         db/log_writer.cc
         db/malloc_stats.cc
+        db/manifest_ops.cc
         db/memtable.cc
         db/memtable_list.cc
         db/merge_helper.cc

HISTORY.md

@@ -1,6 +1,47 @@
 # Rocksdb Change Log
 > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
 
+## 9.10.0 (12/12/2024)
+### New Features
+* Introduce `TransactionOptions::commit_bypass_memtable` to enable transaction commit to bypass memtable insertions. This can be beneficial for transactions with many operations, as it reduces commit time that is mostly spent on memtable insertion.
+
+### Public API Changes
+* Deprecated Remote Compaction APIs (StartV2, WaitForCompleteV2) are completely removed from the codebase
+
+### Behavior Changes
+* DB::KeyMayExist() now follows its function comment, which means `value` parameter can be null, and it will be set only if `value_found` is passed in.
+
+### Bug Fixes
+* Fix the issue where compaction incorrectly drops a key when there is a snapshot with a sequence number of zero.
+* Honor ConfigOptions.ignore_unknown_options in ParseStruct()
+
+### Performance Improvements
+* Enable reuse of file system allocated buffer for synchronous prefetching.
+* In buffered IO mode, try to align writes on power of 2 if checksum handoff is not enabled for the file type being written.
+
+## 9.9.0 (11/18/2024)
+### New Features
+* Multi-Column-Family-Iterator (CoalescingIterator/AttributeGroupIterator) is no longer marked as experimental
+* Adds a new table property "rocksdb.newest.key.time" which records the unix timestamp of the newest key. Uses this table property for FIFO TTL and temperature change compaction.
+
+### Public API Changes
+* Added a new API `Transaction::GetAttributeGroupIterator` that can be used to create a multi-column-family attribute group iterator over the specified column families, including the data from both the transaction and the underlying database. This API is currently supported for optimistic and write-committed pessimistic transactions.
+* Added a new API `Transaction::GetCoalescingIterator` that can be used to create a multi-column-family coalescing iterator over the specified column families, including the data from both the transaction and the underlying database. This API is currently supported for optimistic and write-committed pessimistic transactions.
+
+### Behavior Changes
+* `BaseDeltaIterator` now honors the read option `allow_unprepared_value`.
+
+### Bug Fixes
+* `BaseDeltaIterator` now calls `PrepareValue` on the base iterator in case it has been created with the `allow_unprepared_value` read option set. Earlier, such base iterators could lead to incorrect values being exposed from `BaseDeltaIterator`.
+* Fix a leak of obsolete blob files left open until DB::Close(). This bug was introduced in version 9.4.0.
+* Fix missing cases of corruption retry during DB open and read API processing.
+* Fix a bug for transaction db with 2pc where an old WAL may be retained longer than needed (#13127).
+* Fix leaks of some open SST files (until `DB::Close()`) that are written but never become live due to various failures. (We now have a check for such leaks with no outstanding issues.)
+* Fix a bug for replaying WALs for WriteCommitted transaction DB when its user-defined timestamps setting is toggled on/off between DB sessions.
+
+### Performance Improvements
+* Fix regression in issue #12038 due to `Options::compaction_readahead_size` greater than `max_sectors_kb` (i.e, largest I/O size that the OS issues to a block device defined in linux)
+
 ## 9.8.0 (10/25/2024)
 ### New Features
 * All non-`block_cache` options in `BlockBasedTableOptions` are now mutable with `DB::SetOptions()`. See also Bug Fixes below.

Makefile

@@ -659,7 +659,7 @@ ifneq ($(filter check-headers, $(MAKECMDGOALS)),)
 # TODO: add/support JNI headers
 	DEV_HEADER_DIRS := $(sort include/ $(dir $(ALL_SOURCES)))
 # Some headers like in port/ are platform-specific
-	DEV_HEADERS_TO_CHECK := $(shell $(FIND) $(DEV_HEADER_DIRS) -type f -name '*.h' | grep -E -v 'port/|plugin/|lua/|range_tree/')
+	DEV_HEADERS_TO_CHECK := $(shell $(FIND) $(DEV_HEADER_DIRS) -type f -name '*.h' | grep -E -v 'port/|plugin/|lua/|range_tree/|secondary_index/')
 	PUBLIC_HEADERS_TO_CHECK := $(shell $(FIND) include/ -type f -name '*.h' | grep -E -v 'lua/')
 else
 	DEV_HEADERS_TO_CHECK :=
@@ -2489,7 +2489,7 @@ checkout_folly:
 	fi
 	@# Pin to a particular version for public CI, so that PR authors don't
 	@# need to worry about folly breaking our integration. Update periodically
-	cd third-party/folly && git reset --hard 03041f014b6e6ebb6119ffae8b7a37308f52e913
+	cd third-party/folly && git reset --hard 33f5b67fcaeb8705b04fd1b850873a180dc89aaa
 	@# NOTE: this hack is required for clang in some cases
 	perl -pi -e 's/int rv = syscall/int rv = (int)syscall/' third-party/folly/folly/detail/Futex.cpp
 	@# NOTE: this hack is required for gcc in some cases

USERS.md

@@ -38,12 +38,11 @@ Snowflake [uses](https://www.snowflake.com/blog/how-foundationdb-powers-snowflak
 The Bing search engine from Microsoft uses RocksDB as the storage engine for its web data platform: https://blogs.bing.com/Engineering-Blog/october-2021/RocksDB-in-Microsoft-Bing
 
 ## LinkedIn
-Two different use cases at Linkedin are using RocksDB as a storage engine:
+1. [Venice](https://venicedb.org/) is a derived data platform using RocksDB as its storage engine. It is LinkedIn's ML feature store, powering thousands of recommender use cases, including the Feed, Video recommendations, and People You May Know.
+2. LinkedIn's follow feed for storing user's activities. Check out the blog post: https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-feed-made-faster-and-smarter
+3. Apache Samza, open source framework for stream processing.
 
-1. LinkedIn's follow feed for storing user's activities. Check out the blog post: https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-feed-made-faster-and-smarter
-2. Apache Samza, open source framework for stream processing
-
-Learn more about those use cases in a Tech Talk by Ankit Gupta and Naveen Somasundaram: http://www.youtube.com/watch?v=plqVp_OnSzg
+Learn more about LinkedIn's follow feed and Apache Samza in a Tech Talk by Ankit Gupta and Naveen Somasundaram: http://www.youtube.com/watch?v=plqVp_OnSzg
 
 ## Yahoo
 Yahoo is using RocksDB as a storage engine for their biggest distributed data store Sherpa. Learn more about it here: http://yahooeng.tumblr.com/post/120730204806/sherpa-scales-new-heights

buckifier/buckify_rocksdb.py

@@ -14,10 +14,10 @@ from targets_builder import TARGETSBuilder, LiteralValue
 
 from util import ColorString
 
-# This script generates TARGETS file for Buck.
+# This script generates BUCK file for Buck.
 # Buck is a build tool specifying dependencies among different build targets.
 # User can pass extra dependencies as a JSON object via command line, and this
-# script can include these dependencies in the generate TARGETS file.
+# script can include these dependencies in the generate BUCK file.
 # Usage:
 # $python3 buckifier/buckify_rocksdb.py
 # (This generates a TARGET file without user-specified dependency for unit
@@ -28,7 +28,7 @@ from util import ColorString
 #                      "extra_compiler_flags": ["-DFOO_BAR", "-Os"]
 #                  }
 #         }'
-# (Generated TARGETS file has test_dep and mock1 as dependencies for RocksDB
+# (Generated BUCK file has test_dep and mock1 as dependencies for RocksDB
 # unit tests, and will use the extra_compiler_flags to compile the unit test
 # source.)
 
@@ -114,9 +114,9 @@ def get_dependencies():
     return deps_map
 
 
-# Prepare TARGETS file for buck
-def generate_targets(repo_path, deps_map):
-    print(ColorString.info("Generating TARGETS"))
+# Prepare BUCK file for buck
+def generate_buck(repo_path, deps_map):
+    print(ColorString.info("Generating BUCK"))
     # parsed src.mk file
     src_mk = parse_src_mk(repo_path)
     # get all .cc files
@@ -133,10 +133,10 @@ def generate_targets(repo_path, deps_map):
         # in how the file was generated.
         extra_argv = " '{}'".format(" ".join(sys.argv[1].split()))
 
-    TARGETS = TARGETSBuilder("%s/TARGETS" % repo_path, extra_argv)
+    BUCK = TARGETSBuilder("%s/BUCK" % repo_path, extra_argv)
 
     # rocksdb_lib
-    TARGETS.add_library(
+    BUCK.add_library(
         "rocksdb_lib",
         src_mk["LIB_SOURCES"] +
         # always add range_tree, it's only excluded on ppc64, which we don't use internally
@@ -152,7 +152,7 @@ def generate_targets(repo_path, deps_map):
         headers=LiteralValue("glob([\"**/*.h\"])")
     )
     # rocksdb_whole_archive_lib
-    TARGETS.add_library(
+    BUCK.add_library(
         "rocksdb_whole_archive_lib",
         [],
         deps=[
@@ -161,8 +161,17 @@ def generate_targets(repo_path, deps_map):
         extra_external_deps="",
         link_whole=True,
     )
+    # rocksdb_with_faiss_lib
+    BUCK.add_library(
+        "rocksdb_with_faiss_lib",
+        src_mk.get("WITH_FAISS_LIB_SOURCES", []),
+        deps=[
+            "//faiss:faiss",
+            ":rocksdb_lib",
+        ],
+    )
     # rocksdb_test_lib
-    TARGETS.add_library(
+    BUCK.add_library(
         "rocksdb_test_lib",
         src_mk.get("MOCK_LIB_SOURCES", [])
         + src_mk.get("TEST_LIB_SOURCES", [])
@@ -171,8 +180,20 @@ def generate_targets(repo_path, deps_map):
         [":rocksdb_lib"],
         extra_test_libs=True,
     )
+    # rocksdb_with_faiss_test_lib
+    BUCK.add_library(
+        "rocksdb_with_faiss_test_lib",
+        src_mk.get("MOCK_LIB_SOURCES", [])
+        + src_mk.get("TEST_LIB_SOURCES", [])
+        + src_mk.get("EXP_LIB_SOURCES", [])
+        + src_mk.get("ANALYZER_LIB_SOURCES", []),
+        deps=[
+            ":rocksdb_with_faiss_lib",
+        ],
+        extra_test_libs=True,
+    )
     # rocksdb_tools_lib
-    TARGETS.add_library(
+    BUCK.add_library(
         "rocksdb_tools_lib",
         src_mk.get("BENCH_LIB_SOURCES", [])
         + src_mk.get("ANALYZER_LIB_SOURCES", [])
@@ -180,51 +201,51 @@ def generate_targets(repo_path, deps_map):
         [":rocksdb_lib"],
     )
     # rocksdb_cache_bench_tools_lib
-    TARGETS.add_library(
+    BUCK.add_library(
         "rocksdb_cache_bench_tools_lib",
         src_mk.get("CACHE_BENCH_LIB_SOURCES", []),
         [":rocksdb_lib"],
     )
     # rocksdb_stress_lib
-    TARGETS.add_rocksdb_library(
+    BUCK.add_rocksdb_library(
         "rocksdb_stress_lib",
         src_mk.get("ANALYZER_LIB_SOURCES", [])
         + src_mk.get("STRESS_LIB_SOURCES", [])
         + ["test_util/testutil.cc"],
     )
     # ldb binary
-    TARGETS.add_binary(
+    BUCK.add_binary(
         "ldb", ["tools/ldb.cc"], [":rocksdb_tools_lib"]
     )
     # db_stress binary
-    TARGETS.add_binary(
+    BUCK.add_binary(
         "db_stress", ["db_stress_tool/db_stress.cc"], [":rocksdb_stress_lib"]
     )
     # db_bench binary
-    TARGETS.add_binary(
+    BUCK.add_binary(
         "db_bench", ["tools/db_bench.cc"], [":rocksdb_tools_lib"]
     )
     # cache_bench binary
-    TARGETS.add_binary(
+    BUCK.add_binary(
         "cache_bench", ["cache/cache_bench.cc"], [":rocksdb_cache_bench_tools_lib"]
     )
     # bench binaries
     for src in src_mk.get("MICROBENCH_SOURCES", []):
         name = src.rsplit("/", 1)[1].split(".")[0] if "/" in src else src.split(".")[0]
-        TARGETS.add_binary(name, [src], [], extra_bench_libs=True)
+        BUCK.add_binary(name, [src], [], extra_bench_libs=True)
     print(f"Extra dependencies:\n{json.dumps(deps_map)}")
 
     # Dictionary test executable name -> relative source file path
     test_source_map = {}
 
-    # c_test.c is added through TARGETS.add_c_test(). If there
+    # c_test.c is added through BUCK.add_c_test(). If there
     # are more than one .c test file, we need to extend
-    # TARGETS.add_c_test() to include other C tests too.
+    # BUCK.add_c_test() to include other C tests too.
     for test_src in src_mk.get("TEST_MAIN_SOURCES_C", []):
         if test_src != "db/c_test.c":
             print("Don't know how to deal with " + test_src)
             return False
-    TARGETS.add_c_test()
+    BUCK.add_c_test()
 
     try:
         with open(f"{repo_path}/buckifier/bench.json") as json_file:
@@ -239,7 +260,7 @@ def generate_targets(repo_path, deps_map):
                         for metric in overloaded_metric_list:
                             if not isinstance(metric, dict):
                                 clean_benchmarks[binary][benchmark].append(metric)
-                TARGETS.add_fancy_bench_config(
+                BUCK.add_fancy_bench_config(
                     config_dict["name"],
                     clean_benchmarks,
                     False,
@@ -261,7 +282,7 @@ def generate_targets(repo_path, deps_map):
                             if not isinstance(metric, dict):
                                 clean_benchmarks[binary][benchmark].append(metric)
             for config_dict in slow_fancy_bench_config_list:
-                TARGETS.add_fancy_bench_config(
+                BUCK.add_fancy_bench_config(
                     config_dict["name"] + "_slow",
                     clean_benchmarks,
                     True,
@@ -274,15 +295,20 @@ def generate_targets(repo_path, deps_map):
     except Exception:
         pass
 
-    TARGETS.add_test_header()
+    BUCK.add_test_header()
 
     for test_src in src_mk.get("TEST_MAIN_SOURCES", []):
         test = test_src.split(".c")[0].strip().split("/")[-1].strip()
-        test_source_map[test] = test_src
+        test_source_map[test] = (test_src, False)
         print("" + test + " " + test_src)
 
+    for test_src in src_mk.get("WITH_FAISS_TEST_MAIN_SOURCES", []):
+        test = test_src.split(".c")[0].strip().split("/")[-1].strip()
+        test_source_map[test] = (test_src, True)
+        print("" + test + " " + test_src + " [FAISS]")
+
     for target_alias, deps in deps_map.items():
-        for test, test_src in sorted(test_source_map.items()):
+        for test, (test_src, with_faiss) in sorted(test_source_map.items()):
             if len(test) == 0:
                 print(ColorString.warning("Failed to get test name for %s" % test_src))
                 continue
@@ -291,31 +317,39 @@ def generate_targets(repo_path, deps_map):
 
             if test in _EXPORTED_TEST_LIBS:
                 test_library = "%s_lib" % test_target_name
-                TARGETS.add_library(
+                BUCK.add_library(
                     test_library,
                     [test_src],
                     deps=[":rocksdb_test_lib"],
                     extra_test_libs=True,
                 )
-                TARGETS.register_test(
+                BUCK.register_test(
                     test_target_name,
                     test_src,
                     deps=json.dumps(deps["extra_deps"] + [":" + test_library]),
                     extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
                 )
             else:
-                TARGETS.register_test(
-                    test_target_name,
-                    test_src,
-                    deps=json.dumps(deps["extra_deps"] + [":rocksdb_test_lib"]),
-                    extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
-                )
-    TARGETS.export_file("tools/db_crashtest.py")
+                if with_faiss:
+                    BUCK.register_test(
+                        test_target_name,
+                        test_src,
+                        deps=json.dumps(deps["extra_deps"] + [":rocksdb_with_faiss_test_lib"]),
+                        extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
+                    )
+                else:
+                    BUCK.register_test(
+                        test_target_name,
+                        test_src,
+                        deps=json.dumps(deps["extra_deps"] + [":rocksdb_test_lib"]),
+                        extra_compiler_flags=json.dumps(deps["extra_compiler_flags"]),
+                    )
+    BUCK.export_file("tools/db_crashtest.py")
 
-    print(ColorString.info("Generated TARGETS Summary:"))
-    print(ColorString.info("- %d libs" % TARGETS.total_lib))
-    print(ColorString.info("- %d binarys" % TARGETS.total_bin))
-    print(ColorString.info("- %d tests" % TARGETS.total_test))
+    print(ColorString.info("Generated BUCK Summary:"))
+    print(ColorString.info("- %d libs" % BUCK.total_lib))
+    print(ColorString.info("- %d binarys" % BUCK.total_bin))
+    print(ColorString.info("- %d tests" % BUCK.total_test))
     return True
 
 
@@ -335,10 +369,10 @@ def exit_with_error(msg):
 
 def main():
     deps_map = get_dependencies()
-    # Generate TARGETS file for buck
-    ok = generate_targets(get_rocksdb_path(), deps_map)
+    # Generate BUCK file for buck
+    ok = generate_buck(get_rocksdb_path(), deps_map)
     if not ok:
-        exit_with_error("Failed to generate TARGETS files")
+        exit_with_error("Failed to generate BUCK files")
 
 
 if __name__ == "__main__":

buckifier/check_buck_targets.sh

@@ -1,32 +1,44 @@
 #!/usr/bin/env bash
 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
-# If clang_format_diff.py command is not specfied, we assume we are able to
-# access directly without any path.
 
-TGT_DIFF=`git diff TARGETS | head -n 1`
+if [[ ! -f "BUCK" ]]
+then
+  echo "BUCK file is missing!"
+  echo "Please do not remove / rename BUCK file in your commit(s)."
+  exit 1
+fi
+
+TGT_DIFF=`git diff BUCK | head -n 1`
 
 if [ ! -z "$TGT_DIFF" ]
 then
-  echo "TARGETS file has uncommitted changes. Skip this check."
+  echo "BUCK file has uncommitted changes. Skip this check."
   exit 0
 fi
 
-echo Backup original TARGETS file.
+echo Backup original BUCK file.
 
-cp TARGETS TARGETS.bkp
+cp BUCK BUCK.bkp
 
 ${PYTHON:-python3} buckifier/buckify_rocksdb.py
 
-TGT_DIFF=`git diff TARGETS | head -n 1`
+if [[ ! -f "BUCK" ]]
+then
+  echo "BUCK file went missing after running buckifier/buckify_rocksdb.py!"
+  echo "Please do not remove the BUCK file."
+  exit 1
+fi
+
+TGT_DIFF=`git diff BUCK | head -n 1`
 
 if [ -z "$TGT_DIFF" ]
 then
-  mv TARGETS.bkp TARGETS
+  mv BUCK.bkp BUCK
   exit 0
 else
-  echo "Please run '${PYTHON:-python3} buckifier/buckify_rocksdb.py' to update TARGETS file."
-  echo "Do not manually update TARGETS file."
+  echo "Please run '${PYTHON:-python3} buckifier/buckify_rocksdb.py' to update BUCK file."
+  echo "Do not manually update BUCK file."
   ${PYTHON:-python3} --version
-  mv TARGETS.bkp TARGETS
+  mv BUCK.bkp BUCK
   exit 1
 fi

db/column_family.cc

@@ -1208,7 +1208,7 @@ Status ColumnFamilyData::RangesOverlapWithMemtables(
   MergeIteratorBuilder merge_iter_builder(&internal_comparator_, &arena);
   merge_iter_builder.AddIterator(super_version->mem->NewIterator(
       read_opts, /*seqno_to_time_mapping=*/nullptr, &arena,
-      /*prefix_extractor=*/nullptr));
+      /*prefix_extractor=*/nullptr, /*for_flush=*/false));
   super_version->imm->AddIterators(read_opts, /*seqno_to_time_mapping=*/nullptr,
                                    /*prefix_extractor=*/nullptr,
                                    &merge_iter_builder,

db/column_family.h

@@ -388,10 +388,14 @@ class ColumnFamilyData {
   uint64_t GetTotalBlobFileSize() const;  // REQUIRE: DB mutex held
   // REQUIRE: DB mutex held
   void SetMemtable(MemTable* new_mem) {
-    new_mem->SetID(++last_memtable_id_);
+    AssignMemtableID(new_mem);
     mem_ = new_mem;
   }
 
+  void AssignMemtableID(ReadOnlyMemTable* new_imm) {
+    new_imm->SetID(++last_memtable_id_);
+  }
+
   // calculate the oldest log needed for the durability of this column family
   uint64_t OldestLogToKeep();
 

db/compaction/compaction_iterator.cc

@@ -872,8 +872,8 @@ void CompactionIterator::NextFromInput() {
       if (Valid()) {
         at_next_ = true;
       }
-    } else if (last_snapshot == current_user_key_snapshot_ ||
-               (last_snapshot > 0 &&
+    } else if (last_sequence != kMaxSequenceNumber &&
+               (last_snapshot == current_user_key_snapshot_ ||
                 last_snapshot < current_user_key_snapshot_)) {
       // If the earliest snapshot is which this key is visible in
       // is the same as the visibility of a previous instance of the

db/compaction/compaction_iterator_test.cc

@@ -833,6 +833,14 @@ TEST_P(CompactionIteratorTest, ConvertToPutAtBottom) {
           true /*bottomost_level*/);
 }
 
+TEST_P(CompactionIteratorTest, ZeroSeqOfKeyAndSnapshot) {
+  AddSnapshot(0);
+  const std::vector<std::string> input_keys = {
+      test::KeyStr("a", 0, kTypeValue), test::KeyStr("b", 0, kTypeValue)};
+  const std::vector<std::string> input_values = {"a1", "b1"};
+  RunTest(input_keys, input_values, input_keys, input_values);
+}
+
 INSTANTIATE_TEST_CASE_P(CompactionIteratorTestInstance, CompactionIteratorTest,
                         testing::Values(true, false));
 
@@ -1846,6 +1854,22 @@ TEST_P(CompactionIteratorTsGcTest, SingleDeleteAllKeysOlderThanThreshold) {
   }
 }
 
+TEST_P(CompactionIteratorTsGcTest, ZeroSeqOfKeyAndSnapshot) {
+  AddSnapshot(0);
+  std::string full_history_ts_low;
+  PutFixed64(&full_history_ts_low, std::numeric_limits<uint64_t>::max());
+  const std::vector<std::string> input_keys = {
+      test::KeyStr(101, "a", 0, kTypeValue),
+      test::KeyStr(102, "b", 0, kTypeValue)};
+  const std::vector<std::string> input_values = {"a1", "b1"};
+  RunTest(input_keys, input_values, input_keys, input_values,
+          /*last_committed_seq=*/kMaxSequenceNumber,
+          /*merge_operator=*/nullptr, /*compaction_filter=*/nullptr,
+          /*bottommost_level=*/false,
+          /*earliest_write_conflict_snapshot=*/kMaxSequenceNumber,
+          /*key_not_exists_beyond_output_level=*/false, &full_history_ts_low);
+}
+
 INSTANTIATE_TEST_CASE_P(CompactionIteratorTsGcTestInstance,
                         CompactionIteratorTsGcTest,
                         testing::Values(true, false));

db/compaction/compaction_job.cc

@@ -1586,6 +1586,8 @@ Status CompactionJob::FinishCompactionOutputFile(
   const uint64_t current_entries = outputs.NumEntries();
 
   s = outputs.Finish(s, seqno_to_time_mapping_);
+  TEST_SYNC_POINT_CALLBACK(
+      "CompactionJob::FinishCompactionOutputFile()::AfterFinish", &s);
 
   if (s.ok()) {
     // With accurate smallest and largest key, we can get a slightly more

db/compaction/compaction_service_job.cc

@@ -376,16 +376,18 @@ Status CompactionServiceCompactionJob::Run() {
   // Build Output
   compaction_result_->output_level = compact_->compaction->output_level();
   compaction_result_->output_path = output_path_;
-  for (const auto& output_file : sub_compact->GetOutputs()) {
-    auto& meta = output_file.meta;
-    compaction_result_->output_files.emplace_back(
-        MakeTableFileName(meta.fd.GetNumber()), meta.fd.smallest_seqno,
-        meta.fd.largest_seqno, meta.smallest.Encode().ToString(),
-        meta.largest.Encode().ToString(), meta.oldest_ancester_time,
-        meta.file_creation_time, meta.epoch_number, meta.file_checksum,
-        meta.file_checksum_func_name, output_file.validator.GetHash(),
-        meta.marked_for_compaction, meta.unique_id,
-        *output_file.table_properties);
+  if (status.ok()) {
+    for (const auto& output_file : sub_compact->GetOutputs()) {
+      auto& meta = output_file.meta;
+      compaction_result_->output_files.emplace_back(
+          MakeTableFileName(meta.fd.GetNumber()), meta.fd.smallest_seqno,
+          meta.fd.largest_seqno, meta.smallest.Encode().ToString(),
+          meta.largest.Encode().ToString(), meta.oldest_ancester_time,
+          meta.file_creation_time, meta.epoch_number, meta.file_checksum,
+          meta.file_checksum_func_name, output_file.validator.GetHash(),
+          meta.marked_for_compaction, meta.unique_id,
+          *output_file.table_properties);
+    }
   }
 
   TEST_SYNC_POINT_CALLBACK("CompactionServiceCompactionJob::Run:0",

db/compaction/compaction_service_test.cc

@@ -416,6 +416,38 @@ TEST_F(CompactionServiceTest, ManualCompaction) {
   ASSERT_TRUE(result.stats.is_remote_compaction);
 }
 
+TEST_F(CompactionServiceTest, CompactionOutputFileIOError) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  ReopenWithCompactionService(&options);
+  GenerateTestData();
+
+  auto my_cs = GetCompactionService();
+
+  SyncPoint::GetInstance()->SetCallBack(
+      "CompactionJob::FinishCompactionOutputFile()::AfterFinish",
+      [&](void* status) {
+        // override status
+        auto s = static_cast<Status*>(status);
+        *s = Status::IOError("Injected IOError!");
+      });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  std::string start_str = Key(15);
+  std::string end_str = Key(45);
+  Slice start(start_str);
+  Slice end(end_str);
+  uint64_t comp_num = my_cs->GetCompactionNum();
+  ASSERT_NOK(db_->CompactRange(CompactRangeOptions(), &start, &end));
+  ASSERT_GE(my_cs->GetCompactionNum(), comp_num + 1);
+
+  CompactionServiceResult result;
+  my_cs->GetResult(&result);
+  ASSERT_NOK(result.status);
+  ASSERT_TRUE(result.stats.is_manual_compaction);
+  ASSERT_TRUE(result.stats.is_remote_compaction);
+}
+
 TEST_F(CompactionServiceTest, PreservedOptionsLocalCompaction) {
   Options options = CurrentOptions();
   options.level0_file_num_compaction_trigger = 2;

db/db_impl/db_impl.cc

@@ -2098,7 +2098,8 @@ InternalIterator* DBImpl::NewInternalIterator(
   // Collect iterator for mutable memtable
   auto mem_iter = super_version->mem->NewIterator(
       read_options, super_version->GetSeqnoToTimeMapping(), arena,
-      super_version->mutable_cf_options.prefix_extractor.get());
+      super_version->mutable_cf_options.prefix_extractor.get(),
+      /*for_flush=*/false);
   Status s;
   if (!read_options.ignore_range_deletions) {
     std::unique_ptr<TruncatedRangeDelIterator> mem_tombstone_iter;
@@ -3247,8 +3248,13 @@ Status DBImpl::MultiGetImpl(
       s = Status::Aborted();
       break;
     }
+
     // This could be a long-running operation
-    ROCKSDB_THREAD_YIELD_HOOK();
+    bool aborted = ROCKSDB_THREAD_YIELD_CHECK_ABORT();
+    if (aborted) {
+      s = Status::Aborted("Query abort.");
+      break;
+    }
   }
 
   // Post processing (decrement reference counts and record statistics)
@@ -3802,7 +3808,6 @@ bool DBImpl::KeyMayExist(const ReadOptions& read_options,
                          ColumnFamilyHandle* column_family, const Slice& key,
                          std::string* value, std::string* timestamp,
                          bool* value_found) {
-  assert(value != nullptr);
   assert(read_options.io_activity == Env::IOActivity::kUnknown);
 
   if (value_found != nullptr) {
@@ -3819,7 +3824,9 @@ bool DBImpl::KeyMayExist(const ReadOptions& read_options,
   get_impl_options.value_found = value_found;
   get_impl_options.timestamp = timestamp;
   auto s = GetImpl(roptions, key, get_impl_options);
-  value->assign(pinnable_val.data(), pinnable_val.size());
+  if (value_found && *value_found && value) {
+    value->assign(pinnable_val.data(), pinnable_val.size());
+  }
 
   // If block_cache is enabled and the index block of the table didn't
   // not present in block_cache, the return value will be Status::Incomplete.
@@ -5912,8 +5919,8 @@ Status DBImpl::IngestExternalFiles(
   const size_t num_cfs = args.size();
   for (size_t i = 0; i != num_cfs; ++i) {
     if (args[i].external_files.empty()) {
-      char err_msg[128] = {0};
-      snprintf(err_msg, 128, "external_files[%zu] is empty", i);
+      std::string err_msg =
+          "external_files[" + std::to_string(i) + "] is empty";
       return Status::InvalidArgument(err_msg);
     }
     if (i && args[i].options.fill_cache != args[i - 1].options.fill_cache) {

db/db_impl/db_impl.h

@@ -48,6 +48,7 @@
 #include "db/write_controller.h"
 #include "db/write_thread.h"
 #include "logging/event_logger.h"
+#include "memtable/wbwi_memtable.h"
 #include "monitoring/instrumented_mutex.h"
 #include "options/db_options.h"
 #include "port/port.h"
@@ -60,6 +61,7 @@
 #include "rocksdb/transaction_log.h"
 #include "rocksdb/user_write_callback.h"
 #include "rocksdb/utilities/replayer.h"
+#include "rocksdb/utilities/write_batch_with_index.h"
 #include "rocksdb/write_buffer_manager.h"
 #include "table/merging_iterator.h"
 #include "util/autovector.h"
@@ -1507,6 +1509,23 @@ class DBImpl : public DB {
 
   void EraseThreadStatusDbInfo() const;
 
+  // For CFs that has updates in `wbwi`, their memtable will be switched,
+  // and `wbwi` will be added as the latest immutable memtable.
+  //
+  // REQUIRES: this thread is currently at the front of the main writer queue.
+  // @param prep_log refers to the WAL that contains prepare record
+  // for the transaction based on wbwi.
+  // @param assigned_seqno Sequence numbers for the ingested memtable.
+  // @param last_seqno the value of versions_->LastSequence() after the write
+  // ingests `wbwi` is done.
+  // @param memtable_updated Whether the same write that ingests wbwi has
+  // updated memtable. This is useful for determining whether to set bg
+  // error when IngestWBWI fails.
+  Status IngestWBWI(std::shared_ptr<WriteBatchWithIndex> wbwi,
+                    const WBWIMemTable::SeqnoRange& assigned_seqno,
+                    uint64_t min_prep_log, SequenceNumber last_seqno,
+                    bool memtable_updated, bool ignore_missing_cf);
+
   // If disable_memtable is set the application logic must guarantee that the
   // batch will still be skipped from memtable during the recovery. An excption
   // to this is seq_per_batch_ mode, in which since each batch already takes one
@@ -1522,6 +1541,16 @@ class DBImpl : public DB {
   // batch_cnt is expected to be non-zero in seq_per_batch mode and
   // indicates the number of sub-patches. A sub-patch is a subset of the write
   // batch that does not have duplicate keys.
+  // `callback` is called before WAL write.
+  // See more in comment above WriteCallback::Callback().
+  // pre_release_callback is called after WAL write and before memtable write.
+  // See more in comment above PreReleaseCallback::Callback().
+  // post_memtable_callback is called after memtable write but before publishing
+  // the sequence number to readers.
+  //
+  // The main write queue. This is the only write queue that updates
+  // LastSequence. When using one write queue, the same sequence also indicates
+  // the last published sequence.
   Status WriteImpl(const WriteOptions& options, WriteBatch* updates,
                    WriteCallback* callback = nullptr,
                    UserWriteCallback* user_write_cb = nullptr,
@@ -1529,7 +1558,9 @@ class DBImpl : public DB {
                    bool disable_memtable = false, uint64_t* seq_used = nullptr,
                    size_t batch_cnt = 0,
                    PreReleaseCallback* pre_release_callback = nullptr,
-                   PostMemTableCallback* post_memtable_callback = nullptr);
+                   PostMemTableCallback* post_memtable_callback = nullptr,
+                   std::shared_ptr<WriteBatchWithIndex> wbwi = nullptr,
+                   uint64_t min_prep_log = 0);
 
   Status PipelinedWriteImpl(const WriteOptions& options, WriteBatch* updates,
                             WriteCallback* callback = nullptr,
@@ -2053,7 +2084,19 @@ class DBImpl : public DB {
 
   // Switches the current live memtable to immutable/read-only memtable.
   // A new WAL is created if the current WAL is not empty.
-  Status SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context);
+  // If `new_imm` is not nullptr, it will be added as the newest immutable
+  // memtable, if and only if OK status is returned.
+  // `last_seqno` needs to be provided if `new_imm` is not nullptr. It is
+  // the value of versions_->LastSequence() after the write that ingests new_imm
+  // is done.
+  //
+  // REQUIRES: mutex_ is held
+  // REQUIRES: this thread is currently at the front of the writer queue
+  // REQUIRES: this thread is currently at the front of the 2nd writer queue if
+  // two_write_queues_ is true (This is to simplify the reasoning.)
+  Status SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
+                        ReadOnlyMemTable* new_imm = nullptr,
+                        SequenceNumber last_seqno = 0);
 
   // Select and output column families qualified for atomic flush in
   // `selected_cfds`. If `provided_candidate_cfds` is non-empty, it will be used

db/db_impl/db_impl_debug.cc

@@ -6,8 +6,8 @@
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
-
 #ifndef NDEBUG
+#include <iostream>
 
 #include "db/blob/blob_file_cache.h"
 #include "db/column_family.h"
@@ -379,7 +379,11 @@ void DBImpl::TEST_VerifyNoObsoleteFilesCached(
     uint64_t file_number;
     GetUnaligned(reinterpret_cast<const uint64_t*>(key.data()), &file_number);
     // Assert file is in live/quarantined set
-    assert(live_and_quar_files.find(file_number) != live_and_quar_files.end());
+    if (live_and_quar_files.find(file_number) == live_and_quar_files.end()) {
+      std::cerr << "File " << file_number << " is not live nor quarantined"
+                << std::endl;
+      assert(false);
+    }
   };
   table_cache_->ApplyToAllEntries(fn, {});
 }

db/db_impl/db_impl_open.cc

@@ -1373,6 +1373,9 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& wal_numbers,
         }
       }
     }
+    ROCKS_LOG_INFO(immutable_db_options_.info_log,
+                   "Recovered to log #%" PRIu64 " seq #%" PRIu64, wal_number,
+                   *next_sequence);
 
     if (!status.ok() || old_log_record) {
       if (status.IsNotSupported()) {
@@ -1405,10 +1408,6 @@ Status DBImpl::RecoverLogFiles(const std::vector<uint64_t>& wal_numbers,
         if (corrupted_wal_found != nullptr) {
           *corrupted_wal_found = true;
         }
-        ROCKS_LOG_INFO(immutable_db_options_.info_log,
-                       "Point in time recovered to log #%" PRIu64
-                       " seq #%" PRIu64,
-                       wal_number, *next_sequence);
       } else {
         assert(immutable_db_options_.wal_recovery_mode ==
                    WALRecoveryMode::kTolerateCorruptedTailRecords ||
@@ -1681,7 +1680,8 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
                   ro, /*seqno_to_time_mapping=*/nullptr, &arena,
                   /*prefix_extractor=*/nullptr, ts_sz)
             : mem->NewIterator(ro, /*seqno_to_time_mapping=*/nullptr, &arena,
-                               /*prefix_extractor=*/nullptr));
+                               /*prefix_extractor=*/nullptr,
+                               /*for_flush=*/true));
     ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
                     "[%s] [WriteLevel0TableForRecovery]"
                     " Level-0 table #%" PRIu64 ": started",

db/db_impl/db_impl_readonly.cc

@@ -8,7 +8,7 @@
 #include "db/arena_wrapped_db_iter.h"
 #include "db/db_impl/compacted_db_impl.h"
 #include "db/db_impl/db_impl.h"
-#include "db/db_iter.h"
+#include "db/manifest_ops.h"
 #include "db/merge_context.h"
 #include "logging/logging.h"
 #include "monitoring/perf_context_imp.h"
@@ -265,9 +265,8 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
     const std::shared_ptr<FileSystem>& fs = db_options.env->GetFileSystem();
     std::string manifest_path;
     uint64_t manifest_file_number;
-    s = VersionSet::GetCurrentManifestPath(dbname, fs.get(), /*is_retry=*/false,
-                                           &manifest_path,
-                                           &manifest_file_number);
+    s = GetCurrentManifestPath(dbname, fs.get(), /*is_retry=*/false,
+                               &manifest_path, &manifest_file_number);
   } else {
     // Historic behavior that doesn't necessarily make sense
     s = db_options.env->CreateDirIfMissing(dbname);

db/db_impl/db_impl_write.cc

@@ -12,6 +12,7 @@
 #include "db/error_handler.h"
 #include "db/event_helpers.h"
 #include "logging/logging.h"
+#include "memtable/wbwi_memtable.h"
 #include "monitoring/perf_context_imp.h"
 #include "options/options_helper.h"
 #include "test_util/sync_point.h"
@@ -189,16 +190,137 @@ Status DBImpl::WriteWithCallback(const WriteOptions& write_options,
   return s;
 }
 
-// The main write queue. This is the only write queue that updates LastSequence.
-// When using one write queue, the same sequence also indicates the last
-// published sequence.
+Status DBImpl::IngestWBWI(std::shared_ptr<WriteBatchWithIndex> wbwi,
+                          const WBWIMemTable::SeqnoRange& assigned_seqno,
+                          uint64_t prep_log,
+                          SequenceNumber last_seqno_after_ingest,
+                          bool memtable_updated, bool ignore_missing_cf) {
+  // Keys in new memtable have seqno > last_seqno_after_ingest >= keys in wbwi.
+  assert(assigned_seqno.upper_bound <= last_seqno_after_ingest);
+  // Keys in the current memtable have seqno <= LastSequence() < keys in wbwi.
+  assert(assigned_seqno.lower_bound > versions_->LastSequence());
+  autovector<ReadOnlyMemTable*> memtables;
+  autovector<ColumnFamilyData*> cfds;
+  InstrumentedMutexLock lock(&mutex_);
+  ColumnFamilySet* cf_set = versions_->GetColumnFamilySet();
+
+  // Create WBWIMemTables
+  for (const auto [cf_id, stat] : wbwi->GetCFStats()) {
+    ColumnFamilyData* cfd = cf_set->GetColumnFamily(cf_id);
+    if (!cfd) {
+      if (ignore_missing_cf) {
+        continue;
+      }
+      for (auto mem : memtables) {
+        mem->Unref();
+        delete mem;
+      }
+      for (auto cfd_ptr : cfds) {
+        cfd_ptr->UnrefAndTryDelete();
+      }
+      Status s = Status::InvalidArgument(
+          "Invalid column family id from WriteBatchWithIndex: " +
+          std::to_string(cf_id));
+      if (memtable_updated) {
+        s = Status::Corruption(
+            "Part of the write batch is applied. Memtable is in a inconsistent "
+            "state. " +
+            s.ToString());
+        error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
+      }
+
+      return s;
+    }
+    WBWIMemTable* wbwi_memtable =
+        new WBWIMemTable(wbwi, cfd->user_comparator(), cf_id, cfd->ioptions(),
+                         cfd->GetLatestMutableCFOptions(), stat);
+    wbwi_memtable->Ref();
+    wbwi_memtable->AssignSequenceNumbers(assigned_seqno);
+    // This is needed to keep the WAL that contains Prepare alive until
+    // committed data in this memtable is persisted.
+    wbwi_memtable->SetMinPrepLog(prep_log);
+    memtables.push_back(wbwi_memtable);
+    cfd->Ref();
+    cfds.push_back(cfd);
+  }
+
+  // Stop writes to the DB by entering both write threads
+  WriteThread::Writer nonmem_w;
+  if (two_write_queues_) {
+    nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
+  }
+  WaitForPendingWrites();
+
+  // Switch memtable and add WBWIMemTables
+  Status s;
+  for (size_t i = 0; i < memtables.size(); ++i) {
+    assert(!immutable_db_options_.atomic_flush);
+    // NOTE: to support atomic flush, need to call
+    // SelectColumnFamiliesForAtomicFlush()
+    WriteContext write_context;
+    // TODO: not switch on empty memtable, may need to update metadata
+    //   like NextLogNumber(), earliest_seqno and memtable id.
+    s = SwitchMemtable(cfds[i], &write_context, memtables[i],
+                       last_seqno_after_ingest);
+    if (!s.ok()) {
+      // SwitchMemtable() can only fail if a new WAL is to be created, this
+      // should only happen for the first call to SwitchMemtable(). log will
+      // be empty and no new WAL is created for the rest of the calls.
+      assert(i == 0);
+      if (i != 0 || memtable_updated) {
+        // escalate error to non-recoverable
+        s = Status::Corruption(
+            "Part of the write batch is applied. Memtable is in a inconsistent "
+            "state. " +
+            s.ToString());
+        error_handler_.SetBGError(s, BackgroundErrorReason::kMemTable);
+      } else {
+        // SwitchMemtable() already sets appropriate bg error
+      }
+      for (size_t j = i; j < memtables.size(); j++) {
+        memtables[j]->Unref();
+        delete memtables[j];
+      }
+      break;
+    }
+  }
+  for (size_t i = 0; i < cfds.size(); ++i) {
+    if (cfds[i]->UnrefAndTryDelete()) {
+      cfds[i] = nullptr;
+    }
+  }
+
+  // exit the second queue before returning
+  if (two_write_queues_) {
+    nonmem_write_thread_.ExitUnbatched(&nonmem_w);
+  }
+  if (s.ok()) {
+    // Trigger flushes for the new immutable memtables.
+    for (const auto cfd : cfds) {
+      if (cfd == nullptr) {
+        continue;
+      }
+      cfd->imm()->FlushRequested();
+      FlushRequest flush_req;
+      // TODO: a new flush reason for ingesting memtable
+      GenerateFlushRequest({cfd}, FlushReason::kExternalFileIngestion,
+                           &flush_req);
+      EnqueuePendingFlush(flush_req);
+    }
+    MaybeScheduleFlushOrCompaction();
+  }
+  return s;
+}
+
 Status DBImpl::WriteImpl(const WriteOptions& write_options,
                          WriteBatch* my_batch, WriteCallback* callback,
                          UserWriteCallback* user_write_cb, uint64_t* log_used,
                          uint64_t log_ref, bool disable_memtable,
                          uint64_t* seq_used, size_t batch_cnt,
                          PreReleaseCallback* pre_release_callback,
-                         PostMemTableCallback* post_memtable_callback) {
+                         PostMemTableCallback* post_memtable_callback,
+                         std::shared_ptr<WriteBatchWithIndex> wbwi,
+                         uint64_t prep_log) {
   assert(!seq_per_batch_ || batch_cnt != 0);
   assert(my_batch == nullptr || my_batch->Count() == 0 ||
          write_options.protection_bytes_per_key == 0 ||
@@ -287,6 +409,23 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
     return Status::NotSupported(
         "DeleteRange is not compatible with row cache.");
   }
+  if (wbwi) {
+    assert(prep_log > 0);
+    // Used only in WriteCommittedTxn::CommitInternal() with no `callback`.
+    assert(!callback);
+    if (immutable_db_options_.unordered_write) {
+      return Status::NotSupported(
+          "Ingesting WriteBatch does not support unordered_write");
+    }
+    if (immutable_db_options_.enable_pipelined_write) {
+      return Status::NotSupported(
+          "Ingesting WriteBatch does not support pipelined_write");
+    }
+    if (immutable_db_options_.atomic_flush) {
+      return Status::NotSupported(
+          "Ingesting WriteBatch does not support atomic_flush");
+    }
+  }
   // Otherwise IsLatestPersistentState optimization does not make sense
   assert(!WriteBatchInternal::IsLatestPersistentState(my_batch) ||
          disable_memtable);
@@ -344,7 +483,8 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
   PERF_TIMER_GUARD(write_pre_and_post_process_time);
   WriteThread::Writer w(write_options, my_batch, callback, user_write_cb,
                         log_ref, disable_memtable, batch_cnt,
-                        pre_release_callback, post_memtable_callback);
+                        pre_release_callback, post_memtable_callback,
+                        /*_ingest_wbwi=*/wbwi != nullptr);
   StopWatch write_sw(immutable_db_options_.clock, stats_, DB_WRITE);
 
   write_thread_.JoinBatchGroup(&w);
@@ -441,6 +581,9 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
   TEST_SYNC_POINT("DBImpl::WriteImpl:BeforeLeaderEnters");
   last_batch_group_size_ =
       write_thread_.EnterAsBatchGroupLeader(&w, &write_group);
+  if (wbwi) {
+    assert(write_group.size == 1);
+  }
 
   IOStatus io_s;
   Status pre_release_cb_status;
@@ -494,10 +637,25 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
     // Note about seq_per_batch_: either disableWAL is set for the entire write
     // group or not. In either case we inc seq for each write batch with no
     // failed callback. This means that there could be a batch with
-    // disalbe_memtable in between; although we do not write this batch to
+    // disable_memtable in between; although we do not write this batch to
     // memtable it still consumes a seq. Otherwise, if !seq_per_batch_, we inc
     // the seq per valid written key to mem.
     size_t seq_inc = seq_per_batch_ ? valid_batches : total_count;
+    if (wbwi) {
+      // Reserve sequence numbers for the ingested memtable. We need to reserve
+      // at lease this amount for recovery. During recovery,
+      // transactions do not commit by ingesting WBWI. The sequence number
+      // associated with the commit entry in WAL is used as the starting
+      // sequence number for inserting into memtable. We need to reserve
+      // enough sequence numbers here (at least the number of operations
+      // in write batch) to assign to memtable entries for this transaction.
+      // This prevents updates in different transactions from using out-of-order
+      // sequence numbers or the same key+seqno.
+      //
+      // WBWI ingestion requires not grouping writes, so we don't need to
+      // consider incrementing sequence number for WBWI from other writers.
+      seq_inc += wbwi->GetWriteBatch()->Count();
+    }
 
     const bool concurrent_update = two_write_queues_;
     // Update stats while we are an exclusive group leader, so we know
@@ -674,6 +832,27 @@ Status DBImpl::WriteImpl(const WriteOptions& write_options,
     // handle exit, false means somebody else did
     should_exit_batch_group = write_thread_.CompleteParallelMemTableWriter(&w);
   }
+  if (wbwi) {
+    if (status.ok() && w.status.ok()) {
+      // w.batch contains (potentially empty) commit time batch updates,
+      // only ingest wbwi if w.batch is applied to memtable successfully
+      assert(wbwi->GetWriteBatch()->Count() > 0);
+
+      uint32_t memtable_update_count = w.batch->Count();
+      SequenceNumber lb = versions_->LastSequence() + memtable_update_count + 1;
+      SequenceNumber ub = versions_->LastSequence() + memtable_update_count +
+                          wbwi->GetWriteBatch()->Count();
+      assert(ub == last_sequence);
+      if (two_write_queues_) {
+        assert(ub <= versions_->LastAllocatedSequence());
+      }
+      status = IngestWBWI(wbwi, {/*lower_bound=*/lb, /*upper_bound=*/ub},
+                          prep_log, last_sequence,
+                          /*memtable_updated=*/memtable_update_count > 0,
+                          write_options.ignore_missing_column_families);
+    }
+  }
+
   if (should_exit_batch_group) {
     if (status.ok()) {
       for (auto* tmp_w : write_group) {
@@ -1210,7 +1389,6 @@ void DBImpl::MemTableInsertStatusCheck(const Status& status) {
   if (!status.ok()) {
     mutex_.Lock();
     assert(!error_handler_.IsBGWorkStopped());
-    // Maybe change the return status to void?
     error_handler_.SetBGError(status, BackgroundErrorReason::kMemTable);
     mutex_.Unlock();
   }
@@ -2204,16 +2382,13 @@ void DBImpl::NotifyOnMemTableSealed(ColumnFamilyData* /*cfd*/,
   mutex_.Lock();
 }
 
-// REQUIRES: mutex_ is held
-// REQUIRES: this thread is currently at the front of the writer queue
-// REQUIRES: this thread is currently at the front of the 2nd writer queue if
-// two_write_queues_ is true (This is to simplify the reasoning.)
-Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
+Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context,
+                              ReadOnlyMemTable* new_imm,
+                              SequenceNumber last_seqno) {
   mutex_.AssertHeld();
   assert(lock_wal_count_ == 0);
 
   // TODO: plumb Env::IOActivity, Env::IOPriority
-  const ReadOptions read_options;
   const WriteOptions write_options;
 
   log::Writer* new_log = nullptr;
@@ -2249,6 +2424,7 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
   const MutableCFOptions mutable_cf_options = *cfd->GetLatestMutableCFOptions();
 
   // Set memtable_info for memtable sealed callback
+  // TODO: memtable_info for `new_imm`
   MemTableInfo memtable_info;
   memtable_info.cf_name = cfd->GetName();
   memtable_info.first_seqno = cfd->mem()->GetFirstSequenceNumber();
@@ -2276,8 +2452,20 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
     }
   }
   if (s.ok()) {
-    SequenceNumber seq = versions_->LastSequence();
-    new_mem = cfd->ConstructNewMemtable(mutable_cf_options, seq);
+    // FIXME: from the comment for GetEarliestSequenceNumber(), any key with
+    //  seqno >= earliest_seqno should be in this or later memtable. This means
+    //  we should use LastSequence() + 1 or last_seqno + 1 here. And it needs to
+    //  be incremented with file ingestion and other operations that consumes
+    //  sequence number.
+    SequenceNumber seq;
+    if (new_imm) {
+      assert(last_seqno > versions_->LastSequence());
+      seq = last_seqno;
+    } else {
+      seq = versions_->LastSequence();
+    }
+    new_mem =
+        cfd->ConstructNewMemtable(mutable_cf_options, /*earliest_seq=*/seq);
     context->superversion_context.NewSuperVersion();
 
     ROCKS_LOG_INFO(immutable_db_options_.info_log,
@@ -2359,6 +2547,8 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
         versions_->PreComputeMinLogNumberWithUnflushedData(logfile_number_);
     if (min_wal_number_to_keep >
         versions_->GetWalSet().GetMinWalNumberToKeep()) {
+      // TODO: plumb Env::IOActivity, Env::IOPriority
+      const ReadOptions read_options;
       // Get a snapshot of the empty column families.
       // LogAndApply may release and reacquire db
       // mutex, during that period, column family may become empty (e.g. its
@@ -2416,6 +2606,18 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
   cfd->mem()->SetNextLogNumber(logfile_number_);
   assert(new_mem != nullptr);
   cfd->imm()->Add(cfd->mem(), &context->memtables_to_free_);
+  if (new_imm) {
+    // Need to assign memtable id here before SetMemtable() below assigns id to
+    // the new live memtable
+    cfd->AssignMemtableID(new_imm);
+    // NOTE: new_imm and cfd->mem() references the same WAL and has the same
+    // NextLogNumber(). They should be flushed together. For non-atomic-flush,
+    // we always try to flush all immutable memtable. For atomic flush, these
+    // two memtables will be marked eligible for flush in the same call to
+    // AssignAtomicFlushSeq().
+    new_imm->SetNextLogNumber(logfile_number_);
+    cfd->imm()->Add(new_imm, &context->memtables_to_free_);
+  }
   new_mem->Ref();
   cfd->SetMemtable(new_mem);
   InstallSuperVersionAndScheduleWork(cfd, &context->superversion_context,
@@ -2428,6 +2630,9 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
   // that is okay.  If we did, it most likely means that s was already an error.
   // In any case, ignore any unchecked error for i_os here.
   io_s.PermitUncheckedError();
+  // We guarantee that if a non-ok status is returned, `new_imm` was not added
+  // to the db.
+  assert(s.ok());
   return s;
 }
 

db/db_iter.cc

@@ -581,8 +581,14 @@ bool DBIter::FindNextUserEntryInternal(bool skipping_saved_key,
     } else {
       iter_.Next();
     }
+
     // This could be a long-running operation due to tombstones, etc.
-    ROCKSDB_THREAD_YIELD_HOOK();
+    bool aborted = ROCKSDB_THREAD_YIELD_CHECK_ABORT();
+    if (aborted) {
+      valid_ = false;
+      status_ = Status::Aborted("Query abort.");
+      return false;
+    }
   } while (iter_.Valid());
 
   valid_ = false;

db/db_test2.cc

@@ -16,6 +16,7 @@
 #include "db/db_test_util.h"
 #include "db/read_callback.h"
 #include "db/version_edit.h"
+#include "env/fs_readonly.h"
 #include "options/options_helper.h"
 #include "port/port.h"
 #include "port/stack_trace.h"
@@ -2103,16 +2104,15 @@ TEST_P(PinL0IndexAndFilterBlocksTest,
   ASSERT_EQ(2, TestGetTickerCount(options, BLOCK_CACHE_ADD));
   ASSERT_EQ(0, TestGetTickerCount(options, BLOCK_CACHE_DATA_MISS));
 
-  std::string value;
   // Miss and hit count should remain the same, they're all pinned.
-  ASSERT_TRUE(db_->KeyMayExist(ReadOptions(), handles_[1], "key", &value));
+  ASSERT_TRUE(db_->KeyMayExist(ReadOptions(), handles_[1], "key", nullptr));
   ASSERT_EQ(1, TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS));
   ASSERT_EQ(0, TestGetTickerCount(options, BLOCK_CACHE_FILTER_HIT));
   ASSERT_EQ(1, TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS));
   ASSERT_EQ(0, TestGetTickerCount(options, BLOCK_CACHE_INDEX_HIT));
 
   // Miss and hit count should remain the same, they're all pinned.
-  value = Get(1, "key");
+  std::string value = Get(1, "key");
   ASSERT_EQ(1, TestGetTickerCount(options, BLOCK_CACHE_FILTER_MISS));
   ASSERT_EQ(0, TestGetTickerCount(options, BLOCK_CACHE_FILTER_HIT));
   ASSERT_EQ(1, TestGetTickerCount(options, BLOCK_CACHE_INDEX_MISS));
@@ -8062,6 +8062,63 @@ TEST_F(DBTest2, TableCacheMissDuringReadFromBlockCacheTier) {
   ASSERT_EQ(orig_num_file_opens, TestGetTickerCount(options, NO_FILE_OPENS));
 }
 
+TEST_F(DBTest2, GetFileChecksumsFromCurrentManifest_CRC32) {
+  Options opts = CurrentOptions();
+  opts.create_if_missing = true;
+  opts.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
+  opts.level0_file_num_compaction_trigger = 10;
+
+  // Bootstrap the test database.
+  DB* db = nullptr;
+  std::string dbname = test::PerThreadDBPath("file_chksum");
+  ASSERT_OK(DB::Open(opts, dbname, &db));
+
+  WriteOptions wopts;
+  FlushOptions fopts;
+  fopts.wait = true;
+  Random rnd(test::RandomSeed());
+  for (int i = 0; i < 4; i++) {
+    ASSERT_OK(db->Put(wopts, Key(i), rnd.RandomString(100)));
+    ASSERT_OK(db->Flush(fopts));
+  }
+
+  // Obtain rich files metadata for source of truth.
+  std::vector<LiveFileMetaData> live_files;
+  db->GetLiveFilesMetaData(&live_files);
+
+  ASSERT_OK(db->Close());
+  delete db;
+  db = nullptr;
+
+  // Process current MANIFEST file and build internal file checksum mappings.
+  std::unique_ptr<FileChecksumList> checksum_list(NewFileChecksumList());
+  auto read_only_fs =
+      std::make_shared<ReadOnlyFileSystem>(env_->GetFileSystem());
+  ASSERT_OK(experimental::GetFileChecksumsFromCurrentManifest(
+      read_only_fs.get(), dbname, checksum_list.get()));
+
+  ASSERT_TRUE(checksum_list != nullptr);
+
+  // Retrieve files, related checksums and checksum functions.
+  std::vector<uint64_t> file_numbers;
+  std::vector<std::string> checksums;
+  std::vector<std::string> checksum_func_names;
+  ASSERT_OK(checksum_list->GetAllFileChecksums(&file_numbers, &checksums,
+                                               &checksum_func_names));
+
+  // Compare results.
+  ASSERT_EQ(live_files.size(), checksum_list->size());
+  for (size_t i = 0; i < live_files.size(); i++) {
+    std::string stored_checksum;
+    std::string stored_func_name;
+    ASSERT_OK(checksum_list->SearchOneFileChecksum(
+        live_files[i].file_number, &stored_checksum, &stored_func_name));
+
+    ASSERT_EQ(live_files[i].file_checksum, stored_checksum);
+    ASSERT_EQ(live_files[i].file_checksum_func_name, stored_func_name);
+  }
+}
+
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/db_test_util.cc

@@ -1600,42 +1600,74 @@ std::vector<std::uint64_t> DBTestBase::ListTableFiles(Env* env,
   return file_numbers;
 }
 
-void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
-                                 size_t* total_reads_res, bool tailing_iter,
-                                 std::map<std::string, Status> status) {
-  size_t total_reads = 0;
+void DBTestBase::VerifyDBFromMap(
+    std::map<std::string, std::string> true_data, size_t* total_reads_res,
+    bool tailing_iter, ReadOptions* ro, ColumnFamilyHandle* cf,
+    std::unordered_set<std::string>* not_found) const {
+  ReadOptions temp_ro;
+  if (!ro) {
+    ro = &temp_ro;
+    ro->verify_checksums = true;
+  }
+  if (!cf) {
+    cf = db_->DefaultColumnFamily();
+  }
 
-  for (auto& kv : true_data) {
-    Status s = status[kv.first];
-    if (s.ok()) {
-      ASSERT_EQ(Get(kv.first), kv.second);
-    } else {
-      std::string value;
-      ASSERT_EQ(s, db_->Get(ReadOptions(), kv.first, &value));
-    }
+  // Get
+  size_t total_reads = 0;
+  std::string result;
+  for (auto& [k, v] : true_data) {
+    ASSERT_OK(db_->Get(*ro, cf, k, &result)) << "key is " << k;
+    ASSERT_EQ(v, result);
     total_reads++;
   }
+  if (not_found) {
+    for (const auto& k : *not_found) {
+      ASSERT_TRUE(db_->Get(*ro, cf, k, &result).IsNotFound())
+          << "key is " << k << " val is " << result;
+    }
+  }
+
+  // MultiGet
+  std::vector<Slice> key_slice;
+  for (const auto& [k, _] : true_data) {
+    key_slice.emplace_back(k);
+  }
+  std::vector<std::string> values;
+  std::vector<ColumnFamilyHandle*> cfs(key_slice.size(), cf);
+  std::vector<Status> status = db_->MultiGet(*ro, cfs, key_slice, &values);
+  total_reads += key_slice.size();
+  auto data_iter = true_data.begin();
+  for (size_t i = 0; i < key_slice.size(); ++i, ++data_iter) {
+    ASSERT_OK(status[i]);
+    ASSERT_EQ(values[i], data_iter->second);
+  }
+  // MultiGet - not found
+  if (not_found) {
+    key_slice.clear();
+    for (const auto& k : *not_found) {
+      key_slice.emplace_back(k);
+    }
+    cfs = std::vector<ColumnFamilyHandle*>(key_slice.size(), cf);
+    values.clear();
+    status = db_->MultiGet(*ro, cfs, key_slice, &values);
+    for (const auto& s : status) {
+      ASSERT_TRUE(s.IsNotFound());
+    }
+  }
 
   // Normal Iterator
   {
     int iter_cnt = 0;
-    ReadOptions ro;
-    ro.total_order_seek = true;
-    Iterator* iter = db_->NewIterator(ro);
+    ReadOptions ro_ = *ro;
+    ro_.total_order_seek = true;
+    Iterator* iter = db_->NewIterator(ro_, cf);
     // Verify Iterator::Next()
     iter_cnt = 0;
-    auto data_iter = true_data.begin();
-    Status s;
-    for (iter->SeekToFirst(); iter->Valid(); iter->Next(), data_iter++) {
+    data_iter = true_data.begin();
+    for (iter->SeekToFirst(); iter->Valid(); iter->Next(), ++data_iter) {
       ASSERT_EQ(iter->key().ToString(), data_iter->first);
-      Status current_status = status[data_iter->first];
-      if (!current_status.ok()) {
-        s = current_status;
-      }
-      ASSERT_EQ(iter->status(), s);
-      if (current_status.ok()) {
         ASSERT_EQ(iter->value().ToString(), data_iter->second);
-      }
       iter_cnt++;
       total_reads++;
     }
@@ -1646,20 +1678,12 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
 
     // Verify Iterator::Prev()
     // Use a new iterator to make sure its status is clean.
-    iter = db_->NewIterator(ro);
+    iter = db_->NewIterator(ro_, cf);
     iter_cnt = 0;
-    s = Status::OK();
     auto data_rev = true_data.rbegin();
     for (iter->SeekToLast(); iter->Valid(); iter->Prev(), data_rev++) {
       ASSERT_EQ(iter->key().ToString(), data_rev->first);
-      Status current_status = status[data_rev->first];
-      if (!current_status.ok()) {
-        s = current_status;
-      }
-      ASSERT_EQ(iter->status(), s);
-      if (current_status.ok()) {
         ASSERT_EQ(iter->value().ToString(), data_rev->second);
-      }
       iter_cnt++;
       total_reads++;
     }
@@ -1667,12 +1691,20 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
     ASSERT_EQ(data_rev, true_data.rend())
         << iter_cnt << " / " << true_data.size();
 
-    // Verify Iterator::Seek()
-    for (const auto& kv : true_data) {
-      iter->Seek(kv.first);
-      ASSERT_EQ(kv.first, iter->key().ToString());
-      ASSERT_EQ(kv.second, iter->value().ToString());
-      total_reads++;
+    // Verify Iterator::Seek() and SeekForPrev()
+    for (const auto& [k, v] : true_data) {
+      for (bool prev : {false, true}) {
+        if (prev) {
+          iter->SeekForPrev(k);
+        } else {
+          iter->Seek(k);
+        }
+        ASSERT_TRUE(iter->Valid());
+        ASSERT_OK(iter->status());
+        ASSERT_EQ(iter->key(), k);
+        ASSERT_EQ(iter->value(), v);
+        ++total_reads;
+      }
     }
     delete iter;
   }
@@ -1680,14 +1712,14 @@ void DBTestBase::VerifyDBFromMap(std::map<std::string, std::string> true_data,
   if (tailing_iter) {
     // Tailing iterator
     int iter_cnt = 0;
-    ReadOptions ro;
-    ro.tailing = true;
-    ro.total_order_seek = true;
-    Iterator* iter = db_->NewIterator(ro);
+    ReadOptions ro_ = *ro;
+    ro_.tailing = true;
+    ro_.total_order_seek = true;
+    Iterator* iter = db_->NewIterator(ro_, cf);
 
     // Verify ForwardIterator::Next()
     iter_cnt = 0;
-    auto data_iter = true_data.begin();
+    data_iter = true_data.begin();
     for (iter->SeekToFirst(); iter->Valid(); iter->Next(), data_iter++) {
       ASSERT_EQ(iter->key().ToString(), data_iter->first);
       ASSERT_EQ(iter->value().ToString(), data_iter->second);

db/db_test_util.h

@@ -1367,7 +1367,8 @@ class DBTestBase : public testing::Test {
   void VerifyDBFromMap(
       std::map<std::string, std::string> true_data,
       size_t* total_reads_res = nullptr, bool tailing_iter = false,
-      std::map<std::string, Status> status = std::map<std::string, Status>());
+      ReadOptions* ro = nullptr, ColumnFamilyHandle* cf = nullptr,
+      std::unordered_set<std::string>* not_found = nullptr) const;
 
   void VerifyDBInternal(
       std::vector<std::pair<std::string, std::string>> true_data);

db/dbformat.cc

@@ -178,11 +178,9 @@ std::string ParsedInternalKey::DebugString(bool log_err_key, bool hex,
     result += "<redacted>";
   }
 
-  char buf[50];
-  snprintf(buf, sizeof(buf), "' seq:%" PRIu64 ", type:%d", sequence,
-           static_cast<int>(type));
+  result += "' seq:" + std::to_string(sequence);
+  result += ", type:" + std::to_string(type);
 
-  result += buf;
   return result;
 }
 

db/experimental.cc

@@ -13,6 +13,8 @@
 #include <vector>
 
 #include "db/db_impl/db_impl.h"
+#include "db/manifest_ops.h"
+#include "db/version_edit_handler.h"
 #include "db/version_util.h"
 #include "logging/logging.h"
 #include "util/atomic.h"
@@ -40,6 +42,58 @@ Status SuggestCompactRange(DB* db, const Slice* begin, const Slice* end) {
   return SuggestCompactRange(db, db->DefaultColumnFamily(), begin, end);
 }
 
+Status GetFileChecksumsFromCurrentManifest(FileSystem* fs,
+                                           const std::string& dbname,
+                                           FileChecksumList* checksum_list) {
+  std::string manifest_path;
+  uint64_t manifest_file_number;
+  Status s = GetCurrentManifestPath(dbname, fs, true /* is_retry */,
+                                    &manifest_path, &manifest_file_number);
+  if (!s.ok()) {
+    return s;
+  }
+
+  if (checksum_list == nullptr) {
+    return Status::InvalidArgument("checksum_list is nullptr");
+  }
+  assert(checksum_list);
+
+  const ReadOptions read_options(Env::IOActivity::kReadManifest);
+  checksum_list->reset();
+
+  std::unique_ptr<SequentialFileReader> file_reader;
+  {
+    std::unique_ptr<FSSequentialFile> file;
+    s = fs->NewSequentialFile(manifest_path,
+                              fs->OptimizeForManifestRead(FileOptions()), &file,
+                              nullptr /* dbg */);
+    if (!s.ok()) {
+      return s;
+    }
+    file_reader.reset(new SequentialFileReader(std::move(file), manifest_path));
+  }
+
+  struct LogReporter : public log::Reader::Reporter {
+    Status* status_ptr;
+    void Corruption(size_t /*bytes*/, const Status& st) override {
+      if (status_ptr->ok()) {
+        *status_ptr = st;
+      }
+    }
+  } reporter;
+  reporter.status_ptr = &s;
+  log::Reader reader(nullptr, std::move(file_reader), &reporter,
+                     true /* checksum */, 0 /* log_number */);
+
+  // Read all records from the manifest file...
+  uint64_t manifest_file_size = std::numeric_limits<uint64_t>::max();
+  FileChecksumRetriever retriever(read_options, manifest_file_size,
+                                  *checksum_list);
+  retriever.Iterate(reader, &s);
+
+  return retriever.status();
+}
+
 Status UpdateManifestForFilesState(
     const DBOptions& db_opts, const std::string& db_name,
     const std::vector<ColumnFamilyDescriptor>& column_families,

db/external_sst_file_basic_test.cc

@@ -271,6 +271,79 @@ TEST_F(ExternalSSTFileBasicTest, Basic) {
   DestroyAndRecreateExternalSSTFilesDir();
 }
 
+TEST_F(ExternalSSTFileBasicTest, AlignedBufferedWrite) {
+  class AlignedWriteFS : public FileSystemWrapper {
+   public:
+    explicit AlignedWriteFS(const std::shared_ptr<FileSystem>& _target)
+        : FileSystemWrapper(_target) {}
+    ~AlignedWriteFS() override {}
+    const char* Name() const override { return "AlignedWriteFS"; }
+
+    IOStatus NewWritableFile(const std::string& fname, const FileOptions& opts,
+                             std::unique_ptr<FSWritableFile>* result,
+                             IODebugContext* dbg) override {
+      class AlignedWritableFile : public FSWritableFileOwnerWrapper {
+       public:
+        AlignedWritableFile(std::unique_ptr<FSWritableFile>& file)
+            : FSWritableFileOwnerWrapper(std::move(file)), last_write_(false) {}
+
+        using FSWritableFileOwnerWrapper::Append;
+        IOStatus Append(const Slice& data, const IOOptions& options,
+                        IODebugContext* dbg) override {
+          EXPECT_FALSE(last_write_);
+          if ((data.size() & (data.size() - 1)) != 0) {
+            last_write_ = true;
+          }
+          return target()->Append(data, options, dbg);
+        }
+
+       private:
+        bool last_write_;
+      };
+
+      std::unique_ptr<FSWritableFile> file;
+      IOStatus s = target()->NewWritableFile(fname, opts, &file, dbg);
+      if (s.ok()) {
+        result->reset(new AlignedWritableFile(file));
+      }
+      return s;
+    }
+  };
+
+  Options options = CurrentOptions();
+  std::shared_ptr<AlignedWriteFS> aligned_fs =
+      std::make_shared<AlignedWriteFS>(env_->GetFileSystem());
+  std::unique_ptr<Env> wrap_env(
+      new CompositeEnvWrapper(options.env, aligned_fs));
+  options.env = wrap_env.get();
+
+  EnvOptions env_options;
+  env_options.writable_file_max_buffer_size = 64 * 1024 * 1024;
+
+  SstFileWriter sst_file_writer(env_options, options);
+
+  // Current file size should be 0 after sst_file_writer init and before open a
+  // file.
+  ASSERT_EQ(sst_file_writer.FileSize(), 0);
+
+  // file1.sst (0 => 99)
+  std::string file1 = sst_files_dir_ + "file1.sst";
+  ASSERT_OK(sst_file_writer.Open(file1));
+  Random r(301);
+  for (int k = 0; k < 16 * 1024; k++) {
+    uint32_t num = 4096 + r.Uniform(8192);
+    std::string random_string = r.RandomString(num);
+    ASSERT_OK(sst_file_writer.Put(Key(k), random_string));
+  }
+  Status s = sst_file_writer.Finish();
+  ASSERT_OK(s) << s.ToString();
+
+  // Current file size should be non-zero after success write.
+  ASSERT_GT(sst_file_writer.FileSize(), 0);
+
+  DestroyAndRecreateExternalSSTFilesDir();
+}
+
 class ChecksumVerifyHelper {
  private:
   Options options_;

db/flush_job.cc

@@ -422,7 +422,8 @@ Status FlushJob::MemPurge() {
       range_del_iters;
   for (ReadOnlyMemTable* m : mems_) {
     memtables.push_back(m->NewIterator(ro, /*seqno_to_time_mapping=*/nullptr,
-                                       &arena, /*prefix_extractor=*/nullptr));
+                                       &arena, /*prefix_extractor=*/nullptr,
+                                       /*for_flush=*/true));
     auto* range_del_iter = m->NewRangeTombstoneIterator(
         ro, kMaxSequenceNumber, true /* immutable_memtable */);
     if (range_del_iter != nullptr) {
@@ -624,10 +625,13 @@ Status FlushJob::MemPurge() {
         // Construct fragmented memtable range tombstones without mutex
         new_mem->ConstructFragmentedRangeTombstones();
         db_mutex_->Lock();
-        uint64_t new_mem_id = mems_[0]->GetID();
+        // Take the newest id, so that memtables in MemtableList don't have
+        // out-of-order memtable ids.
+        uint64_t new_mem_id = mems_.back()->GetID();
 
         new_mem->SetID(new_mem_id);
-        new_mem->SetNextLogNumber(mems_[0]->GetNextLogNumber());
+        // Take the latest memtable's next log number.
+        new_mem->SetNextLogNumber(mems_.back()->GetNextLogNumber());
 
         // This addition will not trigger another flush, because
         // we do not call EnqueuePendingFlush().
@@ -907,7 +911,7 @@ Status FlushJob::WriteLevel0Table() {
       } else {
         memtables.push_back(
             m->NewIterator(ro, /*seqno_to_time_mapping=*/nullptr, &arena,
-                           /*prefix_extractor=*/nullptr));
+                           /*prefix_extractor=*/nullptr, /*for_flush=*/true));
       }
       auto* range_del_iter =
           logical_strip_timestamp

db/forward_iterator.cc

@@ -714,7 +714,8 @@ void ForwardIterator::RebuildIterators(bool refresh_sv) {
       sv_->GetSeqnoToTimeMapping();
   mutable_iter_ =
       sv_->mem->NewIterator(read_options_, seqno_to_time_mapping, &arena_,
-                            sv_->mutable_cf_options.prefix_extractor.get());
+                            sv_->mutable_cf_options.prefix_extractor.get(),
+                            /*for_flush=*/false);
   sv_->imm->AddIterators(read_options_, seqno_to_time_mapping,
                          sv_->mutable_cf_options.prefix_extractor.get(),
                          &imm_iters_, &arena_);
@@ -784,7 +785,8 @@ void ForwardIterator::RenewIterators() {
       svnew->GetSeqnoToTimeMapping();
   mutable_iter_ =
       svnew->mem->NewIterator(read_options_, seqno_to_time_mapping, &arena_,
-                              svnew->mutable_cf_options.prefix_extractor.get());
+                              svnew->mutable_cf_options.prefix_extractor.get(),
+                              /*for_flush=*/false);
   svnew->imm->AddIterators(read_options_, seqno_to_time_mapping,
                            svnew->mutable_cf_options.prefix_extractor.get(),
                            &imm_iters_, &arena_);

db/log_reader.cc

@@ -313,11 +313,11 @@ bool Reader::ReadRecord(Slice* record, std::string* scratch,
         break;
 
       default: {
-        char buf[40];
-        snprintf(buf, sizeof(buf), "unknown record type %u", record_type);
+        std::string reason =
+            "unknown record type " + std::to_string(record_type);
         ReportCorruption(
             (fragment.size() + (in_fragmented_record ? scratch->size() : 0)),
-            buf);
+            reason.c_str());
         in_fragmented_record = false;
         scratch->clear();
         break;
@@ -781,12 +781,11 @@ bool FragmentBufferedReader::ReadRecord(Slice* record, std::string* scratch,
         break;
 
       default: {
-        char buf[40];
-        snprintf(buf, sizeof(buf), "unknown record type %u",
-                 fragment_type_or_err);
+        std::string reason =
+            "unknown record type " + std::to_string(fragment_type_or_err);
         ReportCorruption(
             fragment.size() + (in_fragmented_record_ ? fragments_.size() : 0),
-            buf);
+            reason.c_str());
         in_fragmented_record_ = false;
         fragments_.clear();
         break;

db/manifest_ops.cc

@@ -0,0 +1,47 @@
+//  Copyright (c) Meta Platforms, Inc. and affiliates.
+//
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#include "db/manifest_ops.h"
+
+#include "file/filename.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+Status GetCurrentManifestPath(const std::string& dbname, FileSystem* fs,
+                              bool is_retry, std::string* manifest_path,
+                              uint64_t* manifest_file_number) {
+  assert(fs != nullptr);
+  assert(manifest_path != nullptr);
+  assert(manifest_file_number != nullptr);
+
+  IOOptions opts;
+  std::string fname;
+  if (is_retry) {
+    opts.verify_and_reconstruct_read = true;
+  }
+  Status s = ReadFileToString(fs, CurrentFileName(dbname), opts, &fname);
+  if (!s.ok()) {
+    return s;
+  }
+  if (fname.empty() || fname.back() != '\n') {
+    return Status::Corruption("CURRENT file does not end with newline");
+  }
+  // remove the trailing '\n'
+  fname.resize(fname.size() - 1);
+  FileType type;
+  bool parse_ok = ParseFileName(fname, manifest_file_number, &type);
+  if (!parse_ok || type != kDescriptorFile) {
+    return Status::Corruption("CURRENT file corrupted");
+  }
+  *manifest_path = dbname;
+  if (dbname.back() != '/') {
+    manifest_path->push_back('/');
+  }
+  manifest_path->append(fname);
+  return Status::OK();
+}
+
+}  // namespace ROCKSDB_NAMESPACE

db/manifest_ops.h

@@ -0,0 +1,20 @@
+//  Copyright (c) Meta Platforms, Inc. and affiliates.
+//
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#pragma once
+#include <cassert>
+
+#include "rocksdb/env.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+// Parameter is_retry=true sets the verify_and_reconstruct_read flag.
+// It comes handy when caller intends to re-read the data with much stronger
+// data integrity checking - e.g. in case of a perceived file corruption.
+Status GetCurrentManifestPath(const std::string& dbname, FileSystem* fs,
+                              bool is_retry, std::string* manifest_path,
+                              uint64_t* manifest_file_number);
+}  // namespace ROCKSDB_NAMESPACE

db/memtable.cc

@@ -599,7 +599,7 @@ class MemTableIterator : public InternalIterator {
 InternalIterator* MemTable::NewIterator(
     const ReadOptions& read_options,
     UnownedPtr<const SeqnoToTimeMapping> seqno_to_time_mapping, Arena* arena,
-    const SliceTransform* prefix_extractor) {
+    const SliceTransform* prefix_extractor, bool /*for_flush*/) {
   assert(arena != nullptr);
   auto mem = arena->AllocateAligned(sizeof(MemTableIterator));
   return new (mem)

db/memtable.h

@@ -153,7 +153,7 @@ class ReadOnlyMemTable {
   virtual InternalIterator* NewIterator(
       const ReadOptions& read_options,
       UnownedPtr<const SeqnoToTimeMapping> seqno_to_time_mapping, Arena* arena,
-      const SliceTransform* prefix_extractor) = 0;
+      const SliceTransform* prefix_extractor, bool for_flush) = 0;
 
   // Returns an iterator that wraps a MemTableIterator and logically strips the
   // user-defined timestamp of each key. This API is only used by flush when
@@ -541,7 +541,7 @@ class MemTable final : public ReadOnlyMemTable {
   InternalIterator* NewIterator(
       const ReadOptions& read_options,
       UnownedPtr<const SeqnoToTimeMapping> seqno_to_time_mapping, Arena* arena,
-      const SliceTransform* prefix_extractor) override;
+      const SliceTransform* prefix_extractor, bool for_flush) override;
 
   InternalIterator* NewTimestampStrippingIterator(
       const ReadOptions& read_options,

db/memtable_list.cc

@@ -32,6 +32,10 @@ class Mutex;
 class VersionSet;
 
 void MemTableListVersion::AddMemTable(ReadOnlyMemTable* m) {
+  if (!memlist_.empty()) {
+    // ID can be equal for MemPurge
+    assert(m->GetID() >= memlist_.front()->GetID());
+  }
   memlist_.push_front(m);
   *parent_memtable_list_memory_usage_ += m->ApproximateMemoryUsage();
 }
@@ -216,7 +220,8 @@ void MemTableListVersion::AddIterators(
     std::vector<InternalIterator*>* iterator_list, Arena* arena) {
   for (auto& m : memlist_) {
     iterator_list->push_back(m->NewIterator(options, seqno_to_time_mapping,
-                                            arena, prefix_extractor));
+                                            arena, prefix_extractor,
+                                            /*for_flush=*/false));
   }
 }
 
@@ -228,7 +233,8 @@ void MemTableListVersion::AddIterators(
   for (auto& m : memlist_) {
     auto mem_iter =
         m->NewIterator(options, seqno_to_time_mapping,
-                       merge_iter_builder->GetArena(), prefix_extractor);
+                       merge_iter_builder->GetArena(), prefix_extractor,
+                       /*for_flush=*/false);
     if (!add_range_tombstone_iter || options.ignore_range_deletions) {
       merge_iter_builder->AddIterator(mem_iter);
     } else {
@@ -410,7 +416,8 @@ void MemTableList::PickMemtablesToFlush(uint64_t max_memtable_id,
   // ret is filled with memtables already sorted in increasing MemTable ID.
   // However, when the mempurge feature is activated, new memtables with older
   // IDs will be added to the memlist.
-  for (auto it = memlist.rbegin(); it != memlist.rend(); ++it) {
+  auto it = memlist.rbegin();
+  for (; it != memlist.rend(); ++it) {
     ReadOnlyMemTable* m = *it;
     if (!atomic_flush && m->atomic_flush_seqno_ != kMaxSequenceNumber) {
       atomic_flush = true;
@@ -439,6 +446,12 @@ void MemTableList::PickMemtablesToFlush(uint64_t max_memtable_id,
       break;
     }
   }
+  if (!ret->empty() && it != memlist.rend()) {
+    // checks that the first memtable not picked to flush is not ingested wbwi.
+    // Ingested memtable should be flushed together with the memtable before it
+    // since they map to the same WAL and have the same NextLogNumber().
+    assert(strcmp((*it)->Name(), "WBWIMemTable") != 0);
+  }
   if (!atomic_flush || num_flush_not_started_ == 0) {
     flush_requested_ = false;  // start-flush request is complete
   }

db/memtable_list_test.cc

@@ -330,6 +330,7 @@ TEST_F(MemTableListTest, GetTest) {
   // This is to make assert(memtable->IsFragmentedRangeTombstonesConstructed())
   // in MemTableListVersion::GetFromList work.
   mem->ConstructFragmentedRangeTombstones();
+  mem->SetID(1);
   list.Add(mem, &to_delete);
 
   SequenceNumber saved_seq = seq;
@@ -338,6 +339,7 @@ TEST_F(MemTableListTest, GetTest) {
   WriteBufferManager wb2(options.db_write_buffer_size);
   MemTable* mem2 = new MemTable(cmp, ioptions, MutableCFOptions(options), &wb2,
                                 kMaxSequenceNumber, 0 /* column_family_id */);
+  mem2->SetID(2);
   mem2->Ref();
 
   ASSERT_OK(

db/repair.cc

@@ -449,7 +449,7 @@ class Repairer {
       Arena arena;
       ScopedArenaPtr<InternalIterator> iter(
           mem->NewIterator(ro, /*seqno_to_time_mapping=*/nullptr, &arena,
-                           /*prefix_extractor=*/nullptr));
+                           /*prefix_extractor=*/nullptr, /*for_flush=*/true));
       int64_t _current_time = 0;
       immutable_db_options_.clock->GetCurrentTime(&_current_time)
           .PermitUncheckedError();  // ignore error

db/transaction_log_impl.cc

@@ -231,13 +231,14 @@ bool TransactionLogIteratorImpl::IsBatchExpected(
   assert(batch);
   SequenceNumber batchSeq = WriteBatchInternal::Sequence(batch);
   if (batchSeq != expected_seq) {
-    char buf[200];
-    snprintf(buf, sizeof(buf),
-             "Discontinuity in log records. Got seq=%" PRIu64
-             ", Expected seq=%" PRIu64 ", Last flushed seq=%" PRIu64
-             ".Log iterator will reseek the correct batch.",
-             batchSeq, expected_seq, versions_->LastSequence());
-    reporter_.Info(buf);
+    std::ostringstream oss;
+    oss << "Discontinuity in log records. "
+        << "Got seq=" << batchSeq << ", "
+        << "Expected seq=" << expected_seq << ", "
+        << "Last flushed seq=" << versions_->LastSequence() << ". "
+        << "Log iterator will reseek the correct batch.";
+
+    reporter_.Info(oss.str().c_str());
     return false;
   }
   return true;

db/version_set.cc

@@ -31,6 +31,7 @@
 #include "db/internal_stats.h"
 #include "db/log_reader.h"
 #include "db/log_writer.h"
+#include "db/manifest_ops.h"
 #include "db/memtable.h"
 #include "db/merge_context.h"
 #include "db/merge_helper.h"
@@ -6011,41 +6012,6 @@ Status VersionSet::LogAndApplyHelper(ColumnFamilyData* cfd,
   return builder ? builder->Apply(edit) : Status::OK();
 }
 
-Status VersionSet::GetCurrentManifestPath(const std::string& dbname,
-                                          FileSystem* fs, bool is_retry,
-                                          std::string* manifest_path,
-                                          uint64_t* manifest_file_number) {
-  assert(fs != nullptr);
-  assert(manifest_path != nullptr);
-  assert(manifest_file_number != nullptr);
-
-  IOOptions opts;
-  std::string fname;
-  if (is_retry) {
-    opts.verify_and_reconstruct_read = true;
-  }
-  Status s = ReadFileToString(fs, CurrentFileName(dbname), opts, &fname);
-  if (!s.ok()) {
-    return s;
-  }
-  if (fname.empty() || fname.back() != '\n') {
-    return Status::Corruption("CURRENT file does not end with newline");
-  }
-  // remove the trailing '\n'
-  fname.resize(fname.size() - 1);
-  FileType type;
-  bool parse_ok = ParseFileName(fname, manifest_file_number, &type);
-  if (!parse_ok || type != kDescriptorFile) {
-    return Status::Corruption("CURRENT file corrupted");
-  }
-  *manifest_path = dbname;
-  if (dbname.back() != '/') {
-    manifest_path->push_back('/');
-  }
-  manifest_path->append(fname);
-  return Status::OK();
-}
-
 Status VersionSet::Recover(
     const std::vector<ColumnFamilyDescriptor>& column_families, bool read_only,
     std::string* db_id, bool no_error_if_files_missing, bool is_retry,

db/version_set.h

@@ -1277,10 +1277,6 @@ class VersionSet {
       const std::vector<std::function<void(const Status&)>>& manifest_wcbs =
           {});
 
-  static Status GetCurrentManifestPath(const std::string& dbname,
-                                       FileSystem* fs, bool is_retry,
-                                       std::string* manifest_filename,
-                                       uint64_t* manifest_file_number);
   void WakeUpWaitingManifestWriters();
 
   // Recover the last saved descriptor (MANIFEST) from persistent storage.

db/version_set_test.cc

@@ -15,6 +15,7 @@
 #include "db/db_impl/db_impl.h"
 #include "db/db_test_util.h"
 #include "db/log_writer.h"
+#include "db/manifest_ops.h"
 #include "db/version_edit.h"
 #include "rocksdb/advanced_options.h"
 #include "rocksdb/convenience.h"
@@ -1390,18 +1391,16 @@ class VersionSetTestBase {
   void GetManifestPath(std::string* manifest_path) const {
     assert(manifest_path != nullptr);
     uint64_t manifest_file_number = 0;
-    Status s = versions_->GetCurrentManifestPath(
-        dbname_, fs_.get(), /*is_retry=*/false, manifest_path,
-        &manifest_file_number);
+    Status s = GetCurrentManifestPath(dbname_, fs_.get(), /*is_retry=*/false,
+                                      manifest_path, &manifest_file_number);
     ASSERT_OK(s);
   }
 
   void VerifyManifest(std::string* manifest_path) const {
     assert(manifest_path != nullptr);
     uint64_t manifest_file_number = 0;
-    Status s = versions_->GetCurrentManifestPath(
-        dbname_, fs_.get(), /*is_retry=*/false, manifest_path,
-        &manifest_file_number);
+    Status s = GetCurrentManifestPath(dbname_, fs_.get(), /*is_retry=*/false,
+                                      manifest_path, &manifest_file_number);
     ASSERT_OK(s);
     ASSERT_EQ(1, manifest_file_number);
   }

db/wide/wide_column_serialization.cc

@@ -129,21 +129,6 @@ Status WideColumnSerialization::Deserialize(Slice& input,
   return Status::OK();
 }
 
-WideColumns::const_iterator WideColumnSerialization::Find(
-    const WideColumns& columns, const Slice& column_name) {
-  const auto it =
-      std::lower_bound(columns.cbegin(), columns.cend(), column_name,
-                       [](const WideColumn& lhs, const Slice& rhs) {
-                         return lhs.name().compare(rhs) < 0;
-                       });
-
-  if (it == columns.cend() || it->name() != column_name) {
-    return columns.cend();
-  }
-
-  return it;
-}
-
 Status WideColumnSerialization::GetValueOfDefaultColumn(Slice& input,
                                                         Slice& value) {
   WideColumns columns;

db/wide/wide_column_serialization.h

@@ -47,8 +47,6 @@ class WideColumnSerialization {
 
   static Status Deserialize(Slice& input, WideColumns& columns);
 
-  static WideColumns::const_iterator Find(const WideColumns& columns,
-                                          const Slice& column_name);
   static Status GetValueOfDefaultColumn(Slice& input, Slice& value);
 
   static constexpr uint32_t kCurrentVersion = 1;

db/wide/wide_column_serialization_test.cc

@@ -5,6 +5,7 @@
 
 #include "db/wide/wide_column_serialization.h"
 
+#include "db/wide/wide_columns_helper.h"
 #include "test_util/testharness.h"
 #include "util/coding.h"
 
@@ -99,27 +100,28 @@ TEST(WideColumnSerializationTest, SerializeDeserialize) {
   ASSERT_EQ(columns, deserialized_columns);
 
   {
-    const auto it = WideColumnSerialization::Find(deserialized_columns, "foo");
+    const auto it = WideColumnsHelper::Find(deserialized_columns.cbegin(),
+                                            deserialized_columns.cend(), "foo");
     ASSERT_NE(it, deserialized_columns.cend());
     ASSERT_EQ(*it, deserialized_columns.front());
   }
 
   {
-    const auto it =
-        WideColumnSerialization::Find(deserialized_columns, "hello");
+    const auto it = WideColumnsHelper::Find(
+        deserialized_columns.cbegin(), deserialized_columns.cend(), "hello");
     ASSERT_NE(it, deserialized_columns.cend());
     ASSERT_EQ(*it, deserialized_columns.back());
   }
 
   {
-    const auto it =
-        WideColumnSerialization::Find(deserialized_columns, "fubar");
+    const auto it = WideColumnsHelper::Find(
+        deserialized_columns.cbegin(), deserialized_columns.cend(), "fubar");
     ASSERT_EQ(it, deserialized_columns.cend());
   }
 
   {
-    const auto it =
-        WideColumnSerialization::Find(deserialized_columns, "snafu");
+    const auto it = WideColumnsHelper::Find(
+        deserialized_columns.cbegin(), deserialized_columns.cend(), "snafu");
     ASSERT_EQ(it, deserialized_columns.cend());
   }
 }

db/wide/wide_columns_helper.cc

@@ -5,7 +5,6 @@
 
 #include "db/wide/wide_columns_helper.h"
 
-#include <algorithm>
 #include <ios>
 
 #include "db/wide/wide_column_serialization.h"
@@ -42,11 +41,4 @@ Status WideColumnsHelper::DumpSliceAsWideColumns(const Slice& value,
   return s;
 }
 
-void WideColumnsHelper::SortColumns(WideColumns& columns) {
-  std::sort(columns.begin(), columns.end(),
-            [](const WideColumn& lhs, const WideColumn& rhs) {
-              return lhs.name().compare(rhs.name()) < 0;
-            });
-}
-
 }  // namespace ROCKSDB_NAMESPACE

db/wide/wide_columns_helper.h

@@ -4,8 +4,10 @@
 //  (found in the LICENSE.Apache file in the root directory).
 
 #pragma once
+
+#include <algorithm>
+#include <cassert>
 #include <ostream>
-#include <string>
 
 #include "rocksdb/rocksdb_namespace.h"
 #include "rocksdb/wide_columns.h"
@@ -34,7 +36,31 @@ class WideColumnsHelper {
     return columns.front().value();
   }
 
-  static void SortColumns(WideColumns& columns);
+  static void SortColumns(WideColumns& columns) {
+    std::sort(columns.begin(), columns.end(),
+              [](const WideColumn& lhs, const WideColumn& rhs) {
+                return lhs.name().compare(rhs.name()) < 0;
+              });
+  }
+
+  template <typename Iterator>
+  static Iterator Find(Iterator begin, Iterator end, const Slice& column_name) {
+    assert(std::is_sorted(begin, end,
+                          [](const WideColumn& lhs, const WideColumn& rhs) {
+                            return lhs.name().compare(rhs.name()) < 0;
+                          }));
+
+    auto it = std::lower_bound(begin, end, column_name,
+                               [](const WideColumn& lhs, const Slice& rhs) {
+                                 return lhs.name().compare(rhs) < 0;
+                               });
+
+    if (it == end || it->name() != column_name) {
+      return end;
+    }
+
+    return it;
+  }
 };
 
 }  // namespace ROCKSDB_NAMESPACE

db/write_batch_test.cc

@@ -59,7 +59,8 @@ static std::string PrintContents(WriteBatch* b,
     InternalIterator* iter;
     if (i == 0) {
       iter = mem->NewIterator(ReadOptions(), /*seqno_to_time_mapping=*/nullptr,
-                              &arena, /*prefix_extractor=*/nullptr);
+                              &arena, /*prefix_extractor=*/nullptr,
+                              /*for_flush=*/false);
       arena_iter_guard.reset(iter);
     } else {
       iter = mem->NewRangeTombstoneIterator(ReadOptions(),

db/write_thread.cc

@@ -523,8 +523,10 @@ size_t WriteThread::EnterAsBatchGroupLeader(Writer* leader,
         // those are something else. They want to be alone
         (w->callback != nullptr && !w->callback->AllowWriteBatching()) ||
         // dont batch writes that don't want to be batched
-        (size + WriteBatchInternal::ByteSize(w->batch) > max_size)
+        (size + WriteBatchInternal::ByteSize(w->batch) > max_size) ||
         // Do not make batch too big
+        (leader->ingest_wbwi || w->ingest_wbwi)
+        // ingesting WBWI needs to be its own group
     ) {
       // remove from list
       w->link_older->link_newer = w->link_newer;

db/write_thread.h

@@ -148,6 +148,8 @@ class WriteThread {
     Writer* link_older;  // read/write only before linking, or as leader
     Writer* link_newer;  // lazy, read/write only before linking, or as leader
 
+    bool ingest_wbwi;
+
     Writer()
         : batch(nullptr),
           sync(false),
@@ -174,7 +176,8 @@ class WriteThread {
            WriteCallback* _callback, UserWriteCallback* _user_write_cb,
            uint64_t _log_ref, bool _disable_memtable, size_t _batch_cnt = 0,
            PreReleaseCallback* _pre_release_callback = nullptr,
-           PostMemTableCallback* _post_memtable_callback = nullptr)
+           PostMemTableCallback* _post_memtable_callback = nullptr,
+           bool _ingest_wbwi = false)
         : batch(_batch),
           // TODO: store a copy of WriteOptions instead of its seperated data
           // members
@@ -196,7 +199,8 @@ class WriteThread {
           write_group(nullptr),
           sequence(kMaxSequenceNumber),
           link_older(nullptr),
-          link_newer(nullptr) {}
+          link_newer(nullptr),
+          ingest_wbwi(_ingest_wbwi) {}
 
     ~Writer() {
       if (made_waitable) {

db_stress_tool/db_stress_common.h

@@ -419,6 +419,8 @@ DECLARE_bool(inplace_update_support);
 DECLARE_uint32(uncache_aggressiveness);
 DECLARE_int32(test_ingest_standalone_range_deletion_one_in);
 DECLARE_bool(allow_unprepared_value);
+DECLARE_string(file_temperature_age_thresholds);
+DECLARE_uint32(commit_bypass_memtable_one_in);
 
 constexpr long KB = 1024;
 constexpr int kRandomValueMaxFactor = 3;

db_stress_tool/db_stress_gflags.cc

@@ -546,6 +546,10 @@ DEFINE_uint32(use_timed_put_one_in, 0,
               "If greater than zero, TimedPut is used per every N write ops on "
               "on average.");
 
+DEFINE_string(file_temperature_age_thresholds, "",
+              "See CompactionOptionsFIFO::file_temperature_age_thresholds. "
+              "empty == unset");
+
 static const bool FLAGS_subcompactions_dummy __attribute__((__unused__)) =
     RegisterFlagValidator(&FLAGS_subcompactions, &ValidateUint32Range);
 
@@ -1465,4 +1469,7 @@ DEFINE_bool(paranoid_memory_checks,
             ROCKSDB_NAMESPACE::Options().paranoid_memory_checks,
             "Sets CF option paranoid_memory_checks.");
 
+DEFINE_uint32(commit_bypass_memtable_one_in, 0,
+              "If greater than zero, transaction option will set "
+              "commit_bypass_memtable to per every N transactions on average.");
 #endif  // GFLAGS

db_stress_tool/db_stress_test_base.cc

@@ -413,6 +413,16 @@ bool StressTest::BuildOptionsTable() {
                                                  "2147483646", "2147483647"});
   }
 
+  if (!FLAGS_file_temperature_age_thresholds.empty()) {
+    // Modify file_temperature_age_thresholds only if it is set initially
+    // (FIFO tiered storage setup)
+    options_tbl.emplace(
+        "file_temperature_age_thresholds",
+        std::vector<std::string>{
+            "{{temperature=kWarm;age=30}:{temperature=kCold;age=300}}",
+            "{{temperature=kCold;age=100}}", "{}"});
+  }
+
   options_table_ = std::move(options_tbl);
 
   for (const auto& iter : options_table_) {
@@ -795,8 +805,9 @@ void StressTest::ProcessRecoveredPreparedTxnsHelper(Transaction* txn,
   }
 }
 
-Status StressTest::NewTxn(WriteOptions& write_opts,
-                          std::unique_ptr<Transaction>* out_txn) {
+Status StressTest::NewTxn(WriteOptions& write_opts, ThreadState* thread,
+                          std::unique_ptr<Transaction>* out_txn,
+                          bool* commit_bypass_memtable) {
   if (!FLAGS_use_txn) {
     return Status::InvalidArgument("NewTxn when FLAGS_use_txn is not set");
   }
@@ -811,6 +822,15 @@ Status StressTest::NewTxn(WriteOptions& write_opts,
         FLAGS_use_only_the_last_commit_time_batch_for_recovery;
     txn_options.lock_timeout = 600000;  // 10 min
     txn_options.deadlock_detect = true;
+    if (FLAGS_commit_bypass_memtable_one_in > 0) {
+      assert(FLAGS_txn_write_policy == 0);
+      assert(FLAGS_user_timestamp_size == 0);
+      txn_options.commit_bypass_memtable =
+          thread->rand.OneIn(FLAGS_commit_bypass_memtable_one_in);
+      if (commit_bypass_memtable) {
+        *commit_bypass_memtable = txn_options.commit_bypass_memtable;
+      }
+    }
     out_txn->reset(txn_db_->BeginTransaction(write_opts, txn_options));
     auto istr = std::to_string(txn_id.fetch_add(1));
     Status s = (*out_txn)->SetName("xid" + istr);
@@ -866,11 +886,12 @@ Status StressTest::CommitTxn(Transaction& txn, ThreadState* thread) {
   return s;
 }
 
-Status StressTest::ExecuteTransaction(
-    WriteOptions& write_opts, ThreadState* thread,
-    std::function<Status(Transaction&)>&& ops) {
+Status StressTest::ExecuteTransaction(WriteOptions& write_opts,
+                                      ThreadState* thread,
+                                      std::function<Status(Transaction&)>&& ops,
+                                      bool* commit_bypass_memtable) {
   std::unique_ptr<Transaction> txn;
-  Status s = NewTxn(write_opts, &txn);
+  Status s = NewTxn(write_opts, thread, &txn, commit_bypass_memtable);
   std::string try_again_messages;
   if (s.ok()) {
     for (int tries = 1;; ++tries) {
@@ -4156,6 +4177,18 @@ void InitializeOptionsFromFlags(
   options.default_temperature =
       StringToTemperature(FLAGS_default_temperature.c_str());
 
+  if (!FLAGS_file_temperature_age_thresholds.empty()) {
+    Status s = GetColumnFamilyOptionsFromString(
+        {}, options,
+        "compaction_options_fifo={file_temperature_age_thresholds=" +
+            FLAGS_file_temperature_age_thresholds + "}",
+        &options);
+    if (!s.ok()) {
+      fprintf(stderr, "While setting file_temperature_age_thresholds: %s\n",
+              s.ToString().c_str());
+      exit(1);
+    }
+  }
   options.preclude_last_level_data_seconds =
       FLAGS_preclude_last_level_data_seconds;
   options.preserve_internal_time_seconds = FLAGS_preserve_internal_time_seconds;

db_stress_tool/db_stress_test_base.h

@@ -71,14 +71,19 @@ class StressTest {
   void UpdateIfInitialWriteFails(Env* db_stress_env, const Status& write_s,
                                  Status* initial_write_s,
                                  bool* initial_wal_write_may_succeed,
-                                 uint64_t* wait_for_recover_start_time) {
+                                 uint64_t* wait_for_recover_start_time,
+                                 bool commit_bypass_memtable = false) {
     assert(db_stress_env && initial_write_s && initial_wal_write_may_succeed &&
            wait_for_recover_start_time);
     // Only update `initial_write_s`, `initial_wal_write_may_succeed` when the
     // first write fails
     if (!write_s.ok() && (*initial_write_s).ok()) {
       *initial_write_s = write_s;
+      // With commit_bypass_memtable, we create a new WAL after WAL write
+      // succeeds, that wal creation may fail due to injected error. So the
+      // initial wal write may succeed even if status is failed to write to wal
       *initial_wal_write_may_succeed =
+          commit_bypass_memtable ||
           !FaultInjectionTestFS::IsFailedToWriteToWALError(*initial_write_s);
       *wait_for_recover_start_time = db_stress_env->NowMicros();
     }
@@ -138,13 +143,19 @@ class StressTest {
                                                   SharedState* shared);
 
   // ExecuteTransaction is recommended instead
-  Status NewTxn(WriteOptions& write_opts,
-                std::unique_ptr<Transaction>* out_txn);
+  // @param commit_bypass_memtable Whether commit_bypass_memtable is set to
+  // true in transaction options.
+  Status NewTxn(WriteOptions& write_opts, ThreadState* thread,
+                std::unique_ptr<Transaction>* out_txn,
+                bool* commit_bypass_memtable = nullptr);
   Status CommitTxn(Transaction& txn, ThreadState* thread = nullptr);
 
   // Creates a transaction, executes `ops`, and tries to commit
+  // @param commit_bypass_memtable Whether commit_bypass_memtable is set to
+  // true in transaction options.
   Status ExecuteTransaction(WriteOptions& write_opts, ThreadState* thread,
-                            std::function<Status(Transaction&)>&& ops);
+                            std::function<Status(Transaction&)>&& ops,
+                            bool* commit_bypass_memtable = nullptr);
 
   virtual void MaybeClearOneColumnFamily(ThreadState* /* thread */) {}
 

db_stress_tool/multi_ops_txns_stress.cc

@@ -544,6 +544,11 @@ Status MultiOpsTxnsStressTest::TestCustomOperations(
     // Should never reach here.
     assert(false);
   }
+  if (!s.ok()) {
+    fprintf(stderr, "Transaction failed %s\n", s.ToString().c_str());
+    fflush(stderr);
+    thread->shared->SafeTerminate();
+  }
 
   return s;
 }
@@ -579,7 +584,7 @@ Status MultiOpsTxnsStressTest::PrimaryKeyUpdateTxn(ThreadState* thread,
   std::string new_pk = Record::EncodePrimaryKey(new_a);
   std::unique_ptr<Transaction> txn;
   WriteOptions wopts;
-  Status s = NewTxn(wopts, &txn);
+  Status s = NewTxn(wopts, thread, &txn);
   if (!s.ok()) {
     assert(!txn);
     thread->stats.AddErrors(1);
@@ -611,7 +616,12 @@ Status MultiOpsTxnsStressTest::PrimaryKeyUpdateTxn(ThreadState* thread,
     }
     auto& key_gen = key_gen_for_a_[thread->tid];
     key_gen->UndoAllocation(new_a);
-    txn->Rollback().PermitUncheckedError();
+    s = txn->Rollback();
+    if (!s.ok()) {
+      fprintf(stderr, "Transaction rollback failed %s\n", s.ToString().c_str());
+      fflush(stderr);
+      assert(false);
+    }
   });
 
   ReadOptions ropts;
@@ -699,7 +709,7 @@ Status MultiOpsTxnsStressTest::SecondaryKeyUpdateTxn(ThreadState* thread,
                                                      uint32_t new_c) {
   std::unique_ptr<Transaction> txn;
   WriteOptions wopts;
-  Status s = NewTxn(wopts, &txn);
+  Status s = NewTxn(wopts, thread, &txn);
   if (!s.ok()) {
     assert(!txn);
     thread->stats.AddErrors(1);
@@ -735,7 +745,12 @@ Status MultiOpsTxnsStressTest::SecondaryKeyUpdateTxn(ThreadState* thread,
     }
     auto& key_gen = key_gen_for_c_[thread->tid];
     key_gen->UndoAllocation(new_c);
-    txn->Rollback().PermitUncheckedError();
+    s = txn->Rollback();
+    if (!s.ok()) {
+      fprintf(stderr, "Transaction rollback failed %s\n", s.ToString().c_str());
+      fflush(stderr);
+      assert(false);
+    }
   });
 
   // TODO (yanqin) try SetSnapshotOnNextOperation(). We currently need to take
@@ -901,7 +916,7 @@ Status MultiOpsTxnsStressTest::UpdatePrimaryIndexValueTxn(ThreadState* thread,
   std::string pk_str = Record::EncodePrimaryKey(a);
   std::unique_ptr<Transaction> txn;
   WriteOptions wopts;
-  Status s = NewTxn(wopts, &txn);
+  Status s = NewTxn(wopts, thread, &txn);
   if (!s.ok()) {
     assert(!txn);
     thread->stats.AddErrors(1);
@@ -927,7 +942,12 @@ Status MultiOpsTxnsStressTest::UpdatePrimaryIndexValueTxn(ThreadState* thread,
     } else {
       thread->stats.AddErrors(1);
     }
-    txn->Rollback().PermitUncheckedError();
+    s = txn->Rollback();
+    if (!s.ok()) {
+      fprintf(stderr, "Transaction rollback failed %s\n", s.ToString().c_str());
+      fflush(stderr);
+      assert(false);
+    }
   });
   ReadOptions ropts;
   ropts.rate_limiter_priority =
@@ -982,7 +1002,7 @@ Status MultiOpsTxnsStressTest::PointLookupTxn(ThreadState* thread,
 
   std::unique_ptr<Transaction> txn;
   WriteOptions wopts;
-  Status s = NewTxn(wopts, &txn);
+  Status s = NewTxn(wopts, thread, &txn);
   if (!s.ok()) {
     assert(!txn);
     thread->stats.AddErrors(1);
@@ -1026,7 +1046,7 @@ Status MultiOpsTxnsStressTest::RangeScanTxn(ThreadState* thread,
 
   std::unique_ptr<Transaction> txn;
   WriteOptions wopts;
-  Status s = NewTxn(wopts, &txn);
+  Status s = NewTxn(wopts, thread, &txn);
   if (!s.ok()) {
     assert(!txn);
     thread->stats.AddErrors(1);
@@ -1384,6 +1404,12 @@ Status MultiOpsTxnsStressTest::CommitAndCreateTimestampedSnapshotIfNeeded(
   } else {
     s = txn.Commit();
   }
+  if (!s.ok()) {
+    fprintf(stderr, "Txn %s commit failed with %s\n", txn.GetName().c_str(),
+            s.ToString().c_str());
+    fflush(stderr);
+  }
+
   assert(txn_db_);
   if (FLAGS_create_timestamped_snapshot_one_in > 0 &&
       thread->rand.OneInOpt(50000)) {

db_stress_tool/no_batched_ops_stress.cc

@@ -666,7 +666,7 @@ class NonBatchedOpsStressTest : public StressTest {
       if (FLAGS_rate_limit_auto_wal_flush) {
         wo.rate_limiter_priority = Env::IO_USER;
       }
-      Status s = NewTxn(wo, &txn);
+      Status s = NewTxn(wo, thread, &txn);
       if (!s.ok()) {
         fprintf(stderr, "NewTxn error: %s\n", s.ToString().c_str());
         shared->SafeTerminate();
@@ -1127,7 +1127,7 @@ class NonBatchedOpsStressTest : public StressTest {
         write_options.rate_limiter_priority = Env::IO_USER;
       }
 
-      const Status s = NewTxn(write_options, &txn);
+      const Status s = NewTxn(write_options, thread, &txn);
       if (!s.ok()) {
         fprintf(stderr, "NewTxn error: %s\n", s.ToString().c_str());
         thread->shared->SafeTerminate();
@@ -1652,6 +1652,7 @@ class NonBatchedOpsStressTest : public StressTest {
     // To track whether WAL write may have succeeded during the initial failed
     // write
     bool initial_wal_write_may_succeed = true;
+    bool commit_bypass_memtable = false;
 
     PendingExpectedValue pending_expected_value =
         shared->PreparePut(rand_column_family, rand_key);
@@ -1713,9 +1714,10 @@ class NonBatchedOpsStressTest : public StressTest {
             s = db_->Put(write_opts, cfh, k, write_ts, v);
           }
         } else {
-          s = ExecuteTransaction(write_opts, thread, [&](Transaction& txn) {
-            return txn.Put(cfh, k, v);
-          });
+          s = ExecuteTransaction(
+              write_opts, thread,
+              [&](Transaction& txn) { return txn.Put(cfh, k, v); },
+              &commit_bypass_memtable);
         }
       }
       UpdateIfInitialWriteFails(db_stress_env, s, &initial_write_s,
@@ -1779,6 +1781,7 @@ class NonBatchedOpsStressTest : public StressTest {
     // To track whether WAL write may have succeeded during the initial failed
     // write
     bool initial_wal_write_may_succeed = true;
+    bool commit_bypass_memtable = false;
 
     // Use delete if the key may be overwritten and a single deletion
     // otherwise.
@@ -1803,13 +1806,14 @@ class NonBatchedOpsStressTest : public StressTest {
             s = db_->Delete(write_opts, cfh, key, write_ts);
           }
         } else {
-          s = ExecuteTransaction(write_opts, thread, [&](Transaction& txn) {
-            return txn.Delete(cfh, key);
-          });
+          s = ExecuteTransaction(
+              write_opts, thread,
+              [&](Transaction& txn) { return txn.Delete(cfh, key); },
+              &commit_bypass_memtable);
         }
-        UpdateIfInitialWriteFails(db_stress_env, s, &initial_write_s,
-                                  &initial_wal_write_may_succeed,
-                                  &wait_for_recover_start_time);
+        UpdateIfInitialWriteFails(
+            db_stress_env, s, &initial_write_s, &initial_wal_write_may_succeed,
+            &wait_for_recover_start_time, commit_bypass_memtable);
       } while (!s.ok() && IsErrorInjectedAndRetryable(s) &&
                initial_wal_write_may_succeed);
 
@@ -1859,13 +1863,14 @@ class NonBatchedOpsStressTest : public StressTest {
             s = db_->SingleDelete(write_opts, cfh, key, write_ts);
           }
         } else {
-          s = ExecuteTransaction(write_opts, thread, [&](Transaction& txn) {
-            return txn.SingleDelete(cfh, key);
-          });
+          s = ExecuteTransaction(
+              write_opts, thread,
+              [&](Transaction& txn) { return txn.SingleDelete(cfh, key); },
+              &commit_bypass_memtable);
         }
-        UpdateIfInitialWriteFails(db_stress_env, s, &initial_write_s,
-                                  &initial_wal_write_may_succeed,
-                                  &wait_for_recover_start_time);
+        UpdateIfInitialWriteFails(
+            db_stress_env, s, &initial_write_s, &initial_wal_write_may_succeed,
+            &wait_for_recover_start_time, commit_bypass_memtable);
       } while (!s.ok() && IsErrorInjectedAndRetryable(s) &&
                initial_wal_write_may_succeed);
 
@@ -2875,7 +2880,7 @@ class NonBatchedOpsStressTest : public StressTest {
           const size_t sz = GenerateValue(value_base, value, sizeof(value));
           const Slice v(value, sz);
 
-          if (op == Op::PutOrPutEntity) {
+          if (op == Op::PutOrPutEntity || !FLAGS_use_merge) {
             if (FLAGS_use_put_entity_one_in > 0 &&
                 (value_base % FLAGS_use_put_entity_one_in) == 0) {
               s = txn->PutEntity(cfh, k, GenerateWideColumns(value_base, v));

env/io_posix.cc

@@ -28,7 +28,7 @@
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
-#ifdef OS_LINUX
+#if defined(OS_LINUX) || defined(OS_ANDROID)
 #include <sys/statfs.h>
 #include <sys/sysmacros.h>
 #endif
@@ -1421,9 +1421,10 @@ IOStatus PosixWritableFile::Close(const IOOptions& /*opts*/,
     // After ftruncate, we check whether ftruncate has the correct behavior.
     // If not, we should hack it with FALLOC_FL_PUNCH_HOLE
     if (result == 0 &&
-        (file_stats.st_size + file_stats.st_blksize - 1) /
-                file_stats.st_blksize !=
-            file_stats.st_blocks / (file_stats.st_blksize / 512)) {
+        static_cast<size_t>((file_stats.st_size + file_stats.st_blksize - 1) /
+                            file_stats.st_blksize) !=
+            static_cast<size_t>(file_stats.st_blocks /
+                                (file_stats.st_blksize / 512))) {
       IOSTATS_TIMER_GUARD(allocate_nanos);
       if (allow_fallocate_) {
         fallocate(fd_, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, filesize_,

env/io_posix.h

@@ -30,7 +30,7 @@
 // For non linux platform, the following macros are used only as place
 // holder.
 #if !(defined OS_LINUX) && !(defined OS_FREEBSD) && !(defined CYGWIN) && \
-    !(defined OS_AIX)
+    !(defined OS_AIX) && !(defined OS_ANDROID)
 #define POSIX_FADV_NORMAL 0     /* [MC1] no further special treatment */
 #define POSIX_FADV_RANDOM 1     /* [MC1] expect random page refs */
 #define POSIX_FADV_SEQUENTIAL 2 /* [MC1] expect sequential page refs */

file/file_prefetch_buffer.cc

@@ -22,11 +22,9 @@
 
 namespace ROCKSDB_NAMESPACE {
 
-void FilePrefetchBuffer::PrepareBufferForRead(BufferInfo* buf, size_t alignment,
-                                              uint64_t offset,
-                                              size_t roundup_len,
-                                              bool refit_tail,
-                                              uint64_t& aligned_useful_len) {
+void FilePrefetchBuffer::PrepareBufferForRead(
+    BufferInfo* buf, size_t alignment, uint64_t offset, size_t roundup_len,
+    bool refit_tail, bool use_fs_buffer, uint64_t& aligned_useful_len) {
   uint64_t aligned_useful_offset_in_buf = 0;
   bool copy_data_to_new_buffer = false;
   // Check if requested bytes are in the existing buffer_.
@@ -39,6 +37,9 @@ void FilePrefetchBuffer::PrepareBufferForRead(BufferInfo* buf, size_t alignment,
     // new buffer is created.
     aligned_useful_offset_in_buf =
         Rounddown(static_cast<size_t>(offset - buf->offset_), alignment);
+    // aligned_useful_len is passed by reference and used to calculate how much
+    // data needs to be read, so it is needed regardless of whether
+    // use_fs_buffer is true
     aligned_useful_len = static_cast<uint64_t>(buf->CurrentSize()) -
                          aligned_useful_offset_in_buf;
     assert(aligned_useful_offset_in_buf % alignment == 0);
@@ -53,6 +54,16 @@ void FilePrefetchBuffer::PrepareBufferForRead(BufferInfo* buf, size_t alignment,
     }
   }
 
+  // The later buffer allocation / tail refitting does not apply when
+  // use_fs_buffer is true. If we allocate a new buffer, we end up throwing it
+  // away later when we reuse the file system allocated buffer. If we refit
+  // the tail in the main buffer, we don't have a place to put the next chunk of
+  // data provided by the file system (without performing another copy, which we
+  // are trying to avoid in the first place)
+  if (use_fs_buffer) {
+    return;
+  }
+
   // Create a new buffer only if current capacity is not sufficient, and memcopy
   // bytes from old buffer if needed (i.e., if aligned_useful_len is greater
   // than 0).
@@ -62,8 +73,8 @@ void FilePrefetchBuffer::PrepareBufferForRead(BufferInfo* buf, size_t alignment,
         static_cast<size_t>(roundup_len), copy_data_to_new_buffer,
         aligned_useful_offset_in_buf, static_cast<size_t>(aligned_useful_len));
   } else if (aligned_useful_len > 0 && refit_tail) {
-    // New buffer not needed. But memmove bytes from tail to the beginning since
-    // aligned_useful_len is greater than 0.
+    // New buffer not needed. But memmove bytes from tail to the beginning
+    // since aligned_useful_len is greater than 0.
     buf->buffer_.RefitTail(static_cast<size_t>(aligned_useful_offset_in_buf),
                            static_cast<size_t>(aligned_useful_len));
   } else if (aligned_useful_len > 0) {
@@ -82,11 +93,19 @@ void FilePrefetchBuffer::PrepareBufferForRead(BufferInfo* buf, size_t alignment,
 Status FilePrefetchBuffer::Read(BufferInfo* buf, const IOOptions& opts,
                                 RandomAccessFileReader* reader,
                                 uint64_t read_len, uint64_t aligned_useful_len,
-                                uint64_t start_offset) {
+                                uint64_t start_offset, bool use_fs_buffer) {
   Slice result;
-  char* to_buf = buf->buffer_.BufferStart() + aligned_useful_len;
-  Status s = reader->Read(opts, start_offset + aligned_useful_len, read_len,
-                          &result, to_buf, /*aligned_buf=*/nullptr);
+  Status s;
+  char* to_buf = nullptr;
+  if (use_fs_buffer) {
+    s = FSBufferDirectRead(reader, buf, opts, start_offset + aligned_useful_len,
+                           read_len, result);
+  } else {
+    to_buf = buf->buffer_.BufferStart() + aligned_useful_len;
+    s = reader->Read(opts, start_offset + aligned_useful_len, read_len, &result,
+                     to_buf, /*aligned_buf=*/nullptr);
+  }
+
 #ifndef NDEBUG
   if (result.size() < read_len) {
     // Fake an IO error to force db_stress fault injection to ignore
@@ -97,7 +116,7 @@ Status FilePrefetchBuffer::Read(BufferInfo* buf, const IOOptions& opts,
   if (!s.ok()) {
     return s;
   }
-  if (result.data() != to_buf) {
+  if (!use_fs_buffer && result.data() != to_buf) {
     // If the read is coming from some other buffer already in memory (such as
     // mmap) then it would be inefficient to create another copy in this
     // FilePrefetchBuffer. The caller is expected to exclude this case.
@@ -108,8 +127,11 @@ Status FilePrefetchBuffer::Read(BufferInfo* buf, const IOOptions& opts,
   if (usage_ == FilePrefetchBufferUsage::kUserScanPrefetch) {
     RecordTick(stats_, PREFETCH_BYTES, read_len);
   }
-  // Update the buffer size.
-  buf->buffer_.Size(static_cast<size_t>(aligned_useful_len) + result.size());
+  if (!use_fs_buffer) {
+    // Update the buffer size.
+    // We already explicitly set the buffer size when we reuse the FS buffer
+    buf->buffer_.Size(static_cast<size_t>(aligned_useful_len) + result.size());
+  }
   return s;
 }
 
@@ -157,33 +179,42 @@ Status FilePrefetchBuffer::Prefetch(const IOOptions& opts,
     return Status::OK();
   }
 
-  size_t alignment = reader->file()->GetRequiredBufferAlignment();
+  size_t alignment = GetRequiredBufferAlignment(reader);
   uint64_t rounddown_offset = offset, roundup_end = 0, aligned_useful_len = 0;
   size_t read_len = 0;
+  // TODO: Enable file system buffer reuse optimization. Need to incorporate
+  // overlap buffer logic here (similar to what is done in PrefetchInternal).
+  // Currently, if we attempt to use the optimization, it results in an
+  // unsigned integer overflow because the returned buffer's offset ends up
+  // higher than the requested offset.
+  bool use_fs_buffer = false;
 
   ReadAheadSizeTuning(buf, /*read_curr_block=*/true,
-                      /*refit_tail=*/true, rounddown_offset, alignment, 0, n,
-                      rounddown_offset, roundup_end, read_len,
+                      /*refit_tail=*/true, use_fs_buffer, rounddown_offset,
+                      alignment, 0, n, rounddown_offset, roundup_end, read_len,
                       aligned_useful_len);
 
   Status s;
   if (read_len > 0) {
-    s = Read(buf, opts, reader, read_len, aligned_useful_len, rounddown_offset);
+    s = Read(buf, opts, reader, read_len, aligned_useful_len, rounddown_offset,
+             use_fs_buffer);
   }
 
   if (usage_ == FilePrefetchBufferUsage::kTableOpenPrefetchTail && s.ok()) {
     RecordInHistogram(stats_, TABLE_OPEN_PREFETCH_TAIL_READ_BYTES, read_len);
   }
+  assert(buf->offset_ <= offset);
   return s;
 }
 
 // Copy data from src to overlap_buf_.
-void FilePrefetchBuffer::CopyDataToBuffer(BufferInfo* src, uint64_t& offset,
-                                          size_t& length) {
+void FilePrefetchBuffer::CopyDataToOverlapBuffer(BufferInfo* src,
+                                                 uint64_t& offset,
+                                                 size_t& length) {
   if (length == 0) {
     return;
   }
-
+  assert(src->IsOffsetInBuffer(offset));
   uint64_t copy_offset = (offset - src->offset_);
   size_t copy_len = 0;
   if (src->IsDataBlockInBuffer(offset, length)) {
@@ -194,10 +225,8 @@ void FilePrefetchBuffer::CopyDataToBuffer(BufferInfo* src, uint64_t& offset,
   }
 
   BufferInfo* dst = overlap_buf_;
-  memcpy(dst->buffer_.BufferStart() + dst->CurrentSize(),
-         src->buffer_.BufferStart() + copy_offset, copy_len);
-
-  dst->buffer_.Size(dst->CurrentSize() + copy_len);
+  assert(copy_len <= dst->buffer_.Capacity() - dst->buffer_.CurrentSize());
+  dst->buffer_.Append(src->buffer_.BufferStart() + copy_offset, copy_len);
 
   // Update offset and length.
   offset += copy_len;
@@ -208,6 +237,7 @@ void FilePrefetchBuffer::CopyDataToBuffer(BufferInfo* src, uint64_t& offset,
   if (length > 0) {
     FreeFrontBuffer();
   }
+  TEST_SYNC_POINT("FilePrefetchBuffer::CopyDataToOverlapBuffer:Complete");
 }
 
 // Clear the buffers if it contains outdated data. Outdated data can be because
@@ -355,7 +385,7 @@ void FilePrefetchBuffer::PollIfNeeded(uint64_t offset, size_t length) {
 //                       of ReadAsync to make sure it doesn't read anything from
 //                       previous buffer which is already prefetched.
 void FilePrefetchBuffer::ReadAheadSizeTuning(
-    BufferInfo* buf, bool read_curr_block, bool refit_tail,
+    BufferInfo* buf, bool read_curr_block, bool refit_tail, bool use_fs_buffer,
     uint64_t prev_buf_end_offset, size_t alignment, size_t length,
     size_t readahead_size, uint64_t& start_offset, uint64_t& end_offset,
     size_t& read_len, uint64_t& aligned_useful_len) {
@@ -408,7 +438,7 @@ void FilePrefetchBuffer::ReadAheadSizeTuning(
   uint64_t roundup_len = end_offset - start_offset;
 
   PrepareBufferForRead(buf, alignment, start_offset, roundup_len, refit_tail,
-                       aligned_useful_len);
+                       use_fs_buffer, aligned_useful_len);
   assert(roundup_len >= aligned_useful_len);
 
   // Update the buffer offset.
@@ -422,11 +452,43 @@ void FilePrefetchBuffer::ReadAheadSizeTuning(
                              (end_offset - start_offset));
 }
 
+// This is for when num_buffers_ = 1.
+// If we are reusing the file system allocated buffer, and only some of the
+// requested data is in the buffer, we copy the relevant data to overlap_buf_
+void FilePrefetchBuffer::HandleOverlappingSyncData(uint64_t offset,
+                                                   size_t length,
+                                                   uint64_t& tmp_offset,
+                                                   size_t& tmp_length,
+                                                   bool& use_overlap_buffer) {
+  if (IsBufferQueueEmpty()) {
+    return;
+  }
+  BufferInfo* buf = GetFirstBuffer();
+  // We should only be calling this when num_buffers_ = 1, so there should
+  // not be any async reads.
+  assert(!buf->async_read_in_progress_);
+
+  if (!buf->async_read_in_progress_ && buf->DoesBufferContainData() &&
+      buf->IsOffsetInBuffer(offset) &&
+      buf->offset_ + buf->CurrentSize() < offset + length) {
+    // Allocated overlap_buf_ is just enough to hold the result for the user
+    // Alignment does not matter here
+    use_overlap_buffer = true;
+    overlap_buf_->ClearBuffer();
+    overlap_buf_->buffer_.Alignment(1);
+    overlap_buf_->buffer_.AllocateNewBuffer(length);
+    overlap_buf_->offset_ = offset;
+    CopyDataToOverlapBuffer(buf, tmp_offset, tmp_length);
+    UpdateStats(/*found_in_buffer=*/false, overlap_buf_->CurrentSize());
+  }
+}
+
+// This is for when num_buffers_ > 1.
 // If data is overlapping between two buffers then during this call:
 //   - data from first buffer is copied into overlapping buffer,
 //   - first is removed from bufs_ and freed so that it can be used for async
 //     prefetching of further data.
-Status FilePrefetchBuffer::HandleOverlappingData(
+Status FilePrefetchBuffer::HandleOverlappingAsyncData(
     const IOOptions& opts, RandomAccessFileReader* reader, uint64_t offset,
     size_t length, size_t readahead_size, bool& copy_to_overlap_buffer,
     uint64_t& tmp_offset, size_t& tmp_length) {
@@ -436,7 +498,7 @@ Status FilePrefetchBuffer::HandleOverlappingData(
   }
 
   Status s;
-  size_t alignment = reader->file()->GetRequiredBufferAlignment();
+  size_t alignment = GetRequiredBufferAlignment(reader);
 
   BufferInfo* buf = GetFirstBuffer();
 
@@ -470,7 +532,7 @@ Status FilePrefetchBuffer::HandleOverlappingData(
     overlap_buf_->offset_ = offset;
     copy_to_overlap_buffer = true;
 
-    CopyDataToBuffer(buf, tmp_offset, tmp_length);
+    CopyDataToOverlapBuffer(buf, tmp_offset, tmp_length);
     UpdateStats(/*found_in_buffer=*/false, overlap_buf_->CurrentSize());
 
     // Call async prefetching on freed buffer since data has been consumed
@@ -495,8 +557,8 @@ Status FilePrefetchBuffer::HandleOverlappingData(
       uint64_t end_offset = start_offset, aligned_useful_len = 0;
 
       ReadAheadSizeTuning(new_buf, /*read_curr_block=*/false,
-                          /*refit_tail=*/false, next_buf->offset_ + second_size,
-                          alignment,
+                          /*refit_tail=*/false, /*use_fs_buffer=*/false,
+                          next_buf->offset_ + second_size, alignment,
                           /*length=*/0, readahead_size, start_offset,
                           end_offset, read_len, aligned_useful_len);
       if (read_len > 0) {
@@ -537,7 +599,7 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
 
   TEST_SYNC_POINT("FilePrefetchBuffer::Prefetch:Start");
 
-  size_t alignment = reader->file()->GetRequiredBufferAlignment();
+  size_t alignment = GetRequiredBufferAlignment(reader);
   Status s;
   uint64_t tmp_offset = offset;
   size_t tmp_length = length;
@@ -550,12 +612,20 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
   }
   ClearOutdatedData(offset, length);
 
-  // Handle overlapping data over two buffers.
-  s = HandleOverlappingData(opts, reader, offset, length, readahead_size,
-                            copy_to_overlap_buffer, tmp_offset, tmp_length);
+  // Handle overlapping data over two buffers (async prefetching case).
+  s = HandleOverlappingAsyncData(opts, reader, offset, length, readahead_size,
+                                 copy_to_overlap_buffer, tmp_offset,
+                                 tmp_length);
   if (!s.ok()) {
     return s;
   }
+  // Handle partially available data when reusing the file system buffer
+  // and num_buffers_ = 1 (sync prefetching case)
+  bool use_fs_buffer = UseFSBuffer(reader);
+  if (!copy_to_overlap_buffer && use_fs_buffer) {
+    HandleOverlappingSyncData(offset, length, tmp_offset, tmp_length,
+                              copy_to_overlap_buffer);
+  }
 
   AllocateBufferIfEmpty();
   BufferInfo* buf = GetFirstBuffer();
@@ -586,8 +656,18 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
     if (copy_to_overlap_buffer) {
       // Data is overlapping i.e. some of the data has been copied to overlap
       // buffer and remaining will be updated below.
+      // Note: why do we not end up performing a duplicate copy when we already
+      // copy to the overlap buffer in HandleOverlappingAsyncData /
+      // HandleOverlappingSyncData? The reason is that when we call
+      // CopyDataToOverlapBuffer, if the buffer is only a "partial hit", then we
+      // clear it out since it does not have any more useful data once we copy
+      // to the overlap buffer. Once we reallocate a fresh buffer, that buffer
+      // will have no data, and it will be the "first" buffer when num_buffers_
+      // = 1. When num_buffers_ > 1, we call ClearOutdatedData() so we know
+      // that, if we get to this point in the control flow, the "front" buffer
+      // has to have the data we need.
       size_t initial_buf_size = overlap_buf_->CurrentSize();
-      CopyDataToBuffer(buf, offset, length);
+      CopyDataToOverlapBuffer(buf, offset, length);
       UpdateStats(
           /*found_in_buffer=*/false,
           overlap_buf_->CurrentSize() - initial_buf_size);
@@ -636,10 +716,10 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
       UpdateStats(/*found_in_buffer=*/false,
                   (buf->offset_ + buf->CurrentSize() - offset));
     }
-    ReadAheadSizeTuning(buf, /*read_curr_block=*/true, /*refit_tail*/
-                        true, start_offset1, alignment, length, readahead_size,
-                        start_offset1, end_offset1, read_len1,
-                        aligned_useful_len1);
+    ReadAheadSizeTuning(buf, /*read_curr_block=*/true, /*refit_tail=*/
+                        true, /*use_fs_buffer=*/use_fs_buffer, start_offset1,
+                        alignment, length, readahead_size, start_offset1,
+                        end_offset1, read_len1, aligned_useful_len1);
   } else {
     UpdateStats(/*found_in_buffer=*/true, original_length);
   }
@@ -654,7 +734,8 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
   }
 
   if (read_len1 > 0) {
-    s = Read(buf, opts, reader, read_len1, aligned_useful_len1, start_offset1);
+    s = Read(buf, opts, reader, read_len1, aligned_useful_len1, start_offset1,
+             use_fs_buffer);
     if (!s.ok()) {
       AbortAllIOs();
       FreeAllBuffers();
@@ -662,10 +743,10 @@ Status FilePrefetchBuffer::PrefetchInternal(const IOOptions& opts,
     }
   }
 
-  // Copy remaining requested bytes to overlap_buffer. No need to update stats
-  // as data is prefetched during this call.
+  // Copy remaining requested bytes to overlap_buf_. No need to
+  // update stats as data is prefetched during this call.
   if (copy_to_overlap_buffer && length > 0) {
-    CopyDataToBuffer(buf, offset, length);
+    CopyDataToOverlapBuffer(buf, offset, length);
   }
   return s;
 }
@@ -782,6 +863,7 @@ bool FilePrefetchBuffer::TryReadFromCacheUntracked(
   if (copy_to_overlap_buffer) {
     buf = overlap_buf_;
   }
+  assert(buf->offset_ <= offset);
   uint64_t offset_in_buffer = offset - buf->offset_;
   *result = Slice(buf->buffer_.BufferStart() + offset_in_buffer, n);
   if (prefetched) {
@@ -892,7 +974,7 @@ Status FilePrefetchBuffer::PrefetchAsync(const IOOptions& opts,
   std::string msg;
 
   Status s;
-  size_t alignment = reader->file()->GetRequiredBufferAlignment();
+  size_t alignment = GetRequiredBufferAlignment(reader);
   size_t readahead_size = is_eligible_for_prefetching ? readahead_size_ / 2 : 0;
   size_t offset_to_read = static_cast<size_t>(offset);
   uint64_t start_offset1 = offset, end_offset1 = 0, aligned_useful_len1 = 0;
@@ -915,6 +997,7 @@ Status FilePrefetchBuffer::PrefetchAsync(const IOOptions& opts,
     // Prefetch full data + readahead_size in the first buffer.
     if (is_eligible_for_prefetching || reader->use_direct_io()) {
       ReadAheadSizeTuning(buf, /*read_curr_block=*/true, /*refit_tail=*/false,
+                          /*use_fs_buffer=*/false,
                           /*prev_buf_end_offset=*/start_offset1, alignment, n,
                           readahead_size, start_offset1, end_offset1, read_len1,
                           aligned_useful_len1);
@@ -923,7 +1006,8 @@ Status FilePrefetchBuffer::PrefetchAsync(const IOOptions& opts,
       start_offset1 = offset_to_read;
       end_offset1 = offset_to_read + n;
       roundup_len1 = end_offset1 - start_offset1;
-      PrepareBufferForRead(buf, alignment, start_offset1, roundup_len1, false,
+      PrepareBufferForRead(buf, alignment, start_offset1, roundup_len1,
+                           /*refit_tail=*/false, /*use_fs_buffer=*/false,
                            aligned_useful_len1);
       assert(aligned_useful_len1 == 0);
       assert(roundup_len1 >= aligned_useful_len1);
@@ -970,7 +1054,7 @@ Status FilePrefetchBuffer::PrefetchRemBuffers(const IOOptions& opts,
     uint64_t end_offset2 = start_offset2, aligned_useful_len2 = 0;
     size_t read_len2 = 0;
     ReadAheadSizeTuning(new_buf, /*read_curr_block=*/false,
-                        /*refit_tail=*/false,
+                        /*refit_tail=*/false, /*use_fs_buffer=*/false,
                         /*prev_buf_end_offset=*/end_offset1, alignment,
                         /*length=*/0, readahead_size, start_offset2,
                         end_offset2, read_len2, aligned_useful_len2);

file/file_prefetch_buffer.h

@@ -15,7 +15,9 @@
 #include <sstream>
 #include <string>
 
+#include "file/random_access_file_reader.h"
 #include "file/readahead_file_info.h"
+#include "file_util.h"
 #include "monitoring/statistics_impl.h"
 #include "port/port.h"
 #include "rocksdb/env.h"
@@ -149,6 +151,9 @@ enum class FilePrefetchBufferUsage {
 //
 // If num_buffers_ == 1, it's a sequential read flow. Read API will be called on
 // that one buffer whenever the data is requested and is not in the buffer.
+// When reusing the file system allocated buffer, overlap_buf_ is used if the
+// main buffer only contains part of the requested data. It is returned to
+// the caller after the remaining data is fetched.
 // If num_buffers_ > 1, then the data is prefetched asynchronosuly in the
 // buffers whenever the data is consumed from the buffers and that buffer is
 // freed.
@@ -206,10 +211,15 @@ class FilePrefetchBuffer {
     assert((num_file_reads_ >= num_file_reads_for_auto_readahead_ + 1) ||
            (num_file_reads_ == 0));
 
-    // If num_buffers_ > 1, data is asynchronously filled in the
-    // queue. As result, data can be overlapping in two buffers. It copies the
-    // data to overlap_buf_ in order to to return continuous buffer.
-    if (num_buffers_ > 1) {
+    // overlap_buf_ is used whenever the main buffer only has part of the
+    // requested data. The relevant data is copied into overlap_buf_ and the
+    // remaining data is copied in later to satisfy the user's request. This is
+    // used in both the synchronous (num_buffers_ = 1) and asynchronous
+    // (num_buffers_ > 1) cases. In the asynchronous case, the requested data
+    // may be spread out over 2 buffers.
+    if (num_buffers_ > 1 ||
+        (fs_ != nullptr &&
+         CheckFSFeatureSupport(fs_, FSSupportedOps::kFSBuffer))) {
       overlap_buf_ = new BufferInfo();
     }
 
@@ -379,12 +389,21 @@ class FilePrefetchBuffer {
   void PrefetchAsyncCallback(FSReadRequest& req, void* cb_arg);
 
   void TEST_GetBufferOffsetandSize(
-      std::vector<std::pair<uint64_t, size_t>>& buffer_info) {
+      std::vector<std::tuple<uint64_t, size_t, bool>>& buffer_info) {
     for (size_t i = 0; i < bufs_.size(); i++) {
-      buffer_info[i].first = bufs_[i]->offset_;
-      buffer_info[i].second = bufs_[i]->async_read_in_progress_
-                                  ? bufs_[i]->async_req_len_
-                                  : bufs_[i]->CurrentSize();
+      std::get<0>(buffer_info[i]) = bufs_[i]->offset_;
+      std::get<1>(buffer_info[i]) = bufs_[i]->async_read_in_progress_
+                                        ? bufs_[i]->async_req_len_
+                                        : bufs_[i]->CurrentSize();
+      std::get<2>(buffer_info[i]) = bufs_[i]->async_read_in_progress_;
+    }
+  }
+
+  void TEST_GetOverlapBufferOffsetandSize(
+      std::pair<uint64_t, size_t>& buffer_info) {
+    if (overlap_buf_ != nullptr) {
+      buffer_info.first = overlap_buf_->offset_;
+      buffer_info.second = overlap_buf_->CurrentSize();
     }
   }
 
@@ -394,7 +413,7 @@ class FilePrefetchBuffer {
   // required.
   void PrepareBufferForRead(BufferInfo* buf, size_t alignment, uint64_t offset,
                             size_t roundup_len, bool refit_tail,
-                            uint64_t& aligned_useful_len);
+                            bool use_fs_buffer, uint64_t& aligned_useful_len);
 
   void AbortOutdatedIO(uint64_t offset);
 
@@ -411,14 +430,16 @@ class FilePrefetchBuffer {
 
   Status Read(BufferInfo* buf, const IOOptions& opts,
               RandomAccessFileReader* reader, uint64_t read_len,
-              uint64_t aligned_useful_len, uint64_t start_offset);
+              uint64_t aligned_useful_len, uint64_t start_offset,
+              bool use_fs_buffer);
 
   Status ReadAsync(BufferInfo* buf, const IOOptions& opts,
                    RandomAccessFileReader* reader, uint64_t read_len,
                    uint64_t start_offset);
 
   // Copy the data from src to overlap_buf_.
-  void CopyDataToBuffer(BufferInfo* src, uint64_t& offset, size_t& length);
+  void CopyDataToOverlapBuffer(BufferInfo* src, uint64_t& offset,
+                               size_t& length);
 
   bool IsBlockSequential(const size_t& offset) {
     return (prev_len_ == 0 || (prev_offset_ + prev_len_ == offset));
@@ -465,6 +486,49 @@ class FilePrefetchBuffer {
     return true;
   }
 
+  // Whether we reuse the file system provided buffer
+  // Until we also handle the async read case, only enable this optimization
+  // for the synchronous case when num_buffers_ = 1.
+  bool UseFSBuffer(RandomAccessFileReader* reader) {
+    return reader->file() != nullptr && !reader->use_direct_io() &&
+           fs_ != nullptr &&
+           CheckFSFeatureSupport(fs_, FSSupportedOps::kFSBuffer) &&
+           num_buffers_ == 1;
+  }
+
+  // When we are reusing the file system provided buffer, we are not concerned
+  // with alignment. However, quite a bit of prefetch code incorporates
+  // alignment, so we can put in 1 to keep the code simpler.
+  size_t GetRequiredBufferAlignment(RandomAccessFileReader* reader) {
+    if (UseFSBuffer(reader)) {
+      return 1;
+    }
+    return reader->file()->GetRequiredBufferAlignment();
+  }
+
+  // Reuses the file system allocated buffer to avoid an extra copy
+  IOStatus FSBufferDirectRead(RandomAccessFileReader* reader, BufferInfo* buf,
+                              const IOOptions& opts, uint64_t offset, size_t n,
+                              Slice& result) {
+    FSReadRequest read_req;
+    read_req.offset = offset;
+    read_req.len = n;
+    read_req.scratch = nullptr;
+    IOStatus s = reader->MultiRead(opts, &read_req, 1, nullptr);
+    if (!s.ok()) {
+      return s;
+    }
+    s = read_req.status;
+    if (!s.ok()) {
+      return s;
+    }
+    buf->buffer_.SetBuffer(read_req.result, std::move(read_req.fs_scratch));
+    buf->offset_ = offset;
+    buf->initial_end_offset_ = offset + read_req.result.size();
+    result = read_req.result;
+    return s;
+  }
+
   void DestroyAndClearIOHandle(BufferInfo* buf) {
     if (buf->io_handle_ != nullptr && buf->del_fn_ != nullptr) {
       buf->del_fn_(buf->io_handle_);
@@ -474,24 +538,28 @@ class FilePrefetchBuffer {
     buf->async_read_in_progress_ = false;
   }
 
-  Status HandleOverlappingData(const IOOptions& opts,
-                               RandomAccessFileReader* reader, uint64_t offset,
-                               size_t length, size_t readahead_size,
-                               bool& copy_to_third_buffer, uint64_t& tmp_offset,
-                               size_t& tmp_length);
+  void HandleOverlappingSyncData(uint64_t offset, size_t length,
+                                 uint64_t& tmp_offset, size_t& tmp_length,
+                                 bool& use_overlap_buffer);
+
+  Status HandleOverlappingAsyncData(const IOOptions& opts,
+                                    RandomAccessFileReader* reader,
+                                    uint64_t offset, size_t length,
+                                    size_t readahead_size,
+                                    bool& copy_to_third_buffer,
+                                    uint64_t& tmp_offset, size_t& tmp_length);
 
   bool TryReadFromCacheUntracked(const IOOptions& opts,
                                  RandomAccessFileReader* reader,
                                  uint64_t offset, size_t n, Slice* result,
-                                 Status* s,
-                                 bool for_compaction = false);
+                                 Status* s, bool for_compaction = false);
 
   void ReadAheadSizeTuning(BufferInfo* buf, bool read_curr_block,
-                           bool refit_tail, uint64_t prev_buf_end_offset,
-                           size_t alignment, size_t length,
-                           size_t readahead_size, uint64_t& offset,
-                           uint64_t& end_offset, size_t& read_len,
-                           uint64_t& aligned_useful_len);
+                           bool refit_tail, bool use_fs_buffer,
+                           uint64_t prev_buf_end_offset, size_t alignment,
+                           size_t length, size_t readahead_size,
+                           uint64_t& offset, uint64_t& end_offset,
+                           size_t& read_len, uint64_t& aligned_useful_len);
 
   void UpdateStats(bool found_in_buffer, size_t length_found) {
     if (found_in_buffer) {

file/prefetch_test.cc

@@ -3290,6 +3290,482 @@ TEST_F(FilePrefetchBufferTest, SyncReadaheadStats) {
       /* 24576(end offset of the buffer) - 16000(requested offset) =*/8576);
 }
 
+class FSBufferPrefetchTest : public testing::Test,
+                             public ::testing::WithParamInterface<bool> {
+ public:
+  // Mock file system supporting the kFSBuffer buffer reuse operation
+  class BufferReuseFS : public FileSystemWrapper {
+   public:
+    explicit BufferReuseFS(const std::shared_ptr<FileSystem>& _target)
+        : FileSystemWrapper(_target) {}
+    ~BufferReuseFS() override {}
+    const char* Name() const override { return "BufferReuseFS"; }
+
+    IOStatus NewRandomAccessFile(const std::string& fname,
+                                 const FileOptions& opts,
+                                 std::unique_ptr<FSRandomAccessFile>* result,
+                                 IODebugContext* dbg) override {
+      class WrappedRandomAccessFile : public FSRandomAccessFileOwnerWrapper {
+       public:
+        explicit WrappedRandomAccessFile(
+            std::unique_ptr<FSRandomAccessFile>& file)
+            : FSRandomAccessFileOwnerWrapper(std::move(file)) {}
+
+        IOStatus MultiRead(FSReadRequest* reqs, size_t num_reqs,
+                           const IOOptions& options,
+                           IODebugContext* dbg) override {
+          for (size_t i = 0; i < num_reqs; ++i) {
+            FSReadRequest& req = reqs[i];
+            FSAllocationPtr buffer(new char[req.len], [](void* ptr) {
+              delete[] static_cast<char*>(ptr);
+            });
+            req.fs_scratch = std::move(buffer);
+            req.status = Read(req.offset, req.len, options, &req.result,
+                              static_cast<char*>(req.fs_scratch.get()), dbg);
+          }
+          return IOStatus::OK();
+        }
+      };
+
+      std::unique_ptr<FSRandomAccessFile> file;
+      IOStatus s = target()->NewRandomAccessFile(fname, opts, &file, dbg);
+      EXPECT_OK(s);
+      result->reset(new WrappedRandomAccessFile(file));
+      return s;
+    }
+
+    void SupportedOps(int64_t& supported_ops) override {
+      supported_ops = 1 << FSSupportedOps::kAsyncIO;
+      supported_ops |= 1 << FSSupportedOps::kFSBuffer;
+    }
+  };
+
+  void SetUp() override {
+    SetupSyncPointsToMockDirectIO();
+    env_ = Env::Default();
+    bool use_async_prefetch = GetParam();
+    if (use_async_prefetch) {
+      fs_ = FileSystem::Default();
+    } else {
+      fs_ = std::make_shared<BufferReuseFS>(FileSystem::Default());
+    }
+
+    test_dir_ = test::PerThreadDBPath("fs_buffer_prefetch_test");
+    ASSERT_OK(fs_->CreateDir(test_dir_, IOOptions(), nullptr));
+    stats_ = CreateDBStatistics();
+  }
+
+  void TearDown() override { EXPECT_OK(DestroyDir(env_, test_dir_)); }
+
+  void Write(const std::string& fname, const std::string& content) {
+    std::unique_ptr<FSWritableFile> f;
+    ASSERT_OK(fs_->NewWritableFile(Path(fname), FileOptions(), &f, nullptr));
+    ASSERT_OK(f->Append(content, IOOptions(), nullptr));
+    ASSERT_OK(f->Close(IOOptions(), nullptr));
+  }
+
+  void Read(const std::string& fname, const FileOptions& opts,
+            std::unique_ptr<RandomAccessFileReader>* reader) {
+    std::string fpath = Path(fname);
+    std::unique_ptr<FSRandomAccessFile> f;
+    ASSERT_OK(fs_->NewRandomAccessFile(fpath, opts, &f, nullptr));
+    reader->reset(new RandomAccessFileReader(
+        std::move(f), fpath, env_->GetSystemClock().get(),
+        /*io_tracer=*/nullptr, stats_.get()));
+  }
+
+  FileSystem* fs() { return fs_.get(); }
+  Statistics* stats() { return stats_.get(); }
+  SystemClock* clock() { return env_->GetSystemClock().get(); }
+
+ private:
+  Env* env_;
+  std::shared_ptr<FileSystem> fs_;
+  std::string test_dir_;
+  std::shared_ptr<Statistics> stats_;
+
+  std::string Path(const std::string& fname) { return test_dir_ + "/" + fname; }
+};
+
+INSTANTIATE_TEST_CASE_P(FSBufferPrefetchTest, FSBufferPrefetchTest,
+                        ::testing::Bool());
+
+TEST_P(FSBufferPrefetchTest, FSBufferPrefetchStatsInternals) {
+  // Check that the main buffer, the overlap_buf_, and the secondary buffer (in
+  // the case of num_buffers_ > 1) are populated correctly while reading a 32
+  // KiB file
+  std::string fname = "fs-buffer-prefetch-stats-internals";
+  Random rand(0);
+  std::string content = rand.RandomString(32768);
+  Write(fname, content);
+
+  FileOptions opts;
+  std::unique_ptr<RandomAccessFileReader> r;
+  Read(fname, opts, &r);
+
+  std::shared_ptr<Statistics> stats = CreateDBStatistics();
+  ReadaheadParams readahead_params;
+  readahead_params.initial_readahead_size = 8192;
+  readahead_params.max_readahead_size = 8192;
+  bool use_async_prefetch = GetParam();
+  size_t num_buffers = use_async_prefetch ? 2 : 1;
+  readahead_params.num_buffers = num_buffers;
+
+  FilePrefetchBuffer fpb(readahead_params, true, false, fs(), clock(),
+                         stats.get());
+
+  int overlap_buffer_write_ct = 0;
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "FilePrefetchBuffer::CopyDataToOverlapBuffer:Complete",
+      [&](void* /*arg*/) { overlap_buffer_write_ct++; });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  Slice result;
+  // Read 4096 bytes at offset 0.
+  Status s;
+  std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(num_buffers);
+  std::pair<uint64_t, size_t> overlap_buffer_info;
+  bool could_read_from_cache =
+      fpb.TryReadFromCache(IOOptions(), r.get(), 0, 4096, &result, &s);
+  // Platforms that don't have IO uring may not support async IO.
+  if (use_async_prefetch && s.IsNotSupported()) {
+    return;
+  }
+  ASSERT_TRUE(could_read_from_cache);
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_HITS), 0);
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_BYTES_USEFUL), 0);
+  ASSERT_EQ(strncmp(result.data(), content.substr(0, 4096).c_str(), 4096), 0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Cut the readahead of 8192 in half.
+    // Overlap buffer is not used
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // Buffers: 0-8192, 8192-12288
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 4096 + 8192 / 2);
+    ASSERT_EQ(std::get<0>(buffer_info[1]), 4096 + 8192 / 2);
+    ASSERT_EQ(std::get<1>(buffer_info[1]), 8192 / 2);
+  } else {
+    // Read at offset 0 with length 4096 + 8192 = 12288.
+    // Overlap buffer is not used
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // Main buffer contains the requested data + the 8192 of prefetched data
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 4096 + 8192);
+  }
+
+  // Simulate a block cache hit
+  fpb.UpdateReadPattern(4096, 4096, false);
+  ASSERT_TRUE(
+      fpb.TryReadFromCache(IOOptions(), r.get(), 8192, 8192, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_HITS), 0);
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_BYTES_USEFUL),
+            4096);  // 8192-12288
+  ASSERT_EQ(strncmp(result.data(), content.substr(8192, 8192).c_str(), 8192),
+            0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+
+  if (use_async_prefetch) {
+    // Our buffers were 0-8192, 8192-12288 at the start so we had some
+    // overlapping data in the second buffer
+    // We clean up outdated buffers so 0-8192 gets freed for more prefetching.
+    // Our remaining buffer 8192-12288 has data that we want, so we can reuse it
+    // We end up with: 8192-20480, 20480-24576
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 8192);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 8192 + 8192 / 2);
+    ASSERT_EQ(std::get<0>(buffer_info[1]), 8192 + (8192 + 8192 / 2));
+    ASSERT_EQ(std::get<1>(buffer_info[1]), 8192 / 2);
+  } else {
+    // We only have 0-12288 cached, so reading from 8192-16384 will trigger a
+    // prefetch up through 16384 + 8192 = 24576.
+    // Overlap buffer reuses bytes 8192 to 12288
+    ASSERT_EQ(overlap_buffer_info.first, 8192);
+    ASSERT_EQ(overlap_buffer_info.second, 8192);
+    ASSERT_EQ(overlap_buffer_write_ct, 2);
+    // We spill to the overlap buffer so the remaining buffer only has the
+    // missing and prefetched part
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 12288);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 12288);
+  }
+
+  ASSERT_TRUE(
+      fpb.TryReadFromCache(IOOptions(), r.get(), 12288, 4096, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_HITS), 1);
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_BYTES_USEFUL),
+            4096);  // 12288-16384
+  ASSERT_EQ(strncmp(result.data(), content.substr(12288, 4096).c_str(), 4096),
+            0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+
+  if (use_async_prefetch) {
+    // Same as before: 8192-20480, 20480-24576 (cache hit in first buffer)
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 8192);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 8192 + 8192 / 2);
+    ASSERT_EQ(std::get<0>(buffer_info[1]), 8192 + (8192 + 8192 / 2));
+    ASSERT_EQ(std::get<1>(buffer_info[1]), 8192 / 2);
+  } else {
+    // The main buffer has 12288-24576, so 12288-16384 is a cache hit.
+    // Overlap buffer does not get used
+    fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+    ASSERT_EQ(overlap_buffer_info.first, 8192);
+    ASSERT_EQ(overlap_buffer_info.second, 8192);
+    ASSERT_EQ(overlap_buffer_write_ct, 2);
+    // Main buffer stays the same
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 12288);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 12288);
+  }
+
+  // Read from 16000-26000 (start and end do not meet normal alignment)
+  ASSERT_TRUE(
+      fpb.TryReadFromCache(IOOptions(), r.get(), 16000, 10000, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(stats->getAndResetTickerCount(PREFETCH_HITS), 0);
+  ASSERT_EQ(
+      stats->getAndResetTickerCount(PREFETCH_BYTES_USEFUL),
+      /* 24576(end offset of the buffer) - 16000(requested offset) =*/8576);
+  ASSERT_EQ(strncmp(result.data(), content.substr(16000, 10000).c_str(), 10000),
+            0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Overlap buffer reuses bytes 16000 to 20480
+    ASSERT_EQ(overlap_buffer_info.first, 16000);
+    ASSERT_EQ(overlap_buffer_info.second, 10000);
+    // First 2 writes are reusing existing 2 buffers. Last write fills in
+    // what could not be found in either.
+    ASSERT_EQ(overlap_buffer_write_ct, 3);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 24576);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 32768 - 24576);
+    ASSERT_EQ(std::get<0>(buffer_info[1]), 32768);
+    ASSERT_EQ(std::get<1>(buffer_info[1]), 4096);
+    ASSERT_TRUE(std::get<2>(
+        buffer_info[1]));  // in progress async request (otherwise we should not
+                           // be getting 4096 for the size)
+  } else {
+    // Overlap buffer reuses bytes 16000 to 24576
+    ASSERT_EQ(overlap_buffer_info.first, 16000);
+    ASSERT_EQ(overlap_buffer_info.second, 10000);
+    ASSERT_EQ(overlap_buffer_write_ct, 4);
+    // Even if you try to readahead to offset 16000 + 10000 + 8192, there are
+    // only 32768 bytes in the original file
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 12288 + 12288);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 8192);
+  }
+}
+
+TEST_P(FSBufferPrefetchTest, FSBufferPrefetchUnalignedReads) {
+  // Check that the main buffer, the overlap_buf_, and the secondary buffer (in
+  // the case of num_buffers_ > 1) are populated correctly
+  // while reading with no regard to alignment
+  std::string fname = "fs-buffer-prefetch-unaligned-reads";
+  Random rand(0);
+  std::string content = rand.RandomString(1000);
+  Write(fname, content);
+
+  FileOptions opts;
+  std::unique_ptr<RandomAccessFileReader> r;
+  Read(fname, opts, &r);
+
+  std::shared_ptr<Statistics> stats = CreateDBStatistics();
+  ReadaheadParams readahead_params;
+  // Readahead size will double each time
+  readahead_params.initial_readahead_size = 5;
+  readahead_params.max_readahead_size = 100;
+  bool use_async_prefetch = GetParam();
+  size_t num_buffers = use_async_prefetch ? 2 : 1;
+  readahead_params.num_buffers = num_buffers;
+  FilePrefetchBuffer fpb(readahead_params, true, false, fs(), clock(),
+                         stats.get());
+
+  int overlap_buffer_write_ct = 0;
+  ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
+      "FilePrefetchBuffer::CopyDataToOverlapBuffer:Complete",
+      [&](void* /*arg*/) { overlap_buffer_write_ct++; });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  Slice result;
+  // Read 3 bytes at offset 5
+  Status s;
+  std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(num_buffers);
+  std::pair<uint64_t, size_t> overlap_buffer_info;
+  bool could_read_from_cache =
+      fpb.TryReadFromCache(IOOptions(), r.get(), 5, 3, &result, &s);
+  // Platforms that don't have IO uring may not support async IO.
+  if (use_async_prefetch && s.IsNotSupported()) {
+    return;
+  }
+  ASSERT_TRUE(could_read_from_cache);
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(5, 3).c_str(), 3), 0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Overlap buffer is not used
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // With async prefetching, we still try to align to 4096 bytes, so
+    // our main buffer read and secondary buffer prefetch are rounded up
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 1000);
+    // This buffer won't actually get filled up with data since there is nothing
+    // after 1000
+    ASSERT_EQ(std::get<0>(buffer_info[1]), 4096);
+    ASSERT_EQ(std::get<1>(buffer_info[1]), 4096);
+    ASSERT_TRUE(std::get<2>(buffer_info[1]));  // in progress async request
+  } else {
+    // Overlap buffer is not used
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // Main buffer contains the requested data + 5 of prefetched data (5 - 13)
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 5);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 3 + 5);
+  }
+
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 16, 7, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(16, 7).c_str(), 7), 0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Complete hit since we have the entire file loaded in the main buffer
+    // The remaining requests will be the same when use_async_prefetch is true
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 1000);
+  } else {
+    // Complete miss: read 7 bytes at offset 16
+    // Overlap buffer is not used (no partial hit)
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // Main buffer contains the requested data + 10 of prefetched data (16 - 33)
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 16);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 7 + 10);
+  }
+
+  // Go backwards
+  if (use_async_prefetch) {
+    ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 10, 8, &result, &s));
+  } else {
+    // TryReadFromCacheUntracked returns false since the offset
+    // requested is less than the start of our buffer
+    ASSERT_FALSE(
+        fpb.TryReadFromCache(IOOptions(), r.get(), 10, 8, &result, &s));
+  }
+  ASSERT_EQ(s, Status::OK());
+
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 27, 6, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(27, 6).c_str(), 6), 0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Complete hit since we have the entire file loaded in the main buffer
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 1000);
+  } else {
+    // Complete hit
+    // Overlap buffer still not used
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    // Main buffer unchanged
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 16);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 7 + 10);
+  }
+
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 30, 20, &result, &s));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(30, 20).c_str(), 20), 0);
+  fpb.TEST_GetOverlapBufferOffsetandSize(overlap_buffer_info);
+  fpb.TEST_GetBufferOffsetandSize(buffer_info);
+  if (use_async_prefetch) {
+    // Complete hit since we have the entire file loaded in the main buffer
+    ASSERT_EQ(overlap_buffer_info.first, 0);
+    ASSERT_EQ(overlap_buffer_info.second, 0);
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 0);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), 1000);
+  } else {
+    // Partial hit (overlapping with end of main buffer)
+    // Overlap buffer is used because we already had 30-33
+    ASSERT_EQ(overlap_buffer_info.first, 30);
+    ASSERT_EQ(overlap_buffer_info.second, 20);
+    ASSERT_EQ(overlap_buffer_write_ct, 2);
+    // Main buffer has up to offset 50 + 20 of prefetched data
+    ASSERT_EQ(std::get<0>(buffer_info[0]), 33);
+    ASSERT_EQ(std::get<1>(buffer_info[0]), (50 - 33) + 20);
+  }
+}
+
+TEST_P(FSBufferPrefetchTest, FSBufferPrefetchForCompaction) {
+  // Quick test to make sure file system buffer reuse is disabled for compaction
+  // reads. Will update once it is re-enabled
+  // Primarily making sure we do not hit unsigned integer overflow issues
+  std::string fname = "fs-buffer-prefetch-for-compaction";
+  Random rand(0);
+  std::string content = rand.RandomString(32768);
+  Write(fname, content);
+
+  FileOptions opts;
+  std::unique_ptr<RandomAccessFileReader> r;
+  Read(fname, opts, &r);
+
+  std::shared_ptr<Statistics> stats = CreateDBStatistics();
+  ReadaheadParams readahead_params;
+  readahead_params.initial_readahead_size = 8192;
+  readahead_params.max_readahead_size = 8192;
+  bool use_async_prefetch = GetParam();
+  // Async IO is not enabled for compaction prefetching
+  if (use_async_prefetch) {
+    return;
+  }
+  readahead_params.num_buffers = 1;
+
+  FilePrefetchBuffer fpb(readahead_params, true, false, fs(), clock(),
+                         stats.get());
+
+  Slice result;
+  Status s;
+  ASSERT_TRUE(
+      fpb.TryReadFromCache(IOOptions(), r.get(), 0, 4096, &result, &s, true));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(0, 4096).c_str(), 4096), 0);
+
+  fpb.UpdateReadPattern(4096, 4096, false);
+
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 8192, 8192, &result,
+                                   &s, true));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(8192, 8192).c_str(), 8192),
+            0);
+
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 12288, 4096, &result,
+                                   &s, true));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(12288, 4096).c_str(), 4096),
+            0);
+
+  // Read from 16000-26000 (start and end do not meet normal alignment)
+  ASSERT_TRUE(fpb.TryReadFromCache(IOOptions(), r.get(), 16000, 10000, &result,
+                                   &s, true));
+  ASSERT_EQ(s, Status::OK());
+  ASSERT_EQ(strncmp(result.data(), content.substr(16000, 10000).c_str(), 10000),
+            0);
+}
+
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

file/random_access_file_reader.cc

@@ -199,7 +199,7 @@ IOStatus RandomAccessFileReader::Read(const IOOptions& opts, uint64_t offset,
           buf.Read(scratch, offset_advance, res_len);
         } else {
           scratch = buf.BufferStart() + offset_advance;
-          aligned_buf->reset(buf.Release());
+          *aligned_buf = buf.Release();
         }
       }
       *result = Slice(scratch, res_len);
@@ -384,7 +384,7 @@ IOStatus RandomAccessFileReader::MultiRead(const IOOptions& opts,
         scratch += r.len;
       }
 
-      aligned_buf->reset(buf.Release());
+      *aligned_buf = buf.Release();
       fs_reqs = aligned_reqs.data();
       num_fs_reqs = aligned_reqs.size();
     }
@@ -598,8 +598,7 @@ void RandomAccessFileReader::ReadAsyncCallback(FSReadRequest& req,
         // Set aligned_buf provided by user without additional copy.
         user_req.scratch =
             read_async_info->buf_.BufferStart() + offset_advance_len;
-        read_async_info->user_aligned_buf_->reset(
-            read_async_info->buf_.Release());
+        *read_async_info->user_aligned_buf_ = read_async_info->buf_.Release();
       }
       user_req.result = Slice(user_req.scratch, res_len);
     } else {

file/random_access_file_reader.h

@@ -25,7 +25,7 @@ class Statistics;
 class HistogramImpl;
 class SystemClock;
 
-using AlignedBuf = std::unique_ptr<char[]>;
+using AlignedBuf = FSAllocationPtr;
 
 // Align the request r according to alignment and return the aligned result.
 FSReadRequest Align(const FSReadRequest& r, size_t alignment);

file/writable_file_writer.cc

@@ -107,6 +107,14 @@ IOStatus WritableFileWriter::Append(const IOOptions& opts, const Slice& data,
   // Flush only when buffered I/O
   if (!use_direct_io() && (buf_.Capacity() - buf_.CurrentSize()) < left) {
     if (buf_.CurrentSize() > 0) {
+      if (!buffered_data_with_checksum_) {
+        // If we're not calculating checksum of buffered data, fill the
+        // buffer before flushing so that the writes are aligned. This will
+        // benefit file system performance.
+        size_t appended = buf_.Append(src, left);
+        left -= appended;
+        src += appended;
+      }
       s = Flush(io_options);
       if (!s.ok()) {
         set_seen_error(s);

include/rocksdb/db.h

@@ -29,6 +29,7 @@
 #include "rocksdb/transaction_log.h"
 #include "rocksdb/types.h"
 #include "rocksdb/user_write_callback.h"
+#include "rocksdb/utilities/table_properties_collectors.h"
 #include "rocksdb/version.h"
 #include "rocksdb/wide_columns.h"
 
@@ -2012,6 +2013,15 @@ class DB {
       ColumnFamilyHandle* column_family, const Range* range, std::size_t n,
       TablePropertiesCollection* props) = 0;
 
+  // Get the table properties of files per level.
+  virtual Status GetPropertiesOfTablesForLevels(
+      ColumnFamilyHandle* /* column_family */,
+      std::vector<
+          std::unique_ptr<TablePropertiesCollection>>* /* levels_props */) {
+    return Status::NotSupported(
+        "GetPropertiesOfTablesForLevels() is not implemented.");
+  }
+
   virtual Status SuggestCompactRange(ColumnFamilyHandle* /*column_family*/,
                                      const Slice* /*begin*/,
                                      const Slice* /*end*/) {

include/rocksdb/env.h

@@ -458,6 +458,7 @@ class Env : public Customizable {
     kVerifyFileChecksums = 7,
     kGetEntity = 8,
     kMultiGetEntity = 9,
+    kReadManifest = 10,
     kUnknown,  // Keep last for easy array of non-unknowns
   };
 

include/rocksdb/experimental.h

@@ -30,6 +30,13 @@ Status SuggestCompactRange(DB* db, const Slice* begin, const Slice* end);
 Status PromoteL0(DB* db, ColumnFamilyHandle* column_family,
                  int target_level = 1);
 
+// This function is intended to be called with the DB closed and does not write
+// to the DB path. It will usually work on an open DB but may fail spuriously
+// in that case, and results may be out of date on return.
+Status GetFileChecksumsFromCurrentManifest(FileSystem* fs,
+                                           const std::string& dbname,
+                                           FileChecksumList* checksum_list);
+
 struct UpdateManifestForFilesStateOptions {
   // When true, read current file temperatures from FileSystem and update in
   // DB manifest when a temperature other than Unknown is reported and

include/rocksdb/file_system.h

@@ -821,54 +821,47 @@ struct FSReadRequest {
   // and will be passed to underlying FileSystem.
   char* scratch;
 
-  // Output parameter set by MultiRead() to point to the data buffer, and
-  // the number of valid bytes
+  // Output parameter set by MultiRead() to point to the start of the data
+  // buffer.
   //
-  // In case of asynchronous reads, this output parameter is set by Async Read
-  // APIs to point to the data buffer, and
-  // the number of valid bytes.
-  // Slice result should point to scratch i.e the data should
-  // always be read into scratch.
+  // When FSReadRequest::scratch is provided, this should point to
+  // FSReadRequest::scratch. When FSSupportedOps::kFSBuffer is enabled and
+  // FSReadRequest::scratch is nullptr, this points to the start of the
+  // data buffer allocated by the FileSystem.
+  //
+  // WARNING: Even with the FSSupportedOps::kFSBuffer optimization, you must
+  // still use result.data() to get the start of the actual data that was read.
+  // Do NOT treat FSReadRequest::fs_scratch as a char* to the start of a valid
+  // data buffer.
   Slice result;
 
   // Output parameter set by underlying FileSystem that represents status of
   // read request.
   IOStatus status;
 
-  // fs_scratch is a data buffer allocated and provided by underlying FileSystem
-  // to RocksDB during reads, when FS wants to provide its own buffer with data
-  // instead of using RocksDB provided FSReadRequest::scratch.
+  // fs_scratch is a unique pointer to an arbitrary object allocated by the
+  // underlying FileSystem.
   //
-  // FileSystem needs to provide a buffer and custom delete function. The
-  // lifecycle of fs_scratch until data is used by RocksDB. The buffer
-  // should be released by RocksDB using custom delete function provided in
-  // unique_ptr fs_scratch.
+  // Instead of having the FileSystem spend CPU cycles copying data into the
+  // FSReadRequest::scratch buffer provided by RocksDB, RocksDB can directly use
+  // a buffer allocated by the FileSystem.
   //
-  // Optimization benefits:
-  // This is helpful in cases where underlying FileSystem has to do additional
-  // copy of data to RocksDB provided buffer which can consume CPU cycles. It
-  // can be optimized by avoiding copying to RocksDB buffer and directly using
-  // FS provided buffer.
+  // This optimization is enabled for MultiReads (sync and async) with non
+  // direct io, when these conditions hold:
+  // 1. The FileSystem has overriden the SupportedOps() API and set
+  // FSSupportedOps::kFSBuffer.
+  // 2. FSReadRequest::scratch is set to nullptr.
   //
-  // How to enable:
-  // In order to enable this option, FS needs to override SupportedOps() API and
-  // set FSSupportedOps::kFSBuffer in SupportedOps() as:
-  //  {
-  //    supported_ops |= (1 << FSSupportedOps::kFSBuffer);
-  //  }
+  // RocksDB will:
+  // 1. Reuse the buffer allocated by the FileSystem.
+  // 2. Take ownership of the object managed by fs_scratch.
+  // 3. Handle invoking the custom deleter function from the FSAllocationPtr.
   //
-  // Work in progress:
-  // Right now it's only enabled for MultiReads (sync and async
-  // both) with non direct io.
-  // If RocksDB provide its own buffer (scratch) during reads, that's a
-  //  signal for FS to use RocksDB buffer.
-  // If FSSupportedOps::kFSBuffer is enabled and scratch == nullptr,
-  //   then FS have to provide its own buffer in fs_scratch.
-  //
-  // NOTE:
-  // - FSReadRequest::result should point to fs_scratch.
-  // - This is needed only if FSSupportedOps::kFSBuffer support is provided by
-  // underlying FS.
+  // WARNING: Do NOT assume that fs_scratch points to the start of the actual
+  // char* data returned by the read. As the type signature suggests, fs_scratch
+  // is a pointer to any arbitrary data type. Use result.data() to get a valid
+  // start to the real data. See https://github.com/facebook/rocksdb/pull/13189
+  // for more context.
   FSAllocationPtr fs_scratch;
 };
 

include/rocksdb/options.h

@@ -531,25 +531,6 @@ class CompactionService : public Customizable {
   virtual void OnInstallation(const std::string& /*scheduled_job_id*/,
                               CompactionServiceJobStatus /*status*/) {}
 
-  // Deprecated. Please implement Schedule() and Wait() API to handle remote
-  // compaction
-
-  // Start the remote compaction with `compaction_service_input`, which can be
-  // passed to `DB::OpenAndCompact()` on the remote side. `info` provides the
-  // information the user might want to know, which includes `job_id`.
-  virtual CompactionServiceJobStatus StartV2(
-      const CompactionServiceJobInfo& /*info*/,
-      const std::string& /*compaction_service_input*/) {
-    return CompactionServiceJobStatus::kUseLocal;
-  }
-
-  // Wait for remote compaction to finish.
-  virtual CompactionServiceJobStatus WaitForCompleteV2(
-      const CompactionServiceJobInfo& /*info*/,
-      std::string* /*compaction_service_result*/) {
-    return CompactionServiceJobStatus::kUseLocal;
-  }
-
   ~CompactionService() override = default;
 };
 
@@ -590,11 +571,19 @@ struct DBOptions {
   // Default: false
   bool error_if_exists = false;
 
-  // If true, RocksDB will aggressively check consistency of the data.
-  // Also, if any of the  writes to the database fails (Put, Delete, Merge,
-  // Write), the database will switch to read-only mode and fail all other
-  // Write operations.
-  // In most cases you want this to be set to true.
+  // If true, RocksDB does some pro-active and generally inexpensive checks
+  // for DB or data corruption, on top of usual protections such as block
+  // checksums. True also enters a read-only mode when a DB write fails;
+  // see DB::Resume().
+  //
+  // As most workloads value data correctness over availability, this option
+  // is on by default. Note that the name of this old option is potentially
+  // misleading, and other options and operations go further in proactive
+  // checking for corruption, including
+  // * paranoid_file_checks
+  // * paranoid_memory_checks
+  // * DB::VerifyChecksum()
+  //
   // Default: true
   bool paranoid_checks = true;
 

include/rocksdb/thread_status.h

@@ -64,6 +64,7 @@ struct ThreadStatus {
     OP_VERIFY_FILE_CHECKSUMS,
     OP_GETENTITY,
     OP_MULTIGETENTITY,
+    OP_READ_MANIFEST,
     NUM_OP_TYPES
   };
 

include/rocksdb/utilities/secondary_index.h

@@ -0,0 +1,102 @@
+//  Copyright (c) Meta Platforms, Inc. and affiliates.
+//
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+#pragma once
+
+#include <optional>
+#include <string>
+#include <variant>
+
+#include "rocksdb/rocksdb_namespace.h"
+#include "rocksdb/slice.h"
+#include "rocksdb/status.h"
+
+namespace ROCKSDB_NAMESPACE {
+
+class ColumnFamilyHandle;
+
+//       / \     UNDER CONSTRUCTION
+//      / ! \    UNDER CONSTRUCTION
+//     /-----\   UNDER CONSTRUCTION
+
+// A secondary index is an additional data structure built over a set of primary
+// key-values that enables efficiently querying key-values by value instead of
+// key. Both plain and wide-column key-values can be indexed, the latter on a
+// per-column basis. The secondary index then maintains a mapping from (column)
+// value to the list of primary keys that have the corresponding value (in the
+// given column).
+//
+// The primary and secondary key-value pairs can be stored in either the same
+// column family or different ones. It is the application's responsibility to
+// avoid conflicts and ambiguities (for example, by using prefixes to create
+// separate key spaces or using a dedicated column family for each secondary
+// index). Also, note that applications are not expected to manipulate secondary
+// index entries directly.
+//
+// In the general case where there are concurrent writers, maintaining a
+// secondary index requires transactional semantics and concurrency control.
+// Because of this, secondary indices are only supported via the transaction
+// layer. With secondary indices, whenever a (primary) key-value is inserted,
+// updated, or deleted via a transaction (regardless of whether it is an
+// explicit or implicit one), RocksDB will invoke any applicable SecondaryIndex
+// objects based on primary column family and column name, and it will
+// automatically add or remove any secondary index entries as needed (using
+// the same transaction).
+//
+// Note: the methods of SecondaryIndex implementations are expected to be
+// thread-safe with the exception of Set{Primary,Secondary}ColumnFamily (which
+// are not expected to be called after initialization).
+class SecondaryIndex {
+ public:
+  virtual ~SecondaryIndex() = default;
+
+  virtual void SetPrimaryColumnFamily(ColumnFamilyHandle* column_family) = 0;
+  virtual void SetSecondaryColumnFamily(ColumnFamilyHandle* column_family) = 0;
+
+  virtual ColumnFamilyHandle* GetPrimaryColumnFamily() const = 0;
+  virtual ColumnFamilyHandle* GetSecondaryColumnFamily() const = 0;
+
+  // The name of the primary column to index. Plain key-values can be indexed by
+  // specifying kDefaultWideColumnName.
+  virtual Slice GetPrimaryColumnName() const = 0;
+
+  // Optionally update the primary column value during an insert or update of a
+  // primary key-value. Called by the transaction layer before the primary
+  // key-value write is added to the transaction. Returning a non-OK status
+  // rolls back all operations in the transaction related to this primary
+  // key-value.
+  virtual Status UpdatePrimaryColumnValue(
+      const Slice& primary_key, const Slice& primary_column_value,
+      std::optional<std::variant<Slice, std::string>>* updated_column_value)
+      const = 0;
+
+  // Get the secondary key prefix for a given primary key-value. This method is
+  // called by the transaction layer when adding or removing secondary index
+  // entries (which have the form <secondary_key_prefix><primary_key> ->
+  // <secondary_value>) and should be deterministic. The output parameter
+  // secondary_key_prefix is expected to be based on primary_column_value,
+  // potentially with some additional metadata to prevent ambiguities (e.g.
+  // index id or length indicator). Returning a non-OK status rolls back all
+  // operations in the transaction related to this primary key-value.
+  virtual Status GetSecondaryKeyPrefix(
+      const Slice& primary_key, const Slice& primary_column_value,
+      std::variant<Slice, std::string>* secondary_key_prefix) const = 0;
+
+  // Get the optional secondary value for a given primary key-value. This method
+  // is called by the transaction layer when adding secondary index
+  // entries (which have the form <secondary_key_prefix><primary_key> ->
+  // <secondary_value>). previous_column_value contains the previous value of
+  // the primary column in case it was changed by UpdatePrimaryColumnValue.
+  // Returning a non-OK status rolls back all operations in the transaction
+  // related to this primary key-value.
+  virtual Status GetSecondaryValue(
+      const Slice& primary_key, const Slice& primary_column_value,
+      const Slice& previous_column_value,
+      std::optional<std::variant<Slice, std::string>>* secondary_value)
+      const = 0;
+};
+
+}  // namespace ROCKSDB_NAMESPACE

include/rocksdb/utilities/table_properties_collectors.h

@@ -7,6 +7,8 @@
 #include <atomic>
 #include <memory>
 
+#include "rocksdb/status.h"
+#include "rocksdb/system_clock.h"
 #include "rocksdb/table_properties.h"
 
 namespace ROCKSDB_NAMESPACE {
@@ -31,7 +33,7 @@ class CompactOnDeletionCollectorFactory
                                     size_t deletion_trigger,
                                     double deletion_ratio);
 
-  ~CompactOnDeletionCollectorFactory() {}
+  ~CompactOnDeletionCollectorFactory() override {}
 
   TablePropertiesCollector* CreateTablePropertiesCollector(
       TablePropertiesCollectorFactory::Context context) override;
@@ -106,7 +108,7 @@ class CompactForTieringCollectorFactory
   // for what entry is eligible.
   CompactForTieringCollectorFactory(double compaction_trigger_ratio);
 
-  ~CompactForTieringCollectorFactory() {}
+  ~CompactForTieringCollectorFactory() override {}
 
   TablePropertiesCollector* CreateTablePropertiesCollector(
       TablePropertiesCollectorFactory::Context context) override;
@@ -130,4 +132,89 @@ class CompactForTieringCollectorFactory
 
 std::shared_ptr<CompactForTieringCollectorFactory>
 NewCompactForTieringCollectorFactory(double compaction_trigger_ratio);
+
+// Information for the unix write time of a collection of data. Combined with
+// the current unix time, these stats give an overview of how long the data
+// have been written to the DB.
+// Check `DataCollectionIsEmpty` and `TrackedDataRatio` before interpreting
+// the stats.
+struct DataCollectionUnixWriteTimeInfo {
+  // The minimum write time, a.k.a. the write time of the oldest key.
+  uint64_t min_write_time = 0;
+  // The maximum write time, a.k.a. the write time of the newest key.
+  uint64_t max_write_time = 0;
+  // The average write time.
+  uint64_t average_write_time = 0;
+
+  // The number of entries that can be considered infinitely old because their
+  // sequence number are zeroed out. We know they are old entries but do not
+  // know how old exactly. These entries are separately counted and not
+  // aggregated in above stats.
+  uint64_t num_entries_infinitely_old = 0;
+
+  // The number of entries used to create above min, max, average stats.
+  uint64_t num_entries_write_time_aggregated = 0;
+
+  // The number of entries for which write time is untracked.
+  uint64_t num_entries_write_time_untracked = 0;
+
+  DataCollectionUnixWriteTimeInfo() {}
+
+  DataCollectionUnixWriteTimeInfo(uint64_t _min_write_time,
+                                  uint64_t _max_write_time,
+                                  uint64_t _average_write_time,
+                                  uint64_t _num_entries_infinitely_old,
+                                  uint64_t _num_entries_write_time_aggregated,
+                                  uint64_t _num_entries_write_time_untracked)
+      : min_write_time(_min_write_time),
+        max_write_time(_max_write_time),
+        average_write_time(_average_write_time),
+        num_entries_infinitely_old(_num_entries_infinitely_old),
+        num_entries_write_time_aggregated(_num_entries_write_time_aggregated),
+        num_entries_write_time_untracked(_num_entries_write_time_untracked) {}
+
+  // Returns true if the data collection for which this
+  // `DataCollectionUnixWriteTimeInfo` is for is empty.
+  bool DataCollectionIsEmpty() const {
+    return num_entries_infinitely_old == 0 &&
+           num_entries_write_time_aggregated == 0 &&
+           num_entries_write_time_untracked == 0;
+  }
+
+  // The closer the ratio is to 1, the more accurate the stats reflect the
+  // actual write time of this collection of data. If this ratio is 0, there is
+  // no write time information available. It could be either the data collection
+  // is empty, or none of its data has write time info tracked.
+  //
+  // For a single file, its data either has write time info tracked or not
+  // tracked, this ratio would be either 0 or 1. For a level, this ratio
+  // reflects what portion of the data has its write time info tracked in this
+  // struct. 0 is returned if the level is empty.
+  double TrackedDataRatio() const {
+    if (DataCollectionIsEmpty()) {
+      return 0;
+    }
+    uint64_t num_entries_write_time_tracked =
+        num_entries_infinitely_old + num_entries_write_time_aggregated;
+    return num_entries_write_time_tracked /
+           static_cast<double>(num_entries_write_time_tracked +
+                               num_entries_write_time_untracked);
+  }
+
+  // Whether the file or the level has infinitely old data.
+  bool HasInfinitelyOldData() const { return num_entries_infinitely_old > 0; }
+};
+
+// Given the table properties of a file, return data's unix write time stats
+// if available.
+Status GetDataCollectionUnixWriteTimeInfoForFile(
+    const std::shared_ptr<const TableProperties>& table_properties,
+    std::unique_ptr<DataCollectionUnixWriteTimeInfo>* file_info);
+
+// Given the collection of table properties per level, return data unix write
+// time stats if available.
+Status GetDataCollectionUnixWriteTimeInfoForLevels(
+    const std::vector<std::unique_ptr<TablePropertiesCollection>>&
+        levels_table_properties,
+    std::vector<std::unique_ptr<DataCollectionUnixWriteTimeInfo>>* levels_info);
 }  // namespace ROCKSDB_NAMESPACE

include/rocksdb/utilities/transaction.h

@@ -750,6 +750,8 @@ class Transaction {
 
   virtual TxnTimestamp GetCommitTimestamp() const { return kMaxTxnTimestamp; }
 
+  virtual bool GetCommitBypassMemTable() const { return false; }
+
  protected:
   explicit Transaction(const TransactionDB* /*db*/) {}
   Transaction() : log_number_(0), txn_state_(STARTED) {}

include/rocksdb/utilities/transaction_db.h

@@ -20,6 +20,7 @@
 
 namespace ROCKSDB_NAMESPACE {
 
+class SecondaryIndex;
 class TransactionDBMutexFactory;
 
 enum TxnDBWritePolicy {
@@ -240,6 +241,14 @@ struct TransactionDBOptions {
   // user-defined timestamps so this option only applies in this case.
   bool enable_udt_validation = true;
 
+  //       / \     UNDER CONSTRUCTION
+  //      / ! \    UNDER CONSTRUCTION
+  //     /-----\   UNDER CONSTRUCTION
+  //
+  // The secondary indices to be maintained. See the SecondaryIndex interface
+  // for more details.
+  std::vector<std::shared_ptr<SecondaryIndex>> secondary_indices;
+
  private:
   // 128 entries
   // Should the default value change, please also update wp_snapshot_cache_bits
@@ -339,6 +348,23 @@ struct TransactionOptions {
   // size in APIs that MyRocks currently are using, including Put, Merge, Delete
   // DeleteRange, SingleDelete.
   bool write_batch_track_timestamp_size = false;
+
+  // EXPERIMENTAL
+  // Only supports write-committed policy. If set to true, the transaction will
+  // skip memtable write and ingest into the DB directly during Commit(). This
+  // makes Commit() much faster for transactions with many operations.
+  // Transactions with Merge() or PutEntity() is not supported yet.
+  //
+  // Note that the transaction will be ingested as an immutable memtable for
+  // CFs it updates, and the current memtable will be switched to a new one.
+  // So ingesting many transactions in a short period of time may cause stall
+  // due to too many memtables.
+  // Note that the ingestion relies on the transaction's underlying index,
+  // (WriteBatchWithIndex), so updates that are added to the transaction
+  // without indexing (e.g. added directly to the transaction underlying
+  // write batch through Transaction::GetWriteBatch()->GetWriteBatch())
+  // are not supported. They will not be applied to the DB.
+  bool commit_bypass_memtable = false;
 };
 
 // The per-write optimizations that do not involve transactions. TransactionDB

include/rocksdb/utilities/write_batch_with_index.h

@@ -74,6 +74,10 @@ class WBWIIterator {
   // WriteBatchWithIndex
   virtual WriteEntry Entry() const = 0;
 
+  // For this user key, there is a single delete in this write batch,
+  // and it was overwritten by another update.
+  virtual bool HasOverWrittenSingleDel() const { return false; }
+
   virtual Status status() const = 0;
 };
 
@@ -350,6 +354,17 @@ class WriteBatchWithIndex : public WriteBatchBase {
   void SetMaxBytes(size_t max_bytes) override;
   size_t GetDataSize() const;
 
+  struct CFStat {
+    uint32_t entry_count = 0;
+    uint32_t overwritten_sd_count = 0;
+  };
+  // Will track CF ID, per CF entry count and overwritten sd count.
+  // Should be enabled when WBWI is empty for correct tracking.
+  void SetTrackPerCFStat(bool track);
+  const std::unordered_map<uint32_t, CFStat>& GetCFStats() const;
+
+  bool GetOverwriteKey() const;
+
  private:
   friend class PessimisticTransactionDB;
   friend class WritePreparedTxn;

include/rocksdb/version.h

@@ -12,7 +12,7 @@
 // NOTE: in 'main' development branch, this should be the *next*
 // minor or major version number planned for release.
 #define ROCKSDB_MAJOR 9
-#define ROCKSDB_MINOR 9
+#define ROCKSDB_MINOR 10
 #define ROCKSDB_PATCH 0
 
 // Do not use these. We made the mistake of declaring macros starting with

java/rocksjni/write_batch_test.cc

@@ -61,7 +61,7 @@ jbyteArray Java_org_rocksdb_WriteBatchTest_getContents(JNIEnv* env,
   ROCKSDB_NAMESPACE::ScopedArenaPtr<ROCKSDB_NAMESPACE::InternalIterator> iter(
       mem->NewIterator(ROCKSDB_NAMESPACE::ReadOptions(),
                        /*seqno_to_time_mapping=*/nullptr, &arena,
-                       /*prefix_extractor=*/nullptr));
+                       /*prefix_extractor=*/nullptr, /*for_flush=*/false));
   for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
     ROCKSDB_NAMESPACE::ParsedInternalKey ikey;
     ikey.clear();

memtable/wbwi_memtable.cc

@@ -21,6 +21,27 @@ const std::unordered_map<WriteType, ValueType>
         // kLogDataRecord, kXIDRecord, kUnknownRecord
 };
 
+InternalIterator* WBWIMemTable::NewIterator(
+    const ReadOptions&, UnownedPtr<const SeqnoToTimeMapping>, Arena* arena,
+    const SliceTransform* /* prefix_extractor */, bool for_flush) {
+  // Ingested WBWIMemTable should have an assigned seqno
+  assert(assigned_seqno_.upper_bound != kMaxSequenceNumber);
+  assert(assigned_seqno_.lower_bound != kMaxSequenceNumber);
+  assert(arena);
+  auto mem = arena->AllocateAligned(sizeof(WBWIMemTableIterator));
+  return new (mem) WBWIMemTableIterator(
+      std::unique_ptr<WBWIIterator>(wbwi_->NewIterator(cf_id_)),
+      assigned_seqno_, comparator_, for_flush);
+}
+
+inline InternalIterator* WBWIMemTable::NewIterator() const {
+  assert(assigned_seqno_.upper_bound != kMaxSequenceNumber);
+  assert(assigned_seqno_.lower_bound != kMaxSequenceNumber);
+  return new WBWIMemTableIterator(
+      std::unique_ptr<WBWIIterator>(wbwi_->NewIterator(cf_id_)),
+      assigned_seqno_, comparator_, /*for_flush=*/false);
+}
+
 bool WBWIMemTable::Get(const LookupKey& key, std::string* value,
                        PinnableWideColumns* columns, std::string* timestamp,
                        Status* s, MergeContext* merge_context,
@@ -34,7 +55,8 @@ bool WBWIMemTable::Get(const LookupKey& key, std::string* value,
   assert(immutable_memtable);
   assert(!timestamp);  // TODO: support UDT
   assert(!columns);    // TODO: support WideColumn
-  assert(global_seqno_ != kMaxSequenceNumber);
+  assert(assigned_seqno_.upper_bound != kMaxSequenceNumber);
+  assert(assigned_seqno_.lower_bound != kMaxSequenceNumber);
   // WBWI does not support DeleteRange yet.
   assert(!wbwi_->GetWriteBatch()->HasDeleteRange());
 

memtable/wbwi_memtable.h

@@ -8,157 +8,56 @@
 #include "rocksdb/utilities/write_batch_with_index.h"
 
 namespace ROCKSDB_NAMESPACE {
-
-class WBWIMemTableIterator final : public InternalIterator {
- public:
-  WBWIMemTableIterator(std::unique_ptr<WBWIIterator>&& it, SequenceNumber seqno,
-                       const Comparator* comparator)
-      : it_(std::move(it)), global_seqno_(seqno), comparator_(comparator) {
-    assert(seqno != kMaxSequenceNumber);
-    s_.PermitUncheckedError();
-  }
-
-  // No copying allowed
-  WBWIMemTableIterator(const WBWIMemTableIterator&) = delete;
-  WBWIMemTableIterator& operator=(const WBWIMemTableIterator&) = delete;
-
-  bool Valid() const override { return valid_; }
-
-  void SeekToFirst() override {
-    it_->SeekToFirst();
-    UpdateKey();
-  }
-
-  void SeekToLast() override {
-    it_->SeekToLast();
-    UpdateKey();
-  }
-
-  void Seek(const Slice& target) override {
-    Slice target_user_key = ExtractUserKey(target);
-    it_->Seek(target_user_key);
-    if (it_->Valid()) {
-      // compare seqno
-      SequenceNumber seqno = GetInternalKeySeqno(target);
-      if (seqno < global_seqno_ &&
-          comparator_->Compare(it_->Entry().key, target_user_key) == 0) {
-        it_->Next();
-        // TODO: cannot assume distinct keys once Merge is supported
-        if (it_->Valid()) {
-          assert(comparator_->Compare(it_->Entry().key, target_user_key) > 0);
-        }
-      }
-    }
-    UpdateKey();
-  }
-
-  void SeekForPrev(const Slice& target) override {
-    Slice target_user_key = ExtractUserKey(target);
-    it_->SeekForPrev(target_user_key);
-    if (it_->Valid()) {
-      SequenceNumber seqno = GetInternalKeySeqno(target);
-      if (seqno > global_seqno_ &&
-          comparator_->Compare(it_->Entry().key, target_user_key) == 0) {
-        it_->Prev();
-        if (it_->Valid()) {
-          // TODO: cannot assume distinct keys once Merge is supported
-          assert(comparator_->Compare(it_->Entry().key, target_user_key) < 0);
-        }
-      }
-    }
-    UpdateKey();
-  }
-
-  void Next() override {
-    assert(Valid());
-    it_->Next();
-    UpdateKey();
-  }
-
-  bool NextAndGetResult(IterateResult* result) override {
-    assert(Valid());
-    Next();
-    bool is_valid = Valid();
-    if (is_valid) {
-      result->key = key();
-      result->bound_check_result = IterBoundCheck::kUnknown;
-      result->value_prepared = true;
-    }
-    return is_valid;
-  }
-
-  void Prev() override {
-    assert(Valid());
-    it_->Prev();
-    UpdateKey();
-  }
-
-  Slice key() const override {
-    assert(Valid());
-    return key_;
-  }
-
-  Slice value() const override {
-    assert(Valid());
-    return it_->Entry().value;
-  }
-
-  Status status() const override {
-    assert(it_->status().ok());
-    return s_;
-  }
-
- private:
-  static const std::unordered_map<WriteType, ValueType> WriteTypeToValueTypeMap;
-
-  void UpdateKey() {
-    valid_ = it_->Valid();
-    if (!Valid()) {
-      key_.clear();
-      return;
-    }
-    auto t = WriteTypeToValueTypeMap.find(it_->Entry().type);
-    assert(t != WriteTypeToValueTypeMap.end());
-    if (t == WriteTypeToValueTypeMap.end()) {
-      key_.clear();
-      valid_ = false;
-      s_ = Status::Corruption("Unexpected write_batch_with_index entry type " +
-                              std::to_string(t->second));
-      return;
-    }
-    key_buf_.SetInternalKey(it_->Entry().key, global_seqno_, t->second);
-    key_ = key_buf_.GetInternalKey();
-  }
-
-  std::unique_ptr<WBWIIterator> it_;
-  // The sequence number of entries in this write batch.
-  SequenceNumber global_seqno_;
-  const Comparator* comparator_;
-  IterKey key_buf_;
-  // The current internal key.
-  Slice key_;
-  Status s_;
-  bool valid_ = false;
-};
-
+// An implementation of the ReadOnlyMemTable interface based on the content
+// of the given write batch with index (WBWI) object. This can be used to ingest
+// a transaction (which is based on WBWI) into the DB as an immutable memtable.
+//
+// REQUIRE overwrite_key to be true for the WBWI
+// Since the keys in WBWI do not have sequence number, the memtable needs to be
+// assigned a range of sequence numbers through AssignSequenceNumbers(seqno)
+// before being available for reads.
+// With overwrite_key = true, WBWI keeps track of the most recent update for
+// each key, and each such key will be assigned seqno.upper_bound during reads.
+// One exception is during flush, consider the following scenario:
+// - WBWI has SD(k) then PUT(k, v1)
+// - DB has PUT(k, v2) in L1
+// - flush WBWI adds PUT(k, v1) into L0
+// - live memtable contains SD(k)
+// - flush live memtable and compact it with L0 will drop SD(k) and PUT(k, v1)
+// - the PUT(k, v2) in L1 incorrectly becomes visible
+// So during flush, iterator from this memtable will need emit overwritten
+// single deletion. These single deletion entries will be
+// assigned seqno.upper_bound - 1.
 class WBWIMemTable final : public ReadOnlyMemTable {
  public:
+  struct SeqnoRange {
+    SequenceNumber lower_bound = kMaxSequenceNumber;
+    SequenceNumber upper_bound = kMaxSequenceNumber;
+  };
   WBWIMemTable(const std::shared_ptr<WriteBatchWithIndex>& wbwi,
                const Comparator* cmp, uint32_t cf_id,
                const ImmutableOptions* immutable_options,
-               const MutableCFOptions* cf_options)
+               const MutableCFOptions* cf_options,
+               const WriteBatchWithIndex::CFStat& stat)
       : wbwi_(wbwi),
         comparator_(cmp),
         ikey_comparator_(comparator_),
         moptions_(*immutable_options, *cf_options),
         clock_(immutable_options->clock),
-        cf_id_(cf_id) {}
+        // We need to include overwritten_sd_count in num_entries_ since flush
+        // verifies number of entries processed and that iterator for this
+        // memtable will emit overwritten SingleDelete entries during flush, See
+        // comment above WBWIMemTableIterator for more detail.
+        num_entries_(stat.entry_count + stat.overwritten_sd_count),
+        cf_id_(cf_id) {
+    assert(wbwi->GetOverwriteKey());
+  }
 
   // No copying allowed
   WBWIMemTable(const WBWIMemTable&) = delete;
   WBWIMemTable& operator=(const WBWIMemTable&) = delete;
 
-  ~WBWIMemTable() override = default;
+  ~WBWIMemTable() override { assert(refs_ == 0); }
 
   const char* Name() const override { return "WBWIMemTable"; }
 
@@ -182,17 +81,11 @@ class WBWIMemTable final : public ReadOnlyMemTable {
     assert(false);
   }
 
-  InternalIterator* NewIterator(
-      const ReadOptions&, UnownedPtr<const SeqnoToTimeMapping>, Arena* arena,
-      const SliceTransform* /* prefix_extractor */) override {
-    // Ingested WBWIMemTable should have an assigned seqno
-    assert(global_seqno_ != kMaxSequenceNumber);
-    assert(arena);
-    auto mem = arena->AllocateAligned(sizeof(WBWIMemTableIterator));
-    return new (mem) WBWIMemTableIterator(
-        std::unique_ptr<WBWIIterator>(wbwi_->NewIterator(cf_id_)),
-        global_seqno_, comparator_);
-  }
+  InternalIterator* NewIterator(const ReadOptions&,
+                                UnownedPtr<const SeqnoToTimeMapping>,
+                                Arena* arena,
+                                const SliceTransform* /* prefix_extractor */,
+                                bool for_flush) override;
 
   // Returns an iterator that wraps a MemTableIterator and logically strips the
   // user-defined timestamp of each key. This API is only used by flush when
@@ -235,13 +128,7 @@ class WBWIMemTable final : public ReadOnlyMemTable {
   void MultiGet(const ReadOptions& read_options, MultiGetRange* range,
                 ReadCallback* callback, bool immutable_memtable) override;
 
-  uint64_t NumEntries() const override {
-    // FIXME: used in
-    // - verify number of entries processed during flush
-    // - stats for estimate num entries and num entries in immutable memtables
-    // - MemPurgeDecider
-    return 0;
-  }
+  uint64_t NumEntries() const override { return num_entries_; }
 
   uint64_t NumDeletion() const override {
     // FIXME: this is used for stats and event logging
@@ -259,7 +146,9 @@ class WBWIMemTable final : public ReadOnlyMemTable {
     return 0;
   }
 
-  SequenceNumber GetEarliestSequenceNumber() override { return global_seqno_; }
+  SequenceNumber GetEarliestSequenceNumber() override {
+    return assigned_seqno_.lower_bound;
+  }
 
   bool IsEmpty() const override {
     // Ideally also check that wbwi contains updates from this CF. For now, we
@@ -267,7 +156,9 @@ class WBWIMemTable final : public ReadOnlyMemTable {
     return wbwi_->GetWriteBatch()->Count() == 0;
   }
 
-  SequenceNumber GetFirstSequenceNumber() override { return global_seqno_; }
+  SequenceNumber GetFirstSequenceNumber() override {
+    return assigned_seqno_.lower_bound;
+  }
 
   uint64_t GetMinLogContainingPrepSection() override {
     // FIXME: used to retain WAL with pending Prepare
@@ -304,31 +195,206 @@ class WBWIMemTable final : public ReadOnlyMemTable {
   }
 
   // Assign a sequence number to the entries in this memtable.
-  void SetGlobalSequenceNumber(SequenceNumber global_seqno) {
+  void AssignSequenceNumbers(const SeqnoRange& seqno_range) {
     // Not expecting to assign seqno multiple times.
-    assert(global_seqno_ == kMaxSequenceNumber);
-    global_seqno_ = global_seqno;
+    assert(assigned_seqno_.lower_bound == kMaxSequenceNumber);
+    assert(assigned_seqno_.upper_bound == kMaxSequenceNumber);
+
+    assigned_seqno_ = seqno_range;
+
+    assert(assigned_seqno_.lower_bound <= assigned_seqno_.upper_bound);
+    assert(assigned_seqno_.upper_bound != kMaxSequenceNumber);
+  }
+
+  void SetMinPrepLog(uint64_t min_prep_log) {
+    min_prep_log_referenced_ = min_prep_log;
   }
 
  private:
-  InternalIterator* NewIterator() const {
-    assert(global_seqno_ != kMaxSequenceNumber);
-    return new WBWIMemTableIterator(
-        std::unique_ptr<WBWIIterator>(wbwi_->NewIterator(cf_id_)),
-        global_seqno_, comparator_);
-  }
+  inline InternalIterator* NewIterator() const;
 
   Slice newest_udt_;
   std::shared_ptr<WriteBatchWithIndex> wbwi_;
   const Comparator* comparator_;
   InternalKeyComparator ikey_comparator_;
-  SequenceNumber global_seqno_ = kMaxSequenceNumber;
+  SeqnoRange assigned_seqno_;
   const ImmutableMemTableOptions moptions_;
   SystemClock* clock_;
   uint64_t min_prep_log_referenced_{0};
+  uint64_t num_entries_;
   // WBWI can contains updates to multiple CFs. `cf_id_` determines which CF
   // this memtable is for.
   uint32_t cf_id_;
 };
 
+class WBWIMemTableIterator final : public InternalIterator {
+ public:
+  WBWIMemTableIterator(std::unique_ptr<WBWIIterator>&& it,
+                       const WBWIMemTable::SeqnoRange& assigned_seqno,
+                       const Comparator* comparator, bool for_flush)
+      : it_(std::move(it)),
+        assigned_seqno_(assigned_seqno),
+        comparator_(comparator),
+        emit_overwritten_single_del_(for_flush) {
+    assert(assigned_seqno_.lower_bound <= assigned_seqno_.upper_bound);
+    assert(assigned_seqno_.upper_bound < kMaxSequenceNumber);
+    s_.PermitUncheckedError();
+  }
+
+  // No copying allowed
+  WBWIMemTableIterator(const WBWIMemTableIterator&) = delete;
+  WBWIMemTableIterator& operator=(const WBWIMemTableIterator&) = delete;
+
+  bool Valid() const override { return valid_; }
+
+  void SeekToFirst() override {
+    it_->SeekToFirst();
+    UpdateKey();
+  }
+
+  void SeekToLast() override {
+    it_->SeekToLast();
+    UpdateKey();
+  }
+
+  void Seek(const Slice& target) override {
+    Slice target_user_key = ExtractUserKey(target);
+    it_->Seek(target_user_key);
+    if (it_->Valid()) {
+      // compare seqno
+      SequenceNumber seqno = GetInternalKeySeqno(target);
+      assert(!emit_overwritten_single_del_);
+      // For now all keys are assigned seqno_ub_, this may change after merge
+      // is supported.
+      assert(seqno <= assigned_seqno_.lower_bound ||
+             seqno >= assigned_seqno_.upper_bound);
+      if (seqno < assigned_seqno_.upper_bound &&
+          comparator_->Compare(it_->Entry().key, target_user_key) == 0) {
+        it_->Next();
+        // TODO: cannot assume distinct keys once Merge is supported
+        if (it_->Valid()) {
+          assert(comparator_->Compare(it_->Entry().key, target_user_key) > 0);
+        }
+      }
+    }
+    UpdateKey();
+  }
+
+  void SeekForPrev(const Slice& target) override {
+    Slice target_user_key = ExtractUserKey(target);
+    it_->SeekForPrev(target_user_key);
+    if (it_->Valid()) {
+      SequenceNumber seqno = GetInternalKeySeqno(target);
+      assert(seqno <= assigned_seqno_.lower_bound ||
+             seqno >= assigned_seqno_.upper_bound);
+      if (seqno > assigned_seqno_.upper_bound &&
+          comparator_->Compare(it_->Entry().key, target_user_key) == 0) {
+        it_->Prev();
+        if (it_->Valid()) {
+          // TODO: cannot assume distinct keys once Merge is supported
+          assert(comparator_->Compare(it_->Entry().key, target_user_key) < 0);
+        }
+      }
+    }
+    UpdateKey();
+  }
+
+  void Next() override {
+    // Only need to emit single deletion during flush. Since Flush does
+    // sequential forward scan, we only need to emit single deletion in Next(),
+    // and do not need to consider iterator direction change.
+    assert(Valid());
+    if (emit_overwritten_single_del_) {
+      if (it_->HasOverWrittenSingleDel() && !at_overwritten_single_del_) {
+        UpdateSingleDeleteKey();
+        return;
+      }
+      at_overwritten_single_del_ = false;
+    }
+
+    it_->Next();
+    UpdateKey();
+  }
+
+  bool NextAndGetResult(IterateResult* result) override {
+    assert(Valid());
+    Next();
+    bool is_valid = Valid();
+    if (is_valid) {
+      result->key = key();
+      result->bound_check_result = IterBoundCheck::kUnknown;
+      result->value_prepared = true;
+    }
+    return is_valid;
+  }
+
+  void Prev() override {
+    assert(Valid());
+    it_->Prev();
+    UpdateKey();
+  }
+
+  Slice key() const override {
+    assert(Valid());
+    return key_;
+  }
+
+  Slice value() const override {
+    assert(Valid());
+    // TODO: it_->Entry() is not trivial, cache it
+    return it_->Entry().value;
+  }
+
+  Status status() const override {
+    assert(it_->status().ok());
+    return s_;
+  }
+
+  bool IsValuePinned() const override { return true; }
+
+ private:
+  static const std::unordered_map<WriteType, ValueType> WriteTypeToValueTypeMap;
+
+  void UpdateKey() {
+    valid_ = it_->Valid();
+    if (!Valid()) {
+      key_.clear();
+      return;
+    }
+    auto t = WriteTypeToValueTypeMap.find(it_->Entry().type);
+    assert(t != WriteTypeToValueTypeMap.end());
+    if (t == WriteTypeToValueTypeMap.end()) {
+      key_.clear();
+      valid_ = false;
+      s_ = Status::Corruption("Unexpected write_batch_with_index entry type " +
+                              std::to_string(t->second));
+      return;
+    }
+    key_buf_.SetInternalKey(it_->Entry().key, assigned_seqno_.upper_bound,
+                            t->second);
+    key_ = key_buf_.GetInternalKey();
+  }
+
+  void UpdateSingleDeleteKey() {
+    assert(it_->Valid());
+    assert(Valid());
+    assert(assigned_seqno_.lower_bound < assigned_seqno_.upper_bound);
+    key_buf_.SetInternalKey(it_->Entry().key, assigned_seqno_.upper_bound - 1,
+                            kTypeSingleDeletion);
+    key_ = key_buf_.GetInternalKey();
+    at_overwritten_single_del_ = true;
+  }
+
+  std::unique_ptr<WBWIIterator> it_;
+  const WBWIMemTable::SeqnoRange assigned_seqno_;
+  const Comparator* comparator_;
+  IterKey key_buf_;
+  // The current internal key.
+  Slice key_;
+  Status s_;
+  bool valid_ = false;
+  bool at_overwritten_single_del_ = false;
+  bool emit_overwritten_single_del_ = false;
+};
+
 }  // namespace ROCKSDB_NAMESPACE

monitoring/thread_status_util_debug.cc

@@ -50,6 +50,8 @@ Env::IOActivity ThreadStatusUtil::TEST_GetExpectedIOActivity(
       return Env::IOActivity::kGetEntity;
     case ThreadStatus::OperationType::OP_MULTIGETENTITY:
       return Env::IOActivity::kMultiGetEntity;
+    case ThreadStatus::OperationType::OP_READ_MANIFEST:
+      return Env::IOActivity::kReadManifest;
     default:
       return Env::IOActivity::kUnknown;
   }

options/options_helper.cc

@@ -995,7 +995,8 @@ Status OptionTypeInfo::ParseStruct(
     std::unordered_map<std::string, std::string> unused;
     status =
         ParseType(config_options, opt_value, *struct_map, opt_addr, &unused);
-    if (status.ok() && !unused.empty()) {
+    if (status.ok() && !unused.empty() &&
+        !config_options.ignore_unknown_options) {
       status = Status::InvalidArgument(
           "Unrecognized option", struct_name + "." + unused.begin()->first);
     }
@@ -1006,7 +1007,7 @@ Status OptionTypeInfo::ParseStruct(
         Find(opt_name.substr(struct_name.size() + 1), *struct_map, &elem_name);
     if (opt_info != nullptr) {
       status = opt_info->Parse(config_options, elem_name, opt_value, opt_addr);
-    } else {
+    } else if (!config_options.ignore_unknown_options) {
       status = Status::InvalidArgument("Unrecognized option", opt_name);
     }
   } else {
@@ -1015,7 +1016,7 @@ Status OptionTypeInfo::ParseStruct(
     const auto opt_info = Find(opt_name, *struct_map, &elem_name);
     if (opt_info != nullptr) {
       status = opt_info->Parse(config_options, elem_name, opt_value, opt_addr);
-    } else {
+    } else if (!config_options.ignore_unknown_options) {
       status = Status::InvalidArgument("Unrecognized option",
                                        struct_name + "." + opt_name);
     }

options/options_test.cc

@@ -832,6 +832,21 @@ TEST_F(OptionsTest, CompressionOptionsFromString) {
   ASSERT_OK(GetColumnFamilyOptionsFromString(ignore, ColumnFamilyOptions(),
                                              "compression_opts.unknown=bad",
                                              &base_cf_opt));
+
+  // Test with some additional values added
+  config_options.ignore_unknown_options = false;
+  ASSERT_NOK(
+      GetColumnFamilyOptionsFromString(config_options, ColumnFamilyOptions(),
+                                       "new_unknown_field=whatever; "
+                                       "bottommost_compression_opts=4:5:6:7",
+                                       &base_cf_opt));
+  // Test Ignoring Unknown Options
+  config_options.ignore_unknown_options = true;
+  ASSERT_OK(
+      GetColumnFamilyOptionsFromString(config_options, ColumnFamilyOptions(),
+                                       "new_unknown_field=whatever; "
+                                       "bottommost_compression_opts=4:5:6:7",
+                                       &base_cf_opt));
 }
 
 TEST_F(OptionsTest, OldInterfaceTest) {

port/port.h

@@ -24,14 +24,10 @@
 // A temporary hook into long-running RocksDB threads to support modifying their
 // priority etc. This should become a public API hook once the requirements
 // are better understood.
-extern "C" void RocksDbThreadYield() __attribute__((__weak__));
-#define ROCKSDB_THREAD_YIELD_HOOK() \
-  {                                 \
-    if (RocksDbThreadYield) {       \
-      RocksDbThreadYield();         \
-    }                               \
-  }
+// Returns true if query is aborted.
+extern "C" bool RocksDbThreadYieldAndCheckAbort() __attribute__((__weak__));
+#define ROCKSDB_THREAD_YIELD_CHECK_ABORT() \
+  (RocksDbThreadYieldAndCheckAbort ? RocksDbThreadYieldAndCheckAbort() : false)
 #else
-#define ROCKSDB_THREAD_YIELD_HOOK() \
-  {}
+#define ROCKSDB_THREAD_YIELD_CHECK_ABORT() (false)
 #endif

src.mk

@@ -75,6 +75,7 @@ LIB_SOURCES =                                                   \
   db/log_reader.cc                                              \
   db/log_writer.cc                                              \
   db/malloc_stats.cc                                            \
+  db/manifest_ops.cc                                            \
   db/memtable.cc                                                \
   db/memtable_list.cc                                           \
   db/merge_helper.cc                                            \
@@ -340,6 +341,9 @@ LIB_SOURCES_ASM =
 LIB_SOURCES_C =
 endif
 
+WITH_FAISS_LIB_SOURCES = \
+  utilities/secondary_index/faiss_ivf_index.cc                  \
+
 RANGE_TREE_SOURCES =\
   utilities/transactions/lock/range/range_tree/lib/locktree/concurrent_tree.cc \
   utilities/transactions/lock/range/range_tree/lib/locktree/keyrange.cc        \
@@ -650,6 +654,9 @@ TEST_MAIN_SOURCES =                                                     \
 TEST_MAIN_SOURCES_C = \
   db/c_test.c                                                           \
 
+WITH_FAISS_TEST_MAIN_SOURCES = \
+  utilities/secondary_index/faiss_ivf_index_test.cc                     \
+
 MICROBENCH_SOURCES =                                          \
   microbench/ribbon_bench.cc                                  \
   microbench/db_basic_bench.cc                                  \

table/block_based/block_based_table_reader.cc

@@ -660,7 +660,7 @@ Status BlockBasedTable::Open(
   const bool preload_all = !table_options.cache_index_and_filter_blocks;
 
   if (!ioptions.allow_mmap_reads && !env_options.use_mmap_reads) {
-    s = PrefetchTail(ro, file.get(), file_size, force_direct_prefetch,
+    s = PrefetchTail(ro, ioptions, file.get(), file_size, force_direct_prefetch,
                      tail_prefetch_stats, prefetch_all, preload_all,
                      &prefetch_buffer, ioptions.stats, tail_size,
                      ioptions.logger);
@@ -877,7 +877,8 @@ Status BlockBasedTable::Open(
 }
 
 Status BlockBasedTable::PrefetchTail(
-    const ReadOptions& ro, RandomAccessFileReader* file, uint64_t file_size,
+    const ReadOptions& ro, const ImmutableOptions& ioptions,
+    RandomAccessFileReader* file, uint64_t file_size,
     bool force_direct_prefetch, TailPrefetchStats* tail_prefetch_stats,
     const bool prefetch_all, const bool preload_all,
     std::unique_ptr<FilePrefetchBuffer>* prefetch_buffer, Statistics* stats,
@@ -948,7 +949,7 @@ Status BlockBasedTable::PrefetchTail(
   // Use `FilePrefetchBuffer`
   prefetch_buffer->reset(new FilePrefetchBuffer(
       ReadaheadParams(), true /* enable */, true /* track_min_offset */,
-      nullptr /* fs */, nullptr /* clock */, stats,
+      ioptions.fs.get() /* fs */, nullptr /* clock */, stats,
       /* readahead_cb */ nullptr,
       FilePrefetchBufferUsage::kTableOpenPrefetchTail));
 

table/block_based/block_based_table_reader.h

@@ -482,7 +482,8 @@ class BlockBasedTable : public TableReader {
   // If force_direct_prefetch is true, always prefetching to RocksDB
   //    buffer, rather than calling RandomAccessFile::Prefetch().
   static Status PrefetchTail(
-      const ReadOptions& ro, RandomAccessFileReader* file, uint64_t file_size,
+      const ReadOptions& ro, const ImmutableOptions& ioptions,
+      RandomAccessFileReader* file, uint64_t file_size,
       bool force_direct_prefetch, TailPrefetchStats* tail_prefetch_stats,
       const bool prefetch_all, const bool preload_all,
       std::unique_ptr<FilePrefetchBuffer>* prefetch_buffer, Statistics* stats,

table/table_test.cc

@@ -534,7 +534,8 @@ class MemTableConstructor : public Constructor {
       const SliceTransform* /*prefix_extractor*/) const override {
     return new KeyConvertingIterator(
         memtable_->NewIterator(ReadOptions(), /*seqno_to_time_mapping=*/nullptr,
-                               &arena_, /*prefix_extractor=*/nullptr),
+                               &arena_, /*prefix_extractor=*/nullptr,
+                               /*for_flush=*/false),
         true);
   }
 
@@ -3252,14 +3253,14 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupSeqScans) {
       FilePrefetchBuffer* prefetch_buffer =
           (static_cast<BlockBasedTableIterator*>(iter.get()))
               ->prefetch_buffer();
-      std::vector<std::pair<uint64_t, size_t>> buffer_info(1);
+      std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(1);
       prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
       bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first, block_handle);
       // It won't prefetch the data of cache hit.
       // One block data.
-      ASSERT_EQ(buffer_info[0].second, 4096);
-      ASSERT_EQ(buffer_info[0].first, block_handle.offset());
+      ASSERT_EQ(std::get<1>(buffer_info[0]), 4096);
+      ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
 
       ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                 1);
@@ -3290,14 +3291,14 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupSeqScans) {
       FilePrefetchBuffer* prefetch_buffer =
           (static_cast<BlockBasedTableIterator*>(iter.get()))
               ->prefetch_buffer();
-      std::vector<std::pair<uint64_t, size_t>> buffer_info(1);
+      std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(1);
       prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
       bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first, block_handle);
 
       // It won't prefetch the data of cache hit.
       // 3 blocks data.
-      ASSERT_EQ(buffer_info[0].second, 12288);
-      ASSERT_EQ(buffer_info[0].first, block_handle.offset());
+      ASSERT_EQ(std::get<1>(buffer_info[0]), 12288);
+      ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
 
       for (; kv_iter != kvmap.end() && iter->Valid(); kv_iter++) {
         ASSERT_EQ(iter->key(), kv_iter->first);
@@ -3313,8 +3314,8 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupSeqScans) {
       // Second Prefetch.
       prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
       bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first, block_handle);
-      ASSERT_EQ(buffer_info[0].second, 20480);
-      ASSERT_EQ(buffer_info[0].first, block_handle.offset());
+      ASSERT_EQ(std::get<1>(buffer_info[0]), 20480);
+      ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
 
       ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                 1);
@@ -3401,13 +3402,13 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
       FilePrefetchBuffer* prefetch_buffer =
           (static_cast<BlockBasedTableIterator*>(iter.get()))
               ->prefetch_buffer();
-      std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+      std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
       prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
       bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first, block_handle);
-      ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-      ASSERT_EQ(buffer_info[0].second, 4096);
-      ASSERT_EQ(buffer_info[1].second, 0);
+      ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+      ASSERT_EQ(std::get<1>(buffer_info[0]), 4096);
+      ASSERT_EQ(std::get<1>(buffer_info[1]), 0);
 
       ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                 2);
@@ -3440,21 +3441,21 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
       FilePrefetchBuffer* prefetch_buffer =
           (static_cast<BlockBasedTableIterator*>(iter.get()))
               ->prefetch_buffer();
-      std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+      std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
       prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
       {
         // 1st Buffer Verification.
         bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first,
                                      block_handle);
-        ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[0].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[0]), 8192);
 
         // 2nd Buffer Verification.
         InternalKey ikey_tmp("00000360", 0, kTypeValue);
         bbt->TEST_GetDataBlockHandle(read_options, ikey_tmp.Encode().ToString(),
                                      block_handle);
-        ASSERT_EQ(buffer_info[1].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[1].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[1]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[1]), 8192);
 
         ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                   1);
@@ -3493,21 +3494,21 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
               ->prefetch_buffer();
 
       {
-        std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+        std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
         prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
         // 1st Buffer Verification.
         bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first,
                                      block_handle);
-        ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[0].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[0]), 8192);
 
         // 2nd Buffer Verification.
         InternalKey ikey_tmp("00000540", 0, kTypeValue);
         bbt->TEST_GetDataBlockHandle(read_options, ikey_tmp.Encode().ToString(),
                                      block_handle);
-        ASSERT_EQ(buffer_info[1].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[1].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[1]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[1]), 8192);
 
         ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                   1);
@@ -3527,21 +3528,21 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
       }
 
       {
-        std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+        std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
         prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
         // 1st Buffer Verification.
         bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first,
                                      block_handle);
-        ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[0].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[0]), 8192);
 
         // 2nd Buffer Verification.
         InternalKey ikey_tmp("00000585", 0, kTypeValue);
         bbt->TEST_GetDataBlockHandle(read_options, ikey_tmp.Encode().ToString(),
                                      block_handle);
-        ASSERT_EQ(buffer_info[1].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[1].second, 4096);
+        ASSERT_EQ(std::get<0>(buffer_info[1]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[1]), 4096);
 
         ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                   1);
@@ -3561,21 +3562,21 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
       }
 
       {
-        std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+        std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
         prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
         // 1st Buffer Verification.
         bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first,
                                      block_handle);
-        ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[0].second, 4096);
+        ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[0]), 4096);
 
         // 2nd Buffer Verification.
         InternalKey ikey_tmp("00000615", 0, kTypeValue);
         bbt->TEST_GetDataBlockHandle(read_options, ikey_tmp.Encode().ToString(),
                                      block_handle);
-        ASSERT_EQ(buffer_info[1].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[1].second, 4096);
+        ASSERT_EQ(std::get<0>(buffer_info[1]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[1]), 4096);
 
         ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                   1);
@@ -3595,21 +3596,21 @@ TEST_P(BlockBasedTableTest, BlockCacheLookupAsyncScansSeek) {
       }
 
       {
-        std::vector<std::pair<uint64_t, size_t>> buffer_info(2);
+        std::vector<std::tuple<uint64_t, size_t, bool>> buffer_info(2);
         prefetch_buffer->TEST_GetBufferOffsetandSize(buffer_info);
 
         // 1st Buffer Verification.
         bbt->TEST_GetDataBlockHandle(read_options, kv_iter->first,
                                      block_handle);
-        ASSERT_EQ(buffer_info[0].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[0].second, 4096);
+        ASSERT_EQ(std::get<0>(buffer_info[0]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[0]), 4096);
 
         // 2nd Buffer Verification.
         InternalKey ikey_tmp("00000630", 0, kTypeValue);
         bbt->TEST_GetDataBlockHandle(read_options, ikey_tmp.Encode().ToString(),
                                      block_handle);
-        ASSERT_EQ(buffer_info[1].first, block_handle.offset());
-        ASSERT_EQ(buffer_info[1].second, 8192);
+        ASSERT_EQ(std::get<0>(buffer_info[1]), block_handle.offset());
+        ASSERT_EQ(std::get<1>(buffer_info[1]), 8192);
 
         ASSERT_EQ(options.statistics->getAndResetTickerCount(READAHEAD_TRIMMED),
                   0);
@@ -4911,7 +4912,8 @@ TEST_F(MemTableTest, Simple) {
     if (i == 0) {
       iter = GetMemTable()->NewIterator(ReadOptions(),
                                         /*seqno_to_time_mapping=*/nullptr,
-                                        &arena, /*prefix_extractor=*/nullptr);
+                                        &arena, /*prefix_extractor=*/nullptr,
+                                        /*for_flush=*/false);
       arena_iter_guard.reset(iter);
     } else {
       iter = GetMemTable()->NewRangeTombstoneIterator(

tools/check_format_compatible.sh

@@ -125,7 +125,7 @@ EOF
 
 # To check for DB forward compatibility with loading options (old version
 # reading data from new), as well as backward compatibility
-declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb")
+declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb" "9.9.fb")
 # To check for DB forward compatibility without loading options (in addition
 # to the "with loading options" set), as well as backward compatibility
 declare -a db_forward_no_options_refs=() # N/A at the moment

tools/db_crashtest.py

@@ -530,6 +530,9 @@ txn_params = {
     "inplace_update_support": 0,
     # TimedPut is not supported in transaction
     "use_timed_put_one_in": 0,
+    # txn commit with this option will create a new memtable, keep the
+    # frequency low to reduce stalls
+    "commit_bypass_memtable_one_in": random.choice([0] * 2 + [500, 1000]),
 }
 
 # For optimistic transaction db
@@ -593,16 +596,21 @@ ts_params = {
 }
 
 tiered_params = {
-    # Set tiered compaction hot data time as: 1 minute, 1 hour, 10 hour
-    "preclude_last_level_data_seconds": lambda: random.choice([60, 3600, 36000]),
-    # only test universal compaction for now, level has known issue of
-    # endless compaction
-    "compaction_style": 1,
+    # For Leveled/Universal compaction (ignored for FIFO)
+    # Bias toward times that can elapse during a crash test run series
+    "preclude_last_level_data_seconds": lambda: random.choice([10, 60, 1200, 86400]),
+    "last_level_temperature": "kCold",
+    # For FIFO compaction (ignored otherwise)
+    "file_temperature_age_thresholds": lambda: random.choice(
+        [
+            "{{temperature=kWarm;age=30}:{temperature=kCold;age=300}}",
+            "{{temperature=kCold;age=100}}",
+        ]
+    ),
     # tiered storage doesn't support blob db yet
     "enable_blob_files": 0,
     "use_blob_db": 0,
     "default_write_temperature": lambda: random.choice(["kUnknown", "kHot", "kWarm"]),
-    "last_level_temperature": "kCold",
 }
 
 multiops_txn_default_params = {
@@ -669,6 +677,7 @@ multiops_wc_txn_params = {
     "txn_write_policy": 0,
     # TODO re-enable pipelined write. Not well tested atm
     "enable_pipelined_write": 0,
+    "commit_bypass_memtable_one_in": random.choice([0] * 4 + [100]),
 }
 
 multiops_wp_txn_params = {
@@ -780,12 +789,20 @@ def finalize_and_sanitize(src_params):
         # files, which would be problematic when unsynced data can be lost in
         # crash recoveries.
         dest_params["enable_compaction_filter"] = 0
+    # Remove the following once write-prepared/write-unprepared with/without
+    # unordered write supports timestamped snapshots
+    if dest_params.get("create_timestamped_snapshot_one_in", 0) > 0:
+        dest_params["txn_write_policy"] = 0
+        dest_params["unordered_write"] = 0
     # Only under WritePrepared txns, unordered_write would provide the same guarnatees as vanilla rocksdb
     # unordered_write is only enabled with --txn, and txn_params disables inplace_update_support, so
     # setting allow_concurrent_memtable_write=1 won't conflcit with inplace_update_support.
+    # don't overwrite txn_write_policy
     if dest_params.get("unordered_write", 0) == 1:
-        dest_params["txn_write_policy"] = 1
-        dest_params["allow_concurrent_memtable_write"] = 1
+        if dest_params.get("txn_write_policy", 0) == 1:
+            dest_params["allow_concurrent_memtable_write"] = 1
+        else:
+            dest_params["unordered_write"] = 0
     if dest_params.get("disable_wal", 0) == 1:
         dest_params["atomic_flush"] = 1
         dest_params["sync"] = 0
@@ -800,6 +817,12 @@ def finalize_and_sanitize(src_params):
         # now assertion failures are triggered.
         dest_params["compaction_ttl"] = 0
         dest_params["periodic_compaction_seconds"] = 0
+        # Disable irrelevant tiering options
+        dest_params["preclude_last_level_data_seconds"] = 0
+        dest_params["last_level_temperature"] = "kUnknown"
+    else:
+        # Disable irrelevant tiering options
+        dest_params["file_temperature_age_thresholds"] = ""
     if dest_params["partition_filters"] == 1:
         if dest_params["index_type"] != 2:
             dest_params["partition_filters"] = 0
@@ -825,11 +848,6 @@ def finalize_and_sanitize(src_params):
         dest_params["write_fault_one_in"] = 0
         dest_params["skip_verifydb"] = 1
         dest_params["verify_db_one_in"] = 0
-    # Remove the following once write-prepared/write-unprepared with/without
-    # unordered write supports timestamped snapshots
-    if dest_params.get("create_timestamped_snapshot_one_in", 0) > 0:
-        dest_params["txn_write_policy"] = 0
-        dest_params["unordered_write"] = 0
     # For TransactionDB, correctness testing with unsync data loss is currently
     # compatible with only write committed policy
     if dest_params.get("use_txn") == 1 and dest_params.get("txn_write_policy", 0) != 0:
@@ -841,6 +859,8 @@ def finalize_and_sanitize(src_params):
         dest_params["use_put_entity_one_in"] = 0
         # MultiCfIterator is currently only compatible with write committed policy
         dest_params["use_multi_cf_iterator"] = 0
+        # only works with write committed policy
+        dest_params["commit_bypass_memtable_one_in"] = 0
     # TODO(hx235): enable test_multi_ops_txns with fault injection after stabilizing the CI
     if dest_params.get("test_multi_ops_txns") == 1:
         dest_params["write_fault_one_in"] = 0
@@ -924,7 +944,10 @@ def finalize_and_sanitize(src_params):
         dest_params["check_multiget_consistency"] = 0
         dest_params["check_multiget_entity_consistency"] = 0
     if dest_params.get("disable_wal") == 0:
-        if dest_params.get("reopen") > 0 or (dest_params.get("manual_wal_flush_one_in") and dest_params.get("column_families") != 1):
+        if dest_params.get("reopen") > 0 or (
+            dest_params.get("manual_wal_flush_one_in")
+            and dest_params.get("column_families") != 1
+        ):
             # Reopen with WAL currently requires persisting WAL data before closing for reopen.
             # Previous injected WAL write errors may not be cleared by the time of closing and ready
             # for persisting WAL.
@@ -969,8 +992,24 @@ def finalize_and_sanitize(src_params):
     # can cause checkpoint verification to fail. So make the two mutually exclusive.
     if dest_params.get("checkpoint_one_in") != 0:
         dest_params["lock_wal_one_in"] = 0
-    if dest_params.get("ingest_external_file_one_in") == 0 or dest_params.get("delrangepercent") == 0:
+    if (
+        dest_params.get("ingest_external_file_one_in") == 0
+        or dest_params.get("delrangepercent") == 0
+    ):
         dest_params["test_ingest_standalone_range_deletion_one_in"] = 0
+    if dest_params.get("use_txn", 0) == 1 and dest_params.get("commit_bypass_memtable_one_in", 0) > 0:
+        dest_params["enable_blob_files"] = 0
+        dest_params["allow_setting_blob_options_dynamically"] = 0
+        dest_params["atomic_flush"] = 0
+        dest_params["allow_concurrent_memtable_write"] = 0
+        dest_params["use_put_entity_one_in"] = 0
+        dest_params["use_get_entity"] = 0
+        dest_params["use_multi_get_entity"] = 0
+        dest_params["use_merge"] = 0
+        dest_params["use_full_merge_v1"] = 0
+        dest_params["enable_pipelined_write"] = 0
+        dest_params["use_attribute_group"] = 0
+
     return dest_params
 
 
@@ -1017,6 +1056,14 @@ def gen_cmd_params(args):
     ):
         params.update(blob_params)
 
+    if "compaction_style" not in params:
+        # Default to leveled compaction
+        # TODO: Fix "Unsafe to store Seq later" with tiered+leveled and
+        # enable that combination rather than falling back to universal.
+        # TODO: There is also an alleged bug with leveled compaction
+        # infinite looping but that likely would not fail the crash test.
+        params["compaction_style"] = 0 if not args.test_tiered_storage else 1
+
     for k, v in vars(args).items():
         if v is not None:
             params[k] = v
@@ -1145,7 +1192,9 @@ def blackbox_crash_main(args, unknown_args):
     cmd = gen_cmd(
         dict(list(cmd_params.items()) + list({"db": dbname}.items())), unknown_args
     )
-    hit_timeout, retcode, outs, errs = execute_cmd(cmd, cmd_params["verify_timeout"], True)
+    hit_timeout, retcode, outs, errs = execute_cmd(
+        cmd, cmd_params["verify_timeout"], True
+    )
 
     # For the final run
     print_output_and_exit_on_error(outs, errs, args.print_stderr_separately)
@@ -1230,12 +1279,8 @@ def whitebox_crash_main(args, unknown_args):
             # Single level universal has a lot of special logic. Ensure we cover
             # it sometimes.
             if not args.test_tiered_storage and random.randint(0, 1) == 1:
-                additional_opts.update(
-                    {
-                        "num_levels": 1,
-                    }
-                )
-        elif check_mode == 2 and not args.test_tiered_storage:
+                additional_opts["num_levels"] = 1
+        elif check_mode == 2:
             # normal run with FIFO compaction mode
             # ops_per_thread is divided by 5 because FIFO compaction
             # style is quite a bit slower on reads with lot of files
@@ -1244,6 +1289,12 @@ def whitebox_crash_main(args, unknown_args):
                 "ops_per_thread": cmd_params["ops_per_thread"] // 5,
                 "compaction_style": 2,
             }
+            # TODO: test transition from non-FIFO to FIFO with num_levels > 1.
+            # See https://github.com/facebook/rocksdb/pull/10348
+            # For now, tiered storage FIFO (file_temperature_age_thresholds)
+            # requires num_levels == 1 and non-tiered operates that way.
+            if args.test_tiered_storage:
+                additional_opts["num_levels"] = 1
         else:
             # normal run
             additional_opts = {

unreleased_history/behavior_changes/base_delta_iterator_allow_unprepared_value.md

@@ -1 +0,0 @@
-`BaseDeltaIterator` now honors the read option `allow_unprepared_value`.

unreleased_history/bug_fixes/base_delta_iterator_prepare_value.md

@@ -1 +0,0 @@
-`BaseDeltaIterator` now calls `PrepareValue` on the base iterator in case it has been created with the `allow_unprepared_value` read option set. Earlier, such base iterators could lead to incorrect values being exposed from `BaseDeltaIterator`.

unreleased_history/bug_fixes/blob_file_leak.md

@@ -1 +0,0 @@
-* Fix a leak of obsolete blob files left open until DB::Close(). This bug was introduced in version 9.4.0.

unreleased_history/bug_fixes/missing_strong_reads.md

@@ -1 +0,0 @@
-Fix missing cases of corruption retry during DB open and read API processing.

unreleased_history/bug_fixes/old_wal_2pc.md

@@ -1 +0,0 @@
-* Fix a bug for transaction db with 2pc where an old WAL may be retained longer than needed (#13127). 

unreleased_history/bug_fixes/open_sst_file_leaks.md

@@ -1 +0,0 @@
-* Fix leaks of some open SST files (until `DB::Close()`) that are written but never become live due to various failures. (We now have a check for such leaks with no outstanding issues.)

unreleased_history/bug_fixes/write_committed_toggle_udt.md

@@ -1 +0,0 @@
-*Fix a bug for replaying WALs for WriteCommitted transaction DB when its user-defined timestamps setting is toggled on/off between DB sessions.

unreleased_history/new_features/multi_cf_iterator.md

@@ -1 +0,0 @@
-Multi-Column-Family-Iterator (CoalescingIterator/AttributeGroupIterator) is no longer marked as experimental

unreleased_history/new_features/tp_newest_key_time.md

@@ -1 +0,0 @@
-* Adds a new table property "rocksdb.newest.key.time" which records the unix timestamp of the newest key. Uses this table property for FIFO TTL and temperature change compaction. 

unreleased_history/performance_improvements/cra_regression.md

@@ -1 +0,0 @@
-Fix regression in issue #12038 due to `Options::compaction_readahead_size` greater than `max_sectors_kb` (i.e, largest I/O size that the OS issues to a block device defined in linux)

unreleased_history/public_api_changes/txn_get_attribute_group_iterator.md

@@ -1 +0,0 @@
-Added a new API `Transaction::GetAttributeGroupIterator` that can be used to create a multi-column-family attribute group iterator over the specified column families, including the data from both the transaction and the underlying database. This API is currently supported for optimistic and write-committed pessimistic transactions.

unreleased_history/public_api_changes/txn_get_coalescing_iterator.md

@@ -1 +0,0 @@
-Added a new API `Transaction::GetCoalescingIterator` that can be used to create a multi-column-family coalescing iterator over the specified column families, including the data from both the transaction and the underlying database. This API is currently supported for optimistic and write-committed pessimistic transactions.