Don't let error recovery wait forever behind a writer stalled on the failed flush (#15605)

Summary:
Pull Request resolved: https://github.com/meta/rocksdb/pull/15605

**Problem:** automatic error recovery can hang forever, with all writes stopped, when a writer is waiting on a write stop (for example the `max_write_buffer_number` memtable limit) as the flush that would end it fails. Only a restart clears it.

**Why:** the stalled writer leads the write queue. It waits in `DelayWrite()` for the stop to end, and keeps waiting while recovery is in progress, so any recovery step that enters the write queue waits behind it. Two such steps exist:
1. A retryable SST write error is a soft error, and recovery retries the flush of the immutable memtables, which does not need the write queue. But with `two_write_queues = true`, `ResumeImpl()` first enters both write queues to sync `LastSequence()` with the allocated sequence numbers.
2. A hard error with auto recovery, for example a retryable error syncing the closed WALs during the flush of a DB with two column families, is recovered by an atomic flush of all column families. Its memtable switch enters the write queue in every configuration. New writes already fail with the background error at this point (`PreprocessWrite()` checks `IsDBStopped()`); only a writer that was already waiting keeps waiting.

**Fix:**
- The sequence sync at recovery start no longer waits for a write queue whose leader is in a write stall. That leader stalls before it allocates sequence numbers, and it cannot leave the stall while recovery holds the DB mutex, so the sync stays exact. If the stall ends while recovery waits for the other queue, the sync retries. Queues without a stall are still drained first, and the recovery flushes that switch memtables keep their own full fence.
- `DelayWrite()` stops waiting on a write stop once the DB is stopped, and returns the background error, like a new write does.

**Tests:**
- In `error_handler_fs_test`:
  - `RetryableFlushErrorWithWriterStalledOnFlush` (with and without `two_write_queues`): the first flush fails with a retryable SST write error after the writer stops on its memtables; the writer finishes after recovery.
  - `FlushWALSyncErrorWithWriterStalledOnFlush`: with two column families, the closed WAL sync fails after the writer stops; the writer gets the hard error, recovery completes, and the next write succeeds.
- In `write_prepared_transaction_seqno_test`:
  - `RecoverySeqnoSyncRetriesWhenStallEndsDuringNonmemJoin`: the main queue stall ends while recovery joins the nonmem queue, and the writer allocates before recovery syncs; recovery retries and syncs only after the writer publishes.
  - `RecoverySeqnoSyncSkipsStalledWalOnlyCommit`: a WAL-only commit is stalled on the nonmem queue; recovery syncs and finishes while the commit still waits.

Reviewed By: archang19

Differential Revision: D127129804

fbshipit-source-id: 88ed4938be39d7222ceb7c28437187989885b5d0

HISTORY.md

@@ -1,6 +1,81 @@
 # Rocksdb Change Log
 > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
 
+## 11.14.0 (11/05/2026)
+### New Features
+* Added experimental `SetAsyncReadCPUTimeEnabled()`: the next async or coroutine read started on the thread records its CPU time in `PerfContext::get_cpu_nanos` at any perf level except `kDisable`, without turning on the other timers of `kEnableTimeAndCPUTimeExceptForMutex`.
+* Added `SplitDB()`, `DeleteSplitRanges()`, and `MergeDB()` in `rocksdb/utilities/db_split_merge.h`, plus a `db_split_merge` command-line tool, for moving selected column-family key ranges into a new DB and copying ranges into existing column families.
+* Added opt-in sampled logging of long-held DB-wide mutexes under `StatsLevel::kAll`, with a runtime-changeable `mutex_hold_log_threshold_us` DB option defaulting to 0 (disabled).
+* Add experimental live tuning of a DB's options (`StartLiveTuneCycle()`, `GetLiveTuneStatus()`, `StopLiveTuneCycle()` in `include/rocksdb/live_tune.h`): one call runs a cycle that measures the DB, asks an application-provided `LiveTuneAdvisor` for option changes, applies them, and reverts them when they break the application's constraints or do not prove better.
+* Added `CompactionFilter::GetMergeOrder()` and `CompactionFilterFactory::GetMergeOrder()` to optionally run the merge operator before the compaction filter during compaction, so that the filter sees merge results. The default behavior is unchanged.
+* Parallel compression (`CompressionOptions::parallel_threads > 1`) now works with a `user_defined_index_factory` whose builders support it, including the built-in trie index. Previously a custom index always forced single-threaded compression.
+* Added option `reject_too_old_file` to `IngestExternalFileOptions` and `ImportColumnFamilyOptions` to reject files whose data is already older than the column family's `periodic_compaction_seconds` (or `ttl` for FIFO compaction).
+* Added BlobDB support for remote compaction. Blob garbage collection uses the
+cutoff computed by the primary; workers receiving legacy input without a cutoff
+safely disable blob garbage collection for that job. When the primary rejects a
+remote blob result because its metadata, files, or capabilities fail validation,
+it reruns the compaction locally, so `CompactRange` and `CompactFiles` succeed.
+`CompactionService` `kFailure` semantics are unchanged; ZippyDB production also
+maps service failures to local fallback through `HandleFailure(FALLBACK_LOCAL)`.
+* `CompactRangeOptions::skip_missing_blob_files` enables offline repair that deletes values referencing integrated external blob files absent from Version metadata, preserves available wide entity columns, and rewrites selected SSTs locally, bypassing `DBOptions::compaction_service`. It defaults to false; FIFO compaction and legacy BlobDB are unsupported.
+* Added `db_ownership_start_time` and `data_file_provenance` (decoded by `DataFileProvenance`) to `SstFileMetaData`, recording when the DB began holding each SST file and how it got the file (e.g. flush, compaction, ingestion, or import).
+* Added `TableFactory::IsTableBuilderSupported()` and `ExternalTableFactory::IsTableBuilderSupported()` for table factories that only read ingested or imported files. A column family whose factory returns false must set `disallow_memtable_writes`, is never scheduled for automatic compaction, does not stall writes for L0 files or pending compaction bytes, and returns `NotSupported` from `CompactRange()` and `CompactFiles()` instead of failing a background compaction.
+* trace_analyzer: new `--analyze_put_entity` flag analyzes PutEntity queries (previously their statistics were neither enabled nor disabled explicitly).
+* Added `TraceOptions::include_read_write_options`: query traces can record the main `ReadOptions` and `WriteOptions` of each traced call (e.g. `fill_cache`, `disableWAL`), which replay then uses instead of default options.
+
+### Public API Changes
+* Added `GetEncodedEntrySize()` and an `EncodeEntry()` overload that writes internal keys into caller-owned buffers without allocating on success.
+* Added `Env::Priority::READ` and `ThreadStatus::READ_PRIORITY` for a dedicated Env-managed read-work thread pool. `Env::Priority::USER` remains reserved for internal statistics and is not backed by a thread pool.
+* Added experimental `MultiScanArgs::use_parallel_async_io_block_decompression` for materializing asynchronously prefetched data blocks, including decompression, in parallel on the `Env::Priority::READ` thread pool. A zero-sized pool preserves inline materialization.
+* Added the experimental `IndexFactory` interface (`include/rocksdb/index_factory.h`) for pluggable SST indexes; the `UserDefinedIndex*` names remain as aliases. `BlockBasedTableOptions::index_mode` chooses which indexes an SST is built with and which one reads use by default, superseding the now-deprecated `use_udi_as_primary_index` and `fail_if_no_udi_on_open`. `ReadOptions::read_index` overrides that choice per read, and the `standard_index_is_stub` table property marks SSTs built with `kCustomOnly`. Both options are also available in the C API.
+* Added experimental `DB::GetEntityWithMetadata()` and `DB::MultiGetEntityWithMetadata()`, the wide-column entity counterparts of `DB::GetWithMetadata()` and `DB::MultiGetWithMetadata()` with opt-in explicit-snapshot newer-version detection. Requesting key timestamps returns `NotSupported`.
+* Removed support for building on Solaris, which had not compiled since 5.18. Removed RocksJava `Environment.isSolaris()`.
+
+### Behavior Changes
+* With `ReadOptions::async_io`, `MultiGet` now reads data blocks of direct-IO files with asynchronous reads (`FSRandomAccessFile::ReadAsync`) instead of a synchronous `MultiRead`. Each block read is aligned separately rather than merged with adjacent reads. The bytes read are still charged to `DBOptions::rate_limiter` at `ReadOptions::rate_limiter_priority`.
+* Asynchronous file reads (`FSRandomAccessFile::ReadAsync`, used with `ReadOptions::async_io` by `MultiGet` and iterator prefetching) are now charged to `DBOptions::rate_limiter` at `ReadOptions::rate_limiter_priority`, like synchronous reads. A read is charged once the file system accepts it.
+* `CreateColumnFamily()` and `CreateColumnFamilies()` now run the table factory option validation that `DB::Open()` runs, so they return `InvalidArgument` instead of creating a column family that the next `DB::Open()` would reject. External table factories wrapped by `NewExternalTableFactory()` now receive `ValidateOptions()` calls.
+* With `CompactionOptionsFIFO::allow_trivial_copy_when_change_temperature`, each file copied for a temperature change is now reported to `EventListener::OnTableFileCreationStarted()` and `OnTableFileCreated()` (with the original file's table properties) and included in `CompactionJobInfo::table_properties`.
+* `DB::GetMergeOperands()` now resets only the `merge_operands` entries it fills, instead of all `expected_max_number_of_operands` entries. Entries past `*number_of_operands`, and the whole array on a non-OK result, are left as the caller passed them.
+* The JSON written to the info LOG by `EventLogger` (`EVENT_LOG_v1` lines) and by `VersionEdit::DebugJSON()` now escapes special characters in keys and string values, and writes non-finite floating-point values as `null`, so the output is always valid JSON.
+* Improved how `periodic_compaction_phase_recovery_percent` spreads catch-up periodic compactions after it is enabled or changed (or the DB is reopened). This reduces a transient burst in periodic compactions when many DBs make such a change at once.
+* The new DB option `use_separate_remote_compaction_pool` defaults to false and can be changed through `SetDBOptions()` for newly scheduled jobs. When enabled with CompactionService, remote jobs use a dedicated pool and each local subcompaction uses LOW/BOTTOM and counts against the DB compaction limit. Existing jobs finish using their original scheduling model.
+* `DB::SetOptions()` now returns OK without writing a new OPTIONS file or installing a new SuperVersion when the call changes no option value, as `DB::SetDBOptions()` already does. Calls that set an object option, such as `table_factory` or one of its fields, are still applied in full. After a failed OPTIONS file write, the next `SetOptions()` or `SetDBOptions()` call writes the file even if no option value changes.
+* Custom index factories are rejected with user-defined timestamps in every index mode except `kStandardOnly`, which ignores the factory. Custom-index block-cache hits, misses, and adds now contribute to `BLOCK_CACHE_INDEX_*` instead of `BLOCK_CACHE_DATA_*`.
+Trie index construction returns `NotSupported` when a data-block offset or size exceeds `UINT32_MAX`, including when the trie is optional in `kStandardDefault`. Flush, compaction, and `SstFileWriter::Finish()` can therefore fail for such output; an SST larger than 4 GiB is not itself the rejection condition. File-size targets do not split L0 outputs or `SstFileWriter` outputs. Bound those outputs or disable trie construction with `kStandardOnly`.
+
+### Bug Fixes
+* Fixed a race where `IngestExternalFiles()` with `atomic_replace_range` could delete an existing SST file that a concurrently picked compaction (e.g. a universal compaction outputting to L0) used as input, causing the compaction to fail with "Corruption: VersionBuilder: Cannot delete table file ... since it is not in the LSM tree" and putting the DB into read-only mode. Atomic range replacement now returns `NotSupported` with FIFO compaction.
+* Fixed blob direct write when sealing a blob file fails during a flush or a write: the flush left its memtables marked as being flushed, so no later flush picked them up and error recovery waited for them forever, and the blob file was dropped from tracking, so a retried flush could install table files whose blob file was never added to the MANIFEST.
+* Fixed blob direct write with more than one concurrent flush of a column family: a later flush sealed and published the blob files of the oldest pending memtable instead of its own, so flushes failed with `Corruption: VersionBuilder: Blob file #N already added` and the database stopped.
+* Fixed blob direct write when the `SstFileManager` space limit is reached while a blob file is sealed: the sealed blob file was deleted while memtables still referenced it, so reads of its values failed and the values were lost, and reporting the error could deadlock with a concurrent memtable switch. The seal and the write or flush now succeed and keep the file, and writes then stop with a `SpaceLimit` background error, so for blob direct write the limit is soft: usage can exceed it by up to about `blob_file_size` × `blob_direct_write_partitions` for each memtable not flushed yet. Also fixed a write batch that failed after writing some of its values to blob files: those blobs are now counted as garbage, so their blob file can still be deleted.
+* Values that retain external blob references now keep their original timestamps during history compaction, preventing blob key verification failures.
+* Fixed `DB::ClipColumnFamily()` leaving an empty-string key outside the kept range when it is the only key in an L0 SST file.
+* Fixed `format_version=8` with `ReverseBytewiseComparator`: a block's stored common key prefix could extend past a user key (one following longer user keys that extend it), so reads and iteration could miss that key (or its deletion, exposing an older value), and compactions could fail with `Corruption`. Files written by affected versions (11.10 through 11.13) are now read correctly, and new files no longer have this problem.
+* Fixed SST file times that could defer periodic compaction and TTL. Files written by `SstFileWriter`, flushed during WAL recovery, or recovered by `RepairDB` now record a file creation time. Files added by `IngestExternalFile()` or `CreateColumnFamilyWithImport()` now keep the creation and oldest ancestor times from the file or export metadata (capped at the current time) instead of taking the time they were added, so ingesting or importing old data can make it due for periodic compaction or TTL soon afterward. FIFO TTL compaction now also deletes ingested `SstFileWriter` files, which previously blocked TTL deletion of themselves and all newer files. Files copied by FIFO compaction for a temperature change now keep the original file's creation time.
+* Fixed a debug-build assertion failure and incorrect cache charge and block-cache byte statistics in release builds when a full, strict-capacity LRU cache with metadata charging creates an uncharged standalone handle, including during secondary-cache promotion.
+* Fixed atomic flushes of all column families when another column family is dropped while they wait: `CompactRange()` no longer fails with a "Column family dropped" error, and `WaitForCompact()` and DB close no longer return OK without flushing the remaining column families.
+* Fixed a bug where MemPurge returned with the DB mutex unlocked when the compaction filter created for flush did not ignore snapshots.
+* Fixed a crash in MultiScan `Iterator::Prepare()` with a single scan range without a limit when the DB has L0 files, and an assertion failure in debug builds when `Prepare()` is called on an already-positioned iterator. `Prepare()` now leaves the iterator invalid until it is positioned again (e.g. by `Seek()`).
+* Fixed `MultiGet` returning NotFound for existing keys that are outside the prefix extractor's domain, with partitioned filters, `whole_key_filtering=false`, and filter partitions cut together with index partitions (`decouple_partitioned_filters=false`, also the layout of files written by older versions). A filter partition holding only such keys is empty, and `MultiGet` treated every key mapped to it as absent. `Get` was not affected.
+* Fixed prepared external file ingestions retaining unrelated obsolete SSTs until commit or abort. Prepared ingestions now protect only their staged files from deletion.
+* Fixed data loss with `min_tombstones_for_range_conversion` and external file ingestion. An iterator whose view predated the ingestion could convert the deleted keys around the ingested keys into a range tombstone that deleted them. This happened when the file got no new sequence number (sequence number zero, `ingest_behind`, or `allow_db_generated_files`), and when an iterator was created during the ingestion.
+* Fixed a data loss bug where an iterator with `ReadOptions::read_tier = kMemtableTier` could, with `min_tombstones_for_range_conversion` enabled, insert a range tombstone that deleted live keys stored only in SST files. Read-path range tombstone conversion now runs only for iterators with `read_tier = kReadAllTier`.
+* Fixed a race with `recycle_log_file_num > 0` where an obsolete WAL could be reused for a new WAL while its old writer was still open. Closing the old writer then truncated the reused file to the old WAL's size, so writes in the new WAL past that point, including acknowledged `sync` writes, were lost on the next reopen.
+* Fixed remote compaction ignoring `preclude_last_level_data_seconds` and `preserve_internal_time_seconds`, which caused recent data to be written to the last level and recent sequence numbers to be zeroed. Takes effect once both the primary and the compaction worker are upgraded.
+* Fixed `DB::OpenAndCompact()` with `OpenAndCompactOptions::allow_resumption` not calling `EventListener::OnSubcompactionCompleted()` when the compaction fails to resume from its saved progress.
+* Fixed MANIFEST catch-up on secondary instances (`TryCatchUpWithPrimary()`) and follower instances installing a column family with no files and still returning OK. This happened when catch-up switched to a new MANIFEST and the primary deleted a file that MANIFEST lists before catch-up checked it, recording the deletion in a newer MANIFEST. Catch-up now keeps the column family's previous view until a complete state from the new MANIFEST is available.
+* Custom index blocks now reuse cached contents on SST reopen, and deprecated UDI boolean options can be cleared without overriding an explicitly configured index mode.
+Legacy UDI options copied from a factory can now be cleared before constructing a new factory without retaining the derived mode. The C API preserves both that rollback behavior and explicitly selected modes. OPTIONS files serialize equivalent legacy booleans for `kStandardDefault`, `kStandardRequired`, and `kCustomDefault` so older readers retain routing.
+Empty custom-index output retains the standard-index fallback in modes that write both indexes; custom-only output must remain nonempty. Default primary reads retain the table factory's precedence over a legacy per-read factory pointer, while explicit custom selectors still check its name.
+* Preserve legacy UDI routing when loading old OPTIONS files, and reject invalid index modes in standalone SST readers and writers. Table readers now charge the custom index blocks they own; a custom reader whose `ApproximateMemoryUsage()` already includes that block should override `IndexFactoryReader::MemoryUsageIncludesIndexBlock()` to return true so the block is not charged twice.
+Keep cached custom-index Slice views intact when a reader consumes its input. Register custom-index cache helpers without requiring a later change. Trie index size estimates now account for shared prefixes, preventing premature compaction file cuts, and reject data-block offsets or sizes exceeding the 32-bit on-disk limit instead of truncating them.
+
+### Performance Improvements
+* With `ReadOptions::pin_data`, iterators over read-only DBs opened with `max_open_files = -1` and `allow_mmap_reads = true` no longer accumulate memory as they move across SST files.
+* On a read-write primary, `DB::GetSnapshot()` and `DB::ReleaseSnapshot()` no longer acquire the DB mutex in the common case. Secondary and follower snapshot creation retains the DB mutex needed to synchronize with view installation, while release remains sharded. The snapshot list is now split into 16 per-CPU shards, each with its own lock, so primary threads creating and releasing snapshots no longer serialize on one lock or wait behind flush, compaction, or other work holding the DB mutex. Snapshot-release maintenance is deferred to existing flush, compaction, and periodic maintenance paths. Primary snapshot creation serializes with the DB-mutex-held phases of external file ingestion but can proceed while the MANIFEST write temporarily releases that mutex.
+
+
 ## 11.13.0 (10/21/2026)
 ### New Features
 * Added `FileSystem::IsPreallocationSupported()`. A `FileSystem` that never preallocates file space can return false so that DB open skips reopening and truncating the last WAL file.

c_api_gen/c_generated_jobinfo_auto.cc.inc

@@ -109,9 +109,10 @@ int rocksdb_compactionjobinfo_job_id(const rocksdb_compactionjobinfo_t* info) {
   return info->rep.job_id;
 }
 
-unsigned char rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+unsigned char
+rocksdb_compactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_compactionjobinfo_t* info) {
-  return info->rep.use_separate_remote_compaction_pool;
+  return info->rep.separate_remote_compaction_pool_selected;
 }
 
 int rocksdb_compactionjobinfo_num_l0_files(
@@ -177,9 +178,10 @@ int rocksdb_subcompactionjobinfo_job_id(
   return info->rep.job_id;
 }
 
-unsigned char rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+unsigned char
+rocksdb_subcompactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_subcompactionjobinfo_t* info) {
-  return info->rep.use_separate_remote_compaction_pool;
+  return info->rep.separate_remote_compaction_pool_selected;
 }
 
 int rocksdb_subcompactionjobinfo_subcompaction_job_id(

c_api_gen/c_generated_jobinfo_auto.h.inc

@@ -79,7 +79,7 @@ extern ROCKSDB_LIBRARY_API int rocksdb_compactionjobinfo_job_id(
     const rocksdb_compactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API unsigned char
-rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+rocksdb_compactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_compactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API int rocksdb_compactionjobinfo_num_l0_files(
@@ -122,7 +122,7 @@ extern ROCKSDB_LIBRARY_API int rocksdb_subcompactionjobinfo_job_id(
     const rocksdb_subcompactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API unsigned char
-rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+rocksdb_subcompactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_subcompactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API int

c_api_gen/c_generated_option_structs_auto.cc.inc

@@ -2175,6 +2175,16 @@ int rocksdb_ingestexternalfileoptions_get_file_opening_threads(
   return opt->rep.file_opening_threads;
 }
 
+void rocksdb_ingestexternalfileoptions_set_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
+  opt->rep.reject_too_old_file = v;
+}
+
+unsigned char rocksdb_ingestexternalfileoptions_get_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt) {
+  return opt->rep.reject_too_old_file;
+}
+
 /* OpenAndCompactOptions */
 
 void rocksdb_open_and_compact_options_set_max_secondary_open_retries(
@@ -2199,6 +2209,16 @@ unsigned char rocksdb_import_column_family_options_get_move_files(
   return opt->rep.move_files;
 }
 
+void rocksdb_import_column_family_options_set_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt, unsigned char v) {
+  opt->rep.reject_too_old_file = v;
+}
+
+unsigned char rocksdb_import_column_family_options_get_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt) {
+  return opt->rep.reject_too_old_file;
+}
+
 /* SizeApproximationOptions */
 
 void rocksdb_size_approximation_options_set_include_memtables(

c_api_gen/c_generated_option_structs_auto.h.inc

@@ -907,6 +907,14 @@ extern ROCKSDB_LIBRARY_API int
 rocksdb_ingestexternalfileoptions_get_file_opening_threads(
     rocksdb_ingestexternalfileoptions_t* opt);
 
+extern ROCKSDB_LIBRARY_API void
+rocksdb_ingestexternalfileoptions_set_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_ingestexternalfileoptions_get_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt);
+
 /* OpenAndCompactOptions */
 
 extern ROCKSDB_LIBRARY_API void
@@ -927,6 +935,14 @@ extern ROCKSDB_LIBRARY_API unsigned char
 rocksdb_import_column_family_options_get_move_files(
     rocksdb_import_column_family_options_t* opt);
 
+extern ROCKSDB_LIBRARY_API void
+rocksdb_import_column_family_options_set_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_import_column_family_options_get_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt);
+
 /* SizeApproximationOptions */
 
 extern ROCKSDB_LIBRARY_API void

c_api_gen/c_generated_roundtrip_tests.c.inc

@@ -485,6 +485,18 @@ static void rocksdb_roundtrip_test_rocksdb_flushwaloptions(void) {
   rocksdb_flushwaloptions_destroy(obj);
 }
 
+static void rocksdb_roundtrip_test_rocksdb_import_column_family_options(void) {
+  rocksdb_import_column_family_options_t* obj =
+      rocksdb_import_column_family_options_create();
+  rocksdb_import_column_family_options_set_reject_too_old_file(obj, 1);
+  CheckCondition(
+      rocksdb_import_column_family_options_get_reject_too_old_file(obj) == 1);
+  rocksdb_import_column_family_options_set_reject_too_old_file(obj, 0);
+  CheckCondition(
+      rocksdb_import_column_family_options_get_reject_too_old_file(obj) == 0);
+  rocksdb_import_column_family_options_destroy(obj);
+}
+
 static void rocksdb_roundtrip_test_rocksdb_ingestexternalfileoptions(void) {
   rocksdb_ingestexternalfileoptions_t* obj =
       rocksdb_ingestexternalfileoptions_create();
@@ -526,6 +538,12 @@ static void rocksdb_roundtrip_test_rocksdb_ingestexternalfileoptions(void) {
   CheckCondition(
       rocksdb_ingestexternalfileoptions_get_prefetch_lmax_index_and_filter_blocks(
           obj) == 0);
+  rocksdb_ingestexternalfileoptions_set_reject_too_old_file(obj, 1);
+  CheckCondition(
+      rocksdb_ingestexternalfileoptions_get_reject_too_old_file(obj) == 1);
+  rocksdb_ingestexternalfileoptions_set_reject_too_old_file(obj, 0);
+  CheckCondition(
+      rocksdb_ingestexternalfileoptions_get_reject_too_old_file(obj) == 0);
   rocksdb_ingestexternalfileoptions_set_verify_checksums_before_ingest(obj, 1);
   CheckCondition(
       rocksdb_ingestexternalfileoptions_get_verify_checksums_before_ingest(
@@ -1403,6 +1421,7 @@ static void rocksdb_run_generated_roundtrip_tests(void) {
   rocksdb_roundtrip_test_rocksdb_fifo_compaction_options();
   rocksdb_roundtrip_test_rocksdb_flushoptions();
   rocksdb_roundtrip_test_rocksdb_flushwaloptions();
+  rocksdb_roundtrip_test_rocksdb_import_column_family_options();
   rocksdb_roundtrip_test_rocksdb_ingestexternalfileoptions();
   rocksdb_roundtrip_test_rocksdb_livefiles_storage_info_options();
   rocksdb_roundtrip_test_rocksdb_open_and_compact_options();

db/builder.cc

@@ -400,7 +400,7 @@ Status BuildTable(
             std::min(meta->fd.smallest_seqno, smallest_preferred_seqno),
             meta->fd.largest_seqno);
         relevant_mapping.SetCapacity(kMaxSeqnoTimePairsPerSST);
-        relevant_mapping.Enforce(tboptions.file_creation_time);
+        relevant_mapping.Enforce(meta->file_creation_time);
       }
       builder->SetSeqnoTimeTableProperties(
           relevant_mapping,

db/c.cc

@@ -4952,9 +4952,10 @@ int rocksdb_compactionjobinfo_job_id(const rocksdb_compactionjobinfo_t* info) {
   return info->rep.job_id;
 }
 
-unsigned char rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+unsigned char
+rocksdb_compactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_compactionjobinfo_t* info) {
-  return info->rep.use_separate_remote_compaction_pool;
+  return info->rep.separate_remote_compaction_pool_selected;
 }
 
 int rocksdb_compactionjobinfo_num_l0_files(
@@ -5020,9 +5021,10 @@ int rocksdb_subcompactionjobinfo_job_id(
   return info->rep.job_id;
 }
 
-unsigned char rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+unsigned char
+rocksdb_subcompactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_subcompactionjobinfo_t* info) {
-  return info->rep.use_separate_remote_compaction_pool;
+  return info->rep.separate_remote_compaction_pool_selected;
 }
 
 int rocksdb_subcompactionjobinfo_subcompaction_job_id(
@@ -5733,6 +5735,16 @@ uint64_t rocksdb_blob_file_garbage_info_garbage_blob_bytes(
 }
 // END generated: c_generated_metadata_structs_auto.cc.inc
 
+unsigned char rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_compactionjobinfo_t* info) {
+  return info->rep.separate_remote_compaction_pool_selected;
+}
+
+unsigned char rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_subcompactionjobinfo_t* info) {
+  return info->rep.separate_remote_compaction_pool_selected;
+}
+
 rocksdb_compactionjobinfo_t* rocksdb_compactionjobinfo_create() {
   return new rocksdb_compactionjobinfo_t;
 }
@@ -11617,6 +11629,16 @@ int rocksdb_ingestexternalfileoptions_get_file_opening_threads(
   return opt->rep.file_opening_threads;
 }
 
+void rocksdb_ingestexternalfileoptions_set_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt, unsigned char v) {
+  opt->rep.reject_too_old_file = v;
+}
+
+unsigned char rocksdb_ingestexternalfileoptions_get_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt) {
+  return opt->rep.reject_too_old_file;
+}
+
 /* OpenAndCompactOptions */
 
 void rocksdb_open_and_compact_options_set_max_secondary_open_retries(
@@ -11641,6 +11663,16 @@ unsigned char rocksdb_import_column_family_options_get_move_files(
   return opt->rep.move_files;
 }
 
+void rocksdb_import_column_family_options_set_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt, unsigned char v) {
+  opt->rep.reject_too_old_file = v;
+}
+
+unsigned char rocksdb_import_column_family_options_get_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt) {
+  return opt->rep.reject_too_old_file;
+}
+
 /* SizeApproximationOptions */
 
 void rocksdb_size_approximation_options_set_include_memtables(

db/column_family.cc

@@ -635,6 +635,8 @@ ColumnFamilyData::ColumnFamilyData(
       mutable_cf_options_(initial_cf_options_),
       is_delete_range_supported_(
           cf_options.table_factory->IsDeleteRangeSupported()),
+      is_table_builder_supported_(
+          cf_options.table_factory->IsTableBuilderSupported()),
       write_buffer_manager_(write_buffer_manager),
       mem_(nullptr),
       imm_(ioptions_.min_write_buffer_number_to_merge,
@@ -1025,13 +1027,16 @@ ColumnFamilyData::GetWriteStallConditionAndCause(
     int num_unflushed_memtables, int num_l0_files,
     uint64_t num_compaction_needed_bytes,
     const MutableCFOptions& mutable_cf_options,
-    const ImmutableCFOptions& immutable_cf_options) {
+    const ImmutableCFOptions& immutable_cf_options,
+    bool table_builder_supported) {
+  const bool auto_compactions =
+      table_builder_supported && !mutable_cf_options.disable_auto_compactions;
   if (num_unflushed_memtables >= mutable_cf_options.max_write_buffer_number) {
     return {WriteStallCondition::kStopped, WriteStallCause::kMemtableLimit};
-  } else if (!mutable_cf_options.disable_auto_compactions &&
+  } else if (auto_compactions &&
              num_l0_files >= mutable_cf_options.level0_stop_writes_trigger) {
     return {WriteStallCondition::kStopped, WriteStallCause::kL0FileCountLimit};
-  } else if (!mutable_cf_options.disable_auto_compactions &&
+  } else if (auto_compactions &&
              mutable_cf_options.hard_pending_compaction_bytes_limit > 0 &&
              num_compaction_needed_bytes >=
                  mutable_cf_options.hard_pending_compaction_bytes_limit) {
@@ -1043,12 +1048,12 @@ ColumnFamilyData::GetWriteStallConditionAndCause(
              num_unflushed_memtables - 1 >=
                  immutable_cf_options.min_write_buffer_number_to_merge) {
     return {WriteStallCondition::kDelayed, WriteStallCause::kMemtableLimit};
-  } else if (!mutable_cf_options.disable_auto_compactions &&
+  } else if (auto_compactions &&
              mutable_cf_options.level0_slowdown_writes_trigger >= 0 &&
              num_l0_files >=
                  mutable_cf_options.level0_slowdown_writes_trigger) {
     return {WriteStallCondition::kDelayed, WriteStallCause::kL0FileCountLimit};
-  } else if (!mutable_cf_options.disable_auto_compactions &&
+  } else if (auto_compactions &&
              mutable_cf_options.soft_pending_compaction_bytes_limit > 0 &&
              num_compaction_needed_bytes >=
                  mutable_cf_options.soft_pending_compaction_bytes_limit) {
@@ -1070,7 +1075,7 @@ WriteStallCondition ColumnFamilyData::RecalculateWriteStallConditions(
     auto write_stall_condition_and_cause = GetWriteStallConditionAndCause(
         imm()->NumNotFlushed(), vstorage->l0_delay_trigger_count(),
         vstorage->estimated_compaction_needed_bytes(), mutable_cf_options,
-        ioptions());
+        ioptions(), is_table_builder_supported_);
     write_stall_condition = write_stall_condition_and_cause.first;
     auto write_stall_cause = write_stall_condition_and_cause.second;
 
@@ -1276,7 +1281,8 @@ void ColumnFamilyData::CreateNewMemtable(SequenceNumber earliest_seq) {
 }
 
 bool ColumnFamilyData::NeedsCompaction() const {
-  return !mutable_cf_options_.disable_auto_compactions &&
+  return is_table_builder_supported_ &&
+         !mutable_cf_options_.disable_auto_compactions &&
          compaction_picker_->NeedsCompaction(current_->storage_info());
 }
 
@@ -1541,6 +1547,13 @@ Status ColumnFamilyData::ValidateOptions(
     return s;
   }
 
+  if (!cf_options.table_factory->IsTableBuilderSupported() &&
+      !cf_options.disallow_memtable_writes) {
+    return Status::InvalidArgument(
+        "A table factory that cannot build tables requires "
+        "disallow_memtable_writes");
+  }
+
   if (cf_options.ttl > 0 && cf_options.ttl != kDefaultTtl) {
     if (!cf_options.table_factory->IsInstanceOf(
             TableFactory::kBlockBasedTableName())) {

db/column_family.h

@@ -368,6 +368,12 @@ class ColumnFamilyData {
 
   bool is_delete_range_supported() { return is_delete_range_supported_; }
 
+  // False when the table factory cannot write tables, so the column family is
+  // never compacted.
+  bool is_table_builder_supported() const {
+    return is_table_builder_supported_;
+  }
+
   // Validate CF options against DB options
   static Status ValidateOptions(const DBOptions& db_options,
                                 const ColumnFamilyOptions& cf_options);
@@ -562,12 +568,15 @@ class ColumnFamilyData {
     --num_in_flight_compactions_;
   }
 
+  // L0 files and pending compaction bytes stall writes only when the column
+  // family can be compacted automatically, which requires
+  // 'table_builder_supported' and no disable_auto_compactions.
   static std::pair<WriteStallCondition, WriteStallCause>
-  GetWriteStallConditionAndCause(
-      int num_unflushed_memtables, int num_l0_files,
-      uint64_t num_compaction_needed_bytes,
-      const MutableCFOptions& mutable_cf_options,
-      const ImmutableCFOptions& immutable_cf_options);
+  GetWriteStallConditionAndCause(int num_unflushed_memtables, int num_l0_files,
+                                 uint64_t num_compaction_needed_bytes,
+                                 const MutableCFOptions& mutable_cf_options,
+                                 const ImmutableCFOptions& immutable_cf_options,
+                                 bool table_builder_supported);
 
   // Recalculate some stall conditions, which are changed only during
   // compaction, adding new memtable and/or recalculation of compaction score.
@@ -705,6 +714,7 @@ class ColumnFamilyData {
   MutableCFOptions mutable_cf_options_;
 
   const bool is_delete_range_supported_;
+  const bool is_table_builder_supported_;
 
   // True from a successful SetOptions() until MarkOptionsInstalled(). A
   // DB::SetOptions() call that fails on a later column family leaves it set

db/compaction/compaction.h

@@ -279,11 +279,11 @@ class Compaction {
   // Does this compaction include all sst files?
   bool is_full_compaction() const { return is_full_compaction_; }
 
-  bool use_separate_remote_compaction_pool() const {
-    return use_separate_remote_compaction_pool_;
+  bool separate_remote_compaction_pool_selected() const {
+    return separate_remote_compaction_pool_selected_;
   }
-  void SetUseSeparateRemoteCompactionPool(bool enabled) {
-    use_separate_remote_compaction_pool_ = enabled;
+  void SetSeparateRemoteCompactionPoolSelected(bool selected) {
+    separate_remote_compaction_pool_selected_ = selected;
   }
 
   // Was this compaction triggered manually by the client?
@@ -624,7 +624,7 @@ class Compaction {
   const bool bottommost_level_;
   // Does this compaction include all sst files?
   const bool is_full_compaction_;
-  bool use_separate_remote_compaction_pool_ = false;
+  bool separate_remote_compaction_pool_selected_ = false;
 
   // Is this compaction requested by the client?
   const bool is_manual_compaction_;

db/compaction/compaction_job.cc

@@ -1532,8 +1532,8 @@ void CompactionJob::NotifyOnSubcompactionBegin(
   SubcompactionJobInfo info{};
   sub_compact->BuildSubcompactionJobInfo(info);
   info.job_id = static_cast<int>(job_id_);
-  info.use_separate_remote_compaction_pool =
-      compact_->compaction->use_separate_remote_compaction_pool();
+  info.separate_remote_compaction_pool_selected =
+      compact_->compaction->separate_remote_compaction_pool_selected();
   info.thread_id = env_->GetThreadID();
 
   for (const auto& listener : db_options_.listeners) {
@@ -1559,8 +1559,8 @@ void CompactionJob::NotifyOnSubcompactionCompleted(
   SubcompactionJobInfo info{};
   sub_compact->BuildSubcompactionJobInfo(info);
   info.job_id = static_cast<int>(job_id_);
-  info.use_separate_remote_compaction_pool =
-      compact_->compaction->use_separate_remote_compaction_pool();
+  info.separate_remote_compaction_pool_selected =
+      compact_->compaction->separate_remote_compaction_pool_selected();
   info.thread_id = env_->GetThreadID();
 
   for (const auto& listener : db_options_.listeners) {
@@ -2970,6 +2970,9 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
                              sub_compact->compaction->output_path_id(), 0);
     meta.oldest_ancestor_time = oldest_ancestor_time;
     meta.file_creation_time = current_time;
+    meta.db_ownership_start_time = current_time;
+    meta.data_file_provenance =
+        DataFileProvenanceForOrigin(DataFileProvenance::Origin::kCompaction);
     meta.epoch_number = epoch_number;
     meta.temperature = temperature;
     assert(!db_id_.empty());
@@ -3024,7 +3027,7 @@ Status CompactionJob::OpenCompactionOutputFile(SubcompactionState* sub_compact,
       sub_compact->compaction->output_compression_opts(), cfd->GetID(),
       cfd->GetName(), sub_compact->compaction->output_level(), newest_key_time,
       bottommost_level_, TableFileCreationReason::kCompaction,
-      0 /* oldest_key_time */, current_time, db_id_, db_session_id_,
+      0 /* oldest_key_time */, db_id_, db_session_id_,
       sub_compact->compaction->max_output_file_size(), file_number,
       proximal_after_seqno_ /*last_level_inclusive_max_seqno_threshold*/,
       dbname_ /*db_name*/, IsRemoteCompaction() /*is_remote_compaction*/);

db/compaction/compaction_outputs.cc

@@ -235,6 +235,32 @@ uint64_t CompactionOutputs::GetCurrentKeyGrandparentOverlappedBytes(
   return overlapped_bytes;
 }
 
+bool CompactionOutputs::PartitionerRequiresCut(
+    const CompactionIterator& c_iter) const {
+  assert(HasBuilder());
+  if (!partitioner_) {
+    return false;
+  }
+  if (partitioner_prefix_valid_) {
+    // Fast path: no per-key copy or virtual call. A partition is requested at
+    // the first key that no longer shares the established prefix. An empty
+    // prefix means no further partitions (every key "starts with" it).
+    return UNLIKELY(!c_iter.user_key().starts_with(partitioner_prefix_));
+  }
+  // Slow path: per-key callback (and copy elsewhere)
+  return UNLIKELY(partitioner_->ShouldPartition(PartitionerRequest(
+                      last_key_for_partitioner_, c_iter.user_key(),
+                      current_output_file_size_)) == kRequired);
+}
+
+bool CompactionOutputs::LargestUserKeyEquals(const Slice& user_key) {
+  const InternalKey& largest = current_output().meta.largest;
+  return largest.size() > 0 &&
+         compaction_->column_family_data()
+             ->user_comparator()
+             ->EqualWithoutTimestamp(largest.user_key(), user_key);
+}
+
 bool CompactionOutputs::ShouldStopBefore(const CompactionIterator& c_iter) {
   assert(c_iter.Valid());
   const Slice& internal_key = c_iter.key();
@@ -270,22 +296,8 @@ bool CompactionOutputs::ShouldStopBefore(const CompactionIterator& c_iter) {
     return true;
   }
 
-  if (partitioner_) {
-    if (partitioner_prefix_valid_) {
-      // Fast path: no per-key copy or virtual call. A partition is requested at
-      // the first key that no longer shares the established prefix. An empty
-      // prefix means no further partitions (every key "starts with" it).
-      if (UNLIKELY(!c_iter.user_key().starts_with(partitioner_prefix_))) {
-        return true;
-      }
-    } else {
-      // Slow path: per-key callback (and copy elsewhere)
-      if (UNLIKELY(partitioner_->ShouldPartition(PartitionerRequest(
-                       last_key_for_partitioner_, c_iter.user_key(),
-                       current_output_file_size_)) == kRequired)) {
-        return true;
-      }
-    }
+  if (PartitionerRequiresCut(c_iter)) {
+    return true;
   }
 
   // files output to Level 0 won't be split
@@ -400,13 +412,24 @@ Status CompactionOutputs::AddToOutput(
     const ParsedInternalKey& prev_iter_output_internal_key) {
   Status s;
   bool is_range_del = c_iter.IsDeleteRangeSentinelKey();
+  bool cut_before;
   if (is_range_del && compaction_->bottommost_level()) {
-    // We don't consider range tombstone for bottommost level since:
-    // 1. there is no grandparent and hence no overlap to consider
-    // 2. range tombstone may be dropped at bottommost level.
-    return s;
+    // At the bottommost level, a range tombstone is not by itself a reason to
+    // cut: there are no grandparents to limit overlap with, and the tombstone
+    // might be dropped. However, a boundary required by the SstPartitioner
+    // must still be honored; otherwise AddRangeDels() would extend the
+    // current file across it. No boundary can fall inside one user key, and
+    // a point key there may have its seqno zeroed, so that it sorts after a
+    // tombstone starting at the same user key.
+    if (!HasBuilder() || LargestUserKeyEquals(c_iter.user_key()) ||
+        !PartitionerRequiresCut(c_iter)) {
+      return s;
+    }
+    cut_before = true;
+  } else {
+    cut_before = ShouldStopBefore(c_iter) && HasBuilder();
   }
-  if (ShouldStopBefore(c_iter) && HasBuilder()) {
+  if (cut_before) {
     const Slice next_table_min_key = c_iter.key();
     s = close_file_func(c_iter.InputStatus(), prev_iter_output_internal_key,
                         next_table_min_key, &c_iter,

db/compaction/compaction_outputs.h

@@ -245,6 +245,15 @@ class CompactionOutputs {
   // before processing the current key in compaction iterator.
   bool ShouldStopBefore(const CompactionIterator& c_iter);
 
+  // Returns true iff the SST partitioner requires a new output file to start
+  // at the current key. Only meaningful while an output file is open: the
+  // partitioner state describes the current file.
+  bool PartitionerRequiresCut(const CompactionIterator& c_iter) const;
+
+  // Returns true iff the current output file has a largest key and its user
+  // key equals `user_key`, ignoring timestamps.
+  bool LargestUserKeyEquals(const Slice& user_key);
+
   // Re-establishes per-file state that depends on the first key accepted by a
   // freshly opened builder. This keeps caller-requested file cuts equivalent to
   // RocksDB's built-in file-boundary decisions for partition tracking.

db/compaction/compaction_picker_fifo.cc

@@ -110,6 +110,13 @@ Compaction* FIFOCompactionPicker::PickTTLCompaction(
         uint64_t est_newest_key_time = newest_key_time == kUnknownNewestKeyTime
                                            ? oldest_ancestor_time
                                            : newest_key_time;
+        if (est_newest_key_time == kUnknownNewestKeyTime) {
+          // E.g. a file written by SstFileWriter. Every key in the file was
+          // written no later than the file's contents were built.
+          est_newest_key_time = f->TryGetFileCreationTime();
+        }
+        // CheckExternalFileNotTooOld() (reject_too_old_file) mirrors this
+        // estimate
         if (est_newest_key_time == kUnknownNewestKeyTime ||
             est_newest_key_time >= (current_time - mutable_cf_options.ttl)) {
           break;

db/compaction/compaction_service_job.cc

@@ -295,8 +295,8 @@ CompactionJob::ProcessKeyValueCompactionWithCompactionService(
       compaction->is_full_compaction(), compaction->is_manual_compaction(),
       compaction->bottommost_level(), compaction->start_level(),
       compaction->output_level());
-  info.use_separate_remote_compaction_pool =
-      compaction->use_separate_remote_compaction_pool();
+  info.separate_remote_compaction_pool_selected =
+      compaction->separate_remote_compaction_pool_selected();
   info.merge_before_filter = compaction_input.merge_before_filter;
   CompactionServiceScheduleResponse response =
       db_options_.compaction_service->Schedule(info, compaction_input_binary);
@@ -990,6 +990,12 @@ CompactionJob::ProcessKeyValueCompactionWithCompactionService(
     meta.oldest_blob_file_number = file.oldest_blob_file_number;
     meta.oldest_ancestor_time = file.oldest_ancestor_time;
     meta.file_creation_time = file.file_creation_time;
+    // The remote worker wrote the file on this DB's behalf; treat it like a
+    // local compaction output (by the worker's clock, as with its
+    // file_creation_time).
+    meta.db_ownership_start_time = file.file_creation_time;
+    meta.data_file_provenance =
+        DataFileProvenanceForOrigin(DataFileProvenance::Origin::kCompaction);
     meta.epoch_number = file.epoch_number;
     meta.file_checksum = file.file_checksum;
     meta.file_checksum_func_name = file.file_checksum_func_name;

db/compaction/compaction_service_test.cc

@@ -1301,6 +1301,25 @@ TEST_F(CompactionServiceTest, BasicCompactions) {
   SyncPoint::GetInstance()->DisableProcessing();
   VerifyTestData();
 
+  // Remote compaction outputs record DB ownership as local ones would
+  {
+    std::vector<LiveFileMetaData> live_files;
+    db_->GetLiveFilesMetaData(&live_files);
+    int num_compaction_outputs = 0;
+    for (const auto& f : live_files) {
+      const DataFileProvenance info =
+          DataFileProvenance::DecodeFrom(f.data_file_provenance);
+      if (info.origin == DataFileProvenance::Origin::kCompaction) {
+        ++num_compaction_outputs;
+      } else {
+        EXPECT_EQ(info.origin, DataFileProvenance::Origin::kFlush);
+      }
+      EXPECT_GT(f.db_ownership_start_time, 0U);
+      EXPECT_EQ(f.db_ownership_start_time, f.file_creation_time);
+    }
+    EXPECT_GT(num_compaction_outputs, 0);
+  }
+
   auto my_cs = GetCompactionService();
   ASSERT_GE(my_cs->GetCompactionNum(), 1);
   ASSERT_EQ(CompactionServiceJobStatus::kSuccess,
@@ -4862,18 +4881,18 @@ class ShortcutCompactionListener : public EventListener {
   explicit ShortcutCompactionListener(bool gate) : gate_(gate) {}
 
   void OnCompactionBegin(DB*, const CompactionJobInfo& info) override {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     moved_files = info.stats.num_input_files_trivially_moved;
     ++begun;
   }
 
   void OnCompactionPreCommit(DB*, const CompactionJobInfo& info) override {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     ++precommitted;
   }
 
   void OnCompactionCompleted(DB*, const CompactionJobInfo& info) override {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     ++completed;
   }
 
@@ -4896,19 +4915,20 @@ class CompactionContextListener : public EventListener {
  public:
   void OnCompactionBegin(DB*, const CompactionJobInfo& info) override {
     std::lock_guard<std::mutex> lock(mutex_);
-    EXPECT_TRUE(jobs_
-                    .emplace(std::make_pair(info.cf_id, info.job_id),
-                             Counts{info.use_separate_remote_compaction_pool})
-                    .second);
+    EXPECT_TRUE(
+        jobs_
+            .emplace(std::make_pair(info.cf_id, info.job_id),
+                     Counts{info.separate_remote_compaction_pool_selected})
+            .second);
     ++begun_;
-    gated_jobs_ += info.use_separate_remote_compaction_pool ? 1 : 0;
+    gated_jobs_ += info.separate_remote_compaction_pool_selected ? 1 : 0;
   }
 
   void OnSchedule(const CompactionServiceJobInfo& info) {
     std::lock_guard<std::mutex> lock(mutex_);
     auto it = jobs_.find({info.cf_id, static_cast<int>(info.job_id >> 32)});
     ASSERT_NE(it, jobs_.end());
-    EXPECT_EQ(info.use_separate_remote_compaction_pool,
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected,
               it->second.separate_pool);
     ++scheduled_;
   }
@@ -4917,7 +4937,7 @@ class CompactionContextListener : public EventListener {
     std::lock_guard<std::mutex> lock(mutex_);
     auto it = jobs_.find({info.cf_id, info.job_id});
     ASSERT_NE(it, jobs_.end());
-    EXPECT_EQ(info.use_separate_remote_compaction_pool,
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected,
               it->second.separate_pool);
     ++it->second.begun;
   }
@@ -4926,7 +4946,7 @@ class CompactionContextListener : public EventListener {
     std::lock_guard<std::mutex> lock(mutex_);
     auto it = jobs_.find({info.cf_id, info.job_id});
     ASSERT_NE(it, jobs_.end());
-    EXPECT_EQ(info.use_separate_remote_compaction_pool,
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected,
               it->second.separate_pool);
     ++it->second.completed;
   }
@@ -4935,7 +4955,7 @@ class CompactionContextListener : public EventListener {
     std::lock_guard<std::mutex> lock(mutex_);
     auto it = jobs_.find({info.cf_id, info.job_id});
     ASSERT_NE(it, jobs_.end());
-    EXPECT_EQ(info.use_separate_remote_compaction_pool,
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected,
               it->second.separate_pool);
     EXPECT_EQ(it->second.begun, 3);
     EXPECT_EQ(it->second.completed, 3);
@@ -4946,7 +4966,7 @@ class CompactionContextListener : public EventListener {
     std::lock_guard<std::mutex> lock(mutex_);
     auto it = jobs_.find({info.cf_id, info.job_id});
     ASSERT_NE(it, jobs_.end());
-    EXPECT_EQ(info.use_separate_remote_compaction_pool,
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected,
               it->second.separate_pool);
     EXPECT_TRUE(it->second.precommitted);
     jobs_.erase(it);
@@ -4982,15 +5002,15 @@ class CompactFilesPoolGateListener : public EventListener {
   explicit CompactFilesPoolGateListener(bool gate) : gate_(gate) {}
   void OnCompactionBegin(DB*, const CompactionJobInfo&) override { ++parents; }
   void OnSchedule(const CompactionServiceJobInfo& info) {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     ++scheduled;
   }
   void OnSubcompactionBegin(const SubcompactionJobInfo& info) override {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     ++begun;
   }
   void OnSubcompactionCompleted(const SubcompactionJobInfo& info) override {
-    EXPECT_EQ(info.use_separate_remote_compaction_pool, gate_);
+    EXPECT_EQ(info.separate_remote_compaction_pool_selected, gate_);
     ++completed;
   }
   std::atomic<int> parents{0};

db/compaction/tiered_compaction_test.cc

@@ -843,19 +843,20 @@ TEST_F(TieredCompactionTest, LevelOutofBoundaryRangeDelete) {
   ASSERT_OK(Flush());
 
   ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
-  ASSERT_EQ("0,2,10",
-            FilesPerLevel());  // one file is at the proximal level
-                               // which only contains a range delete
+  ASSERT_EQ("0,3,10",
+            FilesPerLevel());  // the partitioner cuts before and after the
+                               // range delete, so it has a file of its own
+                               // at the proximal level
   std::vector<LiveFileMetaData> live_file_meta;
   db_->GetLiveFilesMetaData(&live_file_meta);
   bool found_sst_with_del = false;
   uint64_t sst_with_del_num = 0;
   for (const auto& meta : live_file_meta) {
     if (meta.num_deletions > 0) {
-      // found SST with del, which has 2 entries, one for data one for range del
+      // found SST with del, which only contains the range del
       ASSERT_EQ(meta.level,
                 kNumLevels - 2);  // output to proximal level
-      ASSERT_EQ(meta.num_entries, 2);
+      ASSERT_EQ(meta.num_entries, 1);
       ASSERT_EQ(meta.num_deletions, 1);
       found_sst_with_del = true;
       sst_with_del_num = meta.file_number;
@@ -974,7 +975,9 @@ TEST_F(TieredCompactionTest, UniversalRangeDelete) {
   ASSERT_OK(Flush());
 
   ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
-  ASSERT_EQ("0,0,0,0,0,2,8", FilesPerLevel());
+  // the partitioner cuts before and after the range del, so it has a file of
+  // its own at the proximal level
+  ASSERT_EQ("0,0,0,0,0,3,8", FilesPerLevel());
   // find the SST with range del
   std::vector<LiveFileMetaData> live_file_meta;
   db_->GetLiveFilesMetaData(&live_file_meta);
@@ -982,10 +985,10 @@ TEST_F(TieredCompactionTest, UniversalRangeDelete) {
   uint64_t sst_with_del_num = 0;
   for (const auto& meta : live_file_meta) {
     if (meta.num_deletions > 0) {
-      // found SST with del, which has 2 entries, one for data one for range del
+      // found SST with del, which only contains the range del
       ASSERT_EQ(meta.level,
                 kNumLevels - 2);  // output_to_proximal_level level
-      ASSERT_EQ(meta.num_entries, 2);
+      ASSERT_EQ(meta.num_entries, 1);
       ASSERT_EQ(meta.num_deletions, 1);
       found_sst_with_del = true;
       sst_with_del_num = meta.file_number;

db/db_bloom_filter_test.cc

@@ -928,6 +928,43 @@ INSTANTIATE_TEST_CASE_P(
             BlockBasedTableOptions::kLatestSupportedFormatVersion)));
 #endif  // !defined(ROCKSDB_VALGRIND_RUN) || defined(ROCKSDB_FULL_VALGRIND_RUN)
 
+TEST_F(DBBloomFilterTest, PartitionedPrefixFilterMultiGetKeysOutOfDomain) {
+  for (bool decouple : {false, true}) {
+    SCOPED_TRACE("decouple_partitioned_filters=" + std::to_string(decouple));
+    Options options = CurrentOptions();
+    options.prefix_extractor.reset(NewFixedPrefixTransform(4));
+    BlockBasedTableOptions bbto;
+    bbto.filter_policy.reset(NewBloomFilterPolicy(10));
+    bbto.whole_key_filtering = false;
+    bbto.partition_filters = true;
+    bbto.decouple_partitioned_filters = decouple;
+    bbto.index_type = BlockBasedTableOptions::kTwoLevelIndexSearch;
+    bbto.metadata_block_size = 1;
+    bbto.block_size = 1;
+    options.table_factory.reset(NewBlockBasedTableFactory(bbto));
+    DestroyAndReopen(options);
+
+    std::vector<std::string> keys;
+    std::vector<std::string> expected;
+    for (char c = 'a'; c <= 'j'; ++c) {
+      keys.emplace_back(1, c);
+      expected.push_back("v_" + keys.back());
+      ASSERT_OK(Put(keys.back(), expected.back()));
+    }
+    ASSERT_OK(Put("kkkk1", "v_kkkk1"));
+    ASSERT_OK(Flush());
+    keys.emplace_back("aaaa");
+    expected.emplace_back("NOT_FOUND");
+    keys.emplace_back("kkkk1");
+    expected.emplace_back("v_kkkk1");
+
+    for (size_t i = 0; i < keys.size(); ++i) {
+      ASSERT_EQ(expected[i], Get(keys[i]));
+    }
+    ASSERT_EQ(expected, MultiGet(keys));
+  }
+}
+
 TEST_F(DBBloomFilterTest, BloomFilterRate) {
   while (ChangeFilterOptions()) {
     Options options = CurrentOptions();

db/db_etc3_test.cc

@@ -5,6 +5,7 @@
 
 #include "db/db_test_util.h"
 #include "rocksdb/convenience.h"
+#include "rocksdb/experimental.h"
 #include "rocksdb/metadata.h"
 #include "rocksdb/sst_file_writer.h"
 #include "util/defer.h"
@@ -829,6 +830,301 @@ TEST_F(DBEtc3Test, AutoTuneManifestSize) {
   }
 }
 
+// Tests for the DB ownership metadata recorded for SST files
+// (FileMetaData::db_ownership_start_time and data_file_provenance)
+class DBOwnershipTest : public DBEtc3Test {
+ protected:
+  uint64_t Now() {
+    int64_t now = 0;
+    EXPECT_OK(env_->GetCurrentTime(&now));
+    return static_cast<uint64_t>(now);
+  }
+
+  // The MANIFEST record and level of the file in the default column family
+  // whose smallest user key is `key`
+  std::pair<FileMetaData, int> FileStartingWith(const std::string& key) {
+    std::vector<std::vector<FileMetaData>> metadata;
+    dbfull()->TEST_GetFilesMetaData(db_->DefaultColumnFamily(), &metadata);
+    for (size_t level = 0; level < metadata.size(); ++level) {
+      for (const auto& f : metadata[level]) {
+        if (f.smallest.user_key() == key) {
+          return {f, static_cast<int>(level)};
+        }
+      }
+    }
+    ADD_FAILURE() << "No file starting with " << key;
+    return {FileMetaData(), -1};
+  }
+
+  static DataFileProvenance Decode(const FileMetaData& f) {
+    return DataFileProvenance::DecodeFrom(f.data_file_provenance);
+  }
+
+  static DataFileProvenance OriginOnly(DataFileProvenance::Origin origin) {
+    DataFileProvenance info;
+    info.origin = origin;
+    return info;
+  }
+
+  // Expects `after` to be the same file as `before`, with the same contents
+  // and DB ownership, plus DataFileProvenance flag bits `added_flags`
+  static void ExpectSameFile(const FileMetaData& before,
+                             const FileMetaData& after, uint32_t added_flags) {
+    EXPECT_EQ(after.fd.GetNumber(), before.fd.GetNumber());
+    EXPECT_EQ(after.file_creation_time, before.file_creation_time);
+    EXPECT_EQ(after.db_ownership_start_time, before.db_ownership_start_time);
+    EXPECT_EQ(after.data_file_provenance,
+              before.data_file_provenance | added_flags);
+  }
+};
+
+TEST_F(DBOwnershipTest, FlushAndCompactionOutputs) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  env_->SetMockSleep();
+  DestroyAndReopen(options);
+
+  const uint64_t start = Now();
+  ASSERT_OK(Put("a", "1"));
+  ASSERT_OK(Flush());
+  const FileMetaData flushed = FileStartingWith("a").first;
+  EXPECT_GE(flushed.db_ownership_start_time, start);
+  EXPECT_EQ(flushed.db_ownership_start_time, flushed.file_creation_time);
+  EXPECT_EQ(Decode(flushed), OriginOnly(DataFileProvenance::Origin::kFlush));
+
+  // Reported in public metadata
+  std::vector<LiveFileMetaData> live_files;
+  db_->GetLiveFilesMetaData(&live_files);
+  ASSERT_EQ(live_files.size(), 1U);
+  EXPECT_EQ(live_files[0].db_ownership_start_time,
+            flushed.db_ownership_start_time);
+  EXPECT_EQ(live_files[0].data_file_provenance, flushed.data_file_provenance);
+  ColumnFamilyMetaData cf_meta;
+  db_->GetColumnFamilyMetaData(&cf_meta);
+  ASSERT_EQ(cf_meta.levels[0].files.size(), 1U);
+  EXPECT_EQ(cf_meta.levels[0].files[0].db_ownership_start_time,
+            flushed.db_ownership_start_time);
+  EXPECT_EQ(cf_meta.levels[0].files[0].data_file_provenance,
+            flushed.data_file_provenance);
+
+  // Flushed during WAL recovery
+  ASSERT_OK(Put("b", "1"));
+  Reopen(options);
+  const FileMetaData recovered = FileStartingWith("b").first;
+  EXPECT_GE(recovered.db_ownership_start_time, start);
+  EXPECT_EQ(recovered.db_ownership_start_time, recovered.file_creation_time);
+  EXPECT_EQ(Decode(recovered), OriginOnly(DataFileProvenance::Origin::kFlush));
+
+  // Compactions that take a long time to finish after creating their output
+  ASSERT_OK(Put("a", "2"));
+  ASSERT_OK(Flush());
+  SyncPoint::GetInstance()->SetCallBack(
+      "DBImpl::BackgroundCompaction:NonTrivial:AfterRun",
+      [&](void*) { env_->MockSleepForSeconds(1000); });
+  SyncPoint::GetInstance()->EnableProcessing();
+  CompactRangeOptions cro;
+  cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+  // The DB has held the output since creating it
+  const FileMetaData compacted = FileStartingWith("a").first;
+  EXPECT_EQ(compacted.db_ownership_start_time, compacted.file_creation_time);
+  EXPECT_LE(compacted.db_ownership_start_time + 1000, Now());
+  EXPECT_EQ(Decode(compacted),
+            OriginOnly(DataFileProvenance::Origin::kCompaction));
+}
+
+TEST_F(DBOwnershipTest, MovesPreserveOwnership) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.level_compaction_dynamic_level_bytes = true;
+  env_->SetMockSleep();
+  CompactRangeOptions force_compact;
+  force_compact.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  {
+    SCOPED_TRACE("PromoteL0");
+    DestroyAndReopen(options);
+    ASSERT_OK(Put("a", "1"));
+    ASSERT_OK(Flush());
+    const FileMetaData before = FileStartingWith("a").first;
+    env_->MockSleepForSeconds(1000);
+    ASSERT_OK(
+        experimental::PromoteL0(db_.get(), db_->DefaultColumnFamily(), 2));
+    const auto& [after, level] = FileStartingWith("a");
+    EXPECT_EQ(level, 2);
+    ExpectSameFile(before, after, kProvenanceMovedBetweenLevels);
+  }
+  {
+    SCOPED_TRACE("Trivial move");
+    DestroyAndReopen(options);
+    ASSERT_OK(Put("a", "1"));
+    ASSERT_OK(Flush());
+    const FileMetaData before = FileStartingWith("a").first;
+    env_->MockSleepForSeconds(1000);
+    std::atomic<int> trivial_moves{0};
+    SyncPoint::GetInstance()->SetCallBack(
+        "DBImpl::BackgroundCompaction:TrivialMove",
+        [&](void*) { ++trivial_moves; });
+    SyncPoint::GetInstance()->EnableProcessing();
+    ASSERT_OK(db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
+    SyncPoint::GetInstance()->DisableProcessing();
+    SyncPoint::GetInstance()->ClearAllCallBacks();
+    EXPECT_GT(trivial_moves.load(), 0);
+    const auto& [after, level] = FileStartingWith("a");
+    EXPECT_GT(level, 0);
+    ExpectSameFile(before, after, kProvenanceMovedBetweenLevels);
+  }
+  {
+    SCOPED_TRACE("CompactRange with change_level, and reopen");
+    DestroyAndReopen(options);
+    // A compaction output that has not been moved
+    ASSERT_OK(Put("a", "1"));
+    ASSERT_OK(Flush());
+    ASSERT_OK(Put("a", "2"));
+    ASSERT_OK(Flush());
+    ASSERT_OK(db_->CompactRange(force_compact, nullptr, nullptr));
+    const auto& [before, before_level] = FileStartingWith("a");
+    ASSERT_EQ(Decode(before),
+              OriginOnly(DataFileProvenance::Origin::kCompaction));
+    ASSERT_GT(before_level, 1);
+    env_->MockSleepForSeconds(1000);
+    CompactRangeOptions refit;
+    refit.change_level = true;
+    refit.target_level = 1;
+    ASSERT_OK(db_->CompactRange(refit, nullptr, nullptr));
+    const auto& [after, level] = FileStartingWith("a");
+    EXPECT_EQ(level, 1);
+    ExpectSameFile(before, after, kProvenanceMovedBetweenLevels);
+
+    // Preserved by replaying the MANIFEST and then by the new MANIFEST written
+    // on open
+    for (int i = 0; i < 2; ++i) {
+      Reopen(options);
+      ExpectSameFile(after, FileStartingWith("a").first, 0);
+    }
+  }
+  {
+    SCOPED_TRACE("Level migration on open");
+    options.level_compaction_dynamic_level_bytes = false;
+    DestroyAndReopen(options);
+    ASSERT_OK(Put("a", "1"));
+    ASSERT_OK(Flush());
+    ASSERT_OK(Put("a", "2"));
+    ASSERT_OK(Flush());
+    ASSERT_OK(db_->CompactRange(force_compact, nullptr, nullptr));
+    const auto& [before, before_level] = FileStartingWith("a");
+    ASSERT_EQ(Decode(before),
+              OriginOnly(DataFileProvenance::Origin::kCompaction));
+    ASSERT_LT(before_level, options.num_levels - 1);
+    env_->MockSleepForSeconds(1000);
+    options.level_compaction_dynamic_level_bytes = true;
+    // Required for the migration
+    options.disable_auto_compactions = false;
+    Reopen(options);
+    const auto& [after, level] = FileStartingWith("a");
+    EXPECT_EQ(level, options.num_levels - 1);
+    ExpectSameFile(before, after, kProvenanceMovedBetweenLevels);
+  }
+}
+
+TEST_F(DBOwnershipTest, FIFOTemperatureChangeCopy) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleFIFO;
+  options.num_levels = 1;
+  options.max_open_files = -1;
+  options.level0_file_num_compaction_trigger = 2;
+  CompactionOptionsFIFO fifo_options;
+  fifo_options.file_temperature_age_thresholds = {{Temperature::kCold, 1000}};
+  fifo_options.max_table_files_size = 100000000;
+  fifo_options.allow_trivial_copy_when_change_temperature = true;
+  options.compaction_options_fifo = fifo_options;
+  env_->SetMockSleep();
+  DestroyAndReopen(options);
+
+  ASSERT_OK(Put("a", "1"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(Put("b", "1"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  const FileMetaData before = FileStartingWith("a").first;
+  ASSERT_NE(before.temperature, Temperature::kCold);
+
+  env_->MockSleepForSeconds(2000);
+  ASSERT_OK(Put("c", "1"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+
+  const FileMetaData after = FileStartingWith("a").first;
+  ASSERT_EQ(after.temperature, Temperature::kCold);
+  EXPECT_NE(after.fd.GetNumber(), before.fd.GetNumber());
+  // Same contents...
+  EXPECT_EQ(after.file_creation_time, before.file_creation_time);
+  // ...in a new file for the DB to hold
+  EXPECT_GE(after.db_ownership_start_time,
+            before.db_ownership_start_time + 2000);
+  EXPECT_EQ(after.data_file_provenance,
+            before.data_file_provenance | kProvenanceTemperatureChangedByCopy);
+}
+
+TEST_F(DBOwnershipTest, TemperatureUpdatedFromFileSystem) {
+  auto test_fs = std::make_shared<FileTemperatureTestFS>(env_->GetFileSystem());
+  std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, test_fs));
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.env = env.get();
+  DestroyAndReopen(options);
+  ASSERT_OK(Put("a", "1"));
+  ASSERT_OK(Flush());
+  const FileMetaData before = FileStartingWith("a").first;
+  ASSERT_NE(before.temperature, Temperature::kCold);
+
+  test_fs->OverrideSstFileTemperature(before.fd.GetNumber(),
+                                      Temperature::kCold);
+  Close();
+  experimental::UpdateManifestForFilesStateOptions update_opts;
+  update_opts.update_temperatures = true;
+  ASSERT_OK(experimental::UpdateManifestForFilesState(
+      options, dbname_,
+      {ColumnFamilyDescriptor(kDefaultColumnFamilyName, options)},
+      update_opts));
+  Reopen(options);
+  const FileMetaData after = FileStartingWith("a").first;
+  EXPECT_EQ(after.temperature, Temperature::kCold);
+  ExpectSameFile(before, after, kProvenanceTemperatureUpdatedFromFileSystem);
+  Close();
+}
+
+TEST_F(DBOwnershipTest, Repair) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  DestroyAndReopen(options);
+  ASSERT_OK(Put("a", "1"));
+  ASSERT_OK(Flush());
+  const FileMetaData flushed = FileStartingWith("a").first;
+  // Left in the WAL, for repair to write to a table
+  ASSERT_OK(Put("b", "1"));
+  Close();
+  ASSERT_OK(RepairDB(dbname_, options));
+  Reopen(options);
+
+  // Ownership of tables from before the repair was recorded in the lost
+  // MANIFEST
+  const FileMetaData repaired = FileStartingWith("a").first;
+  // Restored from the table property, recorded when the flush finished
+  EXPECT_GE(repaired.file_creation_time, flushed.file_creation_time);
+  EXPECT_EQ(repaired.db_ownership_start_time, kUnknownDbOwnershipStartTime);
+  DataFileProvenance expected;
+  expected.recorded_by_repair = true;
+  EXPECT_EQ(Decode(repaired), expected);
+
+  const FileMetaData from_wal = FileStartingWith("b").first;
+  EXPECT_GT(from_wal.file_creation_time, 0U);
+  EXPECT_EQ(from_wal.db_ownership_start_time, from_wal.file_creation_time);
+  expected.origin = DataFileProvenance::Origin::kFlush;
+  EXPECT_EQ(Decode(from_wal), expected);
+}
+
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/db_impl/db_impl.cc

@@ -354,6 +354,47 @@ Status DBImpl::Resume() {
   return s;
 }
 
+void DBImpl::SyncLastSequenceWithAllocatedAtRecoveryStart() {
+  mutex_.AssertHeld();
+  assert(two_write_queues_);
+  for (;;) {
+    WriteThread::Writer w;
+    const bool entered = write_thread_.EnterUnbatchedNonBlocking(&w, &mutex_);
+    const uint64_t stall =
+        entered ? 0 : write_thread_.GetBegunCountOfOutstandingStall();
+    WriteThread::Writer nonmem_w;
+    const bool nonmem_entered =
+        nonmem_write_thread_.EnterUnbatchedNonBlocking(&nonmem_w, &mutex_);
+    const uint64_t nonmem_stall =
+        nonmem_entered ? 0
+                       : nonmem_write_thread_.GetBegunCountOfOutstandingStall();
+    WaitForPendingWrites();
+
+    // A queue that was not entered must still be in the same stall, so that
+    // no writer in it can have allocated since.
+    const bool no_writer_in_flight =
+        (entered ||
+         (stall != 0 &&
+          write_thread_.GetBegunCountOfOutstandingStall() == stall)) &&
+        (nonmem_entered ||
+         (nonmem_stall != 0 &&
+          nonmem_write_thread_.GetBegunCountOfOutstandingStall() ==
+              nonmem_stall));
+    if (no_writer_in_flight) {
+      versions_->SyncLastSequenceWithAllocated();
+    }
+    if (nonmem_entered) {
+      nonmem_write_thread_.ExitUnbatched(&nonmem_w);
+    }
+    if (entered) {
+      write_thread_.ExitUnbatched(&w);
+    }
+    if (no_writer_in_flight) {
+      return;
+    }
+  }
+}
+
 // This function implements both manual and automatic background-error
 // recovery. It discards stale queued flush requests, waits for scheduled work
 // to exit, rebuilds and runs the flush work required by the recovery context,
@@ -425,16 +466,7 @@ Status DBImpl::ResumeImpl(DBRecoverContext context,
   // memtable-switch fence, after FlushAllColumnFamilies() has dropped and
   // re-acquired the DB mutex.
   if (two_write_queues_) {
-    WriteThread::Writer w;
-    write_thread_.EnterUnbatched(&w, &mutex_);
-    WriteThread::Writer nonmem_w;
-    nonmem_write_thread_.EnterUnbatched(&nonmem_w, &mutex_);
-    WaitForPendingWrites();
-
-    versions_->SyncLastSequenceWithAllocated();
-
-    nonmem_write_thread_.ExitUnbatched(&nonmem_w);
-    write_thread_.ExitUnbatched(&w);
+    SyncLastSequenceWithAllocatedAtRecoveryStart();
   }
 
   TEST_SYNC_POINT("DBImpl::ResumeImpl:AfterSyncSeq");
@@ -8037,6 +8069,7 @@ class FileIngestionHandleImpl : public FileIngestionHandle {
   FileIngestionHandleImpl& operator=(FileIngestionHandleImpl&&) = delete;
 
   Status Abort() override;
+  std::vector<SstFileMetaData> GetPreparedFiles() const override;
 
   DBImpl* const db_;
   const uint64_t first_file_number_;
@@ -8083,6 +8116,44 @@ Status FileIngestionHandleImpl::Abort() {
   return Status::OK();
 }
 
+std::vector<SstFileMetaData> FileIngestionHandleImpl::GetPreparedFiles() const {
+  std::vector<SstFileMetaData> files;
+  if (consumed_) {
+    return files;
+  }
+  for (const auto& job : jobs_) {
+    const auto& cf_paths = job->GetColumnFamilyData()->ioptions().cf_paths;
+    // Without allow_db_generated_files, commit assigns each file its sequence
+    // numbers, so they are unknown here.
+    const bool seqnos_known =
+        job->GetIngestionOptions().allow_db_generated_files;
+    for (const auto& file : job->files_to_ingest()) {
+      SstFileMetaData meta;
+      meta.directory = file.fd.GetPathId() < cf_paths.size()
+                           ? cf_paths[file.fd.GetPathId()].path
+                           : cf_paths.back().path;
+      meta.db_path = meta.directory;
+      meta.name = MakeTableFileName("", file.fd.GetNumber());
+      meta.relative_filename = meta.name.substr(1);
+      if (seqnos_known) {
+        meta.smallest_seqno = file.smallest_seqno;
+        meta.largest_seqno = file.largest_seqno;
+      }
+      meta.file_number = file.fd.GetNumber();
+      meta.size = file.fd.GetFileSize();
+      meta.temperature = file.file_temperature;
+      meta.file_checksum = file.file_checksum;
+      meta.file_checksum_func_name = file.file_checksum_func_name;
+      meta.smallestkey = file.smallest_internal_key.user_key().ToString();
+      meta.largestkey = file.largest_internal_key.user_key().ToString();
+      meta.num_entries = file.num_entries;
+      meta.num_deletions = file.table_properties.num_deletions;
+      files.push_back(std::move(meta));
+    }
+  }
+  return files;
+}
+
 Status DBImpl::PrepareFileIngestion(
     const std::vector<IngestExternalFileArg>& args,
     std::unique_ptr<FileIngestionHandle>* handle) {

db/db_impl/db_impl.h

@@ -2900,6 +2900,12 @@ class DBImpl : public DB
   // fence that is about to switch memtables.
   void MaybeSyncLastSequenceWithAllocatedForRecovery(FlushReason flush_reason);
 
+  // Like the memtable-switch fence, but does not wait for a write queue whose
+  // leader is in a write stall: that leader has not allocated sequence numbers
+  // yet, and it can be waiting for this recovery.
+  // REQUIRES: mutex locked, two_write_queues
+  void SyncLastSequenceWithAllocatedAtRecoveryStart();
+
   Status RetryFlushesForErrorRecovery(FlushReason flush_reason, bool wait);
 
   // Wait until flushing this column family won't stall writes

db/db_impl/db_impl_compaction_flush.cc

@@ -62,6 +62,11 @@ void RecordAtomicFlushRequestReason(Statistics* stats,
   }
 }
 
+Status TableBuilderNotSupported(const ColumnFamilyData& cfd) {
+  return Status::NotSupported(
+      "Column family's table factory cannot build tables", cfd.GetName());
+}
+
 std::vector<uint64_t> MemTableIds(const autovector<ReadOnlyMemTable*>& mems) {
   std::vector<uint64_t> ids;
   ids.reserve(mems.size());
@@ -158,7 +163,7 @@ bool DBImpl::ShouldRescheduleFlushRequestToRetainUDT(
           cfd->GetUnflushedMemTableCountForWriteStallCheck(),
           /*num_l0_files=*/0,
           /*num_compaction_needed_bytes=*/0, mutable_cf_options,
-          cfd->ioptions())
+          cfd->ioptions(), cfd->is_table_builder_supported())
           .first;
   if (write_stall != WriteStallCondition::kNormal) {
     return false;
@@ -1193,6 +1198,10 @@ Status DBImpl::CompactRange(const CompactRangeOptions& options,
       static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
   assert(cfd);
 
+  if (!cfd->is_table_builder_supported()) {
+    return TableBuilderNotSupported(*cfd);
+  }
+
   if (manual_compaction_paused_.load(std::memory_order_acquire) > 0) {
     return Status::Incomplete(Status::SubCode::kManualCompactionPaused);
   }
@@ -1619,6 +1628,10 @@ Status DBImpl::CompactFiles(const CompactionOptions& compact_options,
       static_cast_with_check<ColumnFamilyHandleImpl>(column_family)->cfd();
   assert(cfd);
 
+  if (!cfd->is_table_builder_supported()) {
+    return TableBuilderNotSupported(*cfd);
+  }
+
   {
     InstrumentedMutexLock l(&mutex_);
     BeginInFlightCompaction(cfd);
@@ -1712,16 +1725,8 @@ void DBImpl::PrepareTrivialMoveEdit(Compaction& c, LogBuffer* log_buffer,
     for (size_t i = 0; i < c.num_input_files(l); i++) {
       FileMetaData* f = c.input(l, i);
       c.edit()->DeleteFile(c.level(l), f->fd.GetNumber());
-      c.edit()->AddFile(
-          c.output_level(), f->fd.GetNumber(), f->fd.GetPathId(),
-          f->fd.GetFileSize(), f->smallest, f->largest, f->fd.smallest_seqno,
-          f->fd.largest_seqno, f->marked_for_compaction, f->temperature,
-          f->oldest_blob_file_number, f->oldest_ancestor_time,
-          f->file_creation_time, f->epoch_number, f->file_checksum,
-          f->file_checksum_func_name, f->unique_id,
-          f->compensated_range_deletion_size, f->tail_size,
-          f->user_defined_timestamps_persisted, f->min_timestamp,
-          f->max_timestamp, f->file_open_metadata);
+      c.edit()->AddExistingFile(c.output_level(), *f,
+                                kProvenanceMovedBetweenLevels);
       moved_bytes += static_cast<size_t>(c.input(l, i)->fd.GetFileSize());
       ROCKS_LOG_BUFFER(
           log_buffer, "[%s] Moved #%" PRIu64 " to level-%d %" PRIu64 " bytes\n",
@@ -1929,7 +1934,7 @@ Status DBImpl::CompactFilesImpl(
   ConfigureCompactionExecutors(&compaction_job, c->column_family_data(),
                                kManualCompactionCanceledFalse_, false,
                                separate_remote_pool);
-  c->SetUseSeparateRemoteCompactionPool(separate_remote_pool);
+  c->SetSeparateRemoteCompactionPoolSelected(separate_remote_pool);
 
   // Creating a compaction influences the compaction score because the score
   // takes running compactions into account (by skipping files that are already
@@ -2302,15 +2307,7 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
 
     for (const auto& f : vstorage->LevelFiles(level)) {
       edit.DeleteFile(level, f->fd.GetNumber());
-      edit.AddFile(
-          to_level, f->fd.GetNumber(), f->fd.GetPathId(), f->fd.GetFileSize(),
-          f->smallest, f->largest, f->fd.smallest_seqno, f->fd.largest_seqno,
-          f->marked_for_compaction, f->temperature, f->oldest_blob_file_number,
-          f->oldest_ancestor_time, f->file_creation_time, f->epoch_number,
-          f->file_checksum, f->file_checksum_func_name, f->unique_id,
-          f->compensated_range_deletion_size, f->tail_size,
-          f->user_defined_timestamps_persisted, f->min_timestamp,
-          f->max_timestamp, f->file_open_metadata);
+      edit.AddExistingFile(to_level, *f, kProvenanceMovedBetweenLevels);
     }
     ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
                     "[%s] Apply version edit:\n%s", cfd->GetName().c_str(),
@@ -3362,7 +3359,7 @@ Status DBImpl::WaitUntilFlushWouldNotStallWrites(ColumnFamilyData* cfd,
               cfd->GetUnflushedMemTableCountForWriteStallCheck(),
               vstorage->l0_delay_trigger_count() + 1,
               vstorage->estimated_compaction_needed_bytes(), mutable_cf_options,
-              cfd->ioptions())
+              cfd->ioptions(), cfd->is_table_builder_supported())
               .first;
     } while (write_stall_condition != WriteStallCondition::kNormal);
   }
@@ -4062,7 +4059,9 @@ IOStatus DBImpl::CopyTrivialCompactionFiles(
                        c->column_family_data()->GetName().c_str(),
                        get_time_status.ToString().c_str());
     }
-    uint64_t out_file_creation_time = static_cast<uint64_t>(tmp_current_time);
+    // The copy is a new file for this DB to hold, but has the same contents
+    // (and file_creation_time) as the input
+    uint64_t copy_time = static_cast<uint64_t>(tmp_current_time);
 
     FileOptions copied_file_options = file_options_;
     copied_file_options.temperature = c->GetOutputTemperature();
@@ -4183,7 +4182,7 @@ IOStatus DBImpl::CopyTrivialCompactionFiles(
                                    c->GetOutputTemperature() /* temperature */,
                                    in_file->oldest_blob_file_number,
                                    in_file->oldest_ancestor_time,
-                                   out_file_creation_time,
+                                   in_file->file_creation_time,
                                    c->MinInputFileEpochNumber(),
                                    dest_writer->GetFileChecksum(),
                                    dest_writer->GetFileChecksumFuncName(),
@@ -4193,6 +4192,9 @@ IOStatus DBImpl::CopyTrivialCompactionFiles(
                                    in_file->user_defined_timestamps_persisted,
                                    in_file->min_timestamp,
                                    in_file->max_timestamp};
+    out_file_metadata.db_ownership_start_time = copy_time;
+    out_file_metadata.data_file_provenance =
+        in_file->data_file_provenance | kProvenanceTemperatureChangedByCopy;
 
     if (out_table_properties) {
       c->SetOutputTableProperties(out_fname, out_table_properties);
@@ -5558,7 +5560,7 @@ Status DBImpl::BackgroundCompaction(
   IOStatus io_s;
   bool compaction_released = false;
   if (c) {
-    c->SetUseSeparateRemoteCompactionPool(separate_remote_pool);
+    c->SetSeparateRemoteCompactionPoolSelected(separate_remote_pool);
   }
   if (!c) {
     // Nothing to do
@@ -6206,8 +6208,8 @@ void DBImpl::BuildCompactionJobInfo(
   compaction_job_info->aborted = st.IsCompactionAborted();
   compaction_job_info->thread_id = env_->GetThreadID();
   compaction_job_info->job_id = job_id;
-  compaction_job_info->use_separate_remote_compaction_pool =
-      c->use_separate_remote_compaction_pool();
+  compaction_job_info->separate_remote_compaction_pool_selected =
+      c->separate_remote_compaction_pool_selected();
   compaction_job_info->base_input_level = c->start_level();
   compaction_job_info->output_level = c->output_level();
   compaction_job_info->stats = compaction_job_stats;

db/db_impl/db_impl_experimental.cc

@@ -137,16 +137,7 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
     edit.SetColumnFamily(cfd->GetID());
     for (const auto& f : l0_files) {
       edit.DeleteFile(0, f->fd.GetNumber());
-      edit.AddFile(target_level, f->fd.GetNumber(), f->fd.GetPathId(),
-                   f->fd.GetFileSize(), f->smallest, f->largest,
-                   f->fd.smallest_seqno, f->fd.largest_seqno,
-                   f->marked_for_compaction, f->temperature,
-                   f->oldest_blob_file_number, f->oldest_ancestor_time,
-                   f->file_creation_time, f->epoch_number, f->file_checksum,
-                   f->file_checksum_func_name, f->unique_id,
-                   f->compensated_range_deletion_size, f->tail_size,
-                   f->user_defined_timestamps_persisted, f->min_timestamp,
-                   f->max_timestamp, f->file_open_metadata);
+      edit.AddExistingFile(target_level, *f, kProvenanceMovedBetweenLevels);
     }
 
     status = versions_->LogAndApply(cfd, read_options, write_options, &edit,

db/db_impl/db_impl_open.cc

@@ -679,19 +679,9 @@ Status DBImpl::Recover(
             edit.SetColumnFamily(cfd->GetID());
             for (const FileMetaData* f : level_files) {
               edit.DeleteFile(from_level, f->fd.GetNumber());
-              edit.AddFile(to_level, f->fd.GetNumber(), f->fd.GetPathId(),
-                           f->fd.GetFileSize(), f->smallest, f->largest,
-                           f->fd.smallest_seqno, f->fd.largest_seqno,
-                           f->marked_for_compaction,
-                           f->temperature,  // this can be different from
-                           // `last_level_temperature`
-                           f->oldest_blob_file_number, f->oldest_ancestor_time,
-                           f->file_creation_time, f->epoch_number,
-                           f->file_checksum, f->file_checksum_func_name,
-                           f->unique_id, f->compensated_range_deletion_size,
-                           f->tail_size, f->user_defined_timestamps_persisted,
-                           f->min_timestamp, f->max_timestamp,
-                           f->file_open_metadata);
+              // Keeps f->temperature, which can be different from
+              // `last_level_temperature`
+              edit.AddExistingFile(to_level, *f, kProvenanceMovedBetweenLevels);
               ROCKS_LOG_WARN(immutable_db_options_.info_log,
                              "[%s] Moving #%" PRIu64
                              " from from_level-%d to from_level-%d %" PRIu64
@@ -2186,6 +2176,10 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
         .PermitUncheckedError();  // ignore error
     const uint64_t current_time = static_cast<uint64_t>(_current_time);
     meta.oldest_ancestor_time = current_time;
+    meta.file_creation_time = current_time;
+    meta.db_ownership_start_time = current_time;
+    meta.data_file_provenance =
+        DataFileProvenanceForOrigin(DataFileProvenance::Origin::kFlush);
     meta.epoch_number = cfd->NewEpochNumber();
     {
       auto write_hint = cfd->current()->storage_info()->CalculateSSTWriteHint(
@@ -2229,9 +2223,9 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
           mutable_cf_options_copy.compression_opts, cfd->GetID(),
           cfd->GetName(), 0 /* level */, current_time /* newest_key_time */,
           false /* is_bottommost */, TableFileCreationReason::kRecovery,
-          0 /* oldest_key_time */, 0 /* file_creation_time */, db_id_,
-          db_session_id_, 0 /* target_file_size */, meta.fd.GetNumber(),
-          kMaxSequenceNumber, dbname_ /* db_name */);
+          0 /* oldest_key_time */, db_id_, db_session_id_,
+          0 /* target_file_size */, meta.fd.GetNumber(), kMaxSequenceNumber,
+          dbname_ /* db_name */);
       Version* version = cfd->current();
       version->Ref();
       TableProperties temp_table_proerties;
@@ -2316,15 +2310,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
   }
 
   if (s.ok() && has_output) {
-    edit->AddFile(level, meta.fd.GetNumber(), meta.fd.GetPathId(),
-                  meta.fd.GetFileSize(), meta.smallest, meta.largest,
-                  meta.fd.smallest_seqno, meta.fd.largest_seqno,
-                  meta.marked_for_compaction, meta.temperature,
-                  meta.oldest_blob_file_number, meta.oldest_ancestor_time,
-                  meta.file_creation_time, meta.epoch_number,
-                  meta.file_checksum, meta.file_checksum_func_name,
-                  meta.unique_id, meta.compensated_range_deletion_size,
-                  meta.tail_size, meta.user_defined_timestamps_persisted);
+    edit->AddFile(level, meta);
 
     for (const auto& blob : blob_file_additions) {
       edit->AddBlobFile(blob);

db/db_impl/db_impl_write.cc

@@ -2945,9 +2945,12 @@ Status DBImpl::DelayWrite(uint64_t num_bytes, WriteThread& write_thread,
     }
 
     // Don't wait if there's a background error that is not pending recovery
-    // since recovery might never be attempted.
+    // since recovery might never be attempted. Don't wait once the DB is
+    // stopped either: its recovery flush needs the write queue this writer
+    // holds.
     while ((error_handler_.GetBGError().ok() ||
-            error_handler_.IsRecoveryInProgress()) &&
+            (error_handler_.IsRecoveryInProgress() &&
+             !error_handler_.IsDBStopped())) &&
            write_controller_.IsStopped() &&
            !shutting_down_.load(std::memory_order_relaxed)) {
       if (write_options.no_slowdown) {

db/db_range_del_test.cc

@@ -1817,6 +1817,148 @@ TEST_F(DBRangeDelTest, CompactionEmitRangeTombstoneToSSTPartitioner) {
   ASSERT_EQ(2, NumTableFilesAtLevel(1));
 }
 
+namespace {
+// Records the compaction output files that were started, and how many of them
+// were empty and not kept.
+class CompactionOutputListener : public EventListener {
+ public:
+  void OnTableFileCreationStarted(
+      const TableFileCreationBriefInfo& info) override {
+    if (info.reason == TableFileCreationReason::kCompaction) {
+      std::lock_guard<std::mutex> lock(mutex_);
+      started_paths_.push_back(info.file_path);
+    }
+  }
+
+  void OnTableFileCreated(const TableFileCreationInfo& info) override {
+    if (info.reason == TableFileCreationReason::kCompaction &&
+        info.status.IsAborted()) {
+      std::lock_guard<std::mutex> lock(mutex_);
+      ++num_not_kept_;
+    }
+  }
+
+  std::vector<std::string> TakeStartedPaths() {
+    std::lock_guard<std::mutex> lock(mutex_);
+    return std::exchange(started_paths_, {});
+  }
+
+  int TakeNumNotKept() {
+    std::lock_guard<std::mutex> lock(mutex_);
+    return std::exchange(num_not_kept_, 0);
+  }
+
+ private:
+  std::mutex mutex_;
+  std::vector<std::string> started_paths_;
+  int num_not_kept_ = 0;
+};
+}  // namespace
+
+TEST_F(DBRangeDelTest, BottommostCompactionEmitRangeTombstoneToSSTPartitioner) {
+  // A range tombstone after the last point key must reach the SSTPartitioner
+  // at the bottommost level too. Otherwise the tombstone is appended to the
+  // file holding the point key and extends that file across the partition
+  // boundary, so no file starting at the boundary fits in the level.
+  // The fixed-prefix factory exercises the partitioner's prefix fast path,
+  // and SingleKeySstPartitionerFactory its per-key ShouldPartition() path.
+  // Each Key(i) is 9 bytes, so a 9-byte prefix puts every key in its own
+  // partition.
+  for (const auto& factory :
+       std::vector<std::shared_ptr<SstPartitionerFactory>>{
+           NewSstPartitionerFixedPrefixFactory(9),
+           std::make_shared<SingleKeySstPartitionerFactory>()}) {
+    SCOPED_TRACE(factory->Name());
+    Options options = CurrentOptions();
+    options.sst_partitioner_factory = factory;
+    options.disable_auto_compactions = true;
+    options.level_compaction_dynamic_level_bytes = true;
+    auto listener = std::make_shared<CompactionOutputListener>();
+    options.listeners.push_back(listener);
+    DestroyAndReopen(options);
+    const int bottommost_level = options.num_levels - 1;
+
+    ASSERT_OK(Put(Key(1), "val"));
+    // Keeps the tombstone from being dropped at the bottommost level
+    const Snapshot* snapshot = db_->GetSnapshot();
+    ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(),
+                               Key(2), Key(5)));
+    ASSERT_OK(Flush());
+
+    CompactRangeOptions cro;
+    cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+    ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+
+    ColumnFamilyMetaData cf_meta;
+    db_->GetColumnFamilyMetaData(&cf_meta);
+    const std::vector<SstFileMetaData>& files =
+        cf_meta.levels[bottommost_level].files;
+    ASSERT_EQ(2, files.size());
+    ASSERT_EQ(Key(1), files[0].largestkey);
+    ASSERT_EQ(Key(2), files[1].smallestkey);
+    ASSERT_EQ(0, listener->TakeNumNotKept());
+    listener->TakeStartedPaths();
+
+    // Once nothing pins it, the tombstone is dropped. The file cut for it is
+    // still created, but it is empty, so the compaction deletes it before
+    // installing its outputs: it is never part of the LSM.
+    db_->ReleaseSnapshot(snapshot);
+    ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+    ASSERT_EQ(1, NumTableFilesAtLevel(bottommost_level));
+    ASSERT_EQ(1, listener->TakeNumNotKept());
+    const std::vector<std::string> started = listener->TakeStartedPaths();
+    ASSERT_EQ(2, started.size());
+    int num_on_disk = 0;
+    for (const auto& path : started) {
+      const Status exists = env_->FileExists(path);
+      if (exists.ok()) {
+        ++num_on_disk;
+      } else {
+        ASSERT_TRUE(exists.IsNotFound()) << exists.ToString();
+      }
+    }
+    ASSERT_EQ(1, num_on_disk);
+  }
+}
+
+TEST_F(DBRangeDelTest, BottommostRangeTombstoneAtSameUserKeyAsPointKey) {
+  // At the bottommost level a point key's seqno can be zeroed, so it sorts
+  // after a range tombstone that starts at the same user key. The partitioner
+  // must not cut between them, even one that requires a cut for every key.
+  Options options = CurrentOptions();
+  options.sst_partitioner_factory =
+      std::make_shared<SingleKeySstPartitionerFactory>();
+  options.disable_auto_compactions = true;
+  options.level_compaction_dynamic_level_bytes = true;
+  CompactRangeOptions cro;
+  cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+
+  // The tombstone is dropped.
+  DestroyAndReopen(options);
+  ASSERT_OK(Put(Key(1), "val"));
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(2),
+                             Key(5)));
+  ASSERT_OK(Put(Key(2), "val"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+  ASSERT_EQ("val", Get(Key(2)));
+
+  // A newer tombstone covering the same user key is kept by a snapshot, so it
+  // is written to every output file that overlaps it.
+  DestroyAndReopen(options);
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(2),
+                             Key(5)));
+  ASSERT_OK(Put(Key(2), "val"));
+  ASSERT_OK(Flush());
+  const Snapshot* snapshot = db_->GetSnapshot();
+  ASSERT_OK(db_->DeleteRange(WriteOptions(), db_->DefaultColumnFamily(), Key(1),
+                             Key(4)));
+  ASSERT_OK(db_->CompactRange(cro, nullptr, nullptr));
+  ASSERT_EQ("NOT_FOUND", Get(Key(2)));
+  ASSERT_EQ("val", Get(Key(2), snapshot));
+  db_->ReleaseSnapshot(snapshot);
+}
+
 TEST_F(DBRangeDelTest, OversizeCompactionGapBetweenPointKeyAndTombstone) {
   // L2 has 2 files
   // L2_0: 0, 1, 2, 3, 4

db/db_table_properties_test.cc

@@ -996,6 +996,70 @@ TEST_F(DBTablePropertiesTest, LsmInfoNotApplicableForSstFileWriter) {
   EXPECT_GT(table_props->uncompressed_data_size, 0U);
 }
 
+TEST_F(DBTablePropertiesTest, FileCreationTimeAlwaysSet) {
+  // file_creation_time is recorded in both table properties and the MANIFEST
+  // for SstFileWriter files, WAL recovery flushes, and after DB repair.
+  Options options = CurrentOptions();
+  DestroyAndReopen(options);
+  int64_t start_time = 0;
+  ASSERT_OK(env_->GetCurrentTime(&start_time));
+
+  std::string file = dbname_ + "/for_ingestion.sst";
+  {
+    // With a column family, so that repair keeps the file
+    SstFileWriter writer(EnvOptions(), options, db_->DefaultColumnFamily());
+    ASSERT_OK(writer.Open(file));
+    ASSERT_OK(writer.Put("a", "1"));
+    ASSERT_OK(writer.Finish());
+  }
+  IngestExternalFileOptions ifo;
+  ifo.move_files = true;
+  ASSERT_OK(db_->IngestExternalFile({file}, ifo));
+
+  // Left in the WAL to be flushed during recovery
+  ASSERT_OK(Put("b", "2"));
+  Reopen(options);
+
+  auto verify = [&]() {
+    int64_t end_time = 0;
+    ASSERT_OK(env_->GetCurrentTime(&end_time));
+    const uint64_t lo = static_cast<uint64_t>(start_time);
+    const uint64_t hi = static_cast<uint64_t>(end_time);
+
+    TablePropertiesCollection props;
+    ASSERT_OK(db_->GetPropertiesOfAllTables(&props));
+    ASSERT_EQ(props.size(), 2U);
+    for (const auto& p : props) {
+      EXPECT_GE(p.second->file_creation_time, lo) << p.first;
+      EXPECT_LE(p.second->file_creation_time, hi) << p.first;
+    }
+
+    std::vector<std::vector<FileMetaData>> metadata;
+    dbfull()->TEST_GetFilesMetaData(db_->DefaultColumnFamily(), &metadata);
+    size_t num_files = 0;
+    for (const auto& level : metadata) {
+      for (const auto& f : level) {
+        ++num_files;
+        EXPECT_GE(f.file_creation_time, lo) << f.fd.GetNumber();
+        EXPECT_LE(f.file_creation_time, hi) << f.fd.GetNumber();
+      }
+    }
+    ASSERT_EQ(num_files, 2U);
+  };
+  {
+    SCOPED_TRACE("before repair");
+    verify();
+  }
+
+  Close();
+  ASSERT_OK(RepairDB(dbname_, options));
+  Reopen(options);
+  {
+    SCOPED_TRACE("after repair");
+    verify();
+  }
+}
+
 TEST_F(DBTablePropertiesTest, CompressionManagerFullDetailsPersisted) {
   // With an explicit CompressionManager, compression_options records the
   // manager's full details via ToString() (id plus any configured options),

db/error_handler_fs_test.cc

@@ -413,6 +413,113 @@ TEST_F(DBErrorHandlingFSTest, FlushWriteFileScopeError) {
   }
 }
 
+// A retryable SST write error during a flush is a soft error that recovery
+// fixes by retrying the flush. A writer that waits for that flush must finish
+// once recovery is done.
+TEST_F(DBErrorHandlingFSTest, RetryableFlushErrorWithWriterStalledOnFlush) {
+  for (bool two_write_queues : {false, true}) {
+    SCOPED_TRACE("two_write_queues=" + std::to_string(two_write_queues));
+    Options options = GetDefaultOptions();
+    options.env = fault_env_.get();
+    options.create_if_missing = true;
+    options.two_write_queues = two_write_queues;
+    options.write_buffer_size = 64 << 10;
+    options.max_write_buffer_number = 2;
+    options.min_write_buffer_number_to_merge = 1;
+    options.bgerror_resume_retry_interval = 100000;
+    DestroyAndReopen(options);
+
+    IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
+    error_msg.SetRetryable(true);
+    std::atomic<bool> injected{false};
+    // The first flush fails only after the writer stops on its memtables.
+    SyncPoint::GetInstance()->LoadDependency(
+        {{"DBImpl::DelayWrite:Wait", "BuildTable:BeforeFinishBuildTable"}});
+    SyncPoint::GetInstance()->SetCallBack(
+        "BuildTable:BeforeFinishBuildTable", [&](void*) {
+          if (!injected.exchange(true)) {
+            fault_fs_->SetFilesystemActive(false, error_msg);
+          }
+        });
+    SyncPoint::GetInstance()->SetCallBack(
+        "RecoverFromRetryableBGIOError:BeforeStart",
+        [&](void*) { fault_fs_->SetFilesystemActive(true); });
+    SyncPoint::GetInstance()->EnableProcessing();
+
+    const std::string value(40 << 10, 'v');
+    for (int i = 0; i < 8; ++i) {
+      ASSERT_OK(Put(Key(i), value));
+    }
+    ASSERT_TRUE(injected);
+    SyncPoint::GetInstance()->DisableProcessing();
+    SyncPoint::GetInstance()->ClearAllCallBacks();
+    SyncPoint::GetInstance()->LoadDependency({});
+    for (int i = 0; i < 8; ++i) {
+      ASSERT_EQ(value, Get(Key(i)));
+    }
+  }
+}
+
+// A retryable error syncing closed WALs during a flush is a hard error. A
+// writer that waits for that flush gets the error, and recovery then flushes
+// and resumes writes.
+TEST_F(DBErrorHandlingFSTest, FlushWALSyncErrorWithWriterStalledOnFlush) {
+  Options options = GetDefaultOptions();
+  options.env = fault_env_.get();
+  options.create_if_missing = true;
+  options.write_buffer_size = 64 << 10;
+  options.max_write_buffer_number = 2;
+  options.min_write_buffer_number_to_merge = 1;
+  options.bgerror_resume_retry_interval = 100000;
+  // With two column families, a flush syncs the closed WALs.
+  CreateAndReopenWithCF({"pikachu"}, options);
+
+  IOStatus error_msg = IOStatus::IOError("Retryable IO Error");
+  error_msg.SetRetryable(true);
+  std::atomic<bool> injected{false};
+  // The first flush fails only after the writer stops on its memtables.
+  SyncPoint::GetInstance()->LoadDependency(
+      {{"DBImpl::DelayWrite:Wait", "DBImpl::SyncClosedWals:Start"},
+       {"RecoverFromRetryableBGIOError:RecoverSuccess",
+        "DBErrorHandlingFSTest::FlushWALSyncErrorWithWriterStalledOnFlush:"
+        "Recovered"}});
+  SyncPoint::GetInstance()->SetCallBack(
+      "DBImpl::SyncClosedWals:Start", [&](void*) {
+        if (!injected.exchange(true)) {
+          fault_fs_->SetFilesystemActive(false, error_msg);
+        }
+      });
+  SyncPoint::GetInstance()->SetCallBack(
+      "RecoverFromRetryableBGIOError:BeforeStart",
+      [&](void*) { fault_fs_->SetFilesystemActive(true); });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  const std::string value(40 << 10, 'v');
+  Status s;
+  int num_written = 0;
+  while (num_written < 8) {
+    s = Put(Key(num_written), value);
+    if (!s.ok()) {
+      break;
+    }
+    ++num_written;
+  }
+  ASSERT_TRUE(injected);
+  ASSERT_TRUE(s.IsIOError()) << s.ToString();
+  ASSERT_EQ(s.severity(), Status::Severity::kHardError);
+
+  TEST_SYNC_POINT(
+      "DBErrorHandlingFSTest::FlushWALSyncErrorWithWriterStalledOnFlush:"
+      "Recovered");
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+  SyncPoint::GetInstance()->LoadDependency({});
+  ASSERT_OK(Put(Key(num_written), value));
+  for (int i = 0; i <= num_written; ++i) {
+    ASSERT_EQ(value, Get(Key(i)));
+  }
+}
+
 TEST_F(DBErrorHandlingFSTest, FlushWALWriteRetryableError) {
   std::shared_ptr<ErrorHandlerFSListener> listener =
       std::make_shared<ErrorHandlerFSListener>();

db/experimental.cc

@@ -158,17 +158,10 @@ Status UpdateManifestForFilesState(
               // Current state inconsistent with manifest
               ++files_updated;
               edit.DeleteFile(level, number);
-              edit.AddFile(
-                  level, lf->fd.GetNumber(), lf->fd.GetPathId(),
-                  lf->fd.GetFileSize(), lf->smallest, lf->largest,
-                  lf->fd.smallest_seqno, lf->fd.largest_seqno,
-                  lf->marked_for_compaction, temp, lf->oldest_blob_file_number,
-                  lf->oldest_ancestor_time, lf->file_creation_time,
-                  lf->epoch_number, lf->file_checksum,
-                  lf->file_checksum_func_name, lf->unique_id,
-                  lf->compensated_range_deletion_size, lf->tail_size,
-                  lf->user_defined_timestamps_persisted, lf->min_timestamp,
-                  lf->max_timestamp, lf->file_open_metadata);
+              FileMetaData updated = *lf;
+              updated.temperature = temp;
+              edit.AddExistingFile(level, updated,
+                                   kProvenanceTemperatureUpdatedFromFileSystem);
             }
           }
         } else {

db/external_sst_file_basic_test.cc

@@ -15,6 +15,7 @@
 #include "rocksdb/options.h"
 #include "rocksdb/perf_context.h"
 #include "rocksdb/sst_file_writer.h"
+#include "test_util/mock_time_env.h"
 #include "test_util/testharness.h"
 #include "test_util/testutil.h"
 #include "util/defer.h"
@@ -2614,7 +2615,14 @@ TEST_F(ExternalSSTFileBasicTest,
       SyncPoint::GetInstance()->ClearAllCallBacks();
     });
 
-    SstFileWriter sst_file_writer(EnvOptions(), options);
+    // Files written while the clock reads 0 record no creation time, like
+    // SstFileWriter files from older versions
+    auto zero_clock =
+        std::make_shared<MockSystemClock>(counting_env.GetSystemClock());
+    CompositeEnvWrapper legacy_writer_env(&counting_env, zero_clock);
+    Options legacy_writer_options = options;
+    legacy_writer_options.env = &legacy_writer_env;
+    SstFileWriter sst_file_writer(EnvOptions(), legacy_writer_options);
     std::vector<std::string> old_files;
     for (const auto& contents :
          std::vector<std::vector<std::pair<std::string, std::string>>>{

db/external_sst_file_ingestion_job.cc

@@ -25,10 +25,97 @@
 #include "table/table_builder.h"
 #include "table/unique_id_impl.h"
 #include "test_util/sync_point.h"
+#include "util/cast_util.h"
 #include "util/udt_util.h"
 
 namespace ROCKSDB_NAMESPACE {
 
+namespace {
+uint64_t ResolveExternalFileTime(const std::string& file_path,
+                                 const char* time_name, uint64_t carried_time,
+                                 uint64_t now, Logger* info_log) {
+  if (carried_time == 0) {
+    return now;
+  }
+  if (now != 0 && carried_time > now) {
+    ROCKS_LOG_WARN(info_log,
+                   "External file %s has %s %" PRIu64
+                   " later than current time %" PRIu64 "; using current time",
+                   file_path.c_str(), time_name, carried_time, now);
+    return now;
+  }
+  return carried_time;
+}
+}  // namespace
+
+void ResolveExternalFileTimes(const std::string& file_path,
+                              uint64_t carried_file_creation_time,
+                              uint64_t carried_oldest_ancestor_time,
+                              uint64_t now, Logger* info_log,
+                              uint64_t* file_creation_time,
+                              uint64_t* oldest_ancestor_time) {
+  *file_creation_time =
+      ResolveExternalFileTime(file_path, "file_creation_time",
+                              carried_file_creation_time, now, info_log);
+  if (carried_oldest_ancestor_time == kUnknownOldestAncestorTime) {
+    *oldest_ancestor_time = *file_creation_time;
+  } else {
+    *oldest_ancestor_time =
+        ResolveExternalFileTime(file_path, "oldest ancestor time",
+                                carried_oldest_ancestor_time, now, info_log);
+  }
+}
+
+Status CheckExternalFileNotTooOld(const std::string& file_path,
+                                  uint64_t file_creation_time,
+                                  const TableProperties& table_properties,
+                                  uint64_t now,
+                                  CompactionStyle compaction_style,
+                                  const MutableCFOptions& mutable_cf_options,
+                                  Logger* info_log) {
+  const char* limit_name;
+  uint64_t max_age;
+  const char* judged_time_name = "file_creation_time";
+  uint64_t judged_time = file_creation_time;
+  switch (compaction_style) {
+    case kCompactionStyleLevel:
+    case kCompactionStyleUniversal:
+      limit_name = "periodic_compaction_seconds";
+      max_age = mutable_cf_options.periodic_compaction_seconds;
+      break;
+    case kCompactionStyleFIFO:
+      limit_name = "ttl";
+      max_age = mutable_cf_options.ttl;
+      // Estimate when the newest data was written, as FIFO TTL compaction
+      // does (FIFOCompactionPicker::PickTTLCompaction()), so that a file it
+      // would treat as expired is rejected
+      if (table_properties.newest_key_time != kUnknownNewestKeyTime) {
+        judged_time_name = "newest_key_time";
+        judged_time = table_properties.newest_key_time;
+      } else if (table_properties.oldest_ancestor_time !=
+                 kUnknownOldestAncestorTime) {
+        judged_time_name = "oldest_ancestor_time";
+        judged_time = table_properties.oldest_ancestor_time;
+      }
+      break;
+    default:
+      return Status::OK();
+  }
+  if (judged_time == 0 || judged_time >= now) {
+    return Status::OK();
+  }
+  const uint64_t age = now - judged_time;
+  if (max_age == 0 || age <= max_age) {
+    return Status::OK();
+  }
+  std::string msg =
+      "External file " + file_path + " is too old: " + judged_time_name +
+      " is " + std::to_string(age) + " seconds ago, exceeding " + limit_name +
+      "=" + std::to_string(max_age) + " (reject_too_old_file)";
+  ROCKS_LOG_WARN(info_log, "%s", msg.c_str());
+  return Status::InvalidArgument(msg);
+}
+
 bool ExternalSstFileIngestionJob::SupportsAtomicReplaceRangeTombstone() const {
   return cfd_->is_delete_range_supported() &&
          cfd_->ioptions().compaction_style == kCompactionStyleUniversal &&
@@ -123,6 +210,11 @@ Status ExternalSstFileIngestionJob::PrepareAtomicReplaceRangeTombstone(
   }
 
   tombstone.internal_file_path = file_path;
+  // Written by this job directly into the DB directory
+  int64_t now = 0;
+  if (clock_->GetCurrentTime(&now).ok()) {
+    tombstone.db_ownership_start_time = lossless_cast<uint64_t>(now);
+  }
   tombstone.copy_file = true;
   tombstone.generated_for_ingestion = true;
   tombstone.file_checksum = external_file_info.file_checksum;
@@ -198,6 +290,21 @@ Status ExternalSstFileIngestionJob::Prepare(
       return Status::Corruption("Generated table have corrupted keys");
     }
 
+    if (ingestion_options_.reject_too_old_file) {
+      int64_t now = 0;
+      if (!clock_->GetCurrentTime(&now).ok()) {
+        now = 0;
+      }
+      status = CheckExternalFileNotTooOld(
+          file_path, file_to_ingest.table_properties.file_creation_time,
+          file_to_ingest.table_properties, lossless_cast<uint64_t>(now),
+          cfd_->ioptions().compaction_style, sv->mutable_cf_options,
+          db_options_.info_log.get());
+      if (!status.ok()) {
+        return status;
+      }
+    }
+
     files_to_ingest_.emplace_back(std::move(file_to_ingest));
   }
 
@@ -353,6 +460,10 @@ Status ExternalSstFileIngestionJob::Prepare(
       break;
     }
     f.internal_file_path = path_inside_db;
+    int64_t now = 0;
+    if (clock_->GetCurrentTime(&now).ok()) {
+      f.db_ownership_start_time = lossless_cast<uint64_t>(now);
+    }
     // Initialize the checksum information of ingested files.
     f.file_checksum = kUnknownFileChecksum;
     f.file_checksum_func_name = kUnknownFileChecksumFuncName;
@@ -863,15 +974,17 @@ Status ExternalSstFileIngestionJob::AssignLevelsForOneBatch(
     }
     max_assigned_seqno_ = std::max(max_assigned_seqno_, assigned_seqno);
 
-    // We use the import time as the ancestor time. This is the time the data
-    // is written to the database.
     int64_t temp_current_time = 0;
-    uint64_t current_time = kUnknownFileCreationTime;
-    uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
+    uint64_t current_time = 0;
     if (clock_->GetCurrentTime(&temp_current_time).ok()) {
-      current_time = oldest_ancestor_time =
-          static_cast<uint64_t>(temp_current_time);
+      current_time = static_cast<uint64_t>(temp_current_time);
     }
+    uint64_t file_creation_time = kUnknownFileCreationTime;
+    uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
+    ResolveExternalFileTimes(
+        file->external_file_path, file->table_properties.file_creation_time,
+        file->table_properties.oldest_ancestor_time, current_time,
+        db_options_.info_log.get(), &file_creation_time, &oldest_ancestor_time);
     uint64_t tail_size = FileMetaData::CalculateTailSize(
         file->fd.GetFileSize(), file->table_properties);
 
@@ -893,7 +1006,7 @@ Status ExternalSstFileIngestionJob::AssignLevelsForOneBatch(
         file->fd.GetNumber(), file->fd.GetPathId(), file->fd.GetFileSize(),
         file->smallest_internal_key, file->largest_internal_key, smallest_seqno,
         largest_seqno, false, file->file_temperature, kInvalidBlobFileNumber,
-        oldest_ancestor_time, current_time,
+        oldest_ancestor_time, file_creation_time,
         ingestion_options_.ingest_behind
             ? kReservedEpochNumberForFileIngestedBehind
             : cfd_->NewEpochNumber(),  // orders files ingested to L0
@@ -901,6 +1014,9 @@ Status ExternalSstFileIngestionJob::AssignLevelsForOneBatch(
         tail_size, file->user_defined_timestamps_persisted, "", "");
     f_metadata.temperature = file->file_temperature;
     f_metadata.marked_for_compaction = marked_for_compaction;
+    f_metadata.db_ownership_start_time = file->db_ownership_start_time;
+    f_metadata.data_file_provenance =
+        DataFileProvenanceForOrigin(DataFileProvenance::Origin::kIngestion);
     f_metadata.skip_index_and_filter_blocks_prefetch =
         !file->prefetch_lmax_index_and_filter_blocks &&
         file->picked_level == cfd_->NumberLevels() - 1;

db/external_sst_file_ingestion_job.h

@@ -150,6 +150,9 @@ struct IngestedFileInfo : public KeyRangeInfo {
   FileDescriptor fd;
   // file path that we picked for file inside the DB
   std::string internal_file_path;
+  // When the file was linked, copied or written into the DB directory, i.e.
+  // when this DB began holding it (FileMetaData::db_ownership_start_time)
+  uint64_t db_ownership_start_time = kUnknownDbOwnershipStartTime;
   // Global sequence number that we picked for the file inside the DB
   SequenceNumber assigned_seqno = 0;
   // Level inside the DB we picked for the external file.
@@ -214,6 +217,41 @@ struct FileBatchInfo : public KeyRangeInfo {
       : track_batch_range(_track_batch_range) {}
 };
 
+// Determines the file_creation_time and oldest_ancestor_time to record in the
+// MANIFEST for an ingested or imported file, from the times carried by the
+// file or its import metadata (0 = unknown) and the current time `now`
+// (0 = unknown). A known carried time is kept unless it is later than `now`,
+// which suggests an unreliable clock, so it is logged as a warning and
+// replaced by `now`. An unknown file_creation_time becomes `now`, and an
+// unknown oldest ancestor time becomes the resolved file_creation_time, an
+// upper bound on when the data was written.
+void ResolveExternalFileTimes(const std::string& file_path,
+                              uint64_t carried_file_creation_time,
+                              uint64_t carried_oldest_ancestor_time,
+                              uint64_t now, Logger* info_log,
+                              uint64_t* file_creation_time,
+                              uint64_t* oldest_ancestor_time);
+
+// Implements `reject_too_old_file` (IngestExternalFileOptions and
+// ImportColumnFamilyOptions) for one file, using the file's creation time
+// carried by the file or its import metadata, and the file's table properties
+// (times of 0 = unknown). Returns InvalidArgument (and logs a warning) if, as
+// of `now`,
+// * for leveled or universal compaction, the file's contents (per
+//   `file_creation_time`) are older than periodic_compaction_seconds, or
+// * for FIFO compaction, its newest data, estimated as FIFO TTL compaction
+//   does (the newest key time table property, else the oldest ancestor time
+//   table property, else `file_creation_time`), is older than ttl.
+// `mutable_cf_options` must be sanitized (as in a SuperVersion), so that these
+// are the effective limits.
+Status CheckExternalFileNotTooOld(const std::string& file_path,
+                                  uint64_t file_creation_time,
+                                  const TableProperties& table_properties,
+                                  uint64_t now,
+                                  CompactionStyle compaction_style,
+                                  const MutableCFOptions& mutable_cf_options,
+                                  Logger* info_log);
+
 class ExternalSstFileIngestionJob {
  public:
   ExternalSstFileIngestionJob(
@@ -259,6 +297,10 @@ class ExternalSstFileIngestionJob {
 
   ColumnFamilyData* GetColumnFamilyData() const { return cfd_; }
 
+  const IngestExternalFileOptions& GetIngestionOptions() const {
+    return ingestion_options_;
+  }
+
   // Prepare the job by copying external files into the DB.
   Status Prepare(const std::vector<std::string>& external_files_paths,
                  const std::vector<std::string>& files_checksums,

db/external_sst_file_test.cc

@@ -8,11 +8,13 @@
 #include <atomic>
 #include <functional>
 #include <memory>
+#include <set>
 #include <sstream>
 
 #include "db/db_test_util.h"
 #include "db/dbformat.h"
 #include "db/job_context.h"
+#include "env/composite_env_wrapper.h"
 #include "file/filename.h"
 #include "options/options_helper.h"
 #include "port/port.h"
@@ -21,6 +23,7 @@
 #include "rocksdb/sst_file_writer.h"
 #include "table/format.h"
 #include "table/prepared_file_info.h"
+#include "test_util/mock_time_env.h"
 #include "test_util/testutil.h"
 #include "util/defer.h"
 #include "util/random.h"
@@ -844,6 +847,189 @@ TEST_F(ExternalSSTFileTest, AbortPreparedIngestion) {
   ASSERT_EQ("1", Get("x"));
 }
 
+static std::string DescribeFile(const SstFileMetaData& meta) {
+  return meta.directory + "/" + meta.relative_filename + " #" +
+         std::to_string(meta.file_number) + " " + std::to_string(meta.size) +
+         "B " + meta.file_checksum_func_name + ":" +
+         Slice(meta.file_checksum).ToString(/*hex=*/true) + " [" +
+         meta.smallestkey + ", " + meta.largestkey + "] temperature " +
+         std::to_string(static_cast<int>(meta.temperature));
+}
+
+static std::set<std::string> DescribeFiles(
+    const std::vector<SstFileMetaData>& files) {
+  std::set<std::string> described;
+  for (const auto& meta : files) {
+    described.insert(DescribeFile(meta));
+  }
+  return described;
+}
+
+static std::set<std::string> DescribeLiveFiles(DB* db) {
+  std::vector<LiveFileMetaData> live;
+  db->GetLiveFilesMetaData(&live);
+  return DescribeFiles({live.begin(), live.end()});
+}
+
+TEST_F(ExternalSSTFileTest, PreparedFilesAreTheFilesThatGoLive) {
+  Options options = CurrentOptions();
+  options.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
+  DestroyAndReopen(options);
+
+  std::string file_a;
+  ASSERT_OK(
+      GenerateExternalFileOnly(options, {{"a1", "1"}, {"a2", "2"}}, &file_a));
+  const std::string file_b = sst_files_dir_ + "put_and_delete.sst";
+  SstFileWriter writer(EnvOptions(), options);
+  ASSERT_OK(writer.Open(file_b));
+  ASSERT_OK(writer.Put("b", "2"));
+  ASSERT_OK(writer.Delete("c"));
+  ASSERT_OK(writer.Finish());
+  IngestExternalFileArg arg;
+  arg.column_family = db_->DefaultColumnFamily();
+  arg.external_files = {file_a, file_b};
+
+  std::unique_ptr<FileIngestionHandle> handle;
+  ASSERT_OK(db_->PrepareFileIngestion({arg}, &handle));
+  const std::vector<SstFileMetaData> prepared = handle->GetPreparedFiles();
+
+  ASSERT_EQ(2, prepared.size());
+  for (const auto& meta : prepared) {
+    const std::string path = meta.directory + "/" + meta.relative_filename;
+    uint64_t size_on_disk = 0;
+    ASSERT_OK(env_->GetFileSize(path, &size_on_disk));
+    ASSERT_EQ(size_on_disk, meta.size);
+    ASSERT_EQ("FileChecksumCrc32c", meta.file_checksum_func_name);
+    ASSERT_FALSE(meta.file_checksum.empty());
+  }
+  std::set<std::string> key_ranges;
+  for (const auto& meta : prepared) {
+    key_ranges.insert(meta.smallestkey + ".." + meta.largestkey + " " +
+                      std::to_string(meta.num_entries) + " entries " +
+                      std::to_string(meta.num_deletions) + " deletions");
+  }
+  ASSERT_EQ((std::set<std::string>{"a1..a2 2 entries 0 deletions",
+                                   "b..c 2 entries 1 deletions"}),
+            key_ranges);
+  ASSERT_EQ("NOT_FOUND", Get("a1"));
+
+  ASSERT_OK(db_->CommitFileIngestionHandle(std::move(handle)));
+
+  ASSERT_EQ(DescribeFiles(prepared), DescribeLiveFiles(db_.get()));
+}
+
+TEST_F(ExternalSSTFileTest, PreparedDbGeneratedFilesCarryTheirSeqnos) {
+  Options options = CurrentOptions();
+  DestroyAndReopen(options);
+
+  const std::string source_dbname = dbname_ + "_source";
+  ASSERT_OK(DestroyDB(source_dbname, options));
+  std::unique_ptr<DB> source_db;
+  ASSERT_OK(DB::Open(options, source_dbname, &source_db));
+  ASSERT_OK(source_db->Put(WriteOptions(), Key(1), "v1"));
+  ASSERT_OK(source_db->Put(WriteOptions(), Key(2), "v2"));
+  ASSERT_OK(source_db->Flush(FlushOptions()));
+  std::vector<LiveFileMetaData> source_meta;
+  source_db->GetLiveFilesMetaData(&source_meta);
+  ASSERT_EQ(1, source_meta.size());
+  ASSERT_LT(source_meta[0].smallest_seqno, source_meta[0].largest_seqno);
+
+  IngestExternalFileArg arg;
+  arg.column_family = db_->DefaultColumnFamily();
+  arg.external_files = {source_meta[0].directory + "/" +
+                        source_meta[0].relative_filename};
+  arg.options.allow_db_generated_files = true;
+  arg.options.snapshot_consistency = false;
+
+  std::unique_ptr<FileIngestionHandle> handle;
+  ASSERT_OK(db_->PrepareFileIngestion({arg}, &handle));
+  const std::vector<SstFileMetaData> prepared = handle->GetPreparedFiles();
+  ASSERT_EQ(1, prepared.size());
+  ASSERT_EQ(source_meta[0].smallest_seqno, prepared[0].smallest_seqno);
+  ASSERT_EQ(source_meta[0].largest_seqno, prepared[0].largest_seqno);
+
+  ASSERT_OK(db_->CommitFileIngestionHandle(std::move(handle)));
+
+  std::vector<LiveFileMetaData> live;
+  db_->GetLiveFilesMetaData(&live);
+  ASSERT_EQ(1, live.size());
+  ASSERT_EQ(live[0].smallest_seqno, prepared[0].smallest_seqno);
+  ASSERT_EQ(live[0].largest_seqno, prepared[0].largest_seqno);
+  ASSERT_EQ(live[0].name, prepared[0].name);
+  ASSERT_EQ(live[0].db_path, prepared[0].db_path);
+
+  ASSERT_OK(source_db->Close());
+  source_db.reset();
+  ASSERT_OK(DestroyDB(source_dbname, options));
+}
+
+TEST_F(ExternalSSTFileTest, PreparedFilesAreGoneAfterAbort) {
+  Options options = CurrentOptions();
+  DestroyAndReopen(options);
+
+  std::string file_path;
+  ASSERT_OK(GenerateExternalFileOnly(options, {{"x", "1"}}, &file_path));
+  IngestExternalFileArg arg;
+  arg.column_family = db_->DefaultColumnFamily();
+  arg.external_files = {file_path};
+
+  std::unique_ptr<FileIngestionHandle> handle;
+  ASSERT_OK(db_->PrepareFileIngestion({arg}, &handle));
+  const std::vector<SstFileMetaData> prepared = handle->GetPreparedFiles();
+  ASSERT_EQ(1, prepared.size());
+  const std::string path =
+      prepared[0].directory + "/" + prepared[0].relative_filename;
+  ASSERT_OK(env_->FileExists(path));
+
+  ASSERT_OK(handle->Abort());
+
+  ASSERT_TRUE(handle->GetPreparedFiles().empty());
+  ASSERT_TRUE(env_->FileExists(path).IsNotFound());
+}
+
+TEST_F(ExternalSSTFileTest, PreparedFilesOfMergedHandlesGoLive) {
+  Options options = CurrentOptions();
+  CreateAndReopenWithCF({"cf1", "cf2", "cf3"}, options);
+
+  std::string f_cf1;
+  std::string f_cf2_h1;
+  std::string f_cf2_h2;
+  std::string f_cf3;
+  ASSERT_OK(GenerateExternalFileOnly(options, {{"a", "1"}}, &f_cf1));
+  ASSERT_OK(GenerateExternalFileOnly(options, {{"b", "h1"}}, &f_cf2_h1));
+  ASSERT_OK(GenerateExternalFileOnly(options, {{"b", "h2"}}, &f_cf2_h2));
+  ASSERT_OK(GenerateExternalFileOnly(options, {{"c", "3"}}, &f_cf3));
+
+  // Each handle spans two column families, and both write cf2, so commit
+  // merges the two cf2 jobs.
+  std::vector<IngestExternalFileArg> args1(2);
+  args1[0].column_family = handles_[1];
+  args1[0].external_files = {f_cf1};
+  args1[1].column_family = handles_[2];
+  args1[1].external_files = {f_cf2_h1};
+  std::vector<IngestExternalFileArg> args2(2);
+  args2[0].column_family = handles_[2];
+  args2[0].external_files = {f_cf2_h2};
+  args2[1].column_family = handles_[3];
+  args2[1].external_files = {f_cf3};
+
+  std::unique_ptr<FileIngestionHandle> h1;
+  std::unique_ptr<FileIngestionHandle> h2;
+  ASSERT_OK(db_->PrepareFileIngestion(args1, &h1));
+  ASSERT_OK(db_->PrepareFileIngestion(args2, &h2));
+  std::set<std::string> prepared = DescribeFiles(h1->GetPreparedFiles());
+  prepared.merge(DescribeFiles(h2->GetPreparedFiles()));
+  ASSERT_EQ(4, prepared.size());
+
+  std::vector<std::unique_ptr<FileIngestionHandle>> batch;
+  batch.push_back(std::move(h1));
+  batch.push_back(std::move(h2));
+  ASSERT_OK(db_->CommitFileIngestionHandles(std::move(batch)));
+
+  ASSERT_EQ("h2", Get(2, "b"));
+  ASSERT_EQ(prepared, DescribeLiveFiles(db_.get()));
+}
+
 TEST_F(ExternalSSTFileTest, MultiHandleAtomicCommit) {
   Options options = CurrentOptions();
   CreateAndReopenWithCF({"cf1", "cf2", "cf3"}, options);
@@ -5812,6 +5998,443 @@ TEST_P(IngestDBGeneratedFileTest2, ZeroAndNonZeroSeqno) {
   } while (ChangeOptions(kSkipPlainTable | kSkipFIFOCompaction));
 }
 
+// Tests for the file times recorded for ingested files and for
+// IngestExternalFileOptions::reject_too_old_file
+class ExternalSSTFileTimeTest : public ExternalSSTFileTestBase {
+ protected:
+  uint64_t Now() {
+    int64_t now = 0;
+    EXPECT_OK(env_->GetCurrentTime(&now));
+    return static_cast<uint64_t>(now);
+  }
+
+  // Writes a single-key SST file with SstFileWriter at `write_time` (0 means
+  // unknown) according to the writer's clock.
+  std::string WriteFileAt(const Options& options, uint64_t write_time,
+                          const std::string& key) {
+    auto clock = std::make_shared<MockSystemClock>(env_->GetSystemClock());
+    clock->SetCurrentTime(write_time);
+    CompositeEnvWrapper writer_env(env_, clock);
+    Options writer_options = options;
+    writer_options.env = &writer_env;
+    std::string file = sst_files_dir_ + key + ".sst";
+    SstFileWriter writer(EnvOptions(), writer_options);
+    EXPECT_OK(writer.Open(file));
+    EXPECT_OK(writer.Put(key, "val"));
+    EXPECT_OK(writer.Finish());
+    return file;
+  }
+
+  // MANIFEST record of the file in `cfh` whose smallest user key is `key`
+  FileMetaData ManifestRecord(ColumnFamilyHandle* cfh, const std::string& key) {
+    std::vector<std::vector<FileMetaData>> metadata;
+    dbfull()->TEST_GetFilesMetaData(cfh, &metadata);
+    for (const auto& level : metadata) {
+      for (const auto& f : level) {
+        if (f.smallest.user_key() == key) {
+          return f;
+        }
+      }
+    }
+    ADD_FAILURE() << "No file starting with " << key;
+    return FileMetaData();
+  }
+
+  // MANIFEST file_creation_time and oldest_ancestor_time of the file in `cfh`
+  // whose smallest user key is `key`
+  std::pair<uint64_t, uint64_t> ManifestTimes(ColumnFamilyHandle* cfh,
+                                              const std::string& key) {
+    const FileMetaData f = ManifestRecord(cfh, key);
+    return {f.file_creation_time, f.oldest_ancestor_time};
+  }
+
+  size_t NumLiveFiles() {
+    std::vector<LiveFileMetaData> files;
+    db_->GetLiveFilesMetaData(&files);
+    return files.size();
+  }
+};
+
+TEST_F(ExternalSSTFileTimeTest, KeepRecordedTimes) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  env_->SetMockSleep();
+  DestroyAndReopen(options);
+
+  const uint64_t start = Now();
+  const uint64_t old_time = start - 5000;
+  std::string old_file = WriteFileAt(options, old_time, "a");
+  std::string unknown_time_file = WriteFileAt(options, 0, "b");
+  std::string future_file = WriteFileAt(options, start + 5000, "c");
+  ASSERT_OK(db_->IngestExternalFile({old_file, unknown_time_file, future_file},
+                                    IngestExternalFileOptions()));
+  const uint64_t end = Now();
+
+  // SstFileWriter files record no oldest ancestor time, so it matches the
+  // resolved file_creation_time
+  auto times = ManifestTimes(db_->DefaultColumnFamily(), "a");
+  EXPECT_EQ(times.first, old_time);
+  EXPECT_EQ(times.second, old_time);
+
+  // Unknown or future times are replaced with the ingestion time
+  for (const char* key : {"b", "c"}) {
+    SCOPED_TRACE(key);
+    times = ManifestTimes(db_->DefaultColumnFamily(), key);
+    EXPECT_GE(times.first, start);
+    EXPECT_LE(times.first, end);
+    EXPECT_EQ(times.second, times.first);
+  }
+}
+
+TEST_F(ExternalSSTFileTimeTest, KeepDBGeneratedFileTimes) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  env_->SetMockSleep();
+  CreateAndReopenWithCF({"src"}, options);
+
+  // Flush and then compact in a source CF so that its file has an oldest
+  // ancestor time older than its file creation time
+  ASSERT_OK(Put(1, "a", "val"));
+  ASSERT_OK(Flush(1));
+  env_->MockSleepForSeconds(1000);
+  CompactRangeOptions cro;
+  cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  ASSERT_OK(db_->CompactRange(cro, handles_[1], nullptr, nullptr));
+  std::vector<LiveFileMetaData> live_meta;
+  db_->GetLiveFilesMetaData(&live_meta);
+  ASSERT_EQ(live_meta.size(), 1U);
+  const std::string src_file =
+      live_meta[0].directory + "/" + live_meta[0].relative_filename;
+  TablePropertiesCollection src_props;
+  ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[1], &src_props));
+  ASSERT_EQ(src_props.size(), 1U);
+  const TableProperties& props = *src_props.begin()->second;
+  ASSERT_GT(props.oldest_ancestor_time, 0U);
+  ASSERT_GE(props.file_creation_time, props.oldest_ancestor_time + 1000);
+
+  env_->MockSleepForSeconds(1000);
+  IngestExternalFileOptions ifo;
+  ifo.allow_db_generated_files = true;
+  ifo.snapshot_consistency = false;
+  ASSERT_OK(db_->IngestExternalFile(handles_[0], {src_file}, ifo));
+
+  auto times = ManifestTimes(handles_[0], "a");
+  EXPECT_EQ(times.first, props.file_creation_time);
+  EXPECT_EQ(times.second, props.oldest_ancestor_time);
+}
+
+TEST_F(ExternalSSTFileTimeTest, PeriodicCompactionFromWriteTime) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleLevel;
+  options.periodic_compaction_seconds = 10000;
+  env_->SetMockSleep();
+  DestroyAndReopen(options);
+
+  std::atomic<int> periodic_compactions{0};
+  SyncPoint::GetInstance()->SetCallBack(
+      "LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
+        Compaction* c = static_cast<Compaction*>(arg);
+        if (c->compaction_reason() == CompactionReason::kPeriodicCompaction) {
+          ++periodic_compactions;
+        }
+      });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  std::string file = WriteFileAt(options, Now(), "a");
+  env_->MockSleepForSeconds(6000);
+  ASSERT_OK(db_->IngestExternalFile({file}, IngestExternalFileOptions()));
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  EXPECT_EQ(periodic_compactions.load(), 0);
+
+  // Due for periodic compaction relative to when the file was written, though
+  // not relative to when it was ingested
+  env_->MockSleepForSeconds(5000);
+  // Recompute files marked for compaction
+  ASSERT_OK(Put("b", "val"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  EXPECT_EQ(periodic_compactions.load(), 1);
+
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+}
+
+TEST_F(ExternalSSTFileTimeTest, RejectTooOldFile) {
+  for (CompactionStyle style :
+       {kCompactionStyleLevel, kCompactionStyleUniversal,
+        kCompactionStyleFIFO}) {
+    SCOPED_TRACE("compaction_style=" + std::to_string(static_cast<int>(style)));
+    Options options = CurrentOptions();
+    options.compaction_style = style;
+    options.disable_auto_compactions = true;
+    if (style == kCompactionStyleFIFO) {
+      options.ttl = 10000;
+      options.max_open_files = -1;
+    } else {
+      options.periodic_compaction_seconds = 10000;
+    }
+    env_->SetMockSleep();
+    DestroyAndReopen(options);
+    DestroyAndRecreateExternalSSTFilesDir();
+
+    const uint64_t now = Now();
+    std::string too_old = WriteFileAt(options, now - 10100, "a");
+    std::string not_too_old = WriteFileAt(options, now - 9900, "b");
+    std::string unknown_time = WriteFileAt(options, 0, "c");
+
+    IngestExternalFileOptions ifo;
+    ifo.reject_too_old_file = true;
+    Status s = db_->IngestExternalFile({not_too_old, too_old}, ifo);
+    ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+    ASSERT_EQ(NumLiveFiles(), 0U);
+
+    ASSERT_OK(db_->IngestExternalFile({not_too_old, unknown_time}, ifo));
+    ASSERT_EQ(NumLiveFiles(), 2U);
+
+    ASSERT_OK(db_->IngestExternalFile({too_old}, IngestExternalFileOptions()));
+    ASSERT_EQ(NumLiveFiles(), 3U);
+  }
+}
+
+// An SstFileWriter file's creation time is when it was finished, so that it is
+// no earlier than when any entry was added
+TEST_F(ExternalSSTFileTimeTest, CreationTimeIsWhenFileIsFinished) {
+  Options options = CurrentOptions();
+  auto clock = std::make_shared<MockSystemClock>(env_->GetSystemClock());
+  clock->SetCurrentTime(1000);
+  CompositeEnvWrapper writer_env(env_, clock);
+  Options writer_options = options;
+  writer_options.env = &writer_env;
+  const std::string file = sst_files_dir_ + "a.sst";
+  SstFileWriter writer(EnvOptions(), writer_options);
+  ASSERT_OK(writer.Open(file));
+  clock->SetCurrentTime(2000);
+  ASSERT_OK(writer.Put("a", "val"));
+  clock->SetCurrentTime(3000);
+  ASSERT_OK(writer.Finish());
+
+  SstFileReader reader(options);
+  ASSERT_OK(reader.Open(file));
+  EXPECT_EQ(reader.GetTableProperties()->file_creation_time, 3000U);
+}
+
+// For FIFO compaction, reject_too_old_file judges a file by its newest data as
+// FIFO TTL compaction estimates it: by its newest key time when known, else
+// its oldest ancestor time, even if the file was built recently
+TEST_F(ExternalSSTFileTimeTest, RejectFIFOFileWithOldNewestKey) {
+  for (bool newest_key_time_unknown : {false, true}) {
+    SCOPED_TRACE("newest_key_time_unknown=" +
+                 std::to_string(newest_key_time_unknown));
+    Options options = CurrentOptions();
+    options.disable_auto_compactions = true;
+    options.max_open_files = -1;
+    env_->SetMockSleep();
+    DestroyAndReopen(options);
+    Options fifo_options = options;
+    fifo_options.compaction_style = kCompactionStyleFIFO;
+    fifo_options.num_levels = 1;
+    fifo_options.ttl = 10000;
+    CreateColumnFamilies({"fifo"}, fifo_options);
+    ReopenWithColumnFamilies({kDefaultColumnFamilyName, "fifo"},
+                             std::vector<Options>{options, fifo_options});
+
+    if (newest_key_time_unknown) {
+      // As in files written by older versions
+      SyncPoint::GetInstance()->SetCallBack(
+          "PropertyBlockBuilder::AddTableProperty:Start", [](void* arg) {
+            static_cast<TableProperties*>(arg)->newest_key_time = 0;
+          });
+      SyncPoint::GetInstance()->EnableProcessing();
+    }
+    // Compacting old data builds a new file with the same newest key time
+    // (when known) and oldest ancestor time
+    ASSERT_OK(Put("a", "val"));
+    ASSERT_OK(Flush());
+    env_->MockSleepForSeconds(20000);
+    CompactRangeOptions cro;
+    cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+    ASSERT_OK(db_->CompactRange(cro, handles_[0], nullptr, nullptr));
+    SyncPoint::GetInstance()->DisableProcessing();
+    SyncPoint::GetInstance()->ClearAllCallBacks();
+    TablePropertiesCollection props;
+    ASSERT_OK(db_->GetPropertiesOfAllTables(handles_[0], &props));
+    ASSERT_EQ(props.size(), 1U);
+    const TableProperties& file_props = *props.begin()->second;
+    if (newest_key_time_unknown) {
+      ASSERT_EQ(file_props.newest_key_time, 0U);
+    } else {
+      ASSERT_GT(file_props.newest_key_time, 0U);
+      ASSERT_GE(file_props.file_creation_time,
+                file_props.newest_key_time + 20000);
+    }
+    ASSERT_GT(file_props.oldest_ancestor_time, 0U);
+    ASSERT_GE(file_props.file_creation_time,
+              file_props.oldest_ancestor_time + 20000);
+    const std::string file = props.begin()->first;
+
+    IngestExternalFileOptions ifo;
+    ifo.allow_db_generated_files = true;
+    ifo.snapshot_consistency = false;
+    ifo.reject_too_old_file = true;
+    Status s = db_->IngestExternalFile(handles_[1], {file}, ifo);
+    ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+    ASSERT_EQ(NumTableFilesAtLevel(0, 1), 0);
+
+    ifo.reject_too_old_file = false;
+    ASSERT_OK(db_->IngestExternalFile(handles_[1], {file}, ifo));
+    ASSERT_EQ(NumTableFilesAtLevel(0, 1), 1);
+  }
+}
+
+// SstFileWriter files carry neither the newest key time nor the oldest
+// ancestor time table property, which FIFO TTL compaction looks at. Such a
+// file, and every newer file, should still expire by TTL.
+TEST_F(ExternalSSTFileTimeTest, FIFOTtlDeletesIngestedFile) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleFIFO;
+  options.ttl = 10000;
+  options.max_open_files = -1;
+  env_->SetMockSleep();
+  DestroyAndReopen(options);
+
+  std::string file = WriteFileAt(options, Now(), "a");
+  ASSERT_OK(db_->IngestExternalFile({file}, IngestExternalFileOptions()));
+  ASSERT_OK(Put("b", "val"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  ASSERT_EQ(NumLiveFiles(), 2U);
+
+  env_->MockSleepForSeconds(11000);
+  // Flush to trigger compaction
+  ASSERT_OK(Put("c", "val"));
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  EXPECT_EQ(NumLiveFiles(), 1U);
+  EXPECT_EQ(Get("a"), "NOT_FOUND");
+  EXPECT_EQ(Get("b"), "NOT_FOUND");
+  EXPECT_EQ(Get("c"), "val");
+}
+
+TEST_F(ExternalSSTFileTimeTest, RecordsDbOwnership) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  env_->SetMockSleep();
+  CreateAndReopenWithCF({"src"}, options);
+  DataFileProvenance ingested;
+  ingested.origin = DataFileProvenance::Origin::kIngestion;
+  const uint64_t old_time = Now() - 5000;
+
+  // The DB began holding the file when it was ingested, regardless of when
+  // the file was written
+  std::string file = WriteFileAt(options, old_time, "a");
+  uint64_t start = Now();
+  ASSERT_OK(db_->IngestExternalFile(handles_[0], {file},
+                                    IngestExternalFileOptions()));
+  FileMetaData f = ManifestRecord(handles_[0], "a");
+  EXPECT_EQ(f.file_creation_time, old_time);
+  EXPECT_GE(f.db_ownership_start_time, start);
+  EXPECT_LE(f.db_ownership_start_time, Now());
+  EXPECT_EQ(DataFileProvenance::DecodeFrom(f.data_file_provenance), ingested);
+
+  // With two-phase ingestion, from the prepare phase
+  file = WriteFileAt(options, old_time, "b");
+  start = Now();
+  std::unique_ptr<FileIngestionHandle> handle;
+  ASSERT_OK(db_->PrepareFileIngestion(handles_[0], {file},
+                                      IngestExternalFileOptions(), &handle));
+  const uint64_t prepared = Now();
+  env_->MockSleepForSeconds(1000);
+  ASSERT_OK(db_->CommitFileIngestionHandle(std::move(handle)));
+  f = ManifestRecord(handles_[0], "b");
+  EXPECT_EQ(f.file_creation_time, old_time);
+  EXPECT_GE(f.db_ownership_start_time, start);
+  EXPECT_LE(f.db_ownership_start_time, prepared);
+  EXPECT_EQ(DataFileProvenance::DecodeFrom(f.data_file_provenance), ingested);
+
+  // A file generated by a DB (here, another column family)
+  ASSERT_OK(Put(1, "c", "val"));
+  ASSERT_OK(Flush(1));
+  const FileMetaData src = ManifestRecord(handles_[1], "c");
+  ColumnFamilyMetaData src_meta;
+  db_->GetColumnFamilyMetaData(handles_[1], &src_meta);
+  ASSERT_EQ(src_meta.levels[0].files.size(), 1U);
+  const SstFileMetaData& src_file = src_meta.levels[0].files[0];
+  env_->MockSleepForSeconds(1000);
+  start = Now();
+  IngestExternalFileOptions ifo;
+  ifo.allow_db_generated_files = true;
+  ifo.snapshot_consistency = false;
+  ASSERT_OK(db_->IngestExternalFile(
+      handles_[0], {src_file.directory + "/" + src_file.relative_filename},
+      ifo));
+  f = ManifestRecord(handles_[0], "c");
+  EXPECT_EQ(f.file_creation_time, src.file_creation_time);
+  EXPECT_GE(f.db_ownership_start_time, start);
+  EXPECT_EQ(DataFileProvenance::DecodeFrom(f.data_file_provenance), ingested);
+}
+
+// bottommost_file_compaction_delay counts from when the DB began holding a
+// file, which for an ingested file is when it was ingested, not when its
+// contents were built
+TEST_F(ExternalSSTFileTimeTest, BottommostCompactionDelayFromIngestion) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleLevel;
+  options.bottommost_file_compaction_delay = 5000;
+  env_->SetMockSleep();
+  CreateAndReopenWithCF({"src"}, options);
+
+  // A DB-generated file with nonzero sequence numbers, which ingestion keeps
+  ASSERT_OK(Put(1, "a", "val"));
+  ASSERT_OK(Put(1, "b", "val"));
+  ASSERT_OK(Flush(1));
+  // Only compactions of the ingested file are of interest
+  ASSERT_OK(
+      db_->SetOptions(handles_[1], {{"disable_auto_compactions", "true"}}));
+  ColumnFamilyMetaData src_meta;
+  db_->GetColumnFamilyMetaData(handles_[1], &src_meta);
+  ASSERT_EQ(src_meta.levels[0].files.size(), 1U);
+  const SstFileMetaData& src_file = src_meta.levels[0].files[0];
+  ASSERT_GT(src_file.largest_seqno, 0U);
+  // Keeps the file from being eligible for bottommost compaction until
+  // released
+  const Snapshot* snapshot = db_->GetSnapshot();
+
+  env_->MockSleepForSeconds(10000);
+  IngestExternalFileOptions ifo;
+  ifo.allow_db_generated_files = true;
+  ifo.snapshot_consistency = false;
+  ASSERT_OK(db_->IngestExternalFile(
+      handles_[0], {src_file.directory + "/" + src_file.relative_filename},
+      ifo));
+  ASSERT_EQ(NumTableFilesAtLevel(options.num_levels - 1, 0), 1);
+
+  std::atomic<int> bottommost_compactions{0};
+  SyncPoint::GetInstance()->SetCallBack(
+      "LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
+        Compaction* c = static_cast<Compaction*>(arg);
+        if (c->compaction_reason() == CompactionReason::kBottommostFiles) {
+          ++bottommost_compactions;
+        }
+      });
+  SyncPoint::GetInstance()->EnableProcessing();
+
+  // Bump the published sequence number so that releasing the snapshot
+  // advances the oldest snapshot past the file's sequence numbers
+  ASSERT_OK(Put("y", "val"));
+  db_->ReleaseSnapshot(snapshot);
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  EXPECT_EQ(bottommost_compactions.load(), 0);
+
+  env_->MockSleepForSeconds(6000);
+  // Recompute files marked for compaction
+  ASSERT_OK(Flush());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  EXPECT_EQ(bottommost_compactions.load(), 1);
+
+  SyncPoint::GetInstance()->DisableProcessing();
+  SyncPoint::GetInstance()->ClearAllCallBacks();
+}
+
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/flush_job.cc

@@ -1044,6 +1044,9 @@ Status FlushJob::WriteLevel0Table() {
           &oldest_ancestor_time);
       meta_.oldest_ancestor_time = oldest_ancestor_time;
       meta_.file_creation_time = current_time;
+      meta_.db_ownership_start_time = current_time;
+      meta_.data_file_provenance =
+          DataFileProvenanceForOrigin(DataFileProvenance::Origin::kFlush);
 
       uint64_t memtable_payload_bytes = 0;
       uint64_t memtable_garbage_bytes = 0;
@@ -1060,9 +1063,8 @@ Status FlushJob::WriteLevel0Table() {
           output_compression_, mutable_cf_options_.compression_opts,
           cfd_->GetID(), cfd_->GetName(), 0 /* level */,
           current_time /* newest_key_time */, false /* is_bottommost */,
-          TableFileCreationReason::kFlush, oldest_key_time, current_time,
-          db_id_, db_session_id_, 0 /* target_file_size */,
-          meta_.fd.GetNumber(),
+          TableFileCreationReason::kFlush, oldest_key_time, db_id_,
+          db_session_id_, 0 /* target_file_size */, meta_.fd.GetNumber(),
           preclude_last_level_min_seqno_ == kMaxSequenceNumber
               ? preclude_last_level_min_seqno_
               : std::min(earliest_snapshot_, preclude_last_level_min_seqno_),

db/import_column_family_job.cc

@@ -20,10 +20,21 @@
 #include "table/sst_file_writer_collectors.h"
 #include "table/table_builder.h"
 #include "table/unique_id_impl.h"
+#include "util/cast_util.h"
 #include "util/stop_watch.h"
 
 namespace ROCKSDB_NAMESPACE {
 
+namespace {
+// A time carried by an imported file: from its import metadata (e.g. the
+// exporting DB's MANIFEST) when known, else from the file's table properties
+// (0 if unknown in both).
+uint64_t CarriedImportTime(uint64_t from_metadata,
+                           uint64_t from_table_properties) {
+  return from_metadata != 0 ? from_metadata : from_table_properties;
+}
+}  // namespace
+
 Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,
                                       SuperVersion* sv) {
   Status status;
@@ -56,6 +67,24 @@ Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,
         return status;
       }
 
+      if (import_options_.reject_too_old_file) {
+        int64_t now = 0;
+        if (!clock_->GetCurrentTime(&now).ok()) {
+          now = 0;
+        }
+        status = CheckExternalFileNotTooOld(
+            file_path,
+            CarriedImportTime(
+                file_metadata.file_creation_time,
+                file_to_import.table_properties.file_creation_time),
+            file_to_import.table_properties, lossless_cast<uint64_t>(now),
+            cfd_->ioptions().compaction_style, sv->mutable_cf_options,
+            db_options_.info_log.get());
+        if (!status.ok()) {
+          return status;
+        }
+      }
+
       files_to_import_per_cf.push_back(file_to_import);
       num_files++;
 
@@ -135,6 +164,10 @@ Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,
       }
       f.copy_file = !hardlink_files;
       f.internal_file_path = path_inside_db;
+      int64_t now = 0;
+      if (clock_->GetCurrentTime(&now).ok()) {
+        f.db_ownership_start_time = lossless_cast<uint64_t>(now);
+      }
     }
     if (!status.ok()) {
       break;
@@ -165,14 +198,10 @@ Status ImportColumnFamilyJob::Prepare(uint64_t next_file_number,
 // REQUIRES: we have become the only writer by entering both write_thread_ and
 // nonmem_write_thread_
 Status ImportColumnFamilyJob::Run() {
-  // We use the import time as the ancestor time. This is the time the data
-  // is written to the database.
   int64_t temp_current_time = 0;
-  uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
-  uint64_t current_time = kUnknownOldestAncestorTime;
+  uint64_t current_time = 0;
   if (clock_->GetCurrentTime(&temp_current_time).ok()) {
-    current_time = oldest_ancestor_time =
-        static_cast<uint64_t>(temp_current_time);
+    current_time = static_cast<uint64_t>(temp_current_time);
   }
 
   Status s;
@@ -206,17 +235,34 @@ Status ImportColumnFamilyJob::Run() {
       uint64_t tail_size = FileMetaData::CalculateTailSize(f.fd.GetFileSize(),
                                                            f.table_properties);
 
-      VersionEdit dummy_version_edit;
-      dummy_version_edit.AddFile(
-          file_metadata.level, f.fd.GetNumber(), f.fd.GetPathId(),
-          f.fd.GetFileSize(), f.smallest_internal_key, f.largest_internal_key,
+      uint64_t file_creation_time = kUnknownFileCreationTime;
+      uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
+      ResolveExternalFileTimes(
+          f.external_file_path,
+          CarriedImportTime(file_metadata.file_creation_time,
+                            f.table_properties.file_creation_time),
+          CarriedImportTime(file_metadata.oldest_ancestor_time,
+                            f.table_properties.oldest_ancestor_time),
+          current_time, db_options_.info_log.get(), &file_creation_time,
+          &oldest_ancestor_time);
+
+      FileMetaData meta(
+          f.fd.GetNumber(), f.fd.GetPathId(), f.fd.GetFileSize(),
+          f.smallest_internal_key, f.largest_internal_key,
           file_metadata.smallest_seqno, file_metadata.largest_seqno, false,
           file_metadata.temperature, kInvalidBlobFileNumber,
-          oldest_ancestor_time, current_time, file_metadata.epoch_number,
+          oldest_ancestor_time, file_creation_time, file_metadata.epoch_number,
           kUnknownFileChecksum, kUnknownFileChecksumFuncName, f.unique_id, 0,
           tail_size,
           static_cast<bool>(
-              f.table_properties.user_defined_timestamps_persisted));
+              f.table_properties.user_defined_timestamps_persisted),
+          /*_min_timestamp=*/"", /*_max_timestamp=*/"");
+      // The export metadata's ownership fields describe the source DB.
+      meta.db_ownership_start_time = f.db_ownership_start_time;
+      meta.data_file_provenance =
+          DataFileProvenanceForOrigin(DataFileProvenance::Origin::kImport);
+      VersionEdit dummy_version_edit;
+      dummy_version_edit.AddFile(file_metadata.level, meta);
       s = dummy_version_builder.Apply(&dummy_version_edit);
     }
     if (s.ok()) {

db/import_column_family_test.cc

@@ -10,6 +10,7 @@
 #include "port/port.h"
 #include "port/stack_trace.h"
 #include "rocksdb/sst_file_writer.h"
+#include "test_util/mock_time_env.h"
 #include "test_util/testutil.h"
 #include "util/random.h"
 
@@ -1053,6 +1054,165 @@ TEST_F(ImportColumnFamilyTest, AssignEpochNumberToMultipleCF) {
   ASSERT_OK(db_->WaitForCompact(o));
   delete checkpoint1;
 }
+
+namespace {
+// MANIFEST record of the file in `cfh` whose smallest user key is `key`
+FileMetaData ManifestRecord(DBImpl* db, ColumnFamilyHandle* cfh,
+                            const std::string& key) {
+  std::vector<std::vector<FileMetaData>> metadata;
+  db->TEST_GetFilesMetaData(cfh, &metadata);
+  for (const auto& level : metadata) {
+    for (const auto& f : level) {
+      if (f.smallest.user_key() == key) {
+        return f;
+      }
+    }
+  }
+  ADD_FAILURE() << "No file starting with " << key;
+  return FileMetaData();
+}
+
+// MANIFEST {file_creation_time, oldest_ancestor_time} of the file in `cfh`
+// whose smallest user key is `key`
+std::pair<uint64_t, uint64_t> ManifestTimes(DBImpl* db, ColumnFamilyHandle* cfh,
+                                            const std::string& key) {
+  const FileMetaData f = ManifestRecord(db, cfh, key);
+  return {f.file_creation_time, f.oldest_ancestor_time};
+}
+}  // namespace
+
+TEST_F(ImportColumnFamilyTest, ImportKeepsExportedFileTimes) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  env_->SetMockSleep();
+  CreateAndReopenWithCF({"koko"}, options);
+
+  // Flush and then compact so that the file has an oldest ancestor time older
+  // than its file creation time
+  ASSERT_OK(Put(1, "a", "val"));
+  ASSERT_OK(Flush(1));
+  env_->MockSleepForSeconds(1000);
+  CompactRangeOptions cro;
+  cro.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  ASSERT_OK(db_->CompactRange(cro, handles_[1], nullptr, nullptr));
+  const auto src_times = ManifestTimes(dbfull(), handles_[1], "a");
+  ASSERT_GT(src_times.second, 0U);
+  ASSERT_GE(src_times.first, src_times.second + 1000);
+
+  Checkpoint* checkpoint;
+  ASSERT_OK(Checkpoint::Create(db_.get(), &checkpoint));
+  ASSERT_OK(checkpoint->ExportColumnFamily(handles_[1], export_files_dir_,
+                                           &metadata_ptr_));
+  delete checkpoint;
+  ASSERT_EQ(metadata_ptr_->files.size(), 1U);
+  EXPECT_EQ(metadata_ptr_->files[0].file_creation_time, src_times.first);
+  EXPECT_EQ(metadata_ptr_->files[0].oldest_ancestor_time, src_times.second);
+  // The source DB's ownership of the file is also exported...
+  EXPECT_EQ(metadata_ptr_->files[0].db_ownership_start_time, src_times.first);
+  EXPECT_EQ(DataFileProvenance::DecodeFrom(
+                metadata_ptr_->files[0].data_file_provenance)
+                .origin,
+            DataFileProvenance::Origin::kCompaction);
+
+  env_->MockSleepForSeconds(1000);
+  int64_t import_start = 0;
+  ASSERT_OK(env_->GetCurrentTime(&import_start));
+  ASSERT_OK(db_->CreateColumnFamilyWithImport(options, "toto",
+                                              ImportColumnFamilyOptions(),
+                                              *metadata_ptr_, &import_cfh_));
+  EXPECT_EQ(ManifestTimes(dbfull(), import_cfh_, "a"), src_times);
+  // ...but this DB began holding the file when importing it
+  const FileMetaData imported = ManifestRecord(dbfull(), import_cfh_, "a");
+  EXPECT_GE(imported.db_ownership_start_time,
+            static_cast<uint64_t>(import_start));
+  DataFileProvenance expected_info;
+  expected_info.origin = DataFileProvenance::Origin::kImport;
+  EXPECT_EQ(DataFileProvenance::DecodeFrom(imported.data_file_provenance),
+            expected_info);
+}
+
+TEST_F(ImportColumnFamilyTest, ImportFileTimesAndRejectTooOldFile) {
+  Options options = CurrentOptions();
+  options.disable_auto_compactions = true;
+  options.periodic_compaction_seconds = 10000;
+  env_->SetMockSleep();
+  CreateAndReopenWithCF({"koko"}, options);
+
+  int64_t start = 0;
+  ASSERT_OK(env_->GetCurrentTime(&start));
+  const uint64_t now = static_cast<uint64_t>(start);
+  // Writes `key`.sst with SstFileWriter, recording creation time `write_time`
+  // (0 = unknown) in its table properties
+  auto write_file = [&](const std::string& key, uint64_t write_time) {
+    auto clock = std::make_shared<MockSystemClock>(env_->GetSystemClock());
+    clock->SetCurrentTime(write_time);
+    CompositeEnvWrapper writer_env(env_, clock);
+    Options writer_options = options;
+    writer_options.env = &writer_env;
+    SstFileWriter writer(EnvOptions(), writer_options);
+    ASSERT_OK(writer.Open(sst_files_dir_ + key + ".sst"));
+    ASSERT_OK(writer.Put(key, "val"));
+    ASSERT_OK(writer.Finish());
+  };
+  write_file("a", now);
+  write_file("b", now);
+  write_file("c", 0);
+  write_file("d", now - 10100);
+  // Times from the import metadata take precedence over the file's
+  LiveFileMetaData too_old =
+      LiveFileMetaDataInit("a.sst", sst_files_dir_, 0, 10, 19);
+  too_old.file_creation_time = now - 10100;
+  LiveFileMetaData future =
+      LiveFileMetaDataInit("b.sst", sst_files_dir_, 0, 20, 29);
+  future.file_creation_time = now + 5000;
+  future.oldest_ancestor_time = now + 5000;
+  // Neither the metadata nor the file has a time
+  LiveFileMetaData unknown_time =
+      LiveFileMetaDataInit("c.sst", sst_files_dir_, 0, 30, 39);
+  // Only the file has a time, which is too old
+  LiveFileMetaData too_old_in_file =
+      LiveFileMetaDataInit("d.sst", sst_files_dir_, 0, 40, 49);
+
+  ExportImportFilesMetaData metadata;
+  metadata.db_comparator_name = options.comparator->Name();
+  ImportColumnFamilyOptions import_options;
+  import_options.reject_too_old_file = true;
+
+  for (const LiveFileMetaData& rejected : {too_old, too_old_in_file}) {
+    SCOPED_TRACE(rejected.name);
+    metadata.files = {future, unknown_time, rejected};
+    Status s = db_->CreateColumnFamilyWithImport(
+        options, "toto", import_options, metadata, &import_cfh_);
+    ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+    ASSERT_EQ(import_cfh_, nullptr);
+  }
+
+  // Future and unknown times are replaced with the import time
+  metadata.files = {future, unknown_time};
+  ASSERT_OK(db_->CreateColumnFamilyWithImport(options, "toto", import_options,
+                                              metadata, &import_cfh_));
+  int64_t end = 0;
+  ASSERT_OK(env_->GetCurrentTime(&end));
+  for (const char* key : {"b", "c"}) {
+    SCOPED_TRACE(key);
+    const auto times = ManifestTimes(dbfull(), import_cfh_, key);
+    EXPECT_GE(times.first, now);
+    EXPECT_LE(times.first, static_cast<uint64_t>(end));
+    EXPECT_EQ(times.second, times.first);
+  }
+
+  // Without the option, too-old files are accepted with their times intact.
+  // An unknown oldest ancestor time falls back on the file creation time.
+  import_options.reject_too_old_file = false;
+  metadata.files = {too_old, too_old_in_file};
+  ASSERT_OK(db_->CreateColumnFamilyWithImport(options, "yoyo", import_options,
+                                              metadata, &import_cfh2_));
+  for (const char* key : {"a", "d"}) {
+    SCOPED_TRACE(key);
+    EXPECT_EQ(ManifestTimes(dbfull(), import_cfh2_, key),
+              std::make_pair(now - 10100, now - 10100));
+  }
+}
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/repair.cc

@@ -226,14 +226,14 @@ class Repairer {
     if (status.ok()) {
       // Need to scan existing SST files first so the column families are
       // created before we process WAL files
-      ExtractMetaData();
+      ExtractMetaData(/*tables_built_from_wals=*/false);
 
       // ExtractMetaData() uses table_fds_ to know which SST files' metadata to
       // extract -- we need to clear it here since metadata for existing SST
       // files has been extracted already
       table_fds_.clear();
       ConvertLogFilesToTables();
-      ExtractMetaData();
+      ExtractMetaData(/*tables_built_from_wals=*/true);
       status = AddTables();
     }
     if (status.ok()) {
@@ -487,8 +487,8 @@ class Repairer {
           default_compression, cfd->GetID(), cfd->GetName(), -1 /* level */,
           current_time /* newest_key_time */, false /* is_bottommost */,
           TableFileCreationReason::kRecovery, 0 /* oldest_key_time */,
-          0 /* file_creation_time */, "DB Repairer" /* db_id */, db_session_id_,
-          0 /*target_file_size*/, meta.fd.GetNumber(),
+          "DB Repairer" /* db_id */, db_session_id_, 0 /*target_file_size*/,
+          meta.fd.GetNumber(),
           kMaxSequenceNumber /*last_level_inclusive_max_seqno_threshold*/,
           dbname_ /*db_name*/);
 
@@ -523,7 +523,9 @@ class Repairer {
     return status;
   }
 
-  void ExtractMetaData() {
+  // `tables_built_from_wals`: whether table_fds_ are tables this repair built
+  // from WAL files, rather than tables that existed before the repair.
+  void ExtractMetaData(bool tables_built_from_wals) {
     for (size_t i = 0; i < table_fds_.size(); i++) {
       TableInfo t;
       t.meta.fd = table_fds_[i];
@@ -538,6 +540,14 @@ class Repairer {
                        file_num_buf, status.ToString().c_str());
         ArchiveFile(fname);
       } else {
+        // The MANIFEST being rebuilt held the ownership start time and origin
+        // of tables that existed before the repair, so those are unknown.
+        t.meta.data_file_provenance = kProvenanceRecordedByRepair;
+        if (tables_built_from_wals) {
+          t.meta.data_file_provenance |=
+              DataFileProvenanceForOrigin(DataFileProvenance::Origin::kFlush);
+          t.meta.db_ownership_start_time = t.meta.file_creation_time;
+        }
         tables_.push_back(t);
       }
     }
@@ -586,6 +596,7 @@ class Repairer {
             AddColumnFamily(props->column_family_name, t->column_family_id);
       }
       t->meta.oldest_ancestor_time = props->oldest_ancestor_time;
+      t->meta.file_creation_time = props->file_creation_time;
       t->meta.user_defined_timestamps_persisted =
           static_cast<bool>(props->user_defined_timestamps_persisted);
     }
@@ -717,17 +728,7 @@ class Repairer {
       VersionEdit dummy_edit;
       for (const auto* table : cf_id_and_tables.second) {
         // TODO(opt): separate out into multiple levels
-        const auto& meta = table->meta;
-        dummy_edit.AddFile(
-            0, meta.fd.GetNumber(), meta.fd.GetPathId(), meta.fd.GetFileSize(),
-            meta.smallest, meta.largest, meta.fd.smallest_seqno,
-            meta.fd.largest_seqno, meta.marked_for_compaction, meta.temperature,
-            meta.oldest_blob_file_number, meta.oldest_ancestor_time,
-            meta.file_creation_time, meta.epoch_number, meta.file_checksum,
-            meta.file_checksum_func_name, meta.unique_id,
-            meta.compensated_range_deletion_size, meta.tail_size,
-            meta.user_defined_timestamps_persisted, meta.min_timestamp,
-            meta.max_timestamp);
+        dummy_edit.AddFile(0, table->meta);
       }
       s = dummy_version_builder.Apply(&dummy_edit);
       if (s.ok()) {

db/version_edit.cc

@@ -21,7 +21,79 @@
 
 namespace ROCKSDB_NAMESPACE {
 
-namespace {}  // anonymous namespace
+namespace {
+// For DebugString() / DebugJSON(), e.g. "ingestion+moved+raw:0x1103" (the raw
+// value shows reserved bits that aren't decoded).
+std::string DataFileProvenanceToString(uint32_t encoded) {
+  const DataFileProvenance info = DataFileProvenance::DecodeFrom(encoded);
+  std::string r;
+  switch (info.origin) {
+    case DataFileProvenance::Origin::kUnknown:
+      r = "unknown_origin";
+      break;
+    case DataFileProvenance::Origin::kFlush:
+      r = "flush";
+      break;
+    case DataFileProvenance::Origin::kCompaction:
+      r = "compaction";
+      break;
+    case DataFileProvenance::Origin::kIngestion:
+      r = "ingestion";
+      break;
+    case DataFileProvenance::Origin::kImport:
+      r = "import";
+      break;
+  }
+  if (info.moved_between_levels) {
+    r += "+moved";
+  }
+  if (info.temperature_changed_by_copy) {
+    r += "+temperature_copied";
+  }
+  if (info.temperature_updated_from_file_system) {
+    r += "+temperature_updated";
+  }
+  if (info.recorded_by_repair) {
+    r += "+repaired";
+  }
+  char raw[24];
+  snprintf(raw, sizeof(raw), "+raw:0x%x", encoded);
+  r += raw;
+  return r;
+}
+}  // anonymous namespace
+
+uint32_t DataFileProvenance::Encode() const {
+  uint32_t encoded = DataFileProvenanceForOrigin(origin);
+  if (moved_between_levels) {
+    encoded |= kProvenanceMovedBetweenLevels;
+  }
+  if (temperature_changed_by_copy) {
+    encoded |= kProvenanceTemperatureChangedByCopy;
+  }
+  if (temperature_updated_from_file_system) {
+    encoded |= kProvenanceTemperatureUpdatedFromFileSystem;
+  }
+  if (recorded_by_repair) {
+    encoded |= kProvenanceRecordedByRepair;
+  }
+  return encoded;
+}
+
+DataFileProvenance DataFileProvenance::DecodeFrom(uint32_t encoded) {
+  DataFileProvenance info;
+  const uint32_t origin_value = encoded & kProvenanceOriginMask;
+  if (origin_value <= DataFileProvenanceForOrigin(Origin::kImport)) {
+    info.origin = static_cast<Origin>(origin_value);
+  }
+  info.moved_between_levels = (encoded & kProvenanceMovedBetweenLevels) != 0;
+  info.temperature_changed_by_copy =
+      (encoded & kProvenanceTemperatureChangedByCopy) != 0;
+  info.temperature_updated_from_file_system =
+      (encoded & kProvenanceTemperatureUpdatedFromFileSystem) != 0;
+  info.recorded_by_repair = (encoded & kProvenanceRecordedByRepair) != 0;
+  return info;
+}
 
 // After copying, we rely on the caller to ensure no double releases. Fragile,
 // but keeps copying cheap.
@@ -126,6 +198,22 @@ Status FileMetaData::UpdateBoundaries(const Slice& key, const Slice& value,
 
 void VersionEdit::Clear() { *this = VersionEdit(); }
 
+void VersionEdit::AddExistingFile(int level, const FileMetaData& f,
+                                  uint32_t provenance_flags_to_set) {
+  assert((provenance_flags_to_set & kProvenanceOriginMask) == 0);
+  AddFile(level, f.fd.GetNumber(), f.fd.GetPathId(), f.fd.GetFileSize(),
+          f.smallest, f.largest, f.fd.smallest_seqno, f.fd.largest_seqno,
+          f.marked_for_compaction, f.temperature, f.oldest_blob_file_number,
+          f.oldest_ancestor_time, f.file_creation_time, f.epoch_number,
+          f.file_checksum, f.file_checksum_func_name, f.unique_id,
+          f.compensated_range_deletion_size, f.tail_size,
+          f.user_defined_timestamps_persisted, f.min_timestamp, f.max_timestamp,
+          f.file_open_metadata);
+  FileMetaData& added = new_files_.back().second;
+  added.db_ownership_start_time = f.db_ownership_start_time;
+  added.data_file_provenance = f.data_file_provenance | provenance_flags_to_set;
+}
+
 bool VersionEdit::ShouldEmitPerColumnFamilyRecoveryEdit(
     uint64_t current_log_number) const {
   return (HasLogNumber() && GetLogNumber() > current_log_number) ||
@@ -407,6 +495,18 @@ void VersionEdit::EncodeToNewFile4(const FileMetaData& f, int level,
     PutVarint32(dst, NewFileCustomTag::kFileOpenMetadata);
     PutLengthPrefixedSlice(dst, Slice(f.file_open_metadata));
   }
+  if (f.db_ownership_start_time != kUnknownDbOwnershipStartTime) {
+    PutVarint32(dst, NewFileCustomTag::kDbOwnershipStartTime);
+    std::string varint_db_ownership_start_time;
+    PutVarint64(&varint_db_ownership_start_time, f.db_ownership_start_time);
+    PutLengthPrefixedSlice(dst, Slice(varint_db_ownership_start_time));
+  }
+  if (f.data_file_provenance != 0) {
+    PutVarint32(dst, NewFileCustomTag::kDataFileProvenance);
+    std::string varint_data_file_provenance;
+    PutVarint32(&varint_data_file_provenance, f.data_file_provenance);
+    PutLengthPrefixedSlice(dst, Slice(varint_data_file_provenance));
+  }
   TEST_SYNC_POINT_CALLBACK("VersionEdit::EncodeTo:NewFile4:CustomizeFields",
                            dst);
 
@@ -555,6 +655,16 @@ const char* VersionEdit::DecodeNewFile4From(Slice* input, int& max_level,
         case kFileOpenMetadata:
           f.file_open_metadata = field.ToString();
           break;
+        case kDbOwnershipStartTime:
+          if (!GetVarint64(&field, &f.db_ownership_start_time)) {
+            return "invalid DB ownership start time";
+          }
+          break;
+        case kDataFileProvenance:
+          if (!GetVarint32(&field, &f.data_file_provenance)) {
+            return "invalid data file provenance";
+          }
+          break;
         default:
           if ((custom_tag & kCustomTagNonSafeIgnoreMask) != 0) {
             // Should not proceed if cannot understand it
@@ -1049,6 +1159,14 @@ std::string VersionEdit::DebugString(bool hex_key) const {
     AppendNumberTo(&r, f.file_creation_time);
     r.append(" epoch_number:");
     AppendNumberTo(&r, f.epoch_number);
+    if (f.db_ownership_start_time != kUnknownDbOwnershipStartTime) {
+      r.append(" db_ownership_start_time:");
+      AppendNumberTo(&r, f.db_ownership_start_time);
+    }
+    if (f.data_file_provenance != 0) {
+      r.append(" data_file_provenance:");
+      r.append(DataFileProvenanceToString(f.data_file_provenance));
+    }
     r.append(" file_checksum:");
     r.append(Slice(f.file_checksum).ToString(true));
     r.append(" file_checksum_func_name: ");
@@ -1190,6 +1308,13 @@ std::string VersionEdit::DebugJSON(int edit_num, bool hex_key) const {
       jw << "OldestAncesterTime" << f.oldest_ancestor_time;
       jw << "FileCreationTime" << f.file_creation_time;
       jw << "EpochNumber" << f.epoch_number;
+      if (f.db_ownership_start_time != kUnknownDbOwnershipStartTime) {
+        jw << "DbOwnershipStartTime" << f.db_ownership_start_time;
+      }
+      if (f.data_file_provenance != 0) {
+        jw << "DataFileProvenance"
+           << DataFileProvenanceToString(f.data_file_provenance);
+      }
       jw << "FileChecksum" << Slice(f.file_checksum).ToString(true);
       jw << "FileChecksumFuncName" << f.file_checksum_func_name;
       if (f.temperature != Temperature::kUnknown) {

db/version_edit.h

@@ -23,10 +23,12 @@
 #include "port/malloc.h"
 #include "rocksdb/advanced_cache.h"
 #include "rocksdb/advanced_options.h"
+#include "rocksdb/metadata.h"
 #include "table/table_reader.h"
 #include "table/unique_id_impl.h"
 #include "test_util/sync_point.h"
 #include "util/autovector.h"
+#include "util/cast_util.h"
 
 namespace ROCKSDB_NAMESPACE {
 
@@ -116,6 +118,8 @@ enum NewFileCustomTag : uint32_t {
   kTailSize = 15,
   kUserDefinedTimestampsPersisted = 16,
   kFileOpenMetadata = 17,
+  kDbOwnershipStartTime = 18,
+  kDataFileProvenance = 19,
 
   // If this bit for the custom tag is set, opening DB should fail if
   // we don't know this field.
@@ -131,11 +135,33 @@ constexpr uint64_t kFileNumberMask = 0x3FFFFFFFFFFFFFFF;
 constexpr uint64_t kUnknownOldestAncestorTime = 0;
 constexpr uint64_t kUnknownNewestKeyTime = 0;
 constexpr uint64_t kUnknownFileCreationTime = 0;
+constexpr uint64_t kUnknownDbOwnershipStartTime = 0;
 constexpr uint64_t kUnknownEpochNumber = 0;
 // If `Options::cf_allow_ingest_behind` is true, this epoch number
 // will be dedicated to files ingested behind.
 constexpr uint64_t kReservedEpochNumberForFileIngestedBehind = 1;
 
+// Bit layout of FileMetaData::data_file_provenance (decoded form:
+// DataFileProvenance), LSB first:
+//  * bits 0-7: DataFileProvenance::Origin (values above kImport reserved)
+//  * bits 8-11: the flags below
+//  * bits 12-31: reserved
+// Readers ignore bits they don't recognize, and code carrying a file's record
+// over (moves, MANIFEST rewrites, copies) preserves them, by updating the
+// packed value rather than re-encoding a decoded one.
+constexpr uint32_t kProvenanceOriginMask = 0xff;
+constexpr uint32_t kProvenanceMovedBetweenLevels = uint32_t{1} << 8;
+constexpr uint32_t kProvenanceTemperatureChangedByCopy = uint32_t{1} << 9;
+constexpr uint32_t kProvenanceTemperatureUpdatedFromFileSystem = uint32_t{1}
+                                                                 << 10;
+constexpr uint32_t kProvenanceRecordedByRepair = uint32_t{1} << 11;
+
+// Packed data_file_provenance for a file this DB got via `origin`, with no
+// flags.
+inline uint32_t DataFileProvenanceForOrigin(DataFileProvenance::Origin origin) {
+  return lossless_cast<uint32_t>(origin);
+}
+
 uint64_t PackFileNumberAndPathId(uint64_t number, uint64_t path_id);
 
 // PinnedTableReader is used to safely access a table reader in a multi-threaded
@@ -280,6 +306,12 @@ struct FileMetaData {
                                        // compact this file.
   Temperature temperature = Temperature::kUnknown;
 
+  // How this DB got the file's contents and what has happened to the file
+  // since, packed as described by DataFileProvenance
+  // (include/rocksdb/metadata.h). 0 means unknown. Persisted in the MANIFEST
+  // only.
+  uint32_t data_file_provenance = 0;
+
   // Used only in BlobDB. The file number of the oldest blob file this SST file
   // refers to. 0 is an invalid value; BlobDB numbers the files starting from 1.
   uint64_t oldest_blob_file_number = kInvalidBlobFileNumber;
@@ -296,9 +328,16 @@ struct FileMetaData {
   // 0 means the information is not available.
   uint64_t oldest_ancestor_time = kUnknownOldestAncestorTime;
 
-  // Unix time when the SST file is created.
+  // Unix time when the SST file's contents were built (by flush or
+  // compaction, in this DB or another, or by SstFileWriter). Preserved when
+  // the file is moved between levels or copied to another temperature.
   uint64_t file_creation_time = kUnknownFileCreationTime;
 
+  // Unix time when this DB began holding this physical file (this file
+  // number); see SstFileMetaData::db_ownership_start_time. Persisted in the
+  // MANIFEST only. kUnknownDbOwnershipStartTime if unknown.
+  uint64_t db_ownership_start_time = kUnknownDbOwnershipStartTime;
+
   // The order of a file being flushed or ingested/imported.
   // Compaction output file will be assigned with the minimum `epoch_number`
   // among input files'.
@@ -422,6 +461,18 @@ struct FileMetaData {
     return kUnknownFileCreationTime;
   }
 
+  // When this DB began holding this file, if recorded. Otherwise falls back on
+  // TryGetFileCreationTime(): for files recorded by older RocksDB versions,
+  // that was the time of ingestion, import or temperature-change copy for
+  // files that arrived that way, and is the creation time for files built by
+  // this DB.
+  uint64_t TryGetDbOwnershipStartTime() {
+    if (db_ownership_start_time != kUnknownDbOwnershipStartTime) {
+      return db_ownership_start_time;
+    }
+    return TryGetFileCreationTime();
+  }
+
   // Tries to get the newest key time from the current file
   // Falls back on oldest ancestor time of previous (newer) file
   uint64_t TryGetNewestKeyTime(FileMetaData* prev_file = nullptr) {
@@ -868,6 +919,15 @@ class VersionEdit {
     }
   }
 
+  // Add a file that is already in the LSM under the same file number, such as
+  // a file being moved to another level (paired with DeleteFile()) or recorded
+  // in a new MANIFEST, preserving all of its persisted metadata. In-memory-only
+  // state (refs, being_compacted, sampled stats, pinned table reader) starts
+  // fresh. `provenance_flags_to_set` are DataFileProvenance flag bits to set
+  // on the added file, recording what is happening to it (0 for none).
+  void AddExistingFile(int level, const FileMetaData& f,
+                       uint32_t provenance_flags_to_set);
+
   const NewFiles& GetNewFiles() const { return new_files_; }
 
   NewFiles& GetMutableNewFiles() { return new_files_; }

db/version_edit_test.cc

@@ -1300,6 +1300,170 @@ TEST_F(VersionEditTest, FileOpenMetadataInApproximateMemoryUsage) {
   ASSERT_GE(with_metadata_usage, base_usage + 500);
 }
 
+namespace {
+// Adds a file with no DB ownership fields set
+void AddTestFile(VersionEdit* edit, uint64_t file_number) {
+  static const uint64_t kBig = 1ull << 50;
+  edit->AddFile(
+      3, file_number, 0, 100, InternalKey("foo", kBig + 500, kTypeValue),
+      InternalKey("zoo", kBig + 600, kTypeDeletion), kBig + 500, kBig + 600,
+      false, Temperature::kUnknown, kInvalidBlobFileNumber,
+      kUnknownOldestAncestorTime, kUnknownFileCreationTime,
+      file_number /* epoch_number */, kUnknownFileChecksum,
+      kUnknownFileChecksumFuncName, kNullUniqueId64x2, 0, 0, true);
+}
+}  // namespace
+
+TEST_F(VersionEditTest, DbOwnershipEncodeDecode) {
+  // Includes a reserved bit, which must be preserved
+  const uint32_t kInfo =
+      DataFileProvenanceForOrigin(DataFileProvenance::Origin::kIngestion) |
+      kProvenanceMovedBetweenLevels | (uint32_t{1} << 20);
+  VersionEdit edit;
+  AddTestFile(&edit, 300);
+  AddTestFile(&edit, 301);
+  auto& new_files = edit.GetMutableNewFiles();
+  new_files[0].second.db_ownership_start_time = 1234567;
+  new_files[0].second.data_file_provenance = kInfo;
+  TestEncodeDecode(edit);
+
+  std::string encoded;
+  edit.EncodeTo(&encoded, 0 /* ts_sz */);
+  VersionEdit parsed;
+  ASSERT_OK(parsed.DecodeFrom(encoded));
+  const auto& parsed_files = parsed.GetNewFiles();
+  ASSERT_EQ(parsed_files.size(), 2u);
+  EXPECT_EQ(parsed_files[0].second.db_ownership_start_time, 1234567u);
+  EXPECT_EQ(parsed_files[0].second.data_file_provenance, kInfo);
+  EXPECT_EQ(parsed_files[1].second.db_ownership_start_time,
+            kUnknownDbOwnershipStartTime);
+  EXPECT_EQ(parsed_files[1].second.data_file_provenance, 0u);
+
+  // Fields are only written when known. (If written for 0, setting a small
+  // value would not change the encoded size.)
+  for (bool set_info : {false, true}) {
+    VersionEdit unknown_edit;
+    AddTestFile(&unknown_edit, 302);
+    std::string unknown_encoded;
+    unknown_edit.EncodeTo(&unknown_encoded, 0 /* ts_sz */);
+    if (set_info) {
+      unknown_edit.GetMutableNewFiles()[0].second.data_file_provenance = 1;
+    } else {
+      unknown_edit.GetMutableNewFiles()[0].second.db_ownership_start_time = 1;
+    }
+    std::string known_encoded;
+    unknown_edit.EncodeTo(&known_encoded, 0 /* ts_sz */);
+    // Tag, length, and one-byte value
+    EXPECT_EQ(known_encoded.size(), unknown_encoded.size() + 3);
+  }
+
+  // Older versions safely ignore the fields
+  EXPECT_EQ(0u, NewFileCustomTag::kDbOwnershipStartTime &
+                    NewFileCustomTag::kCustomTagNonSafeIgnoreMask);
+  EXPECT_EQ(0u, NewFileCustomTag::kDataFileProvenance &
+                    NewFileCustomTag::kCustomTagNonSafeIgnoreMask);
+
+  const std::string debug = edit.DebugString();
+  EXPECT_NE(debug.find(" db_ownership_start_time:1234567 "), std::string::npos)
+      << debug;
+  EXPECT_NE(debug.find(" data_file_provenance:ingestion+moved+raw:0x100103 "),
+            std::string::npos)
+      << debug;
+  const std::string json = edit.DebugJSON(0 /* edit_num */);
+  EXPECT_NE(json.find("\"DbOwnershipStartTime\": 1234567"), std::string::npos)
+      << json;
+  EXPECT_NE(
+      json.find("\"DataFileProvenance\": \"ingestion+moved+raw:0x100103\""),
+      std::string::npos)
+      << json;
+}
+
+TEST_F(VersionEditTest, DataFileProvenanceEncoding) {
+  using Origin = DataFileProvenance::Origin;
+  // The encoding is persisted, so must not change
+  const std::pair<Origin, uint32_t> kOrigins[] = {{Origin::kUnknown, 0},
+                                                  {Origin::kFlush, 1},
+                                                  {Origin::kCompaction, 2},
+                                                  {Origin::kIngestion, 3},
+                                                  {Origin::kImport, 4}};
+  for (const auto& [origin, origin_value] : kOrigins) {
+    for (uint32_t flags = 0; flags < 16; ++flags) {
+      SCOPED_TRACE("origin=" + std::to_string(origin_value) +
+                   " flags=" + std::to_string(flags));
+      DataFileProvenance info;
+      info.origin = origin;
+      info.moved_between_levels = (flags & 1) != 0;
+      info.temperature_changed_by_copy = (flags & 2) != 0;
+      info.temperature_updated_from_file_system = (flags & 4) != 0;
+      info.recorded_by_repair = (flags & 8) != 0;
+      const uint32_t encoded = info.Encode();
+      EXPECT_EQ(encoded, origin_value | (flags << 8));
+      EXPECT_EQ(DataFileProvenance::DecodeFrom(encoded), info);
+      // Reserved bits are ignored
+      EXPECT_EQ(DataFileProvenance::DecodeFrom(encoded | 0xfffff000U), info);
+    }
+  }
+  // Unrecognized origins decode as unknown, without affecting flags
+  const DataFileProvenance decoded =
+      DataFileProvenance::DecodeFrom(0x7f | kProvenanceRecordedByRepair);
+  EXPECT_EQ(decoded.origin, Origin::kUnknown);
+  EXPECT_TRUE(decoded.recorded_by_repair);
+  EXPECT_FALSE(decoded.moved_between_levels);
+}
+
+TEST_F(VersionEditTest, AddExistingFile) {
+  FileMetaData f(300, 0, 100, InternalKey("foo", 500, kTypeValue),
+                 InternalKey("zoo", 600, kTypeValue), 500, 600,
+                 true /* marked_for_compact */, Temperature::kWarm,
+                 7 /* oldest_blob_file */, 1000 /* oldest_ancestor_time */,
+                 2000 /* file_creation_time */, 3 /* epoch_number */,
+                 "checksum", "checksum_func", UniqueId64x2{11, 12},
+                 10 /* compensated_range_deletion_size */, 20 /* tail_size */,
+                 false /* user_defined_timestamps_persisted */, "min_ts",
+                 "max_ts");
+  f.file_open_metadata = "open_metadata";
+  f.db_ownership_start_time = 2500;
+  f.data_file_provenance =
+      DataFileProvenanceForOrigin(DataFileProvenance::Origin::kIngestion) |
+      kProvenanceTemperatureChangedByCopy | (uint32_t{1} << 20);
+  f.refs = 5;
+  f.being_compacted = true;
+
+  VersionEdit edit;
+  edit.AddExistingFile(4, f, kProvenanceMovedBetweenLevels);
+  ASSERT_EQ(edit.GetNewFiles().size(), 1u);
+  EXPECT_EQ(edit.GetNewFiles()[0].first, 4);
+  const FileMetaData& added = edit.GetNewFiles()[0].second;
+
+  // Persisted metadata is preserved; all of it is covered by comparing the
+  // encoded MANIFEST record, once the added flag is accounted for
+  VersionEdit expected_edit;
+  expected_edit.AddFile(4, f);
+  expected_edit.GetMutableNewFiles()[0].second.data_file_provenance |=
+      kProvenanceMovedBetweenLevels;
+  std::string expected_encoded;
+  expected_edit.EncodeTo(&expected_encoded, 0 /* ts_sz */);
+  std::string encoded;
+  edit.EncodeTo(&encoded, 0 /* ts_sz */);
+  EXPECT_EQ(encoded, expected_encoded);
+  EXPECT_EQ(added.db_ownership_start_time, 2500u);
+  EXPECT_EQ(added.data_file_provenance,
+            f.data_file_provenance | kProvenanceMovedBetweenLevels);
+
+  // In-memory-only state starts fresh
+  EXPECT_EQ(added.refs, 0);
+  EXPECT_FALSE(added.being_compacted);
+}
+
+TEST_F(VersionEditTest, TryGetDbOwnershipStartTime) {
+  FileMetaData f;
+  f.file_creation_time = 2000;
+  // Falls back on the file creation time when unknown
+  EXPECT_EQ(f.TryGetDbOwnershipStartTime(), 2000u);
+  f.db_ownership_start_time = 2500;
+  EXPECT_EQ(f.TryGetDbOwnershipStartTime(), 2500u);
+}
+
 }  // namespace ROCKSDB_NAMESPACE
 
 int main(int argc, char** argv) {

db/version_set.cc

@@ -2247,6 +2247,8 @@ void Version::GetColumnFamilyMetaData(ColumnFamilyMetaData* cf_meta) {
           file->file_checksum, file->file_checksum_func_name);
       files.back().num_entries = file->num_entries;
       files.back().num_deletions = file->num_deletions;
+      files.back().db_ownership_start_time = file->db_ownership_start_time;
+      files.back().data_file_provenance = file->data_file_provenance;
       files.back().smallest = file->smallest.Encode().ToString();
       files.back().largest = file->largest.Encode().ToString();
       level_size += file->fd.GetFileSize();
@@ -2330,6 +2332,8 @@ void Version::GetColumnFamilyMetaData(
           file->file_checksum, file->file_checksum_func_name);
       files.back().num_entries = file->num_entries;
       files.back().num_deletions = file->num_deletions;
+      files.back().db_ownership_start_time = file->db_ownership_start_time;
+      files.back().data_file_provenance = file->data_file_provenance;
       files.back().smallest = file->smallest.Encode().ToString();
       files.back().largest = file->largest.Encode().ToString();
       level_size += file->fd.GetFileSize();
@@ -4578,9 +4582,11 @@ void VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction(
   if (allow_ingest_behind) {
     return;
   }
-  // If a file's creation time is larger than creation_time_ub,
-  // it is too new to be marked for compaction.
-  int64_t creation_time_ub = 0;
+  // If this DB began holding a file later than ownership_start_time_ub, it is
+  // too new to be marked for compaction. (Measuring from when the DB took the
+  // file rather than from file creation means a file ingested with old
+  // contents still gets the delay.)
+  int64_t ownership_start_time_ub = 0;
   bool needs_delay = bottommost_file_compaction_delay_ > 0;
   if (needs_delay) {
     int64_t current_time = 0;
@@ -4589,7 +4595,7 @@ void VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction(
     // We will treat it as is and treat all files as too new.
     // The subtraction will not underflow since
     // bottommost_file_compaction_delay_ is of type uint32_t.
-    creation_time_ub =
+    ownership_start_time_ub =
         current_time - static_cast<int64_t>(bottommost_file_compaction_delay_);
   }
 
@@ -4646,11 +4652,11 @@ void VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction(
 
         if (!needs_delay) {
           bottommost_files_marked_for_compaction_.push_back(level_and_file);
-        } else if (creation_time_ub > 0) {
-          int64_t creation_time = static_cast<int64_t>(
-              level_and_file.second->TryGetFileCreationTime());
-          if (creation_time == kUnknownFileCreationTime ||
-              creation_time <= creation_time_ub) {
+        } else if (ownership_start_time_ub > 0) {
+          int64_t ownership_start_time = static_cast<int64_t>(
+              level_and_file.second->TryGetDbOwnershipStartTime());
+          if (ownership_start_time == kUnknownDbOwnershipStartTime ||
+              ownership_start_time <= ownership_start_time_ub) {
             bottommost_files_marked_for_compaction_.push_back(level_and_file);
           } else {
             // Just ignore this file for both
@@ -4662,8 +4668,8 @@ void VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction(
             // bottommost_files_mark_threshold_ is released.
           }
         } else {
-          // creation_time_ub <= 0, all files are too new to be marked for
-          // compaction.
+          // ownership_start_time_ub <= 0, all files are too new to be marked
+          // for compaction.
         }
       } else {
         bottommost_files_mark_threshold_ =
@@ -8018,21 +8024,10 @@ Status VersionSet::WriteCurrentStateToManifest(
 
         for (const auto& f : level_files) {
           assert(f);
-          // Preserve file_open_metadata so live SSTs still benefit from
-          // fast_sst_open after a MANIFEST rotation. Without this, the
-          // rotated MANIFEST snapshot drops kFileOpenMetadata for unchanged
-          // files and their fast-open advantage is lost until they are
-          // rewritten.
-          edit.AddFile(level, f->fd.GetNumber(), f->fd.GetPathId(),
-                       f->fd.GetFileSize(), f->smallest, f->largest,
-                       f->fd.smallest_seqno, f->fd.largest_seqno,
-                       f->marked_for_compaction, f->temperature,
-                       f->oldest_blob_file_number, f->oldest_ancestor_time,
-                       f->file_creation_time, f->epoch_number, f->file_checksum,
-                       f->file_checksum_func_name, f->unique_id,
-                       f->compensated_range_deletion_size, f->tail_size,
-                       f->user_defined_timestamps_persisted, f->min_timestamp,
-                       f->max_timestamp, f->file_open_metadata);
+          // Preserves all persisted metadata, including file_open_metadata
+          // so live SSTs still benefit from fast_sst_open after a MANIFEST
+          // rotation.
+          edit.AddExistingFile(level, *f, /*provenance_flags_to_set=*/0);
         }
       }
 
@@ -8611,6 +8606,8 @@ void VersionSet::GetLiveFilesMetaData(std::vector<LiveFileMetaData>* metadata) {
         filemetadata.temperature = file->temperature;
         filemetadata.oldest_ancestor_time = file->TryGetOldestAncestorTime();
         filemetadata.file_creation_time = file->TryGetFileCreationTime();
+        filemetadata.db_ownership_start_time = file->db_ownership_start_time;
+        filemetadata.data_file_provenance = file->data_file_provenance;
         filemetadata.epoch_number = file->epoch_number;
         metadata->push_back(filemetadata);
       }

include/rocksdb/advanced_options.h

@@ -924,6 +924,12 @@ struct AdvancedColumnFamilyOptions {
   // non-zero values; if not, the age is based on the file's last modified time
   // (given by the underlying Env).
   //
+  // For a file added by IngestExternalFile() or CreateColumnFamilyWithImport(),
+  // the age is based on the creation time recorded in the file or import
+  // metadata (when known and not in the future), not on when it was added, so
+  // such a file can be compacted soon after it is added. See also
+  // IngestExternalFileOptions::reject_too_old_file.
+  //
   // This option only supports block based table format for any compaction
   // style.
   //
@@ -1362,7 +1368,10 @@ struct AdvancedColumnFamilyOptions {
   // "BottommostFiles". Usually such compaction can happen as soon as a
   // relevant snapshot is released. This option allows user to delay
   // such compactions. A file is qualified for "BottommostFiles" compaction
-  // if it is at least "bottommost_file_compaction_delay" seconds old.
+  // if this DB has held it for at least "bottommost_file_compaction_delay"
+  // seconds, i.e. since the DB started writing it or, for an ingested or
+  // imported file, since the file was added to the DB (see
+  // SstFileMetaData::db_ownership_start_time).
   //
   // Default: 0 (no delay)
   // Dynamically changeable through the SetOptions() API.

include/rocksdb/c.h

@@ -1555,7 +1555,7 @@ extern ROCKSDB_LIBRARY_API int rocksdb_compactionjobinfo_job_id(
     const rocksdb_compactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API unsigned char
-rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+rocksdb_compactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_compactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API int rocksdb_compactionjobinfo_num_l0_files(
@@ -1598,7 +1598,7 @@ extern ROCKSDB_LIBRARY_API int rocksdb_subcompactionjobinfo_job_id(
     const rocksdb_subcompactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API unsigned char
-rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+rocksdb_subcompactionjobinfo_separate_remote_compaction_pool_selected(
     const rocksdb_subcompactionjobinfo_t* info);
 
 extern ROCKSDB_LIBRARY_API int
@@ -2106,6 +2106,13 @@ extern ROCKSDB_LIBRARY_API uint64_t
 rocksdb_blob_file_garbage_info_garbage_blob_bytes(
     const rocksdb_blob_file_garbage_info_t* info);
 // END generated: c_generated_metadata_structs_auto.h.inc
+/* Compatibility aliases for the captured compaction pool selection. */
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_compactionjobinfo_t* info);
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_subcompactionjobinfo_t* info);
 extern ROCKSDB_LIBRARY_API rocksdb_compactionjobinfo_t*
 rocksdb_compactionjobinfo_create(void);
 extern ROCKSDB_LIBRARY_API void rocksdb_compactionjobinfo_destroy(
@@ -6008,6 +6015,14 @@ extern ROCKSDB_LIBRARY_API int
 rocksdb_ingestexternalfileoptions_get_file_opening_threads(
     rocksdb_ingestexternalfileoptions_t* opt);
 
+extern ROCKSDB_LIBRARY_API void
+rocksdb_ingestexternalfileoptions_set_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_ingestexternalfileoptions_get_reject_too_old_file(
+    rocksdb_ingestexternalfileoptions_t* opt);
+
 /* OpenAndCompactOptions */
 
 extern ROCKSDB_LIBRARY_API void
@@ -6028,6 +6043,14 @@ extern ROCKSDB_LIBRARY_API unsigned char
 rocksdb_import_column_family_options_get_move_files(
     rocksdb_import_column_family_options_t* opt);
 
+extern ROCKSDB_LIBRARY_API void
+rocksdb_import_column_family_options_set_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_import_column_family_options_get_reject_too_old_file(
+    rocksdb_import_column_family_options_t* opt);
+
 /* SizeApproximationOptions */
 
 extern ROCKSDB_LIBRARY_API void

include/rocksdb/db.h

@@ -192,6 +192,37 @@ class FileIngestionHandle {
   // returns a Status.
   virtual Status Abort() = 0;
 
+  // Metadata of the ingested files this handle will make visible on commit.
+  // The files exist from prepare until commit or abort and keep their path,
+  // number, size and checksum when committed, so a caller can read them
+  // before they go live. Empty once the handle is spent.
+  //
+  // Set fields:
+  //   directory
+  //   relative_filename
+  //   name
+  //   db_path
+  //   file_number
+  //   size
+  //   temperature
+  //   file_checksum
+  //   file_checksum_func_name
+  //   smallestkey
+  //   largestkey
+  //   num_entries
+  //   num_deletions
+  //   smallest_seqno, largest_seqno: only with allow_db_generated_files;
+  //     otherwise commit assigns them
+  //
+  // Left at defaults (commit decides them):
+  //   epoch_number
+  //   file_creation_time
+  //   oldest_ancestor_time
+  //
+  // Does not include the range-tombstone file that an atomic_replace_range
+  // ingestion may add at commit.
+  virtual std::vector<SstFileMetaData> GetPreparedFiles() const = 0;
+
   FileIngestionHandle(const FileIngestionHandle&) = delete;
   FileIngestionHandle& operator=(const FileIngestionHandle&) = delete;
   FileIngestionHandle(FileIngestionHandle&&) = delete;

include/rocksdb/external_table.h

@@ -576,6 +576,11 @@ class ExternalTableFactoryBase : public Customizable {
   // requires the corresponding builder and reader to implement
   // PutRangeDeletionBlock() and GetRangeDeletionBlock(), respectively.
   virtual bool IsDeleteRangeSupported() const { return false; }
+
+  // Returns whether NewTableBuilder() can write tables. A factory that only
+  // reads files written elsewhere returns false; see
+  // TableFactory::IsTableBuilderSupported() for what RocksDB then requires.
+  virtual bool IsTableBuilderSupported() const { return true; }
 };
 
 using ExternalTableFactory =

include/rocksdb/listener.h

@@ -408,7 +408,7 @@ struct SubcompactionJobInfo {
   int job_id;
 
   // The execution pool selected for the parent job.
-  bool use_separate_remote_compaction_pool = false;
+  bool separate_remote_compaction_pool_selected = false;
 
   // sub-compaction job id, which is only unique within the same compaction, so
   // use both 'job_id' and 'subcompaction_job_id' to identify a subcompaction
@@ -448,7 +448,7 @@ struct CompactionJobInfo {
 
   // The execution pool selected for this job, independent of later DB option
   // updates.
-  bool use_separate_remote_compaction_pool = false;
+  bool separate_remote_compaction_pool_selected = false;
 
   // the number of L0 files in the CF right before and after the compaction
   int num_l0_files;

include/rocksdb/metadata.h

@@ -69,6 +69,67 @@ struct LiveFileStorageInfo : public FileStorageInfo {
   bool trim_to_size = false;
 };
 
+// Decoded form of SstFileMetaData::data_file_provenance: how this DB got an SST
+// file's contents, and what has happened to the file since without its
+// contents being rewritten. Recorded in the MANIFEST (not in the file), since
+// the same file contents can be added to several DBs. Round-trips through
+// Encode() / DecodeFrom().
+struct DataFileProvenance {
+  // How this DB got the file's contents. Values may be added in the future;
+  // DecodeFrom() reports values it doesn't recognize as kUnknown.
+  enum class Origin : uint8_t {
+    // Not recorded, e.g. for a file recorded by an older RocksDB version or
+    // found by RepairDB.
+    kUnknown = 0,
+    // Written by a flush of this DB, including flushes of WAL data during DB
+    // open or RepairDB.
+    kFlush = 1,
+    // Written by a compaction of this DB, local or remote.
+    kCompaction = 2,
+    // Added by external file ingestion (one- or two-phase), whether the file
+    // was written by SstFileWriter or by another DB.
+    kIngestion = 3,
+    // Added by CreateColumnFamilyWithImport().
+    kImport = 4,
+  };
+  Origin origin = Origin::kUnknown;
+
+  // The following flags each record that something has happened to the file
+  // at least once without its contents being rewritten. Once set, a flag stays
+  // set when the file is moved or copied again.
+
+  // Moved to another level without being rewritten (e.g. trivial move, or
+  // CompactRange() with change_level).
+  bool moved_between_levels = false;
+  // Copied as-is to a new file with a different temperature, as FIFO
+  // compaction does with allow_trivial_copy_when_change_temperature. The new
+  // file's db_ownership_start_time is the time of the latest such copy.
+  bool temperature_changed_by_copy = false;
+  // The temperature recorded for the file was updated to match where the
+  // FileSystem reports the file to be (UpdateManifestForFilesState()), so
+  // the file was moved at a time unknown to RocksDB.
+  bool temperature_updated_from_file_system = false;
+  // Recorded by RepairDB, which rebuilds the MANIFEST. For files that existed
+  // before the repair, the origin and db_ownership_start_time are lost.
+  bool recorded_by_repair = false;
+
+  // Serialize to / deserialize from the packed
+  // SstFileMetaData::data_file_provenance representation. DecodeFrom() ignores
+  // unrecognized bits so the encoding can be extended in the future, which
+  // means re-encoding a decoded value drops any bits added by newer versions.
+  uint32_t Encode() const;
+  static DataFileProvenance DecodeFrom(uint32_t encoded);
+
+  bool operator==(const DataFileProvenance& other) const {
+    return origin == other.origin &&
+           moved_between_levels == other.moved_between_levels &&
+           temperature_changed_by_copy == other.temperature_changed_by_copy &&
+           temperature_updated_from_file_system ==
+               other.temperature_updated_from_file_system &&
+           recorded_by_repair == other.recorded_by_repair;
+  }
+};
+
 // The metadata that describes an SST file. (Does not need to extend
 // LiveFileStorageInfo because SST files are always immutable.)
 struct SstFileMetaData : public FileStorageInfo {
@@ -144,8 +205,11 @@ struct SstFileMetaData : public FileStorageInfo {
     // removed in a future release.
     uint64_t oldest_ancester_time;
   };
-  // Timestamp when the SST file is created, provided by
-  // SystemClock::GetCurrentTime(). 0 if the information is not available.
+  // Timestamp when the SST file's contents were built, provided by
+  // SystemClock::GetCurrentTime(). For an ingested or imported file, this is
+  // when it was built by SstFileWriter or by the DB it came from. Preserved
+  // when the file is moved between levels or copied to another temperature.
+  // 0 if the information is not available.
   uint64_t file_creation_time = 0;
   // The order of a file being flushed or ingested/imported.
   // Compaction output file will be assigned with the minimum `epoch_number`
@@ -153,6 +217,24 @@ struct SstFileMetaData : public FileStorageInfo {
   // For L0, larger `epoch_number` indicates newer L0 file.
   // 0 if the information is not available.
   uint64_t epoch_number = 0;
+  // Timestamp, provided by SystemClock::GetCurrentTime(), when this DB began
+  // holding this physical file (this file number): when the DB started
+  // writing it (flush or compaction output), when ingestion (in the prepare
+  // phase, for two-phase ingestion) or CreateColumnFamilyWithImport() linked
+  // or copied it into the DB, or when it was copied as a new file with a
+  // different temperature (see DataFileProvenance). Preserved when the file is
+  // moved between levels. A DB opened from a checkpoint or backup keeps the
+  // values recorded by the DB it was copied from. 0 if the information is not
+  // available, e.g. for a file recorded by an older RocksDB version (whose
+  // file_creation_time was the ingestion, import or copy time for files that
+  // arrived that way) or found by RepairDB.
+  uint64_t db_ownership_start_time = 0;
+  // How this DB got the file's contents and what has happened to the file
+  // since, packed as described by DataFileProvenance (decode with
+  // DataFileProvenance::DecodeFrom()). 0 if the information is not available.
+  // (CreateColumnFamilyWithImport() ignores this field and
+  // db_ownership_start_time in its input metadata, recording its own.)
+  uint32_t data_file_provenance = 0;
 
   // These bounds define the effective key range for range tombstones
   // in this file.

include/rocksdb/options.h

@@ -496,7 +496,7 @@ struct CompactionServiceJobInfo {
 
   // The execution pool selected for this job, independent of later DB option
   // updates.
-  bool use_separate_remote_compaction_pool = false;
+  bool separate_remote_compaction_pool_selected = false;
 
   // True if the compaction filter uses
   // CompactionFilter::MergeOrder::kMergeBeforeFilter, so the worker must
@@ -3294,6 +3294,28 @@ struct IngestExternalFileOptions {
   // multiple files at once.
   int file_opening_threads = 1;
 
+  // If true, ingestion fails with Status::InvalidArgument (and no files are
+  // ingested) if any file is already too old for the column family to keep
+  // without recompaction or deletion. This avoids publishing data that, for
+  // example, should already have been processed by a compaction filter. A
+  // file is too old if
+  // * for leveled or universal compaction, the age of its contents, from the
+  //   creation time recorded in the file (see `periodic_compaction_seconds` on
+  //   how ingested files are aged), exceeds the effective
+  //   `periodic_compaction_seconds` (including any RocksDB-picked default and,
+  //   for universal compaction, the effect of `ttl`), or
+  // * for FIFO compaction, the age of its newest data, as estimated by FIFO
+  //   TTL compaction (from the newest key time recorded in the file, or else
+  //   its oldest ancestor time, or else its creation time), exceeds `ttl`, so
+  //   that FIFO compaction would already treat the file as expired.
+  // Nothing is rejected if the applicable `periodic_compaction_seconds` or
+  // `ttl` is 0 (disabled), and a file without the recorded time (e.g. from an
+  // older SstFileWriter) is not rejected. Leveled `ttl` is not considered
+  // because it does not limit how long data can be kept: it only moves data
+  // out of the levels above the last level, where an ingested file is then
+  // compacted down soon after ingestion.
+  bool reject_too_old_file = false;
+
   bool operator==(const IngestExternalFileOptions& rhs) const = default;
 };
 
@@ -3408,6 +3430,11 @@ struct TraceOptions {
 struct ImportColumnFamilyOptions {
   // Can be set to true to move the files instead of copying them.
   bool move_files = false;
+  // Like IngestExternalFileOptions::reject_too_old_file, using the
+  // `file_creation_time` from the import metadata, or from the file itself if
+  // the metadata does not have one. If any file is rejected, the import fails
+  // with Status::InvalidArgument.
+  bool reject_too_old_file = false;
 };
 
 // Options used with DB::GetApproximateSizes()

include/rocksdb/table.h

@@ -1302,6 +1302,12 @@ class TableFactory : public Customizable {
 
   // Return is delete range supported
   virtual bool IsDeleteRangeSupported() const { return false; }
+
+  // Returns whether NewTableBuilder() can write tables. A column family whose
+  // table factory returns false holds only ingested or imported files: it must
+  // set disallow_memtable_writes, automatic compactions are not scheduled for
+  // it, and CompactRange() and CompactFiles() on it return NotSupported.
+  virtual bool IsTableBuilderSupported() const { return true; }
 };
 
 // Create a special table factory that can open either of the supported

include/rocksdb/table_properties.h

@@ -360,7 +360,9 @@ struct TableProperties {
   uint64_t oldest_key_time = 0;
   // Timestamp of the newest key. 0 means unknown.
   uint64_t newest_key_time = 0;
-  // Actual SST file creation time. 0 means unknown.
+  // When the SST file was built (for block-based tables, when the builder
+  // finished, which is no earlier than when any entry was added to it). 0
+  // means unknown.
   uint64_t file_creation_time = 0;
   // Estimated size of data blocks if compressed using a relatively slower
   // compression algorithm (see `ColumnFamilyOptions::sample_for_compression`).

include/rocksdb/version.h

@@ -12,7 +12,7 @@
 // NOTE: in 'main' development branch, this should be the *next*
 // minor or major version number planned for release.
 #define ROCKSDB_MAJOR 11
-#define ROCKSDB_MINOR 14
+#define ROCKSDB_MINOR 15
 #define ROCKSDB_PATCH 0
 
 // Make it easy to do conditional compilation based on version checks, i.e.

java/rocksjni/import_column_family_options.cc

@@ -46,6 +46,30 @@ jboolean Java_org_rocksdb_ImportColumnFamilyOptions_moveFiles(JNIEnv*, jobject,
   return static_cast<jboolean>(options->move_files);
 }
 
+/*
+ * Class:     org_rocksdb_ImportColumnFamilyOptions
+ * Method:    setRejectTooOldFile
+ * Signature: (JZ)V
+ */
+void Java_org_rocksdb_ImportColumnFamilyOptions_setRejectTooOldFile(
+    JNIEnv*, jobject, jlong jhandle, jboolean jreject_too_old_file) {
+  auto* options =
+      reinterpret_cast<ROCKSDB_NAMESPACE::ImportColumnFamilyOptions*>(jhandle);
+  options->reject_too_old_file = static_cast<bool>(jreject_too_old_file);
+}
+
+/*
+ * Class:     org_rocksdb_ImportColumnFamilyOptions
+ * Method:    rejectTooOldFile
+ * Signature: (J)Z
+ */
+jboolean Java_org_rocksdb_ImportColumnFamilyOptions_rejectTooOldFile(
+    JNIEnv*, jobject, jlong jhandle) {
+  auto* options =
+      reinterpret_cast<ROCKSDB_NAMESPACE::ImportColumnFamilyOptions*>(jhandle);
+  return static_cast<jboolean>(options->reject_too_old_file);
+}
+
 /*
  * Class:     org_rocksdb_ImportColumnFamilyOptions
  * Method:    disposeInternal
@@ -56,4 +80,4 @@ void Java_org_rocksdb_ImportColumnFamilyOptions_disposeInternal(JNIEnv*,
                                                                 jlong jhandle) {
   delete reinterpret_cast<ROCKSDB_NAMESPACE::ImportColumnFamilyOptions*>(
       jhandle);
-}
+}

java/rocksjni/ingest_external_file_options.cc

@@ -159,6 +159,30 @@ void Java_org_rocksdb_IngestExternalFileOptions_setIngestBehind(
   options->ingest_behind = jingest_behind == JNI_TRUE;
 }
 
+/*
+ * Class:     org_rocksdb_IngestExternalFileOptions
+ * Method:    rejectTooOldFile
+ * Signature: (J)Z
+ */
+jboolean Java_org_rocksdb_IngestExternalFileOptions_rejectTooOldFile(
+    JNIEnv*, jclass, jlong jhandle) {
+  auto* options =
+      reinterpret_cast<ROCKSDB_NAMESPACE::IngestExternalFileOptions*>(jhandle);
+  return static_cast<jboolean>(options->reject_too_old_file);
+}
+
+/*
+ * Class:     org_rocksdb_IngestExternalFileOptions
+ * Method:    setRejectTooOldFile
+ * Signature: (JZ)V
+ */
+void Java_org_rocksdb_IngestExternalFileOptions_setRejectTooOldFile(
+    JNIEnv*, jclass, jlong jhandle, jboolean jreject_too_old_file) {
+  auto* options =
+      reinterpret_cast<ROCKSDB_NAMESPACE::IngestExternalFileOptions*>(jhandle);
+  options->reject_too_old_file = static_cast<bool>(jreject_too_old_file);
+}
+
 /*
  * Class:     org_rocksdb_IngestExternalFileOptions
  * Method:    disposeInternal

java/src/main/java/org/rocksdb/ImportColumnFamilyOptions.java

@@ -37,8 +37,39 @@ public class ImportColumnFamilyOptions extends RocksObject {
     return this;
   }
 
+  /**
+   * Returns true if the import rejects files that are already too old for
+   * the column family.
+   * <p>
+   * See {@link #setRejectTooOldFile(boolean)}.
+   *
+   * @return true if too-old files are rejected.
+   */
+  public boolean rejectTooOldFile() {
+    return rejectTooOldFile(nativeHandle_);
+  }
+
+  /**
+   * Like {@link IngestExternalFileOptions#setRejectTooOldFile(boolean)}, using
+   * the file creation time from the import metadata, or from the file itself
+   * if the metadata does not have one. If any file is rejected, the import
+   * fails.
+   * <p>
+   * Default: false
+   *
+   * @param rejectTooOldFile true to reject files that are too old
+   *
+   * @return the reference to the current ImportColumnFamilyOptions.
+   */
+  public ImportColumnFamilyOptions setRejectTooOldFile(final boolean rejectTooOldFile) {
+    setRejectTooOldFile(nativeHandle_, rejectTooOldFile);
+    return this;
+  }
+
   private static native long newImportColumnFamilyOptions();
   private native boolean moveFiles(final long handle);
   private native void setMoveFiles(final long handle, final boolean move_files);
+  private native boolean rejectTooOldFile(final long handle);
+  private native void setRejectTooOldFile(final long handle, final boolean rejectTooOldFile);
   @Override protected final native void disposeInternal(final long handle);
 }

java/src/main/java/org/rocksdb/IngestExternalFileOptions.java

@@ -157,6 +157,41 @@ public class IngestExternalFileOptions extends RocksObject {
     return this;
   }
 
+  /**
+   * Returns true if ingestion rejects files that are already too old for
+   * the column family.
+   * <p>
+   * See {@link #setRejectTooOldFile(boolean)}.
+   *
+   * @return true if too-old files are rejected.
+   */
+  public boolean rejectTooOldFile() {
+    return rejectTooOldFile(nativeHandle_);
+  }
+
+  /**
+   * If true, ingestion fails (and no files are ingested) if any file is
+   * already too old for the column family to keep without recompaction or
+   * deletion. A file is too old if the age of its contents, from the creation
+   * time recorded in the file, exceeds the effective
+   * periodic_compaction_seconds (leveled or universal compaction), or if the
+   * age of its newest data, as estimated by FIFO TTL compaction (from the
+   * newest key time recorded in the file, or else its oldest ancestor time, or
+   * else its creation time), exceeds ttl (FIFO compaction). Nothing is
+   * rejected if the applicable periodic_compaction_seconds or ttl is 0
+   * (disabled), and a file without the recorded time is not rejected.
+   * <p>
+   * Default: false
+   *
+   * @param rejectTooOldFile true to reject files that are too old
+   *
+   * @return the reference to the current IngestExternalFileOptions.
+   */
+  public IngestExternalFileOptions setRejectTooOldFile(final boolean rejectTooOldFile) {
+    setRejectTooOldFile(nativeHandle_, rejectTooOldFile);
+    return this;
+  }
+
   private static native long newIngestExternalFileOptions();
   private static native long newIngestExternalFileOptions(final boolean moveFiles,
       final boolean snapshotConsistency, final boolean allowGlobalSeqNo,
@@ -180,4 +215,6 @@ public class IngestExternalFileOptions extends RocksObject {
       final long handle, final boolean allowBlockingFlush);
   private static native boolean ingestBehind(final long handle);
   private static native void setIngestBehind(final long handle, final boolean ingestBehind);
+  private static native boolean rejectTooOldFile(final long handle);
+  private static native void setRejectTooOldFile(final long handle, final boolean rejectTooOldFile);
 }

java/src/test/java/org/rocksdb/ImportColumnFamilyTest.java

@@ -95,4 +95,13 @@ public class ImportColumnFamilyTest {
       }
     }
   }
+
+  @Test
+  public void rejectTooOldFileOption() {
+    try (final ImportColumnFamilyOptions options = new ImportColumnFamilyOptions()) {
+      assertThat(options.rejectTooOldFile()).isFalse();
+      options.setRejectTooOldFile(true);
+      assertThat(options.rejectTooOldFile()).isTrue();
+    }
+  }
 }

java/src/test/java/org/rocksdb/IngestExternalFileOptionsTest.java

@@ -94,4 +94,13 @@ public class IngestExternalFileOptionsTest {
       assertThat(options.ingestBehind()).isTrue();
     }
   }
+
+  @Test
+  public void rejectTooOldFile() {
+    try (final IngestExternalFileOptions options = new IngestExternalFileOptions()) {
+      assertThat(options.rejectTooOldFile()).isFalse();
+      options.setRejectTooOldFile(true);
+      assertThat(options.rejectTooOldFile()).isTrue();
+    }
+  }
 }

table/block_based/block_based_table_builder.cc

@@ -2020,7 +2020,6 @@ struct BlockBasedTableBuilder::Rep {
     props.column_family_name = tbo.column_family_name;
     props.oldest_key_time = tbo.oldest_key_time;
     props.newest_key_time = tbo.newest_key_time;
-    props.file_creation_time = tbo.file_creation_time;
     props.orig_file_number = tbo.cur_file_num;
     props.db_id = tbo.db_id;
     props.db_session_id = tbo.db_session_id;
@@ -4050,6 +4049,14 @@ Status BlockBasedTableBuilder::Finish() {
 
   r->props.tail_start_offset = r->offset.LoadRelaxed();
 
+  // Read the clock when finishing rather than when starting, so that this
+  // time is no earlier than when any entry was added, even for a builder whose
+  // input arrives over a long time (e.g. SstFileWriter).
+  int64_t now = 0;
+  if (r->ioptions.clock->GetCurrentTime(&now).ok()) {
+    r->props.file_creation_time = lossless_cast<uint64_t>(now);
+  }
+
   uint64_t last_estimated_tail_size = EstimatedTailSize();
 
   // Write meta blocks, metaindex block and footer in the following order.

table/block_based/partitioned_filter_block.cc

@@ -571,10 +571,14 @@ void PartitionedFilterBlockReader::MayMatch(
       range->AddSkipsFrom(subrange);
       start_iter_same_handle = iter;
     }
-    if (UNLIKELY(this_filter_handle.size() == 0)) {  // key is out of range
-      // Not reachable with current behavior of GetFilterPartitionHandle
-      assert(false);
-      range->SkipKey(iter);
+    if (UNLIKELY(this_filter_handle.size() == 0)) {
+      // Empty filter partition. Keys outside the prefix extractor's domain
+      // are never added to a prefix filter, so only other keys can be ruled
+      // out.
+      if (prefix_extractor == nullptr ||
+          prefix_extractor->InDomain(iter->ukey_without_ts)) {
+        range->SkipKey(iter);
+      }
       prev_filter_handle = BlockHandle::NullBlockHandle();
     } else {
       prev_filter_handle = this_filter_handle;

table/external_table.cc

@@ -1100,7 +1100,8 @@ class ExternalTableFactoryAdapter : public TableFactory {
   explicit ExternalTableFactoryAdapter(
       std::shared_ptr<ExternalTableFactoryBase<Mode>> inner)
       : inner_(std::move(inner)),
-        supports_range_deletions_(inner_->IsDeleteRangeSupported()) {
+        supports_range_deletions_(inner_->IsDeleteRangeSupported()),
+        supports_table_builder_(inner_->IsTableBuilderSupported()) {
     RegisterOptions(&options_,
                     &GetExternalTableFactoryAdapterOptionsTypeInfo());
   }
@@ -1113,6 +1114,10 @@ class ExternalTableFactoryAdapter : public TableFactory {
     return Mode == ExternalTableMode::kFull && supports_range_deletions_;
   }
 
+  bool IsTableBuilderSupported() const override {
+    return supports_table_builder_;
+  }
+
   bool IsInstanceOf(const std::string& name) const override {
     return name == kClassName() || TableFactory::IsInstanceOf(name);
   }
@@ -1124,6 +1129,7 @@ class ExternalTableFactoryAdapter : public TableFactory {
     }
     if (status.ok()) {
       supports_range_deletions_ = inner_->IsDeleteRangeSupported();
+      supports_table_builder_ = inner_->IsTableBuilderSupported();
     }
     return status;
   }
@@ -1229,9 +1235,10 @@ class ExternalTableFactoryAdapter : public TableFactory {
  private:
   std::shared_ptr<ExternalTableFactoryBase<Mode>> inner_;
   ExternalTableFactoryAdapterOptions options_;
-  // Avoid invoking external code when RocksDB queries this capability while
+  // Avoid invoking external code when RocksDB queries these capabilities while
   // holding the DB mutex.
   bool supports_range_deletions_;
+  bool supports_table_builder_;
 };
 
 template <ExternalTableMode Mode>

table/sst_file_writer.cc

@@ -573,10 +573,9 @@ Status SstFileWriter::Open(const std::string& file_path, Temperature temp) {
       compression_type, compression_opts, cf_id, r->column_family_name,
       unknown_level, kUnknownNewestKeyTime, false /* is_bottommost */,
       TableFileCreationReason::kSstFileWriter, 0 /* oldest_key_time */,
-      0 /* file_creation_time */, "SST Writer" /* db_id */, r->db_session_id,
-      0 /* target_file_size */, r->next_file_number, kMaxSequenceNumber,
-      Slice() /* db_name */, false /* is_remote_compaction */,
-      r->embedded_blob_options.get());
+      "SST Writer" /* db_id */, r->db_session_id, 0 /* target_file_size */,
+      r->next_file_number, kMaxSequenceNumber, Slice() /* db_name */,
+      false /* is_remote_compaction */, r->embedded_blob_options.get());
   // External SST files used to each get a unique session id. Now for
   // slightly better uniqueness probability in constructing cache keys, we
   // assign fake file numbers to each file (into table properties) and keep

table/table_builder.h

@@ -140,8 +140,7 @@ struct TableBuilderOptions : public TablePropertiesCollectorFactory::Context {
       const std::string& _column_family_name, int _level,
       const int64_t _newest_key_time, bool _is_bottommost = false,
       TableFileCreationReason _reason = TableFileCreationReason::kMisc,
-      const int64_t _oldest_key_time = 0,
-      const uint64_t _file_creation_time = 0, const std::string& _db_id = "",
+      const int64_t _oldest_key_time = 0, const std::string& _db_id = "",
       const std::string& _db_session_id = "",
       const uint64_t _target_file_size = 0, const uint64_t _cur_file_num = 0,
       const SequenceNumber _last_level_inclusive_max_seqno_threshold =
@@ -163,7 +162,6 @@ struct TableBuilderOptions : public TablePropertiesCollectorFactory::Context {
         oldest_key_time(_oldest_key_time),
         newest_key_time(_newest_key_time),
         target_file_size(_target_file_size),
-        file_creation_time(_file_creation_time),
         db_id(_db_id),
         db_session_id(_db_session_id),
         is_bottommost(_is_bottommost),
@@ -185,7 +183,6 @@ struct TableBuilderOptions : public TablePropertiesCollectorFactory::Context {
   const int64_t oldest_key_time;
   const int64_t newest_key_time;
   const uint64_t target_file_size;
-  const uint64_t file_creation_time;
   const std::string db_id;
   const std::string db_session_id;
   // BEGIN for FilterBuildingContext

table/table_test.cc

@@ -8384,6 +8384,12 @@ class ExternalTableTest : public DBTestBase {
       : DBTestBase("external_table_test", /*env_do_fsync=*/false) {}
 
  protected:
+  // Reads the files of SimpleExternalTableFactory but cannot write any.
+  class ReadOnlyExternalTableFactory : public SimpleExternalTableFactory {
+   public:
+    bool IsTableBuilderSupported() const override { return false; }
+  };
+
   // A reader that pins values from its internal buffer, exercising the
   // zero-copy path in ExternalTableReaderAdapter::Get().
   class PinnedSimpleExternalTableReader
@@ -8618,6 +8624,144 @@ TEST_F(ExternalTableTest, WrappedFactoryValidatesColumnFamilyOptions) {
       column_families, std::vector<Options>{options, ingest_only}));
 }
 
+TEST_F(ExternalTableTest, FactoryWithoutTableBuilderIsNeverCompacted) {
+  if (encrypted_env_) {
+    ROCKSDB_GTEST_SKIP("Test requires non-encrypted environment");
+    return;
+  }
+  Options options = GetDefaultOptions();
+  // FIFO compaction, used below, requires it.
+  options.max_open_files = -1;
+  Reopen(options);
+  Options writable = options;
+  writable.table_factory =
+      NewExternalTableFactory(std::make_shared<SimpleExternalTableFactory>());
+  writable.disallow_memtable_writes = true;
+  // Any ingested file exceeds the FIFO size limit, so the picker always wants
+  // an automatic compaction.
+  writable.compaction_style = kCompactionStyleFIFO;
+  writable.compaction_options_fifo.max_table_files_size = 1;
+  Options read_only = writable;
+  read_only.table_factory =
+      NewExternalTableFactory(std::make_shared<ReadOnlyExternalTableFactory>());
+  ASSERT_TRUE(writable.table_factory->IsTableBuilderSupported());
+  ASSERT_FALSE(read_only.table_factory->IsTableBuilderSupported());
+
+  // Memtable writes would reach the missing builder at the next flush.
+  Options accepts_writes = read_only;
+  accepts_writes.disallow_memtable_writes = false;
+  ColumnFamilyHandle* handle = nullptr;
+  Status s = db_->CreateColumnFamily(accepts_writes, "accepts_writes", &handle);
+  ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+  ASSERT_EQ(handle, nullptr);
+
+  // handles_[1] can build tables and handles_[2] cannot; both get the same
+  // ingested file.
+  CreateColumnFamilies({"writable", "read_only"}, writable);
+  ReopenWithColumnFamilies({kDefaultColumnFamilyName, "writable", "read_only"},
+                           {options, writable, read_only});
+  const std::string file = dbname_ + "/external.immutable";
+  SstFileWriter writer(EnvOptions(writable), writable);
+  ASSERT_OK(writer.Open(file));
+  ASSERT_OK(writer.Put("a", "va"));
+  ASSERT_OK(writer.Put("b", "vb"));
+  ASSERT_OK(writer.Finish());
+  IngestExternalFileOptions ingest_options;
+  ingest_options.allow_global_seqno = true;
+  const auto live_files = [&](int cf) {
+    ColumnFamilyMetaData metadata;
+    db_->GetColumnFamilyMetaData(handles_[cf], &metadata);
+    std::vector<std::string> names;
+    for (const auto& level : metadata.levels) {
+      for (const auto& file_metadata : level.files) {
+        names.push_back(file_metadata.name);
+      }
+    }
+    return names;
+  };
+  std::vector<std::string> ingested[3];
+  // Otherwise the FIFO compaction of handles_[1] can delete its file before
+  // the check below sees it.
+  ASSERT_OK(db_->PauseBackgroundWork());
+  for (int cf : {1, 2}) {
+    ASSERT_OK(db_->IngestExternalFile(handles_[cf], {file}, ingest_options));
+    ingested[cf] = live_files(cf);
+    ASSERT_EQ(ingested[cf].size(), 1);
+  }
+  ASSERT_OK(db_->ContinueBackgroundWork());
+  ASSERT_OK(dbfull()->TEST_WaitForCompact());
+  ASSERT_TRUE(live_files(1).empty());
+  ASSERT_EQ(live_files(2), ingested[2]);
+  ASSERT_FALSE(static_cast_with_check<ColumnFamilyHandleImpl>(handles_[2])
+                   ->cfd()
+                   ->NeedsCompaction());
+
+  CompactRangeOptions forced;
+  forced.bottommost_level_compaction = BottommostLevelCompaction::kForce;
+  s = db_->CompactRange(CompactRangeOptions(), handles_[2], nullptr, nullptr);
+  ASSERT_TRUE(s.IsNotSupported()) << s.ToString();
+  s = db_->CompactRange(forced, handles_[2], nullptr, nullptr);
+  ASSERT_TRUE(s.IsNotSupported()) << s.ToString();
+  // FIFO compaction keeps every file on level 0.
+  s = db_->CompactFiles(CompactionOptions(), handles_[2], ingested[2],
+                        /*output_level=*/0);
+  ASSERT_TRUE(s.IsNotSupported()) << s.ToString();
+  ASSERT_EQ(live_files(2), ingested[2]);
+
+  // The rejected compactions leave the DB writable and the file readable.
+  ASSERT_OK(Put("key", "value"));
+  ASSERT_OK(Flush());
+  ASSERT_EQ(Get(2, "a"), "va");
+  ASSERT_EQ(Get(2, "b"), "vb");
+}
+
+TEST_F(ExternalTableTest, FactoryWithoutTableBuilderDoesNotStallWrites) {
+  if (encrypted_env_) {
+    ROCKSDB_GTEST_SKIP("Test requires non-encrypted environment");
+    return;
+  }
+  Options options = GetDefaultOptions();
+  Reopen(options);
+  Options writable = options;
+  writable.table_factory =
+      NewExternalTableFactory(std::make_shared<SimpleExternalTableFactory>());
+  Options read_only = writable;
+  read_only.table_factory =
+      NewExternalTableFactory(std::make_shared<ReadOnlyExternalTableFactory>());
+  read_only.disallow_memtable_writes = true;
+  // A second overlapping file can only go to L0, which alone reaches every L0
+  // trigger.
+  read_only.num_levels = 2;
+  read_only.level0_file_num_compaction_trigger = 1;
+  read_only.level0_slowdown_writes_trigger = 1;
+  read_only.level0_stop_writes_trigger = 1;
+  CreateColumnFamilies({"read_only"}, read_only);
+  ReopenWithColumnFamilies({kDefaultColumnFamilyName, "read_only"},
+                           std::vector<Options>{options, read_only});
+
+  IngestExternalFileOptions ingest_options;
+  ingest_options.allow_global_seqno = true;
+  for (const std::string value : {"v1", "v2"}) {
+    const std::string file = dbname_ + "/external_" + value + ".immutable";
+    SstFileWriter writer(EnvOptions(writable), writable);
+    ASSERT_OK(writer.Open(file));
+    ASSERT_OK(writer.Put("a", value));
+    ASSERT_OK(writer.Finish());
+    ASSERT_OK(db_->IngestExternalFile(handles_[1], {file}, ingest_options));
+  }
+  ASSERT_EQ(NumTableFilesAtLevel(0, 1), 1);
+  ASSERT_EQ(NumTableFilesAtLevel(1, 1), 1);
+  ASSERT_EQ(Get(1, "a"), "v2");
+
+  // No compaction can clear that L0 file, so it must not stall other column
+  // families.
+  ASSERT_FALSE(dbfull()->TEST_write_controler().IsStopped());
+  ASSERT_FALSE(dbfull()->TEST_write_controler().NeedsDelay());
+  WriteOptions no_slowdown;
+  no_slowdown.no_slowdown = true;
+  ASSERT_OK(db_->Put(no_slowdown, "key", "value"));
+}
+
 TEST_F(ExternalTableTest, GetWrappedExternalTableFactory) {
   std::shared_ptr<ExternalTableFactory> inner =
       std::make_shared<SimpleExternalTableFactory>();

tools/c_api_gen/c_base.cc

@@ -4555,6 +4555,16 @@ void rocksdb_options_set_block_based_table_factory(
 #include "c_api_gen/c_generated_jobinfo_metadata_subset.cc.inc"
 #include "c_api_gen/c_generated_metadata_structs_auto.cc.inc"
 
+unsigned char rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_compactionjobinfo_t* info) {
+  return info->rep.separate_remote_compaction_pool_selected;
+}
+
+unsigned char rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_subcompactionjobinfo_t* info) {
+  return info->rep.separate_remote_compaction_pool_selected;
+}
+
 rocksdb_compactionjobinfo_t* rocksdb_compactionjobinfo_create() {
   return new rocksdb_compactionjobinfo_t;
 }

tools/c_api_gen/c_base.h

@@ -1270,6 +1270,13 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_sst_file_manager(
 #include "c_api_gen/c_generated_jobinfo_auto.h.inc"
 #include "c_api_gen/c_generated_jobinfo_metadata_subset.h.inc"
 #include "c_api_gen/c_generated_metadata_structs_auto.h.inc"
+/* Compatibility aliases for the captured compaction pool selection. */
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_compactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_compactionjobinfo_t* info);
+extern ROCKSDB_LIBRARY_API unsigned char
+rocksdb_subcompactionjobinfo_use_separate_remote_compaction_pool(
+    const rocksdb_subcompactionjobinfo_t* info);
 extern ROCKSDB_LIBRARY_API rocksdb_compactionjobinfo_t*
 rocksdb_compactionjobinfo_create(void);
 extern ROCKSDB_LIBRARY_API void rocksdb_compactionjobinfo_destroy(

tools/check_format_compatible.sh

@@ -161,7 +161,7 @@ EOF
 
 # To check for DB forward compatibility with loading options (old version
 # reading data from new), as well as backward compatibility
-declare -a db_forward_with_options_refs=("10.4.fb" "10.5.fb" "10.6.fb" "10.7.fb" "10.8.fb" "10.9.fb" "10.10.fb" "10.11.fb" "11.0.fb" "11.1.fb" "11.2.fb" "11.3.fb" "11.4.fb" "11.5.fb" "11.6.fb" "11.7.fb" "11.8.fb" "11.9.fb" "11.10.fb" "11.11.fb" "11.12.fb" "11.13.fb")
+declare -a db_forward_with_options_refs=("10.4.fb" "10.5.fb" "10.6.fb" "10.7.fb" "10.8.fb" "10.9.fb" "10.10.fb" "10.11.fb" "11.0.fb" "11.1.fb" "11.2.fb" "11.3.fb" "11.4.fb" "11.5.fb" "11.6.fb" "11.7.fb" "11.8.fb" "11.9.fb" "11.10.fb" "11.11.fb" "11.12.fb" "11.13.fb" "11.14.fb")
 # To check for DB forward compatibility without loading options (in addition
 # to the "with loading options" set), as well as backward compatibility
 declare -a db_forward_no_options_refs=() # N/A at the moment

unreleased_history/behavior_changes/async_io_direct_io_multiget.md

@@ -1 +0,0 @@
-With `ReadOptions::async_io`, `MultiGet` now reads data blocks of direct-IO files with asynchronous reads (`FSRandomAccessFile::ReadAsync`) instead of a synchronous `MultiRead`. Each block read is aligned separately rather than merged with adjacent reads. The bytes read are still charged to `DBOptions::rate_limiter` at `ReadOptions::rate_limiter_priority`.

unreleased_history/behavior_changes/async_read_rate_limiter.md

@@ -1 +0,0 @@
-Asynchronous file reads (`FSRandomAccessFile::ReadAsync`, used with `ReadOptions::async_io` by `MultiGet` and iterator prefetching) are now charged to `DBOptions::rate_limiter` at `ReadOptions::rate_limiter_priority`, like synchronous reads. A read is charged once the file system accepts it.

unreleased_history/behavior_changes/create_cf_validates_table_factory.md

@@ -1 +0,0 @@
-`CreateColumnFamily()` and `CreateColumnFamilies()` now run the table factory option validation that `DB::Open()` runs, so they return `InvalidArgument` instead of creating a column family that the next `DB::Open()` would reject. External table factories wrapped by `NewExternalTableFactory()` now receive `ValidateOptions()` calls.

unreleased_history/behavior_changes/fifo_temperature_copy_listener_events.md

@@ -1 +0,0 @@
-With `CompactionOptionsFIFO::allow_trivial_copy_when_change_temperature`, each file copied for a temperature change is now reported to `EventListener::OnTableFileCreationStarted()` and `OnTableFileCreated()` (with the original file's table properties) and included in `CompactionJobInfo::table_properties`.

unreleased_history/behavior_changes/get_merge_operands_reset_written_slots.md

@@ -1 +0,0 @@
-`DB::GetMergeOperands()` now resets only the `merge_operands` entries it fills, instead of all `expected_max_number_of_operands` entries. Entries past `*number_of_operands`, and the whole array on a non-OK result, are left as the caller passed them.

unreleased_history/behavior_changes/json_writer_escape.md

@@ -1 +0,0 @@
-The JSON written to the info LOG by `EventLogger` (`EVENT_LOG_v1` lines) and by `VersionEdit::DebugJSON()` now escapes special characters in keys and string values, and writes non-finite floating-point values as `null`, so the output is always valid JSON.

unreleased_history/behavior_changes/periodic_compaction_phase_catch_up.md

@@ -1 +0,0 @@
-Improved how `periodic_compaction_phase_recovery_percent` spreads catch-up periodic compactions after it is enabled or changed (or the DB is reopened). This reduces a transient burst in periodic compactions when many DBs make such a change at once.

unreleased_history/behavior_changes/remote_compaction_pool.md

@@ -1 +0,0 @@
-The new DB option `use_separate_remote_compaction_pool` defaults to false and can be changed through `SetDBOptions()` for newly scheduled jobs. When enabled with CompactionService, remote jobs use a dedicated pool and each local subcompaction uses LOW/BOTTOM and counts against the DB compaction limit. Existing jobs finish using their original scheduling model.

unreleased_history/behavior_changes/set_options_skip_unchanged.md

@@ -1 +0,0 @@
-`DB::SetOptions()` now returns OK without writing a new OPTIONS file or installing a new SuperVersion when the call changes no option value, as `DB::SetDBOptions()` already does. Calls that set an object option, such as `table_factory` or one of its fields, are still applied in full. After a failed OPTIONS file write, the next `SetOptions()` or `SetDBOptions()` call writes the file even if no option value changes.

unreleased_history/behavior_changes/udi_validation_and_cache_statistics.md

@@ -1,3 +0,0 @@
-Custom index factories are rejected with user-defined timestamps in every index mode except `kStandardOnly`, which ignores the factory. Custom-index block-cache hits, misses, and adds now contribute to `BLOCK_CACHE_INDEX_*` instead of `BLOCK_CACHE_DATA_*`.
-
-Trie index construction returns `NotSupported` when a data-block offset or size exceeds `UINT32_MAX`, including when the trie is optional in `kStandardDefault`. Flush, compaction, and `SstFileWriter::Finish()` can therefore fail for such output; an SST larger than 4 GiB is not itself the rejection condition. File-size targets do not split L0 outputs or `SstFileWriter` outputs. Bound those outputs or disable trie construction with `kStandardOnly`.

unreleased_history/bug_fixes/atomic_replace_range_compaction_race.md

@@ -1 +0,0 @@
-Fixed a race where `IngestExternalFiles()` with `atomic_replace_range` could delete an existing SST file that a concurrently picked compaction (e.g. a universal compaction outputting to L0) used as input, causing the compaction to fail with "Corruption: VersionBuilder: Cannot delete table file ... since it is not in the LSM tree" and putting the DB into read-only mode. Atomic range replacement now returns `NotSupported` with FIFO compaction.

unreleased_history/bug_fixes/blob_direct_write_failed_seal_rollback.md

@@ -1 +0,0 @@
-Fixed blob direct write when sealing a blob file fails during a flush or a write: the flush left its memtables marked as being flushed, so no later flush picked them up and error recovery waited for them forever, and the blob file was dropped from tracking, so a retried flush could install table files whose blob file was never added to the MANIFEST.

unreleased_history/bug_fixes/blob_direct_write_overlapping_flushes.md

@@ -1 +0,0 @@
-Fixed blob direct write with more than one concurrent flush of a column family: a later flush sealed and published the blob files of the oldest pending memtable instead of its own, so flushes failed with `Corruption: VersionBuilder: Blob file #N already added` and the database stopped.

unreleased_history/bug_fixes/blob_direct_write_space_limit_seal.md

@@ -1 +0,0 @@
-Fixed blob direct write when the `SstFileManager` space limit is reached while a blob file is sealed: the sealed blob file was deleted while memtables still referenced it, so reads of its values failed and the values were lost, and reporting the error could deadlock with a concurrent memtable switch. The seal and the write or flush now succeed and keep the file, and writes then stop with a `SpaceLimit` background error, so for blob direct write the limit is soft: usage can exceed it by up to about `blob_file_size` × `blob_direct_write_partitions` for each memtable not flushed yet. Also fixed a write batch that failed after writing some of its values to blob files: those blobs are now counted as garbage, so their blob file can still be deleted.

unreleased_history/bug_fixes/blob_timestamp_history.md

@@ -1 +0,0 @@
-Values that retain external blob references now keep their original timestamps during history compaction, preventing blob key verification failures.

unreleased_history/bug_fixes/bottommost_range_del_partitioner_cut.md

@@ -0,0 +1 @@
+Fixed a bug where a bottommost-level compaction could ignore an `SstPartitioner` boundary between the last point key of an output file and a following range tombstone kept alive by a snapshot. The tombstone extended that file across the boundary, so an `IngestExternalFile()` with `fail_if_not_bottommost_level=true` into the range starting at the boundary could fail with `TryAgain`.

unreleased_history/bug_fixes/clip_empty_user_key.md

@@ -1 +0,0 @@
-Fixed `DB::ClipColumnFamily()` leaving an empty-string key outside the kept range when it is the only key in an L0 SST file.

unreleased_history/bug_fixes/common_prefix_reverse_comparator.md

@@ -1 +0,0 @@
-Fixed `format_version=8` with `ReverseBytewiseComparator`: a block's stored common key prefix could extend past a user key (one following longer user keys that extend it), so reads and iteration could miss that key (or its deletion, exposing an older value), and compactions could fail with `Corruption`. Files written by affected versions (11.10 through 11.13) are now read correctly, and new files no longer have this problem.

unreleased_history/bug_fixes/error_recovery_behind_stalled_writer.md

@@ -0,0 +1 @@
+Fixed automatic error recovery hanging forever, with all writes stopped, when a writer was waiting on a write stall (for example the `max_write_buffer_number` memtable limit) as a flush failed. This happened for a retryable SST write error with `two_write_queues = true`, and for a hard error such as a failed WAL sync during the flush in any configuration. A writer that is waiting when the DB stops now returns the background error, like a new write does.

unreleased_history/bug_fixes/lru_uncharged_standalone_get_charge.md

@@ -1 +0,0 @@
-Fixed a debug-build assertion failure and incorrect cache charge and block-cache byte statistics in release builds when a full, strict-capacity LRU cache with metadata charging creates an uncharged standalone handle, including during secondary-cache promotion.

unreleased_history/bug_fixes/manual_compaction_atomic_flush_cf_drop.md

@@ -1 +0,0 @@
-Fixed atomic flushes of all column families when another column family is dropped while they wait: `CompactRange()` no longer fails with a "Column family dropped" error, and `WaitForCompact()` and DB close no longer return OK without flushing the remaining column families.

unreleased_history/bug_fixes/mempurge_filter_error_relock.md

@@ -1 +0,0 @@
-Fixed a bug where MemPurge returned with the DB mutex unlocked when the compaction filter created for flush did not ignore snapshots.

unreleased_history/bug_fixes/multiscan_prepare_l0_crash.md

@@ -1 +0,0 @@
-Fixed a crash in MultiScan `Iterator::Prepare()` with a single scan range without a limit when the DB has L0 files, and an assertion failure in debug builds when `Prepare()` is called on an already-positioned iterator. `Prepare()` now leaves the iterator invalid until it is positioned again (e.g. by `Seek()`).

unreleased_history/bug_fixes/prepared_ingestion_sst_retention.md

@@ -1 +0,0 @@
-Fixed prepared external file ingestions retaining unrelated obsolete SSTs until commit or abort. Prepared ingestions now protect only their staged files from deletion.

unreleased_history/bug_fixes/range_conversion_ingest_older_iterator.md

@@ -1 +0,0 @@
-Fixed data loss with `min_tombstones_for_range_conversion` and external file ingestion. An iterator whose view predated the ingestion could convert the deleted keys around the ingested keys into a range tombstone that deleted them. This happened when the file got no new sequence number (sequence number zero, `ingest_behind`, or `allow_db_generated_files`), and when an iterator was created during the ingestion.

unreleased_history/bug_fixes/range_tombstone_conversion_memtable_tier.md

@@ -1 +0,0 @@
-Fixed a data loss bug where an iterator with `ReadOptions::read_tier = kMemtableTier` could, with `min_tombstones_for_range_conversion` enabled, insert a range tombstone that deleted live keys stored only in SST files. Read-path range tombstone conversion now runs only for iterators with `read_tier = kReadAllTier`.

unreleased_history/bug_fixes/recycled_wal_truncated_by_old_writer.md

@@ -1 +0,0 @@
-Fixed a race with `recycle_log_file_num > 0` where an obsolete WAL could be reused for a new WAL while its old writer was still open. Closing the old writer then truncated the reused file to the old WAL's size, so writes in the new WAL past that point, including acknowledged `sync` writes, were lost on the next reopen.

unreleased_history/bug_fixes/remote_compaction_tiering_options.md

@@ -1 +0,0 @@
-Fixed remote compaction ignoring `preclude_last_level_data_seconds` and `preserve_internal_time_seconds`, which caused recent data to be written to the last level and recent sequence numbers to be zeroed. Takes effect once both the primary and the compaction worker are upgraded.

unreleased_history/bug_fixes/resume_failure_subcompaction_completed.md

@@ -1 +0,0 @@
-Fixed `DB::OpenAndCompact()` with `OpenAndCompactOptions::allow_resumption` not calling `EventListener::OnSubcompactionCompleted()` when the compaction fails to resume from its saved progress.

unreleased_history/bug_fixes/secondary_manifest_switch_empty_cf.md

@@ -1 +0,0 @@
-Fixed MANIFEST catch-up on secondary instances (`TryCatchUpWithPrimary()`) and follower instances installing a column family with no files and still returning OK. This happened when catch-up switched to a new MANIFEST and the primary deleted a file that MANIFEST lists before catch-up checked it, recording the deletion in a newer MANIFEST. Catch-up now keeps the column family's previous view until a complete state from the new MANIFEST is available.

unreleased_history/bug_fixes/udi_cache_and_legacy_options.md

@@ -1,5 +0,0 @@
-Custom index blocks now reuse cached contents on SST reopen, and deprecated UDI boolean options can be cleared without overriding an explicitly configured index mode.
-
-Legacy UDI options copied from a factory can now be cleared before constructing a new factory without retaining the derived mode. The C API preserves both that rollback behavior and explicitly selected modes. OPTIONS files serialize equivalent legacy booleans for `kStandardDefault`, `kStandardRequired`, and `kCustomDefault` so older readers retain routing.
-
-Empty custom-index output retains the standard-index fallback in modes that write both indexes; custom-only output must remain nonempty. Default primary reads retain the table factory's precedence over a legacy per-read factory pointer, while explicit custom selectors still check its name.

unreleased_history/bug_fixes/udi_options_and_memory_compatibility.md

@@ -1,3 +0,0 @@
-Preserve legacy UDI routing when loading old OPTIONS files, and reject invalid index modes in standalone SST readers and writers. Table readers now charge the custom index blocks they own; a custom reader whose `ApproximateMemoryUsage()` already includes that block should override `IndexFactoryReader::MemoryUsageIncludesIndexBlock()` to return true so the block is not charged twice.
-
-Keep cached custom-index Slice views intact when a reader consumes its input. Register custom-index cache helpers without requiring a later change. Trie index size estimates now account for shared prefixes, preventing premature compaction file cuts, and reject data-block offsets or sizes exceeding the 32-bit on-disk limit instead of truncating them.

unreleased_history/new_features/async_read_cpu_time.md

@@ -1 +0,0 @@
-Added experimental `SetAsyncReadCPUTimeEnabled()`: the next async or coroutine read started on the thread records its CPU time in `PerfContext::get_cpu_nanos` at any perf level except `kDisable`, without turning on the other timers of `kEnableTimeAndCPUTimeExceptForMutex`.