mirror of
https://github.com/facebook/rocksdb.git
synced 2026-11-06 02:52:35 -05:00
Peter Dillinger
9d3733d9d8
Summary: Ideally, each age-based feature acts on the time that matches its purpose, which requires tracking three different times for each SST file: * How old its data is (when its keys were written, as bounded by the oldest ancestor time and the `newest_key_time` table property), so that TTL expires data based on its actual age. * When the file's contents were built, i.e. when its data last went through a flush or compaction (`file_creation_time`). Periodic compaction (commonly used so that all data regularly goes through a compaction filter) and `GetCreationTimeOfOldestFile()` should act on this. * When this DB began holding the file. `bottommost_file_compaction_delay` should act on this: it limits how soon a recently written bottommost file is rewritten alone, after snapshot releases, to zero its sequence numbers or drop data no longer protected, so that a series of snapshot releases does not rewrite the same data repeatedly. External storage accounting (how long each file occupies storage) also needs this time. Knowing how the DB got each file (built it, or ingested or imported it) is also useful, e.g. for storage accounting. Previous behavior: A file's age for periodic compaction comes from its MANIFEST `file_creation_time`, then (only if the table reader is pinned) the `file_creation_time` table property, then the oldest ancestor time, then the file's mtime (unchanged by this commit). However: * Files written by `SstFileWriter` had table property `file_creation_time` = 0 (unknown). * Files flushed during WAL recovery had `file_creation_time` = 0 in both the table properties and the MANIFEST. * `RepairDB` restored each file's oldest ancestor time from its table properties but not its `file_creation_time`, so every repaired file lost its MANIFEST creation time. * `IngestExternalFile()` and `CreateColumnFamilyWithImport()` set both `file_creation_time` and the oldest ancestor time to the ingest/import time, ignoring times carried by the file's table properties or the export metadata. Data written or last compacted long before ingestion was therefore not due for periodic compaction (or TTL) until a full period after ingestion. Also, `Checkpoint::ExportColumnFamily()` left the time fields of its export metadata unset. * A FIFO temperature change done as a trivial copy (with `allow_trivial_copy_when_change_temperature`) recorded the copy time as the new file's MANIFEST `file_creation_time`, although its contents and table properties are the original's. * FIFO TTL compaction picks expired files oldest first, judging each by the `newest_key_time` table property or else the oldest ancestor time table property. `SstFileWriter` files have neither, so an ingested `SstFileWriter` file stopped the scan: neither it nor any newer file was ever deleted by TTL. * Nothing recorded when the DB began holding a file, or whether it was ingested, imported, or built by this DB. (For ingested and imported files, `file_creation_time` happened to be the time they were added, which `bottommost_file_compaction_delay` used.) New behavior, for when contents were built: * The block-based table builder now records the `file_creation_time` property itself, reading the clock when it finishes the file, so that the time is no earlier than when any entry was added. So every block-based SST file carries one (unless reading the clock fails), including `SstFileWriter` files, whose entries can be added over a long time. (For flush and compaction outputs, the MANIFEST value is still read when the output file is started, so it can be slightly earlier than the property.) The WAL recovery flush now also sets the MANIFEST value, and `RepairDB` restores it from the table property. * Ingestion and import keep the creation and oldest ancestor times carried by the file when known: for ingestion, from its table properties; for import, from the export metadata, or else the file's table properties. A carried time later than the current time suggests an unreliable clock, so it is logged as a WARNING and replaced with the current time; an unknown time becomes the current time. When no oldest ancestor time is carried (true of all `SstFileWriter` files), it falls back to the resolved creation time, an upper bound on when the data was written, so TTL never fires early. `ExportColumnFamily()` now fills in both time fields. * A FIFO temperature-change copy keeps the original's `file_creation_time`. * FIFO TTL compaction falls back to `file_creation_time` when a file has neither of the table properties above. Every key in a file was written no later than the file's contents were built, so this never deletes early. * New option `reject_too_old_file` (default false) in `IngestExternalFileOptions` and `ImportColumnFamilyOptions` fails the whole ingestion/import with `InvalidArgument`, before any file is linked or copied, if any file is already too old for the column family to keep without recompaction or deletion: for leveled and universal compaction, if its known creation time is older than the effective `periodic_compaction_seconds`; for FIFO compaction, if its newest data (the `newest_key_time` table property when known, else its creation time) is older than `ttl`. Such data should already have been recompacted (e.g. through a compaction filter) or deleted. Leveled `ttl` is not considered because it does not limit how long data can be kept, only how long it stays above the last level. Files without the needed time are not rejected. C and Java bindings included. New behavior, for when the DB began holding a file: * New `SstFileMetaData::db_ownership_start_time` (also in `LiveFileMetaData` and export metadata), backed by a new MANIFEST field: the time this DB began holding the physical file (file number). That is when the DB started writing it (flush or compaction output, local or remote), when ingestion linked or copied it into the DB (in the prepare phase, for two-phase ingestion), when import did, or when a FIFO temperature-change copy wrote it. Moves between levels preserve it. A long compaction's outputs start when they are created, not when the compaction finishes. * New `SstFileMetaData::db_ownership_info`, decoded by the new public struct `DbOwnershipInfo`, backed by another MANIFEST field: how the DB got the contents (flush, compaction, ingestion, or import) plus sticky flags for things that have happened to the file without its contents being rewritten (moved between levels; copied to a new temperature; temperature updated from the file system by `UpdateManifestForFilesState()`; recorded by `RepairDB`). * `bottommost_file_compaction_delay` is now measured from `db_ownership_start_time` (falling back to `file_creation_time` when unknown). For files built by the DB that is the same as before; ingested files again get the delay from when they were ingested, rather than skipping it because of old contents (which only matters for ingested DB-generated files, since normally ingested `SstFileWriter` files have sequence number 0 and are never marked). Neither new field is a table property: they describe one DB's relationship to a file, and the same file contents can be added to several DBs. Times not recorded, for lack of a consumer: when the DB first held the contents across temperature copies, when a file became visible (MANIFEST commit time), and when a file arrived at its current level. Implementation notes: * `TableBuilderOptions::file_creation_time` (and its positional constructor parameter) is removed, since the builder now reads the clock itself. All callers passing arguments past that position were updated. * `ResolveExternalFileTimes()` and `CheckExternalFileNotTooOld()` are shared by ingestion and import. * The new MANIFEST fields use new safe-to-ignore `NewFile` tags, written only when known. Their packed bit layout is defined next to the tags; unknown bits are ignored on read and preserved when a file's record is carried over. * New `VersionEdit::AddExistingFile()` re-adds a file under the same file number with all of its persisted metadata. It replaces field-by-field copies into the positional `AddFile()` at the five places that move files (trivial move, `CompactRange()` with `change_level`, `PromoteL0()`, the level migration at open, and `UpdateManifestForFilesState()`) and in the MANIFEST snapshot writer, so MANIFEST-only fields, including future ones, carry over. The WAL recovery flush, `RepairDB`, and import now use the `FileMetaData` overload of `AddFile()`; as a side effect, recovery flush outputs now also record min/max timestamps and file open metadata in the MANIFEST, as regular flush outputs do. Compatibility and limitations: * Behavior change: ingesting or importing old data can make it due for periodic compaction or TTL soon afterward rather than a full period later, so bulk-ingesting old SSTs can cause a burst of compactions. Users who must not publish such data can set `reject_too_old_file`. * Files already in a DB keep their MANIFEST entries; the new fields are unknown (0) for them. For files ingested, imported, or temperature-copied by older versions, `file_creation_time` was the ingestion, import, or copy time, so the `bottommost_file_compaction_delay` fallback is accurate for them. Files from older `SstFileWriter` versions carry no creation time and still get the ingestion time. * Older RocksDB versions ignore the new MANIFEST fields and drop them when rewriting the MANIFEST. A DB opened from a checkpoint or backup keeps the values recorded by the DB it was copied from. `RepairDB` loses them for files that existed before the repair. * Plain and cuckoo table builders still record no time properties. * The times recorded for remote compaction outputs come from the remote worker's clock. * Possible follow-ups: letting `SstFileWriter` callers supply an authentic oldest ancestor time; and avoiding bottommost-files compactions that rewrite an ingested `SstFileWriter` file only to zero a global sequence number that exists only in the MANIFEST (found by code reading, not yet confirmed by a test). Pull Request resolved: https://github.com/meta/rocksdb/pull/15583 Test Plan: New tests, using mock clocks where time matters: * `DBTablePropertiesTest.FileCreationTimeAlwaysSet`: `SstFileWriter` file plus ingestion, WAL recovery flush, and `RepairDB` all record a creation time, checked in both table properties and the raw MANIFEST `FileMetaData` (`GetLiveFilesMetaData()` would mask a missing MANIFEST value by falling back on pinned table properties). * `ExternalSSTFileTimeTest.*` (external_sst_file_test): `KeepRecordedTimes` covers recorded, unknown, and future creation times; `KeepDBGeneratedFileTimes` checks that `allow_db_generated_files` keeps both times from the source DB; `PeriodicCompactionFromWriteTime` checks that periodic compaction triggers relative to when an `SstFileWriter` file was written, not when it was ingested; `RejectTooOldFile` covers leveled, universal, and FIFO: a batch containing a too-old file is rejected with the DB unchanged, a file just under the limit and a file with unknown creation time are accepted, and the option off accepts the too-old file; `RejectFIFOFileWithOldNewestKey` checks that for FIFO, a recently built file whose newest key time is older than `ttl` is rejected; `CreationTimeIsWhenFileIsFinished` checks that an `SstFileWriter` file's creation time is when it was finished, not opened; `FIFOTtlDeletesIngestedFile` checks that an ingested `SstFileWriter` file and a newer flushed file expire by TTL; `RecordsDbOwnership` checks the ownership start time and origin for one-phase ingestion, two-phase ingestion (from the prepare phase, not the commit), and ingestion of a DB-generated file; `BottommostCompactionDelayFromIngestion` checks that an ingested DB-generated file with old contents is not compacted as a bottommost file until the delay has passed since ingestion. * `ImportColumnFamilyTest.ImportKeepsExportedFileTimes` and `ImportFileTimesAndRejectTooOldFile`: same coverage for export/import, plus falling back on the file's table properties when the import metadata has no time, and ownership metadata in the export and after import. * `DBOwnershipTest.*` (db_etc3_test): ownership start time and origin of flush, WAL recovery flush, and compaction outputs (including a compaction delayed after creating its output), and in public metadata; preservation, with the "moved" flag, through `PromoteL0()`, trivial move, `CompactRange()` with `change_level`, the level migration at open, and reopens (MANIFEST replay and snapshot); FIFO temperature-change copy (same creation time, new ownership start time, "temperature copied" flag); `UpdateManifestForFilesState()`; and `RepairDB` (unknown ownership for pre-existing tables, flush origin for tables built from WALs). * `CompactionServiceTest.BasicCompactions`: remote compaction outputs record ownership like local ones. * `VersionEditTest.*`: MANIFEST encoding of the new fields (round trip, preserved reserved bits, written only when known, safe to ignore, debug output), the persisted `DbOwnershipInfo` bit layout, `AddExistingFile()`, and the fallback for an unknown ownership start time. * Java getter/setter tests for the new option. The JNI code was compiled against `javac`-generated headers, but the Java tests were not run locally. `FIFOTtlDeletesIngestedFile` was verified to fail before the FIFO TTL fix. `BottommostCompactionDelayFromIngestion`, `FIFOTemperatureChangeCopy`, `MovesPreserveOwnership`, `TemperatureUpdatedFromFileSystem`, and `AddExistingFile` were verified to fail with, respectively, the delay measured from `file_creation_time`, the copy time used as creation time, and `AddExistingFile()` dropping the ownership start time. `CreationTimeIsWhenFileIsFinished`, `RejectFIFOFileWithOldNewestKey`, and `ImportFileTimesAndRejectTooOldFile` were verified to fail with, respectively, the creation time read when the builder is created, FIFO rejection based only on creation time, and import ignoring the file's table properties. The creation-time tests were verified to fail with the new logic stubbed out to the old behavior before rebasing onto current main, and `ImportKeepsExportedFileTimes` also fails with only the `ExportColumnFamily()` fix reverted. Adjusted `ExternalSSTFileBasicTest.AtomicReplaceRangeAvoidsRedundantTimestampScans`: it models legacy files with no recorded creation time and relied on `SstFileWriter` files having a zero `file_creation_time` property (pinned readers fall back on it). It now writes those files with a writer clock that reads 0. Reviewed By: xingbowang Differential Revision: D129668538 Pulled By: pdillinger fbshipit-source-id: 1e8ad8e9738d7b0427c408d693bc22c46deca913