tiering: tracking issue #5639
Description
This issue will track the work items related to tiering support.
- Design
- get feedback from SQL foundations
- get feedback from SQL queries
- get feedback from KV
- CockroachDB
-
support configuration of tiered stores
-
SQL
- tiering support in table/index descriptor (prototype)
- implement new syntax for CREATE TABLE
- populate tiering policy in span config (prototype)
- tiering attribute system column (prototype)
- tiering attribute encoding (prototype]
- throttle SQL table statistics on cold data
- implement syntax for ALTER PRIMARY KEY
- implement ALTER PRIMARY KEY tiering schema change
-
KV
- tiering support in span config (prototype]
-
Storage layer
- plumb tiering policy and attribute extraction into span policy
-
Backup
- default to fewer full backups on tiered clusters Backup/DR: default to fewer full backups on cold-tier clusters #5575
- figure out how restoring a tiered cluster will work
-
- Pebble
- API
- extend SpanPolicy to support tiering
- extend SpanPolicy to support separating files at range boundaries
- objstorage
- support cold tier directory
- dual-write of metadata to both hot and cold tier
- integrate and use secondary cache with cold tier
- cold tier throttling
- sstable format
- add tiering attribute and spanID columns (db: add new columns to datablock #5466)
- add "secondary" blob reference columns
- research efficient histogram/quantile sketches, come up with a usable implementation
- store quantile sketch of bytes used across tiering attribute values (tiering: sstable format changes #5566)
- compactions
- unify blob value compaction strategies (valsep: introduce new generalized blob writer #5482)
- implement tiering compaction strategy
- compaction iterator support for tiering metadata
- TODO
- metrics
- TODO
- policy enforcement
- background span policy enforcer background span policy enforcer #5657
- API
Epic CRDB-48982
Jira issue: PEBBLE-1298