[server] throttling deletes to limit disk i/o

[ymuppala]

Problem Statement

Venice storage nodes with the discard mount option enabled are experiencing disk health check failures during large store deletions due to synchronous TRIM operations saturating disk I/O, causing the DiskHealthCheckService to timeout and mark nodes unhealthy.

Solution

Added configurable deletion rate limiting for RocksDB's SstFileManager to throttle file deletion operations during store cleanup:
Added two new configuration parameters to RocksDBServerConfig: rocksdb.sst.file.manager.delete.rate.bytes.per.second & rocksdb.sst.file.manager.max.trash.db.ratio
Updated RocksDBStorageEngineFactory to apply deletion rate limits to the shared SstFileManager instance when configured

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[huangminchn]

Thanks for the quick turnaround Yug!
Can we double check the code path used during backup version deletion? My understanding is that it's going through "RocksDBStoragePartition#drop()".

We can add a test case for backup version deletion if none exists yet, and validate the deletion rate.

[huangminchn]

Can we double check that this code path is used during backup version deletion?

[ymuppala]

Yeah, it is in the code path. The call stack looks like the following

VersionBackend.delete()
  → IngestionBackend.removeStorageEngine(topicName)
    → StorageService.removeStorageEngine(kafkaTopic)
      → StorageEngine.drop()
        → AbstractStorageEngine.drop() [iterates partitions]
          → RocksDBStoragePartition.drop()
            → RocksDB.destroyDB()

[eldernewborn]

Wouldn't this mean that 75% of the DB will be deleted in a throttled manner, but then the last 25% will be immediately deleted ?

We need the throttling to continue to be applied all the way through .

[ymuppala]

I do not think that is what the config value means. It specifies the upper limit on trash size to live DB size ratio that can be tolerated before files are immediately deleted, overriding the rate limit. So, when the ratio of trash to live db hits 25% throttling will be ignored.

[ymuppala]

Thanks for the quick turnaround Yug! Can we double check the code path used during backup version deletion? My understanding is that it's going through "RocksDBStoragePartition#drop()".

We can add a test case for backup version deletion if none exists yet, and validate the deletion rate.

I have added a test to validate the deletion rate