[inventory] Add unhealthy zpools from each sled #9615

karencfv · 2026-01-09T03:44:37Z

This is the second PR for #9412

This commit adds a list of unhealthy zpools to the sled agent inventory's health monitor. In a follow-up PR this information will be added to the DB

Successful unhealthy zpool and SMF service in maintenance retrieval:

$ zpool list -Hpo health,name
DEGRADED        fakepool1
DEGRADED        fakepool2
ONLINE  rpool

$ curl -H "api-version: 16.0.0"  http://[::1]:41364/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [
          {
            "fmri": "svc:/site/fake-service2:default",
            "zone": "global"
          },
          {
            "fmri": "svc:/site/fake-service:default",
            "zone": "global"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-19T06:03:13.713176608Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [
          {
            "zpool": "fakepool1",
            "state": "degraded"
          },
          {
            "zpool": "fakepool2",
            "state": "degraded"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-19T06:03:13.690492960Z"
      }
    }
  }
}

Response contains errors:

$ curl -H "api-version: 16.0.0"  http://[::1]:38614/inventory | jq
<...>
"health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [],
        "errors": [
          <...>
          "Found a service with an unknown state: global",
          "Found a service with an unknown state: global"
        ],
        "time_of_status": "2026-01-19T06:08:11.141551661Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [],
        "errors": [
          "Failed to parse output: Unrecognized zpool 'health': fakepool1",
          "Failed to parse output: Unrecognized zpool 'health': fakepool2",
          "Failed to parse output: Unrecognized zpool 'health': rpool"
        ],
        "time_of_status": "2026-01-19T06:08:11.133593564Z"
      }
    }
  }
}

All zpools and services are healthy

$ zpool list -Hpo health,name
ONLINE  rpool

$ curl -H "api-version: 16.0.0"  http://[::1]:39273/inventory | jq
<...>
"health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [],
        "errors": [],
        "time_of_status": "2026-01-19T06:14:24.219297126Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [],
        "errors": [],
        "time_of_status": "2026-01-19T06:14:24.194600484Z"
      }
    }
  }
}

Health monitor checks aren't running:

$ curl -H "api-version: 16.0.0"  http://[::1]:40386/inventory | jq
<...>
"health_monitor": {
  "smf_services_in_maintenance": null,
  "unhealthy_zpools": null
}

jgallagher · 2026-01-13T18:33:14Z

illumos-utils/src/zpool.rs

+pub struct UnhealthyZpoolsResult {
+    pub zpools: Vec<String>,
+    pub errors: Vec<String>,
+    pub time_of_status: Option<DateTime<Utc>>,


Does this need to be optional? If we have a result, it must have been collected at some time, right?

This is meant for when this runs on non-illumos environements

https://github.com/oxidecomputer/omicron/pull/9615/files#diff-07cfb91c0643fac6671e8b81317c92cbe806ab07e2af6fdce16db75315245e32R446-R453

The idea is that there is no time of status because no command was ever run.

Also there is a point in time where health monitor may not have run yet -> #9589 (comment)

illumos-utils/src/zpool.rs

sled-agent/health-monitor/src/health_checks.rs

karencfv · 2026-01-14T03:09:26Z

illumos-utils/src/zpool.rs

+#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize, JsonSchema)]
+#[serde(rename_all = "snake_case")]
+pub struct UnhealthyZpoolsResult {
+    pub zpools: Vec<String>,


I went back and forth between just having a list of unhealthy zpools, or associating each zpool with it's state. In the end I went with listing the zpools only, but I'm not convinced. We'll be including the information of the health checks in the support bundle, and it'd be useful for them to be able to see what state each zpool is in. Thoughts? @davepacheco @jgallagher

Associating each zpool with its state sounds good to me; having an explicit entry for "this zpool was healthy" seems safer than inferring "any zpool that isn't explicitly listed as unhealthy must have been healthy".

Hmmm, I was thinking of only including the "unhealthy" zpools with their associated statuses in this list. Similarly with the svcs_in_maintenance I only added the services in maintenance with their associated zones. If I were to include the "healthy" zpools then it wouldn't really be consistent with the services in maintenance no?

My take on the health checks is to only report on things that are in an unhealthy state. Thoughts?

karencfv · 2026-01-19T02:38:22Z

Thanks for taking a look @jgallagher @davepacheco ! I think I've addressed all of the comments and this is ready for another review. I also incorporated the feedback from #9589 (comment) here.

I have updated the PR's description to show the output of /inventory after the changes

karencfv · 2026-01-19T06:30:15Z

In the end I decided to make the SMF services in maintenance as an Option<T> as part of this PR. Otherwise the versioning between the 2 PRs was getting annoying. I edited the PR's description to include testing for them as well

karencfv · 2026-01-19T06:34:35Z

nexus/types/src/inventory/display.rs

+            if let Some(svcs) = &health_monitor.smf_services_in_maintenance {
+                match svcs {
+                    Ok(svcs) => {
+                        if !svcs.is_empty() {
+                            if let Some(time_of_status) = &svcs.time_of_status {
+                                writeln!(
+                                    indent2,
+                                    "SMF services in maintenance at {}:",
+                                    time_of_status.to_rfc3339_opts(
+                                        SecondsFormat::Millis,
+                                        /* use_z */ true,
+                                    )
+                                )?;
+                            }
+                            let mut indent3 =
+                                IndentWriter::new("  ", &mut indent2);
+                            for svc in &svcs.services {
+                                writeln!(indent3, "{svc}")?;
+                            }
                        }
                    }
-                }
-                Err(e) => {
-                    writeln!(
-                        indent2,
-                        "failed to retrieve SMF services in maintenance: {e}"
-                    )?;
+                    Err(e) => {
+                        writeln!(
+                            indent2,
+                            "failed to retrieve SMF services in maintenance: {e}"
+                        )?;
+                    }


I am changing all of this in https://github.com/oxidecomputer/omicron/pull/9589/files#diff-7aa92821ab1567814797c276e521417c9ecb6ee700d2b76679ea5e3daedc5904R1105-R1183 anyway. These changes are only to not have any compilation errors with the new Option<Result<SvcsInMaintenanceResult, String>> type

karencfv added 8 commits December 4, 2025 20:55

pseudo-code

ed4d1ee

add some thoughts

b30e767

Merge branch 'main' into zpool-health-check

27204f4

[inventory] Add unhealthy zpools from each sled

b038f6e

add some tests

25db966

Add unhealthy zpool functionality

cc6d397

add zpool health info to API

219918c

fix cmd order

407a068

karencfv marked this pull request as ready for review January 12, 2026 07:29

karencfv requested review from davepacheco and jgallagher January 12, 2026 07:29

jgallagher reviewed Jan 13, 2026

View reviewed changes

jgallagher mentioned this pull request Jan 13, 2026

[inventory] Add svcs in maintenance to DB #9589

Draft

address comments

c817b9c

karencfv commented Jan 14, 2026

View reviewed changes

karencfv added 12 commits January 15, 2026 16:29

get serde/schemars working with Option<Result<T,E>>

d01216d

clean up

b1fecc8

make time_of_status required

8eb5001

add json file and fmt

7cd8f4b

merge main

c5baf76

generate openapi doc

e6944dc

sigh...

0458330

untagged so it works on none

4c7cad3

linter

2771225

report zpool health state as well

b8157ea

add generated openapi spec

a2e3e90

Remove manual testing code

c734654

karencfv requested a review from jgallagher January 19, 2026 02:38

Make smf_services_in_maintenance optional too

030b11f

karencfv commented Jan 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[inventory] Add unhealthy zpools from each sled #9615

[inventory] Add unhealthy zpools from each sled #9615

Uh oh!

karencfv commented Jan 9, 2026 •

edited

Loading

Uh oh!

jgallagher Jan 13, 2026

Uh oh!

karencfv Jan 13, 2026

Uh oh!

karencfv Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

karencfv Jan 14, 2026

Uh oh!

jgallagher Jan 14, 2026

Uh oh!

karencfv Jan 15, 2026

Uh oh!

karencfv commented Jan 19, 2026 •

edited

Loading

Uh oh!

karencfv commented Jan 19, 2026 •

edited

Loading

Uh oh!

karencfv Jan 19, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[inventory] Add unhealthy zpools from each sled #9615

Are you sure you want to change the base?

[inventory] Add unhealthy zpools from each sled #9615

Uh oh!

Conversation

karencfv commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgallagher Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

karencfv Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

karencfv Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

karencfv Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

jgallagher Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

karencfv Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

karencfv commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karencfv commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karencfv Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

karencfv commented Jan 9, 2026 •

edited

Loading

karencfv commented Jan 19, 2026 •

edited

Loading

karencfv commented Jan 19, 2026 •

edited

Loading

karencfv Jan 19, 2026 •

edited

Loading