Skip to content

Running a scrub on a pool with a spare in use sends the pool offline #17472

Open
@cyberpower678

Description

@cyberpower678

System information

Distribution Name | TrueNAS SCALE (Debian Bookworm)
Distribution Version | 25.04.1 (12)
Kernel Version | 6.12.15-production+truenas
Architecture | x64
OpenZFS Version | 2.3.0-1

Describe the problem you're observing

It would appear if a drive in a mirror vdev faults and a hot spare kicks in, initiating a scrub will make the mirror vdev inaccessible, despite there being 2 ONLINE drives in it, and make the entire pool UNAVAIL.

A zpool clear clears the error, but also the FAULTED drive and the pool is ONLINE again, but it shouldn't go offline the moment a zpool scrub command is issued while the pool is in that state.

Zpool Status before scrub:

zpool status

pool: HomeBackup
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: resilvered 3.68T in 08:17:47 with 0 errors on Wed Jun 18 07:56:59 2025
config:

    NAME                                        STATE     READ WRITE CKSUM
    HomeBackup                                  DEGRADED     0     0     0
      mirror-0                                  ONLINE       0     0     0
        885a4df8-3a2d-489f-9459-c0e9c24b7320    ONLINE       0     0     0
        9dca1645-9099-4777-b727-d392954f2f0c    ONLINE       0     0     0
      mirror-1                                  ONLINE       0     0     0
        46480e4c-8076-4de8-b5b5-475c77a02826    ONLINE       0     0     0
        e203dc2a-3037-4904-85c0-bbc2ea542e57    ONLINE       0     0     0
      mirror-2                                  ONLINE       0     0     0
        0672cd9e-82cc-4907-9b6c-1e3d3f385ba6    ONLINE       0     0     0
        abc1c2e1-2844-4021-bac7-8afbea64254f    ONLINE       0     0     0
      mirror-3                                  ONLINE       0     0     0
        cb6b7724-8d54-482f-b0d1-894c06bf0d47    ONLINE       0     0     0
        9cbe3f26-1a7f-4349-9653-3ab2366cee30    ONLINE       0     0     0
      mirror-4                                  DEGRADED     0     0     0
        d3409bdf-9c34-48f7-8350-9a4fb05dfa83    ONLINE       0     0     0
        spare-1                                 DEGRADED     0     0     0
          54172042-d04d-4add-88dd-960c93cc5dc5  FAULTED     27     4    39  too many errors
          5dbac47a-f2ae-4571-a1b5-6e36addee7f4  ONLINE       0     0   401
      mirror-5                                  ONLINE       0     0     0
        b4305d01-a758-40a0-bfe9-c36d7569dea0    ONLINE       0     0     0
        e33c3303-5ff4-4575-b9a9-4e451dcdfc39    ONLINE       0     0     0
      mirror-6                                  ONLINE       0     0     0
        0780131e-8af7-4dea-b217-43ffc2b45e6d    ONLINE       0     0     0
        06fbf8d0-335f-4bdb-ac6d-6b3c5f698a8b    ONLINE       0     0     0
      mirror-7                                  ONLINE       0     0     0
        afac7e0a-de4c-47e1-92f5-0ed91d962434    ONLINE       0     0     0
        1d9921c0-72da-4a5d-b326-0dcb01c67cf0    ONLINE       0     0     0
      mirror-8                                  ONLINE       0     0     0
        697c2aee-1834-4621-a8ef-eb07bdb5b633    ONLINE       0     0     0
        12764d57-5e24-4888-a9aa-eaff1998118b    ONLINE       0     0     0
    spares
      5dbac47a-f2ae-4571-a1b5-6e36addee7f4      INUSE     currently in use
      a12ed467-6daa-456a-96a5-14d1705fbe51      AVAIL   

errors: No known data errors

Zpool Status after scrub starts and immediately errors out:

zpool scrub HomeBackup

zpool status

pool: HomeBackup
state: UNAVAIL
status: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool to
continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking the device
repaired using 'zpool clear' may allow some data to be recovered.
scan: scrub in progress since Wed Jun 18 09:14:54 2025
58.9G / 33.7T scanned at 11.8G/s, 0B / 33.7T issued
0B repaired, 0.00% done, no estimated completion time
config:

    NAME                                        STATE     READ WRITE CKSUM
    HomeBackup                                  UNAVAIL      0     0     0  insufficient replicas
      mirror-0                                  ONLINE       0     0     0
        885a4df8-3a2d-489f-9459-c0e9c24b7320    ONLINE       0     0     0
        9dca1645-9099-4777-b727-d392954f2f0c    ONLINE       0     0     0
      mirror-1                                  ONLINE       0     0     0
        46480e4c-8076-4de8-b5b5-475c77a02826    ONLINE       0     0     0
        e203dc2a-3037-4904-85c0-bbc2ea542e57    ONLINE       0     0     0
      mirror-2                                  ONLINE       0     0     0
        0672cd9e-82cc-4907-9b6c-1e3d3f385ba6    ONLINE       0     0     0
        abc1c2e1-2844-4021-bac7-8afbea64254f    ONLINE       0     0     0
      mirror-3                                  ONLINE       0     0     0
        cb6b7724-8d54-482f-b0d1-894c06bf0d47    ONLINE       0     0     0
        9cbe3f26-1a7f-4349-9653-3ab2366cee30    ONLINE       0     0     0
      mirror-4                                  UNAVAIL      0     0     0  cannot open
        d3409bdf-9c34-48f7-8350-9a4fb05dfa83    ONLINE       0     0     0
        spare-1                                 DEGRADED     0     0     0
          54172042-d04d-4add-88dd-960c93cc5dc5  FAULTED     27     4    39  too many errors
          5dbac47a-f2ae-4571-a1b5-6e36addee7f4  ONLINE       0     0   401
      mirror-5                                  ONLINE       0     0     0
        b4305d01-a758-40a0-bfe9-c36d7569dea0    ONLINE       0     0     0
        e33c3303-5ff4-4575-b9a9-4e451dcdfc39    ONLINE       0     0     0
      mirror-6                                  ONLINE       0     0     0
        0780131e-8af7-4dea-b217-43ffc2b45e6d    ONLINE       0     0     0
        06fbf8d0-335f-4bdb-ac6d-6b3c5f698a8b    ONLINE       0     0     0
      mirror-7                                  ONLINE       0     0     0
        afac7e0a-de4c-47e1-92f5-0ed91d962434    ONLINE       0     0     0
        1d9921c0-72da-4a5d-b326-0dcb01c67cf0    ONLINE       0     0     0
      mirror-8                                  ONLINE       0     0     0
        697c2aee-1834-4621-a8ef-eb07bdb5b633    ONLINE       0     0     0
        12764d57-5e24-4888-a9aa-eaff1998118b    ONLINE       0     0     0
    spares
      5dbac47a-f2ae-4571-a1b5-6e36addee7f4      INUSE     currently in use
      a12ed467-6daa-456a-96a5-14d1705fbe51      AVAIL   

errors: No known data errors

Describe how to reproduce the problem

Begin a scrub on a pool with a spare INUSE after resolver completed.

Include any warning/errors/backtraces from the system logs

No relevant logs were generated

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: DefectIncorrect behavior (e.g. crash, hang)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions