Open
Description
Below is full command how to reproduce and full dmesg log.
As you can see there is a lot of space
uname -a
Linux work 6.8.0-51-generic #52~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Dec 9 15:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
btrfs-progs v5.16.2
kes@work ~/work/projects/raid $ dd if=/dev/zero of=disk1.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0,359154 s, 1,5 GB/s
kes@work ~/work/projects/raid $ dd if=/dev/zero of=disk2.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0,299275 s, 1,8 GB/s
kes@work ~/work/projects/raid $ sudo losetup /dev/loop10 disk1.img
sudo losetup /dev/loop11 disk2.img
kes@work ~/work/projects/raid $ sudo mkfs.btrfs -f -m raid1 -d raid1 /dev/loop10 /dev/loop11
btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.
Performing full device TRIM /dev/loop10 (500.00MiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Performing full device TRIM /dev/loop11 (500.00MiB) ...
Label: (null)
UUID: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
Node size: 16384
Sector size: 4096
Filesystem size: 1000.00MiB
Block group profiles:
Data: RAID1 64.00MiB
Metadata: RAID1 50.00MiB
System: RAID1 8.00MiB
SSD detected: yes
Zoned device: no
Incompat features: extref, skinny-metadata, no-holes
Runtime features: free-space-tree
Checksum: crc32c
Number of devices: 2
Devices:
ID SIZE PATH
1 500.00MiB /dev/loop10
2 500.00MiB /dev/loop11
kes@work ~/work/projects/raid $ sudo mkdir -p /mnt/raid_test
sudo mount /dev/loop10 /mnt/raid_test
kes@work ~/work/projects/raid $ sudo btrfs filesystem usage /mnt/raid_test
sudo btrfs filesystem show /mnt/raid_test
sudo btrfs filesystem df /mnt/raid_test
sudo btrfs device stats /mnt/raid_test
Overall:
Device size: 1000.00MiB
Device allocated: 244.00MiB
Device unallocated: 756.00MiB
Device missing: 0.00B
Used: 288.00KiB
Free (estimated): 442.00MiB (min: 442.00MiB)
Free (statfs, df): 441.00MiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
Data,RAID1: Size:64.00MiB, Used:0.00B (0.00%)
/dev/loop10 64.00MiB
/dev/loop11 64.00MiB
Metadata,RAID1: Size:50.00MiB, Used:128.00KiB (0.25%)
/dev/loop10 50.00MiB
/dev/loop11 50.00MiB
System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/loop10 8.00MiB
/dev/loop11 8.00MiB
Unallocated:
/dev/loop10 378.00MiB
/dev/loop11 378.00MiB
Label: none uuid: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
Total devices 2 FS bytes used 144.00KiB
devid 1 size 500.00MiB used 122.00MiB path /dev/loop10
devid 2 size 500.00MiB used 122.00MiB path /dev/loop11
Data, RAID1: total=64.00MiB, used=0.00B
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=50.00MiB, used=128.00KiB
GlobalReserve, single: total=5.50MiB, used=0.00B
[/dev/loop10].write_io_errs 0
[/dev/loop10].read_io_errs 0
[/dev/loop10].flush_io_errs 0
[/dev/loop10].corruption_errs 0
[/dev/loop10].generation_errs 0
[/dev/loop11].write_io_errs 0
[/dev/loop11].read_io_errs 0
[/dev/loop11].flush_io_errs 0
[/dev/loop11].corruption_errs 0
[/dev/loop11].generation_errs 0
kes@work ~/work/projects/raid $ cd /mnt/raid_test/
kes@work /mnt/raid_test $ dd^Cf=/dev/zero of=disk2.img bs=1M count=500
kes@work /mnt/raid_test $ sudo dd if=/dev/zero of=disk2.img^Cs=1M count=500
kes@work /mnt/raid_test $ sudo dd if=/dev/urandom of=test bs=1M count=400
400+0 records in
400+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 0,792103 s, 530 MB/s
kes@work /mnt/raid_test $ md5sum test | sudo tee -a md5sum
71770059735903f7af5c4d49848e87be test
kes@work /mnt/raid_test $ cat md5sum
71770059735903f7af5c4d49848e87be test
kes@work /mnt/raid_test $ sudo btrfs filesystem usage /mnt/raid_test
sudo btrfs filesystem show /mnt/raid_test
sudo btrfs filesystem df /mnt/raid_test
sudo btrfs device stats /mnt/raid_test
Overall:
Device size: 1000.00MiB
Device allocated: 958.00MiB
Device unallocated: 42.00MiB
Device missing: 0.00B
Used: 801.16MiB
Free (estimated): 42.00MiB (min: 42.00MiB)
Free (statfs, df): 41.00MiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
Data,RAID1: Size:421.00MiB, Used:400.00MiB (95.01%)
/dev/loop10 421.00MiB
/dev/loop11 421.00MiB
Metadata,RAID1: Size:50.00MiB, Used:576.00KiB (1.12%)
/dev/loop10 50.00MiB
/dev/loop11 50.00MiB
System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/loop10 8.00MiB
/dev/loop11 8.00MiB
Unallocated:
/dev/loop10 21.00MiB
/dev/loop11 21.00MiB
Label: none uuid: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
Total devices 2 FS bytes used 400.58MiB
devid 1 size 500.00MiB used 479.00MiB path /dev/loop10
devid 2 size 500.00MiB used 479.00MiB path /dev/loop11
Data, RAID1: total=421.00MiB, used=400.00MiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=50.00MiB, used=576.00KiB
GlobalReserve, single: total=5.50MiB, used=0.00B
[/dev/loop10].write_io_errs 0
[/dev/loop10].read_io_errs 0
[/dev/loop10].flush_io_errs 0
[/dev/loop10].corruption_errs 0
[/dev/loop10].generation_errs 0
[/dev/loop11].write_io_errs 0
[/dev/loop11].read_io_errs 0
[/dev/loop11].flush_io_errs 0
[/dev/loop11].corruption_errs 0
[/dev/loop11].generation_errs 0
kes@work /mnt/raid_test $ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1,6G 2,7M 1,6G 1% /run
efivarfs 192K 135K 53K 73% /sys/firmware/efi/efivars
/dev/nvme0n1p5 911G 180G 724G 20% /
tmpfs 7,7G 3,1M 7,7G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/nvme0n1p3 866M 389M 417M 49% /boot
/dev/nvme0n1p5 911G 180G 724G 20% /home
/dev/nvme0n1p1 127M 7,6M 119M 7% /boot/efi
/dev/sdd3 1,3T 959G 309G 76% /media/kes/STORAGE
tmpfs 1,6G 152K 1,6G 1% /run/user/1000
/dev/sdc1 932G 189G 722G 21% /media/kes/WORK-DATA
/dev/loop10 500M 407M 41M 91% /mnt/raid_test
kes@work /mnt/raid_test $ sudo umount /mnt/raid_test
sudo losetup -d /dev/loop10
sudo losetup -d /dev/loop11
umount: /mnt/raid_test: target is busy.
kes@work /mnt/raid_test $ cd ..
kes@work /mnt $ sudo umount /mnt/raid_test
sudo losetup -d /dev/loop10
sudo losetup -d /dev/loop11
losetup: /dev/loop10: detach failed: No such device or address
losetup: /dev/loop11: detach failed: No such device or address
kes@work /mnt $ sudo umount /mnt/raid_test
sudo losetup -d /dev/loop10
sudo losetup -d /dev/loop11
umount: /mnt/raid_test: not mounted.
losetup: /dev/loop10: detach failed: No such device or address
losetup: /dev/loop11: detach failed: No such device or address
kes@work /mnt $
kes@work /mnt $ sudo losetup -a
/dev/loop2: [0042]:5036995 (/home/kes/work/projects/raid/disk3.img)
kes@work /mnt $ sudo losetup -d /dev/loop2
kes@work /mnt $ sudo losetup -d /dev/loop2
kes@work /mnt $ sudo losetup -d /dev/loop2
kes@work /mnt $
kes@work ~/work/projects/raid $
kes@work ~/work/projects/raid $ dd if=/dev/urandom of=replace.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0,990082 s, 530 MB/s
kes@work ~/work/projects/raid $ sudo losetup /dev/loop10 disk1.img
kes@work ~/work/projects/raid $ sudo btrfs check /dev/loop10
Opening filesystem to check...
warning, device 2 is missing
Checking filesystem on /dev/loop10
UUID: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 420036608 bytes used, no error found
total csum bytes: 409600
total tree bytes: 606208
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 174146
file data blocks allocated: 419430400
referenced 419430400
kes@work ~/work/projects/raid $ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1,6G 2,7M 1,6G 1% /run
efivarfs 192K 135K 53K 73% /sys/firmware/efi/efivars
/dev/nvme0n1p5 911G 181G 724G 20% /
tmpfs 7,7G 3,1M 7,7G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/nvme0n1p3 866M 389M 417M 49% /boot
/dev/nvme0n1p5 911G 181G 724G 20% /home
/dev/nvme0n1p1 127M 7,6M 119M 7% /boot/efi
/dev/sdd3 1,3T 959G 309G 76% /media/kes/STORAGE
tmpfs 1,6G 152K 1,6G 1% /run/user/1000
/dev/sdc1 932G 189G 722G 21% /media/kes/WORK-DATA
kes@work ~/work/projects/raid $ sudo mount -o degraded /dev/loop10 /mnt/raid_test
kes@work ~/work/projects/raid $ mount | grep loop
/dev/loop10 on /mnt/raid_test type btrfs (rw,relatime,degraded,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
kes@work ~/work/projects/raid $ sudo btrfs filesystem show /mnt/raid_test
Label: none uuid: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
Total devices 2 FS bytes used 400.58MiB
devid 1 size 500.00MiB used 479.00MiB path /dev/loop10
*** Some devices missing
kes@work ~/work/projects/raid $ sudo btrfs replace start -B 2 /dev/loop12 /mnt/raid_test/
ERROR: cannot check mount status of /dev/loop12: No such device or address
kes@work ~/work/projects/raid $ sudo btrfs device remove missing /mnt/raid_test
ERROR: error removing device 'missing': unable to go below two devices on raid1
kes@work ~/work/projects/raid $
kes@work ~/work/projects/raid $ sudo btrfs device add /dev/loop12 /mnt/raid_test
ERROR: cannot check mount status of /dev/loop12: No such device or address
kes@work ~/work/projects/raid $
kes@work ~/work/projects/raid $ sudo losetup -a
/dev/loop2: [0042]:5036995 (/home/kes/work/projects/raid/disk3.img)
/dev/loop10: [0042]:5036973 (/home/kes/work/projects/raid/disk1.img)
kes@work ~/work/projects/raid $ sudo losetup /dev/loop12 replace.img
kes@work ~/work/projects/raid $ ls -la replace.img
-rw-rw-r-- 1 kes kes 524288000 Feb 5 00:09 replace.img
kes@work ~/work/projects/raid $ ls -la disk1
ls: cannot access 'disk1': No such file or directory
kes@work ~/work/projects/raid $ ls -la disk1.img
-rw-rw-r-- 1 kes kes 524288000 Feb 5 00:07 disk1.img
kes@work ~/work/projects/raid $ sudo btrfs replace start -B 2 /dev/loop12 /mnt/raid_test/
Performing full device TRIM /dev/loop12 (500.00MiB) ...
ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt/raid_test/": No space left on device
kes@work ~/work/projects/raid $ mount | grep loop
/dev/loop10 on /mnt/raid_test type btrfs (rw,relatime,degraded,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
kes@work ~/work/projects/raid $ sudo btrfs device add /dev/loop12 /mnt/raid_test
Performing full device TRIM /dev/loop12 (500.00MiB) ...
kes@work ~/work/projects/raid $
kes@work ~/work/projects/raid $ sudo btrfs device remove missing /mnt/raid_test
ERROR: error removing device 'missing': Read-only file system
kes@work ~/work/projects/raid $ btrfs fi usage -T
btrfs filesystem usage: not enough arguments: 0 but at least 1 expected
kes@work ~/work/projects/raid $ btrfs fi usage -T /mnt/raid_test/
WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
Overall:
Device size: 1.46GiB
Device allocated: 998.00MiB
Device unallocated: 502.00MiB
Device missing: 500.00MiB
Used: 801.16MiB
Free (estimated): 292.00MiB (min: 292.00MiB)
Free (statfs, df): 41.00MiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- -------------- --------- --------- -------- -----------
1 /dev/loop10 - - - 500.00MiB
2 <missing disk> - - - 0.00B
3 /dev/loop12 - - - 500.00MiB
-- -------------- --------- --------- -------- -----------
Total 441.00MiB 50.00MiB 8.00MiB 1000.00MiB
Used 400.00MiB 576.00KiB 16.00KiB
kes@work ~/work/projects/raid $ btrfs --version
btrfs-progs v5.16.2
kes@work ~/work/projects/raid $ sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/raid_test
ERROR: error during balancing '/mnt/raid_test': Read-only file system
There may be more info in syslog - try dmesg | tail
kes@work ~/work/projects/raid $
kes@work ~/work/projects/raid $ sudo umount /mnt/raid_test
kes@work ~/work/projects/raid $ sudo umount /mnt/raid_test
umount: /mnt/raid_test: not mounted.
kes@work ~/work/projects/raid $ sudo mount -o degraded /dev/loop10 /mnt/raid_test
kes@work ~/work/projects/raid $ mount | grep loop
/dev/loop10 on /mnt/raid_test type btrfs (rw,relatime,degraded,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
kes@work ~/work/projects/raid $ sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/raid_test
ERROR: error during balancing '/mnt/raid_test': Read-only file system
There may be more info in syslog - try dmesg | tail
kes@work ~/work/projects/raid $ sudo btrfs filesystem usage /mnt/raid_test
sudo btrfs filesystem show /mnt/raid_test
sudo btrfs filesystem df /mnt/raid_test
sudo btrfs device stats /mnt/raid_test
Overall:
Device size: 1.46GiB
Device allocated: 998.00MiB
Device unallocated: 502.00MiB
Device missing: 500.00MiB
Used: 801.16MiB
Free (estimated): 292.00MiB (min: 292.00MiB)
Free (statfs, df): 41.00MiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
Data,RAID1: Size:441.00MiB, Used:400.00MiB (90.70%)
/dev/loop10 441.00MiB
<missing disk> 421.00MiB
/dev/loop12 20.00MiB
Metadata,RAID1: Size:50.00MiB, Used:576.00KiB (1.12%)
/dev/loop10 50.00MiB
<missing disk> 50.00MiB
System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
/dev/loop10 8.00MiB
<missing disk> 8.00MiB
Unallocated:
/dev/loop10 1.00MiB
<missing disk> 21.00MiB
/dev/loop12 480.00MiB
Label: none uuid: b0ddefad-4c61-4db1-b5a6-545aa4753cc8
Total devices 3 FS bytes used 400.58MiB
devid 1 size 500.00MiB used 499.00MiB path /dev/loop10
devid 3 size 500.00MiB used 20.00MiB path /dev/loop12
*** Some devices missing
Data, RAID1: total=441.00MiB, used=400.00MiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=50.00MiB, used=576.00KiB
GlobalReserve, single: total=5.50MiB, used=0.00B
[/dev/loop10].write_io_errs 0
[/dev/loop10].read_io_errs 0
[/dev/loop10].flush_io_errs 0
[/dev/loop10].corruption_errs 0
[/dev/loop10].generation_errs 0
[<missing disk>].write_io_errs 0
[<missing disk>].read_io_errs 0
[<missing disk>].flush_io_errs 0
[<missing disk>].corruption_errs 0
[<missing disk>].generation_errs 0
[/dev/loop12].write_io_errs 0
[/dev/loop12].read_io_errs 0
[/dev/loop12].flush_io_errs 0
[/dev/loop12].corruption_errs 0
[/dev/loop12].generation_errs 0
START
[21820.579514] BTRFS info (device loop10: state EA): last unmount of filesystem 1829435a-a68c-464f-9e78-e9ad71ade889
[21897.424158] loop10: detected capacity change from 0 to 1024000
[21897.437951] loop11: detected capacity change from 0 to 1024000
[21907.601411] BTRFS: device fsid b0ddefad-4c61-4db1-b5a6-545aa4753cc8 devid 1 transid 6 /dev/loop10 scanned by mkfs.btrfs (29026)
[21907.601429] BTRFS: device fsid b0ddefad-4c61-4db1-b5a6-545aa4753cc8 devid 2 transid 6 /dev/loop11 scanned by mkfs.btrfs (29026)
[21915.131911] BTRFS info (device loop10): first mount of filesystem b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[21915.131933] BTRFS info (device loop10): using crc32c (crc32c-intel) checksum algorithm
[21915.131939] BTRFS info (device loop10): using free-space-tree
[21915.132979] BTRFS info (device loop10): checking UUID tree
[22073.404594] BTRFS info (device loop10): last unmount of filesystem b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[22204.219691] loop10: detected capacity change from 0 to 1024000
[22247.809452] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.809467] ata7.00: irq_stat 0x40000001
[22247.809470] ata7.00: failed command: READ DMA EXT
[22247.809472] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 0 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.809479] ata7.00: status: { DRDY DF ERR }
[22247.809482] ata7.00: error: { ABRT }
[22247.810649] ata7.00: failed to enable AA (error_mask=0x1)
[22247.812216] ata7.00: failed to enable AA (error_mask=0x1)
[22247.812228] ata7.00: configured for UDMA/133 (device error ignored)
[22247.812258] ata7: EH complete
[22247.826476] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.826484] ata7.00: irq_stat 0x40000001
[22247.826487] ata7.00: failed command: READ DMA EXT
[22247.826489] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 27 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.826497] ata7.00: status: { DRDY DF ERR }
[22247.826499] ata7.00: error: { ABRT }
[22247.827595] ata7.00: failed to enable AA (error_mask=0x1)
[22247.829136] ata7.00: failed to enable AA (error_mask=0x1)
[22247.829146] ata7.00: configured for UDMA/133 (device error ignored)
[22247.830241] ata7: EH complete
[22247.844444] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.844454] ata7.00: irq_stat 0x40000001
[22247.844457] ata7.00: failed command: READ DMA EXT
[22247.844459] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 6 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.844465] ata7.00: status: { DRDY DF ERR }
[22247.844467] ata7.00: error: { ABRT }
[22247.845539] ata7.00: failed to enable AA (error_mask=0x1)
[22247.847086] ata7.00: failed to enable AA (error_mask=0x1)
[22247.847102] ata7.00: configured for UDMA/133 (device error ignored)
[22247.847134] ata7: EH complete
[22247.863448] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.863462] ata7.00: irq_stat 0x40000001
[22247.863465] ata7.00: failed command: READ DMA EXT
[22247.863467] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 11 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.863474] ata7.00: status: { DRDY DF ERR }
[22247.863476] ata7.00: error: { ABRT }
[22247.864567] ata7.00: failed to enable AA (error_mask=0x1)
[22247.865775] ata7.00: failed to enable AA (error_mask=0x1)
[22247.865779] ata7.00: configured for UDMA/133 (device error ignored)
[22247.865792] ata7: EH complete
[22247.888445] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.888453] ata7.00: irq_stat 0x40000001
[22247.888458] ata7.00: failed command: READ DMA EXT
[22247.888462] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 14 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.888469] ata7.00: status: { DRDY DF ERR }
[22247.888471] ata7.00: error: { ABRT }
[22247.889566] ata7.00: failed to enable AA (error_mask=0x1)
[22247.891082] ata7.00: failed to enable AA (error_mask=0x1)
[22247.891091] ata7.00: configured for UDMA/133 (device error ignored)
[22247.891115] ata7: EH complete
[22247.906446] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22247.906457] ata7.00: irq_stat 0x40000001
[22247.906459] ata7.00: failed command: READ DMA EXT
[22247.906460] ata7.00: cmd 25/00:08:80:67:70/00:00:74:00:00/e0 tag 19 dma 4096 in
res 61/04:08:80:67:70/00:00:74:00:00/e0 Emask 0x1 (device error)
[22247.906465] ata7.00: status: { DRDY DF ERR }
[22247.906467] ata7.00: error: { ABRT }
[22247.907575] ata7.00: failed to enable AA (error_mask=0x1)
[22247.909091] ata7.00: failed to enable AA (error_mask=0x1)
[22247.909099] ata7.00: configured for UDMA/133 (device error ignored)
[22247.909127] sd 6:0:0:0: [sdc] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[22247.909132] sd 6:0:0:0: [sdc] tag#19 Sense Key : Illegal Request [current]
[22247.909135] sd 6:0:0:0: [sdc] tag#19 Add. Sense: Unaligned write command
[22247.909138] sd 6:0:0:0: [sdc] tag#19 CDB: Read(10) 28 00 74 70 67 80 00 00 08 00
[22247.909140] I/O error, dev sdc, sector 1953523584 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[22247.909146] Buffer I/O error on dev sdc1, logical block 244190192, async page read
[22247.909155] ata7: EH complete
This happened after sudo btrfs check /dev/loop10
^^^
[22335.754843] BTRFS info (device loop10): first mount of filesystem b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[22335.754864] BTRFS info (device loop10): using crc32c (crc32c-intel) checksum algorithm
[22335.754870] BTRFS info (device loop10): using free-space-tree
[22335.755445] BTRFS warning (device loop10): devid 2 uuid 67b13035-2b37-4f47-9fda-5c3e29a18846 is missing
replace
no info after the command
...
trying to add/remove
[22519.633087] loop12: detected capacity change from 0 to 1024000
[22552.403638] BTRFS info (device loop10): dev_replace from <missing disk> (devid 2) to /dev/loop12 started
[22552.411753] BTRFS warning (device loop10): failed setting block group ro: -28
[22552.413647] BTRFS error (device loop10): btrfs_scrub_dev(<missing disk>, 2, /dev/loop12) failed -28
This output above is after: sudo btrfs replace start -B 2 /dev/loop12 /mnt/raid_test/
[22624.944454] BTRFS info (device loop10): disk added /dev/loop12
[22642.509945] BTRFS info (device loop10): relocating block group 502267904 flags data|raid1
[22642.514864] BTRFS info (device loop10): relocating block group 384827392 flags data|raid1
[22642.519928] BTRFS info (device loop10): relocating block group 267386880 flags data|raid1
[22642.526213] BTRFS info (device loop10): relocating block group 149946368 flags data|raid1
[22642.531831] BTRFS info (device loop10): relocating block group 82837504 flags data|raid1
[22642.537982] BTRFS info (device loop10): relocating block group 30408704 flags metadata|raid1
[22642.538000] ------------[ cut here ]------------
[22642.538002] BTRFS: Transaction aborted (error -28)
[22642.538060] WARNING: CPU: 2 PID: 30356 at fs/btrfs/extent-tree.c:3257 __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22642.538167] Modules linked in: xfs jfs nls_ucs2_utils overlay binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof intel_rapl_msr intel_rapl_common snd_sof_utils snd_hda_codec_realtek intel_uncore_frequency snd_soc_hdac_hda intel_uncore_frequency_common snd_hda_codec_generic snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_generic_allocation soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine x86_pkg_temp_thermal uvcvideo intel_powerclamp snd_hda_intel videobuf2_vmalloc coretemp snd_intel_dspcfg uvc snd_intel_sdw_acpi videobuf2_memops videobuf2_v4l2 snd_usb_audio snd_hda_codec kvm_intel videodev snd_hda_core snd_usbmidi_lib 8821au(OE) snd_hwdep videobuf2_common snd_ump mei_pxp mei_hdcp mc nls_iso8859_1 snd_seq_midi snd_pcm cfg80211 kvm input_leds joydev snd_seq_midi_event irqbypass snd_rawmidi snd_seq rapl intel_cstate snd_seq_device snd_timer
[22642.538227] eeepc_wmi cmdlinepart snd spi_nor wmi_bmof mtd ee1004 soundcore mei_me mei intel_pmc_core intel_vsec pmt_telemetry acpi_tad acpi_pad pmt_class mac_hid sch_fq_codel nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport efi_pstore sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 dm_mirror dm_region_hash dm_log hid_logitech_hidpp hid_logitech_dj xe drm_gpuvm drm_exec hid_generic gpu_sched drm_suballoc_helper usbhid drm_ttm_helper hid i915 crct10dif_pclmul crc32_pclmul nvme mfd_aaeon polyval_clmulni drm_buddy asus_wmi i2c_algo_bit polyval_generic ledtrig_audio nvme_core ghash_clmulni_intel ttm sparse_keymap sha256_ssse3 spi_intel_pci platform_profile sha1_ssse3 drm_display_helper i2c_i801 spi_intel intel_lpss_pci nvme_auth e1000e cec ahci intel_lpss i2c_smbus xhci_pci libahci idma64 vmd xhci_pci_renesas rc_core video wmi pinctrl_alderlake aesni_intel crypto_simd cryptd
[22642.538348] CPU: 2 PID: 30356 Comm: btrfs Tainted: G W OE 6.8.0-51-generic #52~22.04.1-Ubuntu
[22642.538354] Hardware name: ASUS System Product Name/PRIME H610M-A D4, BIOS 0412 09/29/2021
[22642.538357] RIP: 0010:__btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22642.538441] Code: 9d 60 ff ff ff 44 8b 85 7c ff ff ff ba fb 0c 00 00 89 d9 e9 e7 f8 ff ff 8b b5 7c ff ff ff 48 c7 c7 18 8c 0b c1 e8 7f 22 5e f0 <0f> 0b e9 4c f9 ff ff 8b bd 58 ff ff ff e8 bd d7 fe ff 84 c0 0f 85
[22642.538444] RSP: 0018:ffffad3e89e8f830 EFLAGS: 00010246
[22642.538447] RAX: 0000000000000000 RBX: 0000000001e60000 RCX: 0000000000000000
[22642.538449] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[22642.538451] RBP: ffffad3e89e8f8f8 R08: 0000000000000000 R09: 0000000000000000
[22642.538452] R10: 0000000000000000 R11: 0000000000000000 R12: fffffffffffffff7
[22642.538454] R13: 0000000000000001 R14: ffff9bd962d67068 R15: ffff9bd9580f0620
[22642.538455] FS: 0000774fc01538c0(0000) GS:ffff9bdaef500000(0000) knlGS:0000000000000000
[22642.538458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[22642.538460] CR2: 00007d7dc70ca000 CR3: 0000000450466004 CR4: 0000000000f70ef0
[22642.538462] PKRU: 55555554
[22642.538464] Call Trace:
[22642.538468] <TASK>
[22642.538475] ? show_regs+0x6d/0x80
[22642.538485] ? __warn+0x89/0x160
[22642.538490] ? __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22642.538566] ? report_bug+0x17e/0x1b0
[22642.538575] ? handle_bug+0x46/0x90
[22642.538584] ? exc_invalid_op+0x18/0x80
[22642.538590] ? asm_exc_invalid_op+0x1b/0x20
[22642.538602] ? __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22642.538669] run_delayed_tree_ref+0x92/0x200 [btrfs]
[22642.538736] btrfs_run_delayed_refs_for_head+0x2db/0x530 [btrfs]
[22642.538814] __btrfs_run_delayed_refs+0xf9/0x1a0 [btrfs]
[22642.538878] btrfs_run_delayed_refs+0x84/0x130 [btrfs]
[22642.538941] btrfs_commit_transaction+0x6a/0xbe0 [btrfs]
[22642.539024] prepare_to_relocate+0x141/0x1d0 [btrfs]
[22642.539130] relocate_block_group+0x6a/0x540 [btrfs]
[22642.539228] ? btrfs_wait_ordered_roots+0x1f8/0x230 [btrfs]
[22642.539340] ? btrfs_wait_nocow_writers+0x29/0xd0 [btrfs]
[22642.539470] btrfs_relocate_block_group+0x28c/0x3e0 [btrfs]
[22642.539580] btrfs_relocate_chunk+0x40/0x1b0 [btrfs]
[22642.539685] btrfs_shrink_device+0x27e/0x600 [btrfs]
[22642.539786] btrfs_rm_device+0x18f/0x700 [btrfs]
[22642.539881] ? __check_object_size.part.0+0x72/0x150
[22642.539889] ? btrfs_get_dev_args_from_path+0x5b/0x240 [btrfs]
[22642.539982] btrfs_ioctl_rm_dev_v2+0x1a6/0x250 [btrfs]
[22642.540080] btrfs_ioctl+0xe9f/0x13a0 [btrfs]
[22642.540179] ? __count_memcg_events+0x80/0x130
[22642.540184] __x64_sys_ioctl+0xa0/0xf0
[22642.540191] x64_sys_call+0xa71/0x2480
[22642.540195] do_syscall_64+0x81/0x170
[22642.540199] ? irqentry_exit+0x43/0x50
[22642.540203] ? exc_page_fault+0x94/0x1b0
[22642.540207] entry_SYSCALL_64_after_hwframe+0x78/0x80
[22642.540212] RIP: 0033:0x774fbff1a94f
[22642.540252] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41> 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00
[22642.540255] RSP: 002b:00007fffd4b8d5c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[22642.540259] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000774fbff1a94f
[22642.540261] RDX: 00007fffd4b8d638 RSI: 000000005000943a RDI: 0000000000000003
[22642.540263] RBP: 00007fffd4b8f7e8 R08: 00007fffd4b8d670 R09: 000000007fffffff
[22642.540265] R10: 0000000000000017 R11: 0000000000000246 R12: 0000000000000003
[22642.540267] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[22642.540270] </TASK>
[22642.540272] ---[ end trace 0000000000000000 ]---
[22642.540275] BTRFS info (device loop10: state A): dumping space info:
[22642.540279] BTRFS info (device loop10: state A): space_info DATA has 42926080 free, is full
[22642.540282] BTRFS info (device loop10: state A): space_info total=462422016, used=419430400, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[22642.540286] BTRFS info (device loop10: state A): space_info METADATA has -11010048 free, is full
[22642.540289] BTRFS info (device loop10: state A): space_info total=52428800, used=589824, pinned=0, reserved=16384, may_use=11010048, readonly=51822592 zone_unusable=0
[22642.540293] BTRFS info (device loop10: state A): space_info SYSTEM has 8372224 free, is not full
[22642.540297] BTRFS info (device loop10: state A): space_info total=8388608, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
[22642.540302] BTRFS info (device loop10: state A): global_block_rsv: size 5767168 reserved 5767168
[22642.540306] BTRFS info (device loop10: state A): trans_block_rsv: size 0 reserved 0
[22642.540309] BTRFS info (device loop10: state A): chunk_block_rsv: size 0 reserved 0
[22642.540311] BTRFS info (device loop10: state A): delayed_block_rsv: size 0 reserved 0
[22642.540313] BTRFS info (device loop10: state A): delayed_refs_rsv: size 1048576 reserved 1048576
[22642.540316] BTRFS: error (device loop10: state A) in __btrfs_free_extent:3257: errno=-28 No space left
[22642.540322] BTRFS info (device loop10: state EA): forced readonly
[22642.540325] BTRFS error (device loop10: state EA): failed to run delayed ref for logical 31850496 num_bytes 16384 type 176 action 2 ref_mod 1: -28
[22642.540334] BTRFS: error (device loop10: state EA) in btrfs_run_delayed_refs:2261: errno=-28 No space left
The output above is after this command: sudo btrfs device remove missing /mnt/raid_test
[22882.909822] BTRFS info (device loop10: state EA): last unmount of filesystem b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[22900.689380] BTRFS info (device loop10): first mount of filesystem b0ddefad-4c61-4db1-b5a6-545aa4753cc8
[22900.689407] BTRFS info (device loop10): using crc32c (crc32c-intel) checksum algorithm
[22900.689413] BTRFS info (device loop10): using free-space-tree
[22900.689911] BTRFS warning (device loop10): devid 2 uuid 67b13035-2b37-4f47-9fda-5c3e29a18846 is missing
[22900.691215] BTRFS info (device loop10): checking UUID tree
[22916.495490] BTRFS info (device loop10): balance: start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1
[22916.495532] BTRFS info (device loop10): relocating block group 524288000 flags data|raid1
[22916.502036] BTRFS info (device loop10): relocating block group 502267904 flags data|raid1
[22916.506804] BTRFS info (device loop10): relocating block group 384827392 flags data|raid1
[22916.512855] BTRFS info (device loop10): relocating block group 267386880 flags data|raid1
[22916.518271] BTRFS info (device loop10): relocating block group 149946368 flags data|raid1
[22916.523236] BTRFS info (device loop10): relocating block group 82837504 flags data|raid1
[22916.529147] BTRFS info (device loop10): relocating block group 30408704 flags metadata|raid1
[22916.529163] ------------[ cut here ]------------
[22916.529165] BTRFS: Transaction aborted (error -28)
[22916.529216] WARNING: CPU: 0 PID: 30725 at fs/btrfs/extent-tree.c:3257 __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22916.529316] Modules linked in: xfs jfs nls_ucs2_utils overlay binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof intel_rapl_msr intel_rapl_common snd_sof_utils snd_hda_codec_realtek intel_uncore_frequency snd_soc_hdac_hda intel_uncore_frequency_common snd_hda_codec_generic snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_generic_allocation soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine x86_pkg_temp_thermal uvcvideo intel_powerclamp snd_hda_intel videobuf2_vmalloc coretemp snd_intel_dspcfg uvc snd_intel_sdw_acpi videobuf2_memops videobuf2_v4l2 snd_usb_audio snd_hda_codec kvm_intel videodev snd_hda_core snd_usbmidi_lib 8821au(OE) snd_hwdep videobuf2_common snd_ump mei_pxp mei_hdcp mc nls_iso8859_1 snd_seq_midi snd_pcm cfg80211 kvm input_leds joydev snd_seq_midi_event irqbypass snd_rawmidi snd_seq rapl intel_cstate snd_seq_device snd_timer
[22916.529375] eeepc_wmi cmdlinepart snd spi_nor wmi_bmof mtd ee1004 soundcore mei_me mei intel_pmc_core intel_vsec pmt_telemetry acpi_tad acpi_pad pmt_class mac_hid sch_fq_codel nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport efi_pstore sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 dm_mirror dm_region_hash dm_log hid_logitech_hidpp hid_logitech_dj xe drm_gpuvm drm_exec hid_generic gpu_sched drm_suballoc_helper usbhid drm_ttm_helper hid i915 crct10dif_pclmul crc32_pclmul nvme mfd_aaeon polyval_clmulni drm_buddy asus_wmi i2c_algo_bit polyval_generic ledtrig_audio nvme_core ghash_clmulni_intel ttm sparse_keymap sha256_ssse3 spi_intel_pci platform_profile sha1_ssse3 drm_display_helper i2c_i801 spi_intel intel_lpss_pci nvme_auth e1000e cec ahci intel_lpss i2c_smbus xhci_pci libahci idma64 vmd xhci_pci_renesas rc_core video wmi pinctrl_alderlake aesni_intel crypto_simd cryptd
[22916.529456] CPU: 0 PID: 30725 Comm: btrfs Tainted: G W OE 6.8.0-51-generic #52~22.04.1-Ubuntu
[22916.529461] Hardware name: ASUS System Product Name/PRIME H610M-A D4, BIOS 0412 09/29/2021
[22916.529463] RIP: 0010:__btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22916.529538] Code: 9d 60 ff ff ff 44 8b 85 7c ff ff ff ba fb 0c 00 00 89 d9 e9 e7 f8 ff ff 8b b5 7c ff ff ff 48 c7 c7 18 8c 0b c1 e8 7f 22 5e f0 <0f> 0b e9 4c f9 ff ff 8b bd 58 ff ff ff e8 bd d7 fe ff 84 c0 0f 85
[22916.529540] RSP: 0018:ffffad3e8a7b75d8 EFLAGS: 00010246
[22916.529544] RAX: 0000000000000000 RBX: 0000000001e58000 RCX: 0000000000000000
[22916.529546] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[22916.529547] RBP: ffffad3e8a7b76a0 R08: 0000000000000000 R09: 0000000000000000
[22916.529549] R10: 0000000000000000 R11: 0000000000000000 R12: fffffffffffffff7
[22916.529550] R13: 0000000000000001 R14: ffff9bd9c0108bc8 R15: ffff9bd876e8dd20
[22916.529552] FS: 00007889a3a8e8c0(0000) GS:ffff9bdaef400000(0000) knlGS:0000000000000000
[22916.529555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[22916.529557] CR2: 00007d7dc70cb000 CR3: 0000000314d5e002 CR4: 0000000000f70ef0
[22916.529559] PKRU: 55555554
[22916.529560] Call Trace:
[22916.529562] <TASK>
[22916.529566] ? show_regs+0x6d/0x80
[22916.529572] ? __warn+0x89/0x160
[22916.529576] ? __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22916.529647] ? report_bug+0x17e/0x1b0
[22916.529654] ? handle_bug+0x46/0x90
[22916.529660] ? exc_invalid_op+0x18/0x80
[22916.529675] ? asm_exc_invalid_op+0x1b/0x20
[22916.529683] ? __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22916.529768] ? __btrfs_free_extent.isra.0+0xc21/0x11a0 [btrfs]
[22916.529871] run_delayed_tree_ref+0x92/0x200 [btrfs]
[22916.529941] btrfs_run_delayed_refs_for_head+0x2db/0x530 [btrfs]
[22916.530008] __btrfs_run_delayed_refs+0xf9/0x1a0 [btrfs]
[22916.530077] btrfs_run_delayed_refs+0x84/0x130 [btrfs]
[22916.530144] btrfs_commit_transaction+0x6a/0xbe0 [btrfs]
[22916.530232] prepare_to_relocate+0x141/0x1d0 [btrfs]
[22916.530347] relocate_block_group+0x6a/0x540 [btrfs]
[22916.530451] ? btrfs_wait_ordered_roots+0x1f8/0x230 [btrfs]
[22916.530552] ? btrfs_wait_nocow_writers+0x29/0xd0 [btrfs]
[22916.530657] btrfs_relocate_block_group+0x28c/0x3e0 [btrfs]
[22916.530777] btrfs_relocate_chunk+0x40/0x1b0 [btrfs]
[22916.530905] __btrfs_balance+0x325/0x550 [btrfs]
[22916.531007] btrfs_balance+0x52e/0x990 [btrfs]
[22916.531102] btrfs_ioctl_balance+0x24e/0x3c0 [btrfs]
[22916.531199] btrfs_ioctl+0x5de/0x13a0 [btrfs]
[22916.531296] ? mod_memcg_lruvec_state+0x2b/0x60
[22916.531301] ? mod_objcg_state+0x1a7/0x2c0
[22916.531307] ? xa_load+0x87/0xf0
[22916.531310] __x64_sys_ioctl+0xa0/0xf0
[22916.531316] x64_sys_call+0xa71/0x2480
[22916.531320] do_syscall_64+0x81/0x170
[22916.531324] ? __fput+0x15e/0x2e0
[22916.531329] ? kmem_cache_free+0x36c/0x400
[22916.531333] ? __fput+0x15e/0x2e0
[22916.531337] ? __fput_sync+0x1c/0x30
[22916.531341] ? syscall_exit_to_user_mode+0x83/0x260
[22916.531346] ? do_syscall_64+0x8d/0x170
[22916.531350] ? generic_file_llseek+0x24/0x40
[22916.531353] ? kernfs_fop_llseek+0x70/0xb0
[22916.531357] ? ksys_lseek+0x7d/0xd0
[22916.531360] ? syscall_exit_to_user_mode+0x83/0x260
[22916.531364] ? do_syscall_64+0x8d/0x170
[22916.531367] ? __x64_sys_openat+0x6c/0xa0
[22916.531372] ? syscall_exit_to_user_mode+0x83/0x260
[22916.531376] ? do_syscall_64+0x8d/0x170
[22916.531380] ? irqentry_exit+0x43/0x50
[22916.531383] ? exc_page_fault+0x94/0x1b0
[22916.531388] entry_SYSCALL_64_after_hwframe+0x78/0x80
[22916.531392] RIP: 0033:0x7889a391a94f
[22916.531424] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41> 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00
[22916.531426] RSP: 002b:00007ffd5b02d740 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[22916.531430] RAX: ffffffffffffffda RBX: 00007ffd5b02d838 RCX: 00007889a391a94f
[22916.531432] RDX: 00007ffd5b02d838 RSI: 00000000c4009420 RDI: 0000000000000003
[22916.531433] RBP: 0000000000000003 R08: 000000000000006e R09: 000000007fffffff
[22916.531435] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[22916.531437] R13: 0000000000000000 R14: 00007ffd5b02f75a R15: 0000000000000001
[22916.531440] </TASK>
[22916.531441] ---[ end trace 0000000000000000 ]---
[22916.531444] BTRFS info (device loop10: state A): dumping space info:
[22916.531447] BTRFS info (device loop10: state A): space_info DATA has 42926080 free, is full
[22916.531450] BTRFS info (device loop10: state A): space_info total=462422016, used=419430400, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[22916.531454] BTRFS info (device loop10: state A): space_info METADATA has -11010048 free, is full
[22916.531456] BTRFS info (device loop10: state A): space_info total=52428800, used=589824, pinned=0, reserved=16384, may_use=11010048, readonly=51822592 zone_unusable=0
[22916.531460] BTRFS info (device loop10: state A): space_info SYSTEM has 8372224 free, is not full
[22916.531462] BTRFS info (device loop10: state A): space_info total=8388608, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
[22916.531465] BTRFS info (device loop10: state A): global_block_rsv: size 5767168 reserved 5767168
[22916.531467] BTRFS info (device loop10: state A): trans_block_rsv: size 0 reserved 0
[22916.531469] BTRFS info (device loop10: state A): chunk_block_rsv: size 0 reserved 0
[22916.531470] BTRFS info (device loop10: state A): delayed_block_rsv: size 0 reserved 0
[22916.531472] BTRFS info (device loop10: state A): delayed_refs_rsv: size 1048576 reserved 1048576
[22916.531474] BTRFS: error (device loop10: state A) in __btrfs_free_extent:3257: errno=-28 No space left
[22916.531479] BTRFS info (device loop10: state EA): forced readonly
[22916.531481] BTRFS error (device loop10: state EA): failed to run delayed ref for logical 31817728 num_bytes 16384 type 176 action 2 ref_mod 1: -28
[22916.531486] BTRFS: error (device loop10: state EA) in btrfs_run_delayed_refs:2261: errno=-28 No space left
[22916.531502] BTRFS info (device loop10: state EA): 6 enospc errors during balance
[22916.531505] BTRFS info (device loop10: state EA): balance: ended with status: -30