{"id":1942,"date":"2017-03-14T21:32:04","date_gmt":"2017-03-14T21:32:04","guid":{"rendered":"http:\/\/rglinuxtech.com\/?p=1942"},"modified":"2017-03-14T21:32:04","modified_gmt":"2017-03-14T21:32:04","slug":"kernel-4-11-rc1-and-btrfs-a-warning","status":"publish","type":"post","link":"https:\/\/rglinuxtech.com\/?p=1942","title":{"rendered":"Kernel – 4.11-rc1 and BTRFS – a Warning..?"},"content":{"rendered":"

I encountered a catastrophic problem with btrfs<\/strong> formatted system disk partition corruption, shortly after updating a test system to Kernel 4.11-rc1<\/strong>.<\/p>\n

To put this in context, this may<\/em> be an isolated incident, and I have not seen problems – so far – mentioned elsewhere..\u00a0\u00a0\u00a0 But.. there a quite a lot of BTRFS changes on 4.11, and – after the usual tests – I confirmed that system memory was OK (running memtest86+ several times..) and the disk was OK (running smartmontools<\/em> utilities etc. several times).\u00a0\u00a0\u00a0 The disk is only one year old, in any case (WD – 2GB)..<\/p>\n

Unfortunately, I did not have a serial console attached to this particular system\u00a0 at the time, and so had to photograph the screen at various times, and then re-type the details below from those photos (!)..<\/p>\n

The scenario:<\/p>\n

System disk (\/dev\/sda2) is formatted btrfs..
\nUpdated to Kernel 4.11-rc1.
\nWithin 24 hours, the system froze, and on rebooting, got a critical btrfs error:<\/p>\n

...................<\/span>\r\n Mounting \/sysroot....<\/strong><\/span>\r\nBTRFS critical (device sda2): corrupt node, bad key order: block=368346054656, root=1, slot=192<\/strong><\/span>\r\nBTRFS critical (device sda2): corrupt node, bad key order: block=368346054656, root=1, slot=192<\/strong><\/span>\r\nBTRFS error (device sda2): failed to read block groups: -5<\/strong><\/span>\r\nBTRFS error (device sda2): open_ctree failed<\/strong><\/span>\r\n[FAILED] Failed to mount \/sysroot<\/strong><\/span>\r\nSee 'systemctl status sysroon.mount' for details<\/strong><\/span>\r\n[DEPEND] Dependency failed for Initrd Root File System.<\/strong><\/span>\r\n[DEPEND] Dependency failed for Reload Configuration from the Real Root.<\/strong><\/span>\r\n......................<\/span>\r\n Starting Emergency Shell<\/strong>...<\/span><\/pre>\n
Ran:<\/p>\n
:\/# btrfs check \/dev\/sda2<\/span><\/strong>\r\nChecking filesystem on \/dev\/sda2<\/span><\/strong>\r\nUUID xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx<\/span><\/strong>\r\nchecking extents<\/span><\/strong>\r\nbad block 33488896<\/span><\/strong>\r\nErrors found in extent allocation tree or chunk allocation<\/span><\/strong>\r\n^C<\/span><\/strong>\r\n:\/# btrfs rescue chunk-recover \/dev\/sda2<\/span><\/strong>\r\nScanning: 66126696448 in dev0<\/span><\/strong><\/pre>\n
Then got scrolling screens full of:<\/p>\n
...........\r\nDeleting bad dir index [537997,96,160] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,168] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,170] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,180] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,186] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,190] root 5<\/strong><\/span>\r\nDeleting bad dir index [537997,96,198] root 5<\/strong><\/span>\r\n................ continuing...<\/pre>\n
Then more scrolling screens full of:<\/p>\n
...............\r\nTrying to rebuild inode:1658592<\/span><\/strong>\r\nroot 5 inode 1658592 error 2001, no inode item, link count wrong<\/span><\/strong>\r\n unresolved ref dir 198531 index 0 namelen 18 name gtk-indent-ltr.png filetype 7 errors 6, no dir index, no inode ref<\/span><\/strong>\r\nTrying to rebuild inode:1658593<\/span><\/strong>\r\nroot 5 inode 1658593 error 2001, no inode item, link count wrong<\/span><\/strong>\r\n unresolved ref dir 198531 index 0 namelen 18 name gtk-indent-rtl.png filetype 7 errors 6, no dir index, no inode ref<\/span><\/strong>\r\n............... continuing....<\/pre>\n
Then more scrolling screens full of:<\/p>\n
..............\r\nrepairing missing dir index item for inode 30734433<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734434<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734435<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734436<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734437<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734438<\/span><\/strong>\r\nrepairing missing dir index item for inode 30734439<\/span><\/strong>\r\n............ continuing...<\/pre>\n
Then more scrolling screens full of:<\/p>\n
..............\r\nDeleting bad dir index [363308,96,63443] root 5<\/span><\/strong>\r\nDeleting bad dir index [363396,96,12025] root 5<\/span><\/strong>\r\nThe following tree block(s) is corrupted in tree 5:<\/span><\/strong>\r\n tree block bytenr: 44061900800, level: 1, node key: (365821249776, 168, 45056)<\/span><\/strong>\r\nTry to repair the btree for root 5<\/span><\/strong>\r\nBtree for root 5 is fixed<\/span><\/strong>\r\nDeleting bad dir index [364225,96,3382] root 5<\/span><\/strong>\r\nDeleting bad dir index [363308,96,63425] root 5<\/span><\/strong>\r\nDeleting bad dir index [363308,96,63427] root 5<\/span><\/strong>\r\nDeleting bad dir index [363308,96,63427] root 5<\/span><\/strong>\r\n........... continuing ...........<\/pre>\n
The ‘recovery’ kept running, although appearing to stall, several times, but I just left it and it ran for almost three days<\/strong> in total, and finally ended:<\/p>\n
..............\r\nBtree for root 5 is fixed<\/strong><\/span>\r\nroot 5 root dir 256 error<\/strong><\/span>\r\nroot 5 inode 256 errors 200, dir isize wrong<\/strong><\/span>\r\nreset isize for dir 2139040 root 5<\/strong><\/span>\r\nreset isize for dir 2139040 root 5<\/strong><\/span>\r\n..................\r\nTrying to rebuild inode:33318043<\/span><\/strong>\r\nmoving file 'lost+found' to 'lost+found' dir since it has no valid backref<\/span><\/strong>\r\nFixed the nlink of inode 33318043<\/span><\/strong>\r\nfound 583909376 bytes used err is 1<\/span><\/strong>\r\ntotal csum bytes: 0<\/span><\/strong>\r\ntotal tree bytes: 1884160<\/span><\/strong>\r\ntotal fs tree bytes: 0<\/span><\/strong>\r\ntotal extent tree bytes: 1474560<\/span><\/strong>\r\nbtree space waste bytes: 758524<\/span><\/strong>\r\nfile data blocks allocated: 201064448<\/span><\/strong>\r\n referenced 201064448<\/span><\/strong>\r\n:\/#<\/span><\/strong>\r\n:\/#<\/span><\/strong><\/pre>\n
After all that, I rebooted:<\/p>\n
...................\r\n Mounting \/sysroot....<\/strong><\/span>\r\nBTRFS critical (device sda2): corrupt node, bad key order: block=44061900800, root=1, slot=192<\/strong><\/span>\r\nBTRFS critical (device sda2): corrupt node, bad key order: block=44061900800, root=1, slot=192<\/strong><\/span>\r\nBTRFS error (device sda2): failed to read block groups: -5<\/strong><\/span>\r\nBTRFS error (device sda2): open_ctree failed<\/strong><\/span>\r\n[FAILED] Failed to mount \/sysroot<\/strong><\/span>\r\nSee 'systemctl status sysroon.mount' for details<\/strong><\/span>\r\n[DEPEND] Dependency failed for Initrd Root File System.<\/strong><\/span>\r\n[DEPEND] Dependency failed for Reload Configuration from the Real Root.<\/strong><\/span>\r\n....................\r\nEntering emergency mode<\/strong><\/span>..............<\/pre>\n
So… The btrfs recovery was a waste of three days, and did not fix the problem.. I will now have to re-create the data from another source.. Fortunately this was ‘just’ the system disk partition and no irreplaceable recent user data was lost, apart from \/root<\/span><\/strong>…<\/p>\n
I should mention as well, that all other btrfs-formatted partitions on other systems, running 4.10.x and earlier, have been error-free, and reliable.\u00a0\u00a0 And… I have not<\/span> tried 4.11-rc2<\/strong> with BTRFS, yet..<\/p>\n
Robert Gadsdon.\u00a0\u00a0 March 14, 2017.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
I encountered a catastrophic problem with btrfs formatted system disk partition corruption, shortly after updating a test system to Kernel 4.11-rc1. To put this in context, this may be an isolated incident, and I have not seen problems – so far – mentioned elsewhere..\u00a0\u00a0\u00a0 But.. there a quite a lot of BTRFS changes on 4.11, and – …<\/span> Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1280,1566,6,11,14,15,19,20],"tags":[1569,1567,1552,1570,1568],"class_list":["post-1942","post","type-post","status-publish","format-standard","hentry","category-boot","category-btrfs","category-crash-and-burn","category-hacks","category-kernel","category-linux-2","category-opinion","category-performance-2","tag-bad-block","tag-btrfs-errors","tag-kernel-4-11-rc1","tag-open_ctree-failed","tag-recovery-failure"],"_links":{"self":[{"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/posts\/1942","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1942"}],"version-history":[{"count":1,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/posts\/1942\/revisions"}],"predecessor-version":[{"id":1943,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=\/wp\/v2\/posts\/1942\/revisions\/1943"}],"wp:attachment":[{"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1942"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1942"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rglinuxtech.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1942"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}