View Issue Details

IDProjectCategoryView StatusLast Update
0005549CentOS-6mdadmpublic2012-06-22 22:41
Reportersconway 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Platform64 BitOSCentosOS Version6
Product Version6.2 
Target VersionFixed in Version 
Summary0005549: mdmadm removal and re-addition of drive(s)
DescriptionWas having problems on a newly installed and updated system as described in this post http://bugs.centos.org/view.php?id=5400

I was following the instructions according to tru to get the system to boot with just one drive present which eventually succeeded. when I tried to re-add all the partitions of /dev/sdb back into the radi1 array, mdadm only allowed /dev/md1 to be readded for somereason and I had to zero out the other two and re-add them as a spare (?) as displayed below

[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[2] sda3[0]
      10238908 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sdb2[2] sda2[0]
      527348 blocks super 1.0 [2/2] [UU]
      
md1 : active raid1 sdb1[1] sda1[0]
      30718908 blocks super 1.1 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

[root@localhost ~]# fdisk -l

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004a898

   Device Boot Start End Blocks Id System
/dev/sdb1 1 3825 30720000 fd Linux raid autodetect
/dev/sdb2 * 3825 3891 527360 fd Linux raid autodetect
/dev/sdb3 3891 5166 10240000 fd Linux raid autodetect
/dev/sdb4 5166 30401 202707648+ 5 Extended
/dev/sdb5 5166 6471 10489892 83 Linux

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001d486

   Device Boot Start End Blocks Id System
/dev/sda1 1 3825 30720000 fd Linux raid autodetect
/dev/sda2 * 3825 3891 527360 fd Linux raid autodetect
/dev/sda3 3891 5166 10240000 fd Linux raid autodetect
/dev/sda4 5166 30401 202707648+ 5 Extended
/dev/sda5 5166 6471 10489892 83 Linux

Disk /dev/md1: 31.5 GB, 31456161792 bytes
2 heads, 4 sectors/track, 7679727 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md0: 540 MB, 540004352 bytes
2 heads, 4 sectors/track, 131837 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md2: 10.5 GB, 10484641792 bytes
2 heads, 4 sectors/track, 2559727 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table
Steps To Reproduce
[root@localhost ~]# mdadm --manage /dev/md0 --fail /dev/sdb2

Personalities : [raid1]
md2 : active raid1 sdb3[2] sda3[0]
      10238908 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sdb2[2](F) sda2[0]
      527348 blocks super 1.0 [2/1] [U_]
      
md1 : active raid1 sdb1[1] sda1[0]
      30718908 blocks super 1.1 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

[root@localhost ~]# mdadm --manage /dev/md0 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md0
You have new mail in /var/spool/mail/root
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[2] sda3[0]
      10238908 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sda2[0]
      527348 blocks super 1.0 [2/1] [U_]
      
md1 : active raid1 sdb1[1] sda1[0]
      30718908 blocks super 1.1 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

[root@localhost ~]# mdadm --manage /dev/md0 --re-add /dev/sdb2
mdadm: --re-add for /dev/sdb2 to /dev/md0 is not possible

mdadm /dev/md0 -a /dev/sdb2
mdadm: /dev/sdb2 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdb2 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdb2" first.

[root@localhost ~]# mdadm --zero-superblock /dev/sdb2
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[2] sda3[0]
      10238908 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sda2[0]
      527348 blocks super 1.0 [2/1] [U_]
      
md1 : active raid1 sdb1[1] sda1[0]
      30718908 blocks super 1.1 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

[root@localhost ~]# mdadm /dev/md0 -a /dev/sdb2
mdadm: added /dev/sdb2

[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb3[2] sda3[0]
      10238908 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sdb2[2] sda2[0]
      527348 blocks super 1.0 [2/2] [UU]
      
md1 : active raid1 sdb1[1] sda1[0]
      30718908 blocks super 1.1 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
TagsNo tags attached.

Activities

brak44

brak44

2012-02-28 04:54

reporter   ~0014570

This is related to mdadm version 3.2.2

[root@localhost ~]# mdadm -V
mdadm - v3.2.2 - 17th June 2011
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[2] sdb1[1]
      204788 blocks super 1.0 [2/2] [UU]
      
md2 : active raid1 sdb3[1] sda3[0]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 1/3 pages [4KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda2[0]
      4094968 blocks super 1.1 [2/2] [UU]
      
unused devices: <none>
[root@localhost ~]# mdadm /dev/md0 --fail /dev/sda1
mdadm: set /dev/sda1 faulty in /dev/md0
[root@localhost ~]# mdadm /dev/md0 --remove /dev/sda1
mdadm: hot removed /dev/sda1 from /dev/md0
You have new mail in /var/spool/mail/root
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1]
      204788 blocks super 1.0 [2/1] [_U]
      
md2 : active raid1 sdb3[1] sda3[0]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 2/3 pages [8KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda2[0]
      4094968 blocks super 1.1 [2/2] [UU]
      
unused devices: <none>
[root@localhost ~]# mdadm /dev/md0 --add /dev/sda1
mdadm: /dev/sda1 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sda1 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sda1" first.
[root@localhost ~]# mdadm --zero-superblock /dev/sda1
[root@localhost ~]# mdadm /dev/md0 --add /dev/sda1
mdadm: added /dev/sda1
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[2] sdb1[1]
      204788 blocks super 1.0 [2/2] [UU]
      
md2 : active raid1 sdb3[1] sda3[0]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 2/3 pages [8KB], 65536KB chunk

md1 : active raid1 sdb2[1] sda2[0]
      4094968 blocks super 1.1 [2/2] [UU]

---------------------------------------------------------

with mdadm version 3.2.1 it works as expected


[root@localhost ~]# mdadm -V
mdadm - v3.2.1 - 28th March 2011
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[0] sdb3[1]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 1/3 pages [4KB], 65536KB chunk

md1 : active raid1 sda2[0] sdb2[1]
      4094968 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sda1[0] sdb1[1]
      204788 blocks super 1.0 [2/2] [UU]
      
unused devices: <none>
[root@localhost ~]# mdadm /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
[root@localhost ~]# mdadm /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[0] sdb3[1]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 0/3 pages [0KB], 65536KB chunk

md1 : active raid1 sda2[0] sdb2[1]
      4094968 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sda1[0]
      204788 blocks super 1.0 [2/1] [U_]
      
unused devices: <none>
[root@localhost ~]# mdadm /dev/md0 --add /dev/sdb1
mdadm: re-added /dev/sdb1
You have new mail in /var/spool/mail/root
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[0] sdb3[1]
      308266876 blocks super 1.1 [2/2] [UU]
      bitmap: 1/3 pages [4KB], 65536KB chunk

md1 : active raid1 sda2[0] sdb2[1]
      4094968 blocks super 1.1 [2/2] [UU]
      
md0 : active raid1 sdb1[1] sda1[0]
      204788 blocks super 1.0 [2/2] [UU]
      
unused devices: <none>
sconway

sconway

2012-02-28 15:50

reporter   ~0014572

Thanks brak44, it's indeed version 3.2.2 that is currently installed on the system. Do you know if the problem in mdadm 3.2.2 is going to be addressed anytime soon or is it just better to downgrade to 3.2.1
sconway

sconway

2012-02-28 16:23

reporter   ~0014573

and how does one downgrade to 3.2.1 using yum?
brak44

brak44

2012-02-28 23:12

reporter   ~0014578

I used SL 6.1 (updated to the latest which still has mdadm 3.2.1) to test with.

A bug fix was released by Red Hat on 6 DEC 2011
which brought mdadm up to version mdadm-3.2.2-9.el6

http://rhn.redhat.com/errata/RHBA-2011-1520.html

In the 6.2_Technical_Notes/mdadm.html#RHBA-2011-1520 you will find this

BZ#706500
    Due to incorrect internal accounting of disks, the mdadm utility failed to re-add a disk, which was previously marked as faulty and removed. With this update, the underlying code has been modified and such disk is re-added as expected.

However a `yum info mdadm' confirms that the installed version is mdadm-3.2.2-9.el6 on the Centos 6.2 machines.

Further investigation is required.
brak44

brak44

2012-02-29 09:46

reporter   ~0014581

With a brand new install of Centos 6.2 and going back to mdadm v3.2.1 does not fix the problem in Centos.
The previous post was done with SL 6.1 (kernel 3.2.1)
What needs to be tested now is mdadm v3.2.1 with kernel 3.2.1

[root@localhost ~]# mdadm -V
mdadm - v3.2.2 - 17th June 2011
[root@localhost ~]# mdadm /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
[root@localhost ~]# mdadm /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[0]
      204788 blocks super 1.0 [2/1] [U_]
      
md1 : active raid1 sda2[0] sdb2[1]
      4094968 blocks super 1.1 [2/2] [UU]
      
md2 : active raid1 sda3[0] sdb3[1]
      972458876 blocks super 1.1 [2/2] [UU]
      bitmap: 2/8 pages [8KB], 65536KB chunk

unused devices: <none>
[root@localhost ~]# mdadm /dev/md0 --add /dev/sdb1
mdadm: /dev/sdb1 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdb1 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdb1" first.

[root@localhost ~]# rpm -Uv --oldpackage /home/bizux/mdadm-3.2.1-1.el6.x86_64.rpm
Preparing packages for installation...
mdadm-3.2.1-1.el6
You have new mail in /var/spool/mail/root

[root@localhost ~]# mdadm -V
mdadm - v3.2.1 - 28th March 2011
[root@localhost ~]# mdadm /dev/md0 --add /dev/sdb1
mdadm: /dev/sdb1 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdb1 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdb1" first.
brak44

brak44

2012-03-03 03:25

reporter   ~0014611

Loaded Centos 6.1 64bit in a raid 1 configuration.
mdadm - v3.2.1 - 28th March 2011
Kernel 2.6.32-131.0.15
Failed then removed sdb1 drive.
Re-added drive with no problems as expected.
then did a yum update kernel
Kernel now 2.6.32-220.4.2
Failed then removed sdb1 drive.
Re-added drive and re-add fails as per

[root@localhost ~]# mdadm /dev/md0 --add /dev/sdb1
mdadm: /dev/sdb1 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdb1 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdb1" first.
aaron

aaron

2012-06-21 21:15

reporter   ~0015309

Happening to me with a fresh CentOS 6.2 install (latest updates applied). I'm pretty sure this is different behavior than before, and that I was able to easily test fault, removal and re-add with quick incremental sync following test procedures here https://raid.wiki.kernel.org/index.php/Detecting,_querying_and_testing in CentOS 5.

Advice on the internets "helpfully" suggests zeroing the superblock but after doing so it appears the RAID is re-syncing the WHOLE drive, which is understandable assuming mdadm thinks we're just plugging in a totally foreign spare disk. I didn't notice when testing my RAID1 boot partition, but my main LVM partition is now estimated at over an hour for recovery of ~30s downtime.

This issue is also reported on Ubuntu https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/925280

but I'm not clear what the new procedure is to re-add a temporarily removed member (something to do with an intent bitmap or using explicit --run option???).
aaron

aaron

2012-06-22 22:41

reporter   ~0015321

After explicitly setting write-intent bitmaps with the command below, anecdotal evidence shows may RAID1 arrays building MUCH faster, so I assume incremental syncing is in place.

mdadm --grow --bitmap=internal /dev/mdX

I wonder if this was perhaps just a default in previous versions of mdadm??

Issue History

Date Modified Username Field Change
2012-02-26 17:32 sconway New Issue
2012-02-28 04:54 brak44 Note Added: 0014570
2012-02-28 15:50 sconway Note Added: 0014572
2012-02-28 16:23 sconway Note Added: 0014573
2012-02-28 23:12 brak44 Note Added: 0014578
2012-02-29 09:46 brak44 Note Added: 0014581
2012-03-03 03:25 brak44 Note Added: 0014611
2012-06-21 21:15 aaron Note Added: 0015309
2012-06-22 22:41 aaron Note Added: 0015321