View Issue Details

IDProjectCategoryView StatusLast Update
0008271CentOS-7kernelpublic2015-04-02 17:32
Reportermetalcated 
PriorityhighSeverityminorReproducibilityunable to reproduce
Status newResolutionopen 
PlatformVMWare OSCentOSOS Version7.0.1406 (Core)
Product Version7.0-1406 
Target VersionFixed in Version 
Summary0008271: Failed disk on a raid array caused all file systems to go into read-only
Description2 weeks ago in our Data Center's a disk failed on one of the Raid Array's. We had a tech go in and replace the failed disk and being the array rebuild. During this time all CentOS 7 based OS's fell into read only mode on all mounted file systems, even those not located on the failed array.

Current kernel is: 3.10.0-123.13.2.el7.x86_64 on all CentOS 7 systems.

I know a similar issue showed up back on kernel: 2.6.22 but was patched, could this be a resurface of the bug or is this by design?

Steps To ReproduceUnfortunately it be hard to reproduce a failed disk in this environment.
Tags3.10.0-123.13.2.el7.x86_64, centos 7, file system, kernel, read only, ro
abrt_hash
URL

Activities

metalcated

metalcated

2015-03-02 20:17

reporter  

dmesg_dump.txt (33,214 bytes)
[3820055.341676] INFO: rcu_sched detected stalls on CPUs/tasks: { 0 2 3 4 5} (detected by 6, t=61714 jiffies, g=18486394, c=18486393, q=0)
[3820055.341752] sending NMI to all CPUs:
[3820055.341835] NMI backtrace for cpu 6
[3820055.341839] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820055.341841] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820055.341843] task: ffff8804295ee660 ti: ffff8804295fe000 task.ti: ffff8804295fe000
[3820055.341845] RIP: 0010:[<ffffffff8103fd7f>]  [<ffffffff8103fd7f>] flat_send_IPI_all+0x8f/0xa0
[3820055.341852] RSP: 0018:ffff88043fd83dc8  EFLAGS: 00000006
[3820055.341853] RAX: 0000000000000000 RBX: 0000000000000046 RCX: 00000000ff000000
[3820055.341855] RDX: 0000000000000c00 RSI: ffffffff818e6fe0 RDI: 0000000000000300
[3820055.341856] RBP: ffff88043fd83de0 R08: 0000000000000092 R09: 000000000000058a
[3820055.341857] R10: 61206f7420494d4e R11: 3a73555043206c6c R12: 0000000000000800
[3820055.341858] R13: 00000000000000ff R14: ffff88043fd8dda0 R15: ffffffff81926e40
[3820055.341860] FS:  0000000000000000(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
[3820055.341861] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3820055.341862] CR2: 00007f56f1152000 CR3: 000000011812c000 CR4: 00000000000407e0
[3820055.341886] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820055.341922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820055.341923] Stack:
[3820055.341924]  0000000000002710 ffffffff81926d40 ffffffff81926d40 ffff88043fd83df8
[3820055.341928]  ffffffff8103adc5 ffffffff819e4aa0 ffff88043fd83e58 ffffffff811012e8
[3820055.341931]  00000005126eb8e0 0000000000000000 0000000000000005 ffff8804295fe000
[3820055.341933] Call Trace:
[3820055.341935]  <IRQ>

[3820055.341941]  [<ffffffff8103adc5>] arch_trigger_all_cpu_backtrace+0x65/0xa0
[3820055.341945]  [<ffffffff811012e8>] rcu_check_callbacks+0x5a8/0x5f0
[3820055.341949]  [<ffffffff8106fe67>] update_process_times+0x47/0x80
[3820055.341954]  [<ffffffff810be055>] tick_sched_handle.isra.16+0x25/0x60
[3820055.341957]  [<ffffffff810be0d1>] tick_sched_timer+0x41/0x60
[3820055.341962]  [<ffffffff81089a57>] __run_hrtimer+0x77/0x1d0
[3820055.341965]  [<ffffffff810be090>] ? tick_sched_handle.isra.16+0x60/0x60
[3820055.341968]  [<ffffffff8108a297>] hrtimer_interrupt+0xf7/0x240
[3820055.341972]  [<ffffffff81039717>] local_apic_timer_interrupt+0x37/0x60
[3820055.341977]  [<ffffffff815f4d2f>] smp_apic_timer_interrupt+0x3f/0x60
[3820055.341980]  [<ffffffff815f369d>] apic_timer_interrupt+0x6d/0x80
[3820055.341981]  <EOI>

[3820055.341983]  [<ffffffff810392bd>] ? lapic_next_event+0x1d/0x30
[3820055.341987]  [<ffffffff81092d43>] ? finish_task_switch+0x53/0x170
[3820055.341991]  [<ffffffff815e75d5>] __schedule+0x2c5/0x790
[3820055.341993]  [<ffffffff815e8ce9>] schedule_preempt_disabled+0x29/0x70
[3820055.341996]  [<ffffffff810b493a>] cpu_startup_entry+0x24a/0x290
[3820055.342001]  [<ffffffff815d028e>] start_secondary+0x1c4/0x1da
[3820055.342002] Code: 48 8b 35 5d 24 9a 00 44 8b 66 3c ff 96 50 01 00 00 44 89 e9 c1 e1 18 89 0c 25 10 a3 5a ff 44 89 e2 80 ce 04 89 14 25 00 a3 5a ff <48> 89 df 57 9d 66 66 90 66 90 5b 41 5c 41 5d 5d c3 66 66 66 66
[3820159.786307] NMI backtrace for cpu 4
[3820159.786315] CPU: 4 PID: 15918 Comm: kworker/4:0 Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.786317] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.786395] BUG: soft lockup - CPU#6 stuck for 88s! [systemd-journal:534]
[3820159.786397] Workqueue: events e1000_watchdog [e1000]
[3820159.786420] Modules linked in: dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables binfmt_misc bridge stp llc sg coretemp crct10dif_pclmul crc32_pclmul xfs crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev vmw_balloon libcrc32c serio_raw cryptd pcspkr shpchp parport_pc parport i2c_piix4 vmw_vmci mperf ext4 mbcache jbd2 sd_mod sr_mod cdrom crc_t10dif ata_generic
[3820159.786426]  crct10dif_common pata_acpi vmwgfx ttm mptspi drm ata_piix i2c_core scsi_transport_spi libata mptscsih e1000 mptbase floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
[3820159.786428] CPU: 6 PID: 534 Comm: systemd-journal Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.786429] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.786430] task: ffff880425f08000 ti: ffff880425034000 task.ti: ffff880425034000
[3820159.786435] RIP: 0033:[<00007fe5362daafa>]  [<00007fe5362daafa>] 0x7fe5362daaf9
[3820159.786438] RSP: 002b:00007fff496cb608  EFLAGS: 00010206
[3820159.786439] RAX: 00007fe5362ecba0 RBX: ffffffff815f2a19 RCX: 0000000000000000
[3820159.786441] RDX: 00007fff496cb879 RSI: 2cf7a1e2f0bd068d RDI: 0000000000000006
[3820159.786442] RBP: 0000000000000061 R08: 0000000000000000 R09: 0000000000000000
[3820159.786444] R10: 0000000000000002 R11: 00007fe536d8e830 R12: 00050fa4eff87b57
[3820159.786445] R13: 00000000000003e8 R14: 0000000000000000 R15: 00007fff496ce510
[3820159.786447] FS:  00007fe5362ad840(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
[3820159.786448] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3820159.786449] CR2: 00007f56f1152000 CR3: 0000000425b11000 CR4: 00000000000407e0

[3820159.786455] task: ffff88013cb7a220 ti: ffff880146d44000 task.ti: ffff880146d44000
[3820159.786456] RIP: 0010:[<ffffffffa0079506>]  [<ffffffffa0079506>] e1000_update_stats+0x96/0x990 [e1000]
[3820159.786463] RSP: 0018:ffff880146d45da0  EFLAGS: 00000046
[3820159.786465] RAX: 0000000000000000 RBX: ffff88042691e8c0 RCX: 0000000000002542
[3820159.786466] RDX: 0000000000000000 RSI: 8240000000000000 RDI: ffff88042691eadc
[3820159.786467] RBP: ffff880146d45dd0 R08: 0000000000000246 R09: dfeeedbc6571f048
[3820159.786468] R10: dfeeedbc6571f048 R11: 0000000000000004 R12: ffff88042691e000
[3820159.786469] R13: ffff88042691eadc R14: 0000000000000246 R15: ffff88042691ec28
[3820159.786471] FS:  0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
[3820159.786472] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3820159.786473] CR2: 00007f0ce8be4050 CR3: 00000000083d9000 CR4: 00000000000407e0
[3820159.786501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.786506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.786522] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

[3820159.786531] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.786533] Stack:
[3820159.786534]  ffffffff810115d6 0000000012e157d1 ffff88042691f048 ffff88042691e8c0
[3820159.786538]  ffff88042691e000 ffff8804297a42c0 ffff880146d45e18 ffffffffa0079e6d
[3820159.786540]  ffff880425decb00 0000000012e157d1 ffff88042691f048 ffff880426034700
[3820159.786543] Call Trace:
[3820159.786549]  [<ffffffff810115d6>] ? __switch_to+0x136/0x490
[3820159.786558]  [<ffffffffa0079e6d>] e1000_watchdog+0x6d/0x550 [e1000]
[3820159.786562]  [<ffffffff8107e03b>] process_one_work+0x17b/0x460
[3820159.786565]  [<ffffffff8107ee0b>] worker_thread+0x11b/0x400
[3820159.786568]  [<ffffffff8107ecf0>] ? rescuer_thread+0x400/0x400
[3820159.786570]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[3820159.786573]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.786577]  [<ffffffff815f296c>] ret_from_fork+0x7c/0xb0
[3820159.786579]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.786581] Code: 00 00 00 00 00 4c 8d af 1c 02 00 00 4c 89 ef e8 71 05 57 e1 48 8b 93 88 04 00 00 49 89 c6 48 8b 83 68 03 00 00 8b 80 00 40 00 00 <89> c0 48 01 d0 48 8b 93 60 05 00 00 48 89 83 88 04 00 00 48 8b
[3820159.786603] NMI backtrace for cpu 2
[3820159.786608] CPU: 2 PID: 31440 Comm: kworker/2:2 Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.786610] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.786615] Workqueue: events_freezable disk_events_workfn
[3820159.786617] task: ffff88013cbcad80 ti: ffff8804269e0000 task.ti: ffff8804269e0000
[3820159.786619] RIP: 0010:[<ffffffff812cd4a6>]  [<ffffffff812cd4a6>] ioread8+0x16/0x50
[3820159.786623] RSP: 0018:ffff8804269e1990  EFLAGS: 00000006
[3820159.786625] RAX: ffffffffa011da50 RBX: ffff880036b0c000 RCX: 0000000000000006
[3820159.786626] RDX: 0000000000000177 RSI: 0000000000000000 RDI: 0000000000010177
[3820159.786627] RBP: ffff8804269e1998 R08: 000000100000014a R09: 0008000000100000
[3820159.786629] R10: ffff880036b0c850 R11: 0000000000000008 R12: 00000000000003e8
[3820159.786630] R13: 0000000000000000 R14: 0000000000000001 R15: ffff880036b0c818
[3820159.786631] FS:  0000000000000000(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
[3820159.786633] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3820159.786634] CR2: 00007fa2c7045000 CR3: 00000000083d9000 CR4: 00000000000407e0
[3820159.786659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.786689] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.786690] Stack:
[3820159.786691]  ffffffffa00c2b22 ffff8804269e19c0 ffffffffa00c5b69 ffff880036b0c818
[3820159.786694]  ffff880036b0c000 ffff880036b0dc70 ffff8804269e19f0 ffffffffa00c5c15
[3820159.786696]  ffff880036b0c818 ffff880036b0c000 ffff880036b0dc70 0000000000000001
[3820159.786699] Call Trace:
[3820159.786715]  [<ffffffffa00c2b22>] ? ata_sff_check_status+0x12/0x20 [libata]
[3820159.786724]  [<ffffffffa00c5b69>] ata_dev_select.constprop.19+0x49/0xc0 [libata]
[3820159.786733]  [<ffffffffa00c5c15>] ata_sff_qc_issue+0x35/0x1c0 [libata]
[3820159.786740]  [<ffffffffa00c5dd3>] ata_bmdma_qc_issue+0x33/0x190 [libata]
[3820159.786749]  [<ffffffffa00b1a62>] ata_qc_issue+0x172/0x380 [libata]
[3820159.786757]  [<ffffffffa00b6c3b>] ? atapi_xlat+0x15b/0x180 [libata]
[3820159.786765]  [<ffffffffa00ba9b5>] ata_scsi_queuecmd+0x135/0x420 [libata]
[3820159.786773]  [<ffffffffa00b6ae0>] ? ata_scsiop_inq_std+0xa0/0xa0 [libata]
[3820159.786777]  [<ffffffff813d3ae7>] scsi_dispatch_cmd+0xb7/0x290
[3820159.786779]  [<ffffffff813dc599>] scsi_request_fn+0x349/0x5a0
[3820159.786783]  [<ffffffff81188349>] ? alloc_pages_current+0xa9/0x170
[3820159.786785]  [<ffffffff812905f3>] __blk_run_queue+0x33/0x40
[3820159.786788]  [<ffffffff8129843d>] blk_execute_rq_nowait+0xad/0x160
[3820159.786791]  [<ffffffff8129856a>] blk_execute_rq+0x7a/0xe0
[3820159.786793]  [<ffffffff81293de8>] ? blk_rq_bio_prep+0x68/0xc0
[3820159.786796]  [<ffffffff812982b9>] ? blk_rq_map_kern+0xb9/0x160
[3820159.786798]  [<ffffffff8129276b>] ? blk_get_request+0x8b/0x110
[3820159.786800]  [<ffffffff813da417>] scsi_execute+0xd7/0x160
[3820159.786802]  [<ffffffff813db68c>] scsi_execute_req_flags+0x8c/0x100
[3820159.786807]  [<ffffffffa01b482b>] sr_check_events+0xbb/0x2d0 [sr_mod]
[3820159.786812]  [<ffffffffa01a8058>] cdrom_check_events+0x18/0x30 [cdrom]
[3820159.786816]  [<ffffffffa01b4cb9>] sr_block_check_events+0x39/0x50 [sr_mod]
[3820159.786818]  [<ffffffff812a043b>] disk_check_events+0x5b/0x1b0
[3820159.786821]  [<ffffffff812a05a6>] disk_events_workfn+0x16/0x20
[3820159.786824]  [<ffffffff8107e03b>] process_one_work+0x17b/0x460
[3820159.786826]  [<ffffffff8107ee0b>] worker_thread+0x11b/0x400
[3820159.786829]  [<ffffffff8107ecf0>] ? rescuer_thread+0x400/0x400
[3820159.786831]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[3820159.786834]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.786836]  [<ffffffff815f296c>] ret_from_fork+0x7c/0xb0
[3820159.786838]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.786840] Code: 2b 00 00 00 48 c7 c7 2a 49 82 81 31 c0 e8 73 0a d9 ff 5d c3 90 48 81 ff ff ff 03 00 77 37 48 81 ff 00 00 01 00 76 0e 0f b7 d7 ec <0f> b6 c0 c3 66 0f 1f 44 00 00 55 48 c7 c6 36 49 82 81 48 89 e5
[3820159.786861] NMI backtrace for cpu 0
[3820159.786866] CPU: 0 PID: 24102 Comm: kworker/0:1 Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.786868] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.786872] Workqueue: events sync_cmos_clock
[3820159.786874] task: ffff880012700b60 ti: ffff880110cb6000 task.ti: ffff880110cb6000
[3820159.786876] RIP: 0010:[<ffffffff8101aefa>]  [<ffffffff8101aefa>] mach_set_rtc_mmss+0x14a/0x200
[3820159.786881] RSP: 0018:ffff880110cb7d58  EFLAGS: 00000006
[3820159.786882] RAX: 0000000000000003 RBX: 000000000000000f RCX: 0000000000000002
[3820159.786883] RDX: 00000000000000a6 RSI: 00000000000000f6 RDI: 0000000000000015
[3820159.786884] RBP: ffff880110cb7dc0 R08: 0000000000000297 R09: 000000000000001c
[3820159.786885] R10: 00000000fffffff9 R11: 0000000000000001 R12: 0000000000000022
[3820159.786886] R13: 0000000000000003 R14: 0000000000000042 R15: 0000000000000036
[3820159.786888] FS:  0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
[3820159.786889] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3820159.786891] CR2: 00007f0c86539000 CR3: 00000001005ab000 CR4: 00000000000407f0
[3820159.786915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.786964] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.786966] Stack:
[3820159.786968]  0000000000000297 02ff880110cb7d70 000000248101a679 000000030000002a
[3820159.786973]  0000000100000016 0000000000000073 0000000000000035 000000000eeb6043
[3820159.786976]  ffffffff8190be00 ffff880118e17900 ffff88043fc12e00 ffff88043fc17000
[3820159.786980] Call Trace:
[3820159.786993]  [<ffffffff8101b18f>] update_persistent_clock+0x1f/0x30
[3820159.786997]  [<ffffffff810b7a64>] sync_cmos_clock+0x94/0x120
[3820159.787002]  [<ffffffff8107e03b>] process_one_work+0x17b/0x460
[3820159.787005]  [<ffffffff8107ee0b>] worker_thread+0x11b/0x400
[3820159.787008]  [<ffffffff8107ecf0>] ? rescuer_thread+0x400/0x400
[3820159.787011]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[3820159.787014]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.787016]  [<ffffffff815f296c>] ret_from_fork+0x7c/0xb0
[3820159.787018]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3820159.787019] Code: e6 70 89 f8 e6 71 b8 08 00 00 00 e6 70 0f b6 45 a7 e6 71 b8 07 00 00 00 e6 70 44 89 e0 e6 71 b8 04 00 00 00 e6 70 44 89 e8 e6 71 <b8> 02 00 00 00 e6 70 44 89 f0 e6 71 31 c0 e6 70 44 89 f8 e6 71
[3820159.787044] NMI backtrace for cpu 1
[3820159.787049] CPU: 1 PID: 1539 Comm: snmpd Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.787052] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.787054] task: ffff8804261296c0 ti: ffff8804268d2000 task.ti: ffff8804268d2000
[3820159.787056] RIP: 0010:[<ffffffffa0084024>]  [<ffffffffa0084024>] e1000_get_settings+0x64/0x190 [e1000]
[3820159.787070] RSP: 0018:ffff8804268d3cc8  EFLAGS: 00000206
[3820159.787072] RAX: 000000000000cb83 RBX: ffff8804268d3d60 RCX: 0000000000000000
[3820159.787073] RDX: 0000000000000004 RSI: ffff8804268d3d60 RDI: 0000000000000000
[3820159.787075] RBP: ffff8804268d3cd8 R08: 0000000000000000 R09: 0000000000200000
[3820159.787077] R10: 0000000000000000 R11: 0000000000000246 R12: ffff88042691e000
[3820159.787078] R13: ffff88042691e000 R14: 0000000000000001 R15: ffffffff819a29c0
[3820159.787083] FS:  00007f0c8651b840(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
[3820159.787085] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3820159.787087] CR2: 00007f0c86538000 CR3: 0000000428fbf000 CR4: 00000000000407e0
[3820159.787124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.787165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.787167] Stack:
[3820159.787168]  ffff8804268d3d60 ffff88042691e000 ffff8804268d3cf8 ffffffff814d37e6
[3820159.787171]  00007fff7d067750 00000000ffffffed ffff8804268d3dd0 ffffffff814d5a62
[3820159.787174]  0000000200014b89 ffffea000240c560 ffff8804268d3e68 ffff8804268d3d90
[3820159.787176] Call Trace:
[3820159.787182]  [<ffffffff814d37e6>] __ethtool_get_settings+0x86/0x140
[3820159.787185]  [<ffffffff814d5a62>] dev_ethtool+0x1242/0x1a80
[3820159.787187]  [<ffffffff814cd309>] ? dev_get_by_name_rcu+0x69/0x90
[3820159.787191]  [<ffffffff814e4b4b>] dev_ioctl+0x1ab/0x560
[3820159.787195]  [<ffffffff811ca918>] ? destroy_inode+0x38/0x60
[3820159.787198]  [<ffffffff814b4c85>] sock_do_ioctl+0x45/0x50
[3820159.787200]  [<ffffffff814b5380>] sock_ioctl+0x1f0/0x2c0
[3820159.787203]  [<ffffffff811c2b25>] do_vfs_ioctl+0x2e5/0x4c0
[3820159.787208]  [<ffffffff811b150e>] ? ____fput+0xe/0x10
[3820159.787210]  [<ffffffff811c2da1>] SyS_ioctl+0xa1/0xc0
[3820159.787213]  [<ffffffff815f2a19>] system_call_fastpath+0x16/0x1b
[3820159.787214] Code: 2b 01 00 00 c6 43 0f 00 41 8b 84 24 b0 0c 00 00 88 43 10 41 83 bc 24 40 0c 00 00 03 0f 94 43 11 49 8b 84 24 28 0c 00 00 8b 40 08 <a8> 02 0f 85 b4 00 00 00 ba ff ff ff ff b9 ff ff ff ff c6 43 0e
[3820159.787239] NMI backtrace for cpu 5
[3820159.787244] CPU: 5 PID: 837 Comm: vmtoolsd Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.787245] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.787247] task: ffff8804252dc440 ti: ffff880425e7c000 task.ti: ffff880425e7c000
[3820159.787249] RIP: 0033:[<00007f7d00667ead>]  [<00007f7d00667ead>] 0x7f7d00667eac
[3820159.787257] RSP: 002b:00007fffa06e7d48  EFLAGS: 00000246
[3820159.787258] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000001001e
[3820159.787262] RDX: 00007fff00005658 RSI: 000000005105e59e RDI: 00007f7dd1a454ab
[3820159.787263] RBP: 00007f7d019f72b0 R08: 000000000000001e R09: 0000000000000345
[3820159.787264] R10: 0000000000000005 R11: 0000000000000246 R12: 00007fffa06e7d70
[3820159.787265] R13: 0000000000000000 R14: 0000000000000000 R15: 00007f7d019f7470
[3820159.787267] FS:  00007f7d00adf740(0000) GS:ffff88043fd40000(0000) knlGS:0000000000000000
[3820159.787268] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3820159.787269] CR2: 00007f0ce8be83e8 CR3: 0000000425fb0000 CR4: 00000000000407e0
[3820159.787294] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.787324] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

[3820159.787327] NMI backtrace for cpu 3
[3820159.787332] CPU: 3 PID: 2535 Comm: qmgr Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.787334] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.787336] task: ffff880427062220 ti: ffff88042803a000 task.ti: ffff88042803a000
[3820159.787338] RIP: 0010:[<ffffffff811e14c8>]  [<ffffffff811e14c8>] __find_get_block_slow+0xf8/0x170
[3820159.787344] RSP: 0018:ffff88042803bb10  EFLAGS: 00010246
[3820159.787346] RAX: 002fffff00000864 RBX: ffff88016b985ea0 RCX: ffff88016b985ea0
[3820159.787347] RDX: 0000000000000020 RSI: 0000000000000001 RDI: ffff880424820c90
[3820159.787349] RBP: ffff88042803bb68 R08: ffff880424820c90 R09: ffffea000626e11c
[3820159.787350] R10: ffff88042803bac0 R11: 0000000000000246 R12: ffff880424820ab0
[3820159.787351] R13: ffff8804248209c0 R14: 000000000020257f R15: ffffea000626e100
[3820159.787353] FS:  00007fc5e2719840(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
[3820159.787354] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3820159.787356] CR2: 00007f4df40f2ff0 CR3: 000000042627a000 CR4: 00000000000407e0
[3820159.787381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.787411] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.787412] Stack:
[3820159.787413]  ffff880424820c90 ffff88042803bb60 ffffffff810976b6 ffffea0004014b80
[3820159.787416]  ffff880100000003 000000000e475dbb 0000000000000000 0000000000000000
[3820159.787419]  0000000000001000 0000000000000000 0000000000000300 ffff88042803bbe0
[3820159.787421] Call Trace:
[3820159.787426]  [<ffffffff810976b6>] ? try_to_wake_up+0x1b6/0x280
[3820159.787429]  [<ffffffff811e25ec>] __find_get_block+0x11c/0x260
[3820159.787447]  [<ffffffffa021b731>] ? ext4_es_lookup_extent+0x91/0x190 [ext4]
[3820159.787451]  [<ffffffff811e2755>] __getblk+0x25/0x2e0
[3820159.787460]  [<ffffffffa01ddf26>] ext4_getblk+0xa6/0x200 [ext4]
[3820159.787468]  [<ffffffffa01de0a7>] ext4_bread+0x27/0xb0 [ext4]
[3820159.787477]  [<ffffffffa01e631a>] __ext4_read_dirblock+0x4a/0x310 [ext4]
[3820159.787485]  [<ffffffffa01e69a0>] htree_dirblock_to_tree+0x40/0x190 [ext4]
[3820159.787493]  [<ffffffffa01e7812>] ext4_htree_fill_tree+0xe2/0x2a0 [ext4]
[3820159.787495]  [<ffffffff811c2ffd>] ? filldir+0x7d/0xf0
[3820159.787498]  [<ffffffff81194d7e>] ? kmem_cache_alloc_trace+0x1ce/0x1f0
[3820159.787506]  [<ffffffffa01d6da5>] ext4_readdir+0x185/0x7e0 [ext4]
[3820159.787508]  [<ffffffff811c2f80>] ? fillonedir+0xe0/0xe0
[3820159.787510]  [<ffffffff811c2f80>] ? fillonedir+0xe0/0xe0
[3820159.787512]  [<ffffffff811c2f80>] ? fillonedir+0xe0/0xe0
[3820159.787514]  [<ffffffff811c2e70>] vfs_readdir+0xb0/0xe0
[3820159.787516]  [<ffffffff811c3295>] SyS_getdents+0x95/0x120
[3820159.787519]  [<ffffffff815f2a19>] system_call_fastpath+0x16/0x1b
[3820159.787520] Code: d5 f6 ff 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 48 89 d8 75 20 48 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 80 00 00 00 00 <f0> ff 43 60 eb c2 31 db eb ce e8 09 c8 e7 ff 0f 0b 48 8b 53 18
[3820159.787542] NMI backtrace for cpu 7
[3820159.787546] CPU: 7 PID: 12888 Comm: httpd Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[3820159.787548] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[3820159.787550] task: ffff8800369d4fa0 ti: ffff880110caa000 task.ti: ffff880110caa000
[3820159.787551] RIP: 0010:[<ffffffffa02e4f48>]  [<ffffffffa02e4f48>] ipt_do_table+0x168/0x6d1 [ip_tables]
[3820159.787558] RSP: 0018:ffff880110cabab0  EFLAGS: 00010246
[3820159.787560] RAX: ffff8804251f1150 RBX: ffff88013cb9d130 RCX: ffff88011e326c00
[3820159.787561] RDX: 000060fbc0003078 RSI: ffff88013cb9d000 RDI: ffff8801e281faf8
[3820159.787562] RBP: ffff880110cabbc8 R08: ffff8804251f1120 R09: 0000000000000003
[3820159.787563] R10: ffff8801e281faf8 R11: 0000000000000001 R12: ffff880005c93cdc
[3820159.787564] R13: ffffffffa02e7598 R14: ffffffffa02e7598 R15: ffff8801e281faf8
[3820159.787566] FS:  00007f56f1019740(0000) GS:ffff88043fdc0000(0000) knlGS:0000000000000000
[3820159.787567] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3820159.787568] CR2: 00007f0ce8be41a8 CR3: 0000000118da6000 CR4: 00000000000407e0
[3820159.787594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3820159.787624] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3820159.787625] Stack:
[3820159.787626]  0000000000000246 ffff88014f6df0c0 ffff880110cabae8 ffffffff81086911
[3820159.787629]  0000000000000001 0000000000000001 ffff880110cabcc0 ffff880110cabb18
[3820159.787631]  ffffffff811c3897 ffff880110cabba4 0000000000000000 0000000000000003
[3820159.787634] Call Trace:
[3820159.787638]  [<ffffffff81086911>] ? remove_wait_queue+0x31/0x40
[3820159.787641]  [<ffffffff811c3897>] ? poll_freewait+0x47/0xa0
[3820159.787645]  [<ffffffff8150cce0>] ? ip_forward_options+0x1c0/0x1c0
[3820159.787650]  [<ffffffffa02f4039>] iptable_raw_hook+0x39/0x70 [iptable_raw]
[3820159.787653]  [<ffffffff8150070a>] nf_iterate+0xaa/0xc0
[3820159.787655]  [<ffffffff8150cce0>] ? ip_forward_options+0x1c0/0x1c0
[3820159.787658]  [<ffffffff815007a4>] nf_hook_slow+0x84/0x140
[3820159.787660]  [<ffffffff8150cce0>] ? ip_forward_options+0x1c0/0x1c0
[3820159.787662]  [<ffffffff8150ebb2>] __ip_local_out+0xa2/0xb0
[3820159.787664]  [<ffffffff8150ebd2>] ip_local_out+0x12/0x30
[3820159.787666]  [<ffffffff8150ef3d>] ip_queue_xmit+0x13d/0x3e0
[3820159.787669]  [<ffffffff81526951>] tcp_transmit_skb+0x471/0x8f0
[3820159.787672]  [<ffffffff811974b8>] ? __kmalloc_node_track_caller+0x58/0x270
[3820159.787674]  [<ffffffff81526f76>] tcp_write_xmit+0x1a6/0xb80
[3820159.787677]  [<ffffffff81527bae>] __tcp_push_pending_frames+0x2e/0xc0
[3820159.787679]  [<ffffffff815289b2>] tcp_send_fin+0x62/0x180
[3820159.787682]  [<ffffffff8151b808>] tcp_close+0x3a8/0x400
[3820159.787685]  [<ffffffff81541737>] inet_release+0x77/0x80
[3820159.787688]  [<ffffffff814b6e2f>] sock_release+0x1f/0x80
[3820159.787690]  [<ffffffff814b6ea2>] sock_close+0x12/0x20
[3820159.787693]  [<ffffffff811b1239>] __fput+0xe9/0x270
[3820159.787696]  [<ffffffff811b150e>] ____fput+0xe/0x10
[3820159.787699]  [<ffffffff810823c7>] task_work_run+0xa7/0xe0
[3820159.787702]  [<ffffffff81012a77>] do_notify_resume+0x97/0xb0
[3820159.787705]  [<ffffffff815f2cd2>] int_signal+0x12/0x17
[3820159.787706] Code: ff 48 01 f3 f6 05 3a 22 00 00 04 0f 85 4e 03 00 00 49 8d 45 30 c7 45 94 00 00 00 00 4d 89 f5 48 89 85 68 ff ff ff 0f 1f 44 00 00 <0f> b6 53 53 41 8b 44 24 0c 23 43 08 8b 75 c0 3b 03 41 0f b6 4c
[3820159.788388] sd 2:0:0:0: timing out command, waited 180s
[3820159.789042] sd 2:0:0:0: [sda] Unhandled error code
[3820159.789045] sd 2:0:0:0: [sda]
[3820159.789046] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[3820159.789048] sd 2:0:0:0: [sda] CDB:
[3820159.789049] Write(10): 2a 00 01 57 72 38 00 00 08 00
[3820159.789075] end_request: I/O error, dev sda, sector 22508088
[3820159.789570] Buffer I/O error on device dm-1, logical block 2160199
[3820159.789605] EXT4-fs warning (device dm-1): ext4_end_bio:287: I/O error writing to inode 530595 (offset 16384 size 4096 starting block 2160199)
[3820159.789616] sd 2:0:0:0: timing out command, waited 180s
[3820159.789640] sd 2:0:0:0: [sda] Unhandled error code
[3820159.789642] sd 2:0:0:0: [sda]
[3820159.789643] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[3820159.789646] sd 2:0:0:0: [sda] CDB:
[3820159.789647] Write(10): 2a 00 00 ba 9e c0 00 00 10 00
[3820159.789652] end_request: I/O error, dev sda, sector 12230336
[3820159.789708] sd 2:0:0:0: timing out command, waited 180s
[3820159.790174] sd 2:0:0:0: [sda] Unhandled error code
[3820159.790176] sd 2:0:0:0: [sda]
[3820159.790177] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[3820159.790178] sd 2:0:0:0: [sda] CDB:
[3820159.790179] Write(10): 2a 00 01 57 72 20 00 00 08 00
[3820159.790183] end_request: I/O error, dev sda, sector 22508064
[3820159.790213] Buffer I/O error on device dm-1, logical block 2160196
[3820159.790241] EXT4-fs warning (device dm-1): ext4_end_bio:287: I/O error writing to inode 530595 (offset 4096 size 4096 starting block 2160196)
[3820159.791796] Aborting journal on device dm-1-8.
[3820159.799446] ata2: lost interrupt (Status 0x58)
[3820160.478721] ata2: drained 65536 bytes to clear DRQ
[3820160.478864] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[3820160.478922] sr 1:0:0:0: CDB:
[3820160.478925] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
[3820160.478936] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 8 pio 16392 in
         res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[3820160.478993] ata2.00: status: { DRDY }
[3820160.479397] ata2: soft resetting link
[3820160.479340] EXT4-fs error (device dm-1): __ext4_journal_start_sb:62: Detected aborted journal
[3820160.479403] EXT4-fs (dm-1): Remounting filesystem read-only
[3820160.636799] ata2.00: configured for UDMA/33
[3820160.637444] ata2: EH complete
[3906546.775747] EXT4-fs (dm-1): error count: 2
[3906546.775807] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[3906546.775820] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[3943184.471958] systemd-journald[534]: Vacuuming done, freed 0 bytes
[3992968.802387] EXT4-fs (dm-1): error count: 2
[3992968.802415] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[3992968.802422] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4050044.389045] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4079390.813151] EXT4-fs (dm-1): error count: 2
[4079390.813196] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4079390.813208] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4100903.288856] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4134257.244736] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4155179.608152] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4165812.829813] EXT4-fs (dm-1): error count: 2
[4165812.829842] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4165812.829855] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4174042.938321] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4192063.269108] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4208537.459056] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4223297.948314] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4237315.460823] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4250842.497696] systemd-journald[534]: Vacuuming done, freed 0 bytes
[4252234.770833] EXT4-fs (dm-1): error count: 2
[4252234.770861] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4252234.770875] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4263964.890670] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4276971.640054] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4289600.816479] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4300718.900960] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4311532.947532] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4322347.359873] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4333017.990512] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4338656.704936] EXT4-fs (dm-1): error count: 2
[4338656.704943] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4338656.704946] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4343662.547242] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4354307.342875] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4364891.223116] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4375370.371981] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4385656.956087] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4395578.149183] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4405444.752628] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4415310.959665] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4425078.649392] EXT4-fs (dm-1): error count: 2
[4425078.649410] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4425078.649417] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4425179.892176] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4435047.977759] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4444797.835240] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4454521.404086] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4464210.907195] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4473740.814064] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4483108.515412] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4492254.940677] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4501250.941313] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4510202.286559] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4511500.569367] EXT4-fs (dm-1): error count: 2
[4511500.569409] EXT4-fs (dm-1): initial error at 1424576724: __ext4_journal_start_sb:62
[4511500.569417] EXT4-fs (dm-1): last error at 1424576724: __ext4_journal_start_sb:62
[4519038.532605] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4527865.091642] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4536504.185291] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4544974.349117] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4553289.561942] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4561482.515324] systemd-journald[534]: Vacuuming done, freed 75497472 bytes
[4566758.998129] EXT4-fs error (device dm-1): ext4_remount:4711: Abort forced by user
dmesg_dump.txt (33,214 bytes)
tigalch

tigalch

2015-03-02 20:31

manager   ~0022438

Just for clarification: A RAID-Array fails, and a OS, which is not at all connected to that array in any way, switches its mount points to read only?
metalcated

metalcated

2015-03-02 21:17

reporter   ~0022439

The OS did not fail, it just ended up going into read-only mode.
tigalch

tigalch

2015-03-02 21:32

manager   ~0022440

But you are indeed saying a OS went into read-only mode while having no connection/relation whatever to the RAID-array in question?
metalcated

metalcated

2015-03-10 12:37

reporter   ~0022502

Sorry if I confused you in any way. No direct connections on the OS itself were open to the SAN per say, the VMDK's are located on a storage SAN in a RAID-array where a disk failed. When the disk failed 6 VM's all with VMDK's stored this SAN went into read-only mode. Only CentOS 7 VM's with the kernel shown above were affected. CentOS 5.x/6.x VM's were not affected.
metalcated

metalcated

2015-03-27 15:05

reporter   ~0022606

Any updates on this issue? Do you require additional information form me?

Issue History

Date Modified Username Field Change
2015-03-02 20:17 metalcated New Issue
2015-03-02 20:17 metalcated File Added: dmesg_dump.txt
2015-03-02 20:31 tigalch Note Added: 0022438
2015-03-02 21:17 metalcated Note Added: 0022439
2015-03-02 21:32 tigalch Note Added: 0022440
2015-03-10 12:37 metalcated Note Added: 0022502
2015-03-27 15:05 metalcated Note Added: 0022606
2015-04-02 17:31 metalcated Tag Attached: kernel
2015-04-02 17:31 metalcated Tag Attached: 3.10.0-123.13.2.el7.x86_64
2015-04-02 17:31 metalcated Tag Attached: centos 7
2015-04-02 17:32 metalcated Tag Attached: ro
2015-04-02 17:32 metalcated Tag Attached: read only
2015-04-02 17:32 metalcated Tag Attached: file system