0012857: BUG: unable to handle kernel paging request at <address> - CentOS Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0012857	CentOS-7	centos-release	public	2017-02-21 16:06	2020-03-10 04:29

Reporter	oleksandr.mykhalskyi	Assigned To
Priority	normal	Severity	crash	Reproducibility	sometimes
Status	new	Resolution	open
Platform	Dell PowerEdge R630	OS	Centos	OS Version	7.2
Product Version	7.2.1511

Summary	0012857: BUG: unable to handle kernel paging request at <address>
Description	Hello, Time to time our CentOS 7 hypervisors (with several KVM VMs), which are used as compute nodes for openstack clouds, crash with next error "BUG: unable to handle kernel paging request at <address>". These servers are also used as ceph storage nodes (with ceph-osd processes) Kernel - 3.10.0-327.36.3.el7.x86_64 Examples of stack for crashes - in the attachments. BUG: unable to handle kernel paging request at 00000000fc047000 IP: [<ffffffffa064cad4>] kvm_zap_rmapp+0x34/0x60 [kvm] BUG: unable to handle kernel paging request at 00000000fc022010 IP: [<ffffffff81190948>] anon_vma_interval_tree_remove+0x28/0x250
Tags	centos 7

abrt_hash
URL

oleksandr.mykhalskyi 2017-02-21 16:06 reporter	vmcore-dmesg-1.txt (1,034,410 bytes)

oleksandr.mykhalskyi 2017-02-21 16:17 reporter	vmcore-dmesg-2.txt (728,313 bytes)

crrzhao 2017-09-04 12:01 reporter ~0029993	Hello I met the same bug (kvm_zap_rmapp) on 5 machines. They happened on different time. The kernel is 3.10.0-327.36.3.el7.x86_64 --------------------------------- call trace: [12641660.871619] BUG: unable to handle kernel paging request at 00000000fd00f000 [12641660.871990] IP: [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.872210] PGD 0 [12641660.872392] Oops: 0000 [#1] SMP [12641660.872582] Modules linked in: xt_u32 fuse 8021q garp mrp binfmt_misc vport_vxlan(OE) vhost_net vhost macvtap macvlan tun openvswitch(OE) nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_nat nf_conntrack gre ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc dm_mirror dm_region_hash dm_log dm_mod coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel iTCO_wdt iTCO_vendor_support lrw gf128mul glue_helper ablk_helper cryptd mei_me mei i2c_i801 sg lpc_ich mfd_core i2c_core pcspkr sb_edac edac_core shpchp acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ixgbe crc32c_intel mpt3sas [12641660.874710] ahci mdio libahci ptp raid_class libata pps_core scsi_transport_sas dca [last unloaded: nbd] [12641660.875115] CPU: 8 PID: 52423 Comm: qemu-kvm Tainted: G OE ------------ 3.10.0-327.36.3.el7.x86_64 #1 [12641660.875483] Hardware name: Huawei RH1288 V3/BC11HGSC0, BIOS 3.50 11/23/2016 [12641660.875847] task: ffff887f06d04500 ti: ffff887ae3204000 task.ti: ffff887ae3204000 [12641660.876205] RIP: 0010:[<ffffffffa052fad4>] [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.876594] RSP: 0018:ffff887ae3207c08 EFLAGS: 00010206 [12641660.876781] RAX: 0000000000000000 RBX: ffffc9005b716f80 RCX: 00000000000194f0 [12641660.877147] RDX: 00000000fd00f000 RSI: 00000000fd00f000 RDI: ffff887ae32b0000 [12641660.877516] RBP: ffff887ae3207c18 R08: 0000000000000001 R09: 0000000000000000 [12641660.877877] R10: ffffea00c5cbca00 R11: ffffffff812f2a89 R12: ffff887ae32b0000 [12641660.878240] R13: ffffffffa052fb00 R14: 0000000000000000 R15: ffffc9004727c198 [12641660.878606] FS: 00007f9d38624ac0(0000) GS:ffff883f7fc00000(0000) knlGS:0000000000000000 [12641660.878972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12641660.879160] CR2: 00000000fd00f000 CR3: 0000006ccd888000 CR4: 00000000003427e0 [12641660.879518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12641660.879874] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12641660.880227] Stack: [12641660.880410] 0000000000000000 ffff887ae32b0000 ffff887ae3207c28 ffffffffa052fb0e [12641660.880788] ffff887ae3207cc8 ffffffffa052bfa4 00000000e3207c98 00007f9cc1c00000 [12641660.881166] 00007f9cc0000000 ffffc90047286008 ffffc9004727c198 0000000000018c00 [12641660.881541] Call Trace: [12641660.881751] [<ffffffffa052fb0e>] kvm_unmap_rmapp+0xe/0x20 [kvm] [12641660.881958] [<ffffffffa052bfa4>] kvm_handle_hva_range+0x134/0x1a0 [kvm] [12641660.882169] [<ffffffffa0537e07>] kvm_unmap_hva_range+0x17/0x20 [kvm] [12641660.882373] [<ffffffffa050eb73>] kvm_mmu_notifier_invalidate_range_start+0x53/0x90 [kvm] [12641660.882720] [<ffffffff811b9ba4>] __mmu_notifier_invalidate_range_start+0x64/0xc0 [12641660.883031] [<ffffffff8119fdb1>] change_protection_range+0x811/0x820 [12641660.883197] [<ffffffff8119fe25>] change_protection+0x65/0xa0 [12641660.883361] [<ffffffff811b68db>] change_prot_numa+0x1b/0x40 [12641660.883526] [<ffffffff810bd4b6>] task_numa_work+0x1f6/0x320 [12641660.883692] [<ffffffff810a2377>] task_work_run+0xa7/0xe0 [12641660.883857] [<ffffffff81014b12>] do_notify_resume+0x92/0xb0 [12641660.884022] [<ffffffff81646dfd>] int_signal+0x12/0x17 [12641660.884182] Code: 41 54 53 48 8b 16 48 89 f3 48 85 d2 74 3c 49 89 fc 31 c0 0f 1f 40 00 f6 c2 01 48 89 d6 74 07 48 83 e2 fe 48 8b 32 48 85 f6 74 1a <f6> 06 01 74 1e 4c 89 e7 e8 2f ff ff ff 48 8b 13 b8 01 00 00 00 [12641660.885002] RIP [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.885207] RSP <ffff887ae3207c08> [12641660.885394] CR2: 00000000fd00f000 some crash info: crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 131947033 503.3 GB ---- FREE 107747967 411 GB 81% of TOTAL MEM USED 24199066 92.3 GB 18% of TOTAL MEM SHARED 2696712 10.3 GB 2% of TOTAL MEM BUFFERS 794 3.1 MB 0% of TOTAL MEM CACHED 2929837 11.2 GB 2% of TOTAL MEM SLAB 245167 957.7 MB 0% of TOTAL MEM TOTAL SWAP 4194303 16 GB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 4194303 16 GB 100% of TOTAL SWAP COMMIT LIMIT 70167819 267.7 GB ---- COMMITTED 27778182 106 GB 39% of TOTAL LIMIT crash> crash> dis -l kvm_zap_rmapp+52 0xffffffffa052fad4 <kvm_zap_rmapp+52>: testb $0x1,(%rsi) crash> search fd00f000 ffff887925b6df80: fd00f000 ffff887925b6dff8: fd00f000 ffff887ae3207830: fd00f000 ffff887ae3207838: fd00f000 ffff887ae3207898: fd00f000 ffff887ae3207990: fd00f000 ffff887ae3207998: fd00f000 ffff887ae3207a50: fd00f000 ffff887ae3207a90: fd00f000 ffff887ae3207bb8: fd00f000 ffff887ae3207bc0: fd00f000 ffff887f027e0f98: fd00f000 ffff887f06d04c10: fd00f000 ffff887f097e4f80: fd00f000 ffffc9005b716f80: fd00f000 ffffc9005b716ff8: fd00f000 crash> crash> rd ffffc9005b716f80 20 ffffc9005b716f80: 00000000fd00f000 00000000fd00e000 ................ ffffc9005b716f90: 00000000fd00d000 00000000fd00c000 ................ ffffc9005b716fa0: 00000000fd00b000 00000000fd00a000 ................ ffffc9005b716fb0: 00000000fd009000 00000000fd008000 ................ ffffc9005b716fc0: 00000000fd007000 00000000fd006000 .p.......`...... ffffc9005b716fd0: 00000000fd005000 00000000fd004000 .P.......@...... ffffc9005b716fe0: 00000000fd003000 00000000fd002000 .0....... ...... ffffc9005b716ff0: 00000000fd001000 00000000fd00f000 ................ ffffc9005b717000: 0000000000000000 0000000000000000 ................ ffffc9005b717010: 0000000000000000 0000000000000000 ................ crash> crash> kmem ffffc9005b716f80 VMAP_AREA VM_STRUCT ADDRESS RANGE SIZE ffff887f2359f200 ffff886d81445a40 ffffc9005b64d000 - ffffc9005ba4e000 4198400 PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea01e496db40 7925b6d000 0 0 1 6fffff00000000 crash> vm_struct ffff886d81445a40 struct vm_struct { next = 0x0, addr = 0xffffc9005b64d000, size = 4198400, flags = 18, pages = 0xffffc9004717f000, nr_pages = 1024, phys_addr = 0, caller = 0xffffffffa050ed25 <kvm_kvzalloc+37> } crash>

hamidane 2020-03-10 04:29 reporter ~0036484	no solutions yet ?

Date Modified	Username	Field	Change
2017-02-21 16:06	oleksandr.mykhalskyi	New Issue
2017-02-21 16:06	oleksandr.mykhalskyi	File Added: vmcore-dmesg-1.txt
2017-02-21 16:17	oleksandr.mykhalskyi	File Added: vmcore-dmesg-2.txt
2017-02-21 16:39	oleksandr.mykhalskyi	Tag Attached: centos 7
2017-09-04 12:01	crrzhao	Note Added: 0029993
2020-03-10 04:29	hamidane	Note Added: 0036484