0012857: BUG: unable to handle kernel paging request at <address> - CentOS Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0012857	CentOS-7	centos-release	public	2017-02-21 16:06	2020-03-10 04:29

Reporter	oleksandr.mykhalskyi
Priority	normal	Severity	crash	Reproducibility	sometimes
Status	new	Resolution	open
Platform	Dell PowerEdge R630	OS	Centos	OS Version	7.2
Product Version	7.2.1511
Target Version		Fixed in Version

Summary	0012857: BUG: unable to handle kernel paging request at <address>
Description	Hello, Time to time our CentOS 7 hypervisors (with several KVM VMs), which are used as compute nodes for openstack clouds, crash with next error "BUG: unable to handle kernel paging request at <address>". These servers are also used as ceph storage nodes (with ceph-osd processes) Kernel - 3.10.0-327.36.3.el7.x86_64 Examples of stack for crashes - in the attachments. BUG: unable to handle kernel paging request at 00000000fc047000 IP: [<ffffffffa064cad4>] kvm_zap_rmapp+0x34/0x60 [kvm] BUG: unable to handle kernel paging request at 00000000fc022010 IP: [<ffffffff81190948>] anon_vma_interval_tree_remove+0x28/0x250
Tags	centos 7

abrt_hash
URL

oleksandr.mykhalskyi 2017-02-21 16:06 reporter	vmcore-dmesg-1.txt (1,034,410 bytes)

oleksandr.mykhalskyi 2017-02-21 16:17 reporter	vmcore-dmesg-2.txt (728,313 bytes)

crrzhao 2017-09-04 12:01 reporter ~0029993	Hello I met the same bug (kvm_zap_rmapp) on 5 machines. They happened on different time. The kernel is 3.10.0-327.36.3.el7.x86_64 --------------------------------- call trace: [12641660.871619] BUG: unable to handle kernel paging request at 00000000fd00f000 [12641660.871990] IP: [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.872210] PGD 0 [12641660.872392] Oops: 0000 [#1] SMP [12641660.872582] Modules linked in: xt_u32 fuse 8021q garp mrp binfmt_misc vport_vxlan(OE) vhost_net vhost macvtap macvlan tun openvswitch(OE) nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_nat nf_conntrack gre ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc dm_mirror dm_region_hash dm_log dm_mod coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel iTCO_wdt iTCO_vendor_support lrw gf128mul glue_helper ablk_helper cryptd mei_me mei i2c_i801 sg lpc_ich mfd_core i2c_core pcspkr sb_edac edac_core shpchp acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ixgbe crc32c_intel mpt3sas [12641660.874710] ahci mdio libahci ptp raid_class libata pps_core scsi_transport_sas dca [last unloaded: nbd] [12641660.875115] CPU: 8 PID: 52423 Comm: qemu-kvm Tainted: G OE ------------ 3.10.0-327.36.3.el7.x86_64 #1 [12641660.875483] Hardware name: Huawei RH1288 V3/BC11HGSC0, BIOS 3.50 11/23/2016 [12641660.875847] task: ffff887f06d04500 ti: ffff887ae3204000 task.ti: ffff887ae3204000 [12641660.876205] RIP: 0010:[<ffffffffa052fad4>] [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.876594] RSP: 0018:ffff887ae3207c08 EFLAGS: 00010206 [12641660.876781] RAX: 0000000000000000 RBX: ffffc9005b716f80 RCX: 00000000000194f0 [12641660.877147] RDX: 00000000fd00f000 RSI: 00000000fd00f000 RDI: ffff887ae32b0000 [12641660.877516] RBP: ffff887ae3207c18 R08: 0000000000000001 R09: 0000000000000000 [12641660.877877] R10: ffffea00c5cbca00 R11: ffffffff812f2a89 R12: ffff887ae32b0000 [12641660.878240] R13: ffffffffa052fb00 R14: 0000000000000000 R15: ffffc9004727c198 [12641660.878606] FS: 00007f9d38624ac0(0000) GS:ffff883f7fc00000(0000) knlGS:0000000000000000 [12641660.878972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12641660.879160] CR2: 00000000fd00f000 CR3: 0000006ccd888000 CR4: 00000000003427e0 [12641660.879518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12641660.879874] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12641660.880227] Stack: [12641660.880410] 0000000000000000 ffff887ae32b0000 ffff887ae3207c28 ffffffffa052fb0e [12641660.880788] ffff887ae3207cc8 ffffffffa052bfa4 00000000e3207c98 00007f9cc1c00000 [12641660.881166] 00007f9cc0000000 ffffc90047286008 ffffc9004727c198 0000000000018c00 [12641660.881541] Call Trace: [12641660.881751] [<ffffffffa052fb0e>] kvm_unmap_rmapp+0xe/0x20 [kvm] [12641660.881958] [<ffffffffa052bfa4>] kvm_handle_hva_range+0x134/0x1a0 [kvm] [12641660.882169] [<ffffffffa0537e07>] kvm_unmap_hva_range+0x17/0x20 [kvm] [12641660.882373] [<ffffffffa050eb73>] kvm_mmu_notifier_invalidate_range_start+0x53/0x90 [kvm] [12641660.882720] [<ffffffff811b9ba4>] __mmu_notifier_invalidate_range_start+0x64/0xc0 [12641660.883031] [<ffffffff8119fdb1>] change_protection_range+0x811/0x820 [12641660.883197] [<ffffffff8119fe25>] change_protection+0x65/0xa0 [12641660.883361] [<ffffffff811b68db>] change_prot_numa+0x1b/0x40 [12641660.883526] [<ffffffff810bd4b6>] task_numa_work+0x1f6/0x320 [12641660.883692] [<ffffffff810a2377>] task_work_run+0xa7/0xe0 [12641660.883857] [<ffffffff81014b12>] do_notify_resume+0x92/0xb0 [12641660.884022] [<ffffffff81646dfd>] int_signal+0x12/0x17 [12641660.884182] Code: 41 54 53 48 8b 16 48 89 f3 48 85 d2 74 3c 49 89 fc 31 c0 0f 1f 40 00 f6 c2 01 48 89 d6 74 07 48 83 e2 fe 48 8b 32 48 85 f6 74 1a <f6> 06 01 74 1e 4c 89 e7 e8 2f ff ff ff 48 8b 13 b8 01 00 00 00 [12641660.885002] RIP [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm] [12641660.885207] RSP <ffff887ae3207c08> [12641660.885394] CR2: 00000000fd00f000 some crash info: crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 131947033 503.3 GB ---- FREE 107747967 411 GB 81% of TOTAL MEM USED 24199066 92.3 GB 18% of TOTAL MEM SHARED 2696712 10.3 GB 2% of TOTAL MEM BUFFERS 794 3.1 MB 0% of TOTAL MEM CACHED 2929837 11.2 GB 2% of TOTAL MEM SLAB 245167 957.7 MB 0% of TOTAL MEM TOTAL SWAP 4194303 16 GB ---- SWAP USED 0 0 0% of TOTAL SWAP SWAP FREE 4194303 16 GB 100% of TOTAL SWAP COMMIT LIMIT 70167819 267.7 GB ---- COMMITTED 27778182 106 GB 39% of TOTAL LIMIT crash> crash> dis -l kvm_zap_rmapp+52 0xffffffffa052fad4 <kvm_zap_rmapp+52>: testb $0x1,(%rsi) crash> search fd00f000 ffff887925b6df80: fd00f000 ffff887925b6dff8: fd00f000 ffff887ae3207830: fd00f000 ffff887ae3207838: fd00f000 ffff887ae3207898: fd00f000 ffff887ae3207990: fd00f000 ffff887ae3207998: fd00f000 ffff887ae3207a50: fd00f000 ffff887ae3207a90: fd00f000 ffff887ae3207bb8: fd00f000 ffff887ae3207bc0: fd00f000 ffff887f027e0f98: fd00f000 ffff887f06d04c10: fd00f000 ffff887f097e4f80: fd00f000 ffffc9005b716f80: fd00f000 ffffc9005b716ff8: fd00f000 crash> crash> rd ffffc9005b716f80 20 ffffc9005b716f80: 00000000fd00f000 00000000fd00e000 ................ ffffc9005b716f90: 00000000fd00d000 00000000fd00c000 ................ ffffc9005b716fa0: 00000000fd00b000 00000000fd00a000 ................ ffffc9005b716fb0: 00000000fd009000 00000000fd008000 ................ ffffc9005b716fc0: 00000000fd007000 00000000fd006000 .p.......`...... ffffc9005b716fd0: 00000000fd005000 00000000fd004000 .P.......@...... ffffc9005b716fe0: 00000000fd003000 00000000fd002000 .0....... ...... ffffc9005b716ff0: 00000000fd001000 00000000fd00f000 ................ ffffc9005b717000: 0000000000000000 0000000000000000 ................ ffffc9005b717010: 0000000000000000 0000000000000000 ................ crash> crash> kmem ffffc9005b716f80 VMAP_AREA VM_STRUCT ADDRESS RANGE SIZE ffff887f2359f200 ffff886d81445a40 ffffc9005b64d000 - ffffc9005ba4e000 4198400 PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea01e496db40 7925b6d000 0 0 1 6fffff00000000 crash> vm_struct ffff886d81445a40 struct vm_struct { next = 0x0, addr = 0xffffc9005b64d000, size = 4198400, flags = 18, pages = 0xffffc9004717f000, nr_pages = 1024, phys_addr = 0, caller = 0xffffffffa050ed25 <kvm_kvzalloc+37> } crash>

hamidane 2020-03-10 04:29 reporter ~0036484	no solutions yet ?

Date Modified	Username	Field	Change
2017-02-21 16:06	oleksandr.mykhalskyi	New Issue
2017-02-21 16:06	oleksandr.mykhalskyi	File Added: vmcore-dmesg-1.txt
2017-02-21 16:17	oleksandr.mykhalskyi	File Added: vmcore-dmesg-2.txt
2017-02-21 16:39	oleksandr.mykhalskyi	Tag Attached: centos 7
2017-09-04 12:01	crrzhao	Note Added: 0029993
2020-03-10 04:29	hamidane	Note Added: 0036484