View Issue Details

IDProjectCategoryView StatusLast Update
0012857CentOS-7centos-releasepublic2020-03-10 04:29
Reporteroleksandr.mykhalskyi 
PrioritynormalSeveritycrashReproducibilitysometimes
Status newResolutionopen 
PlatformDell PowerEdge R630OSCentos OS Version7.2
Product Version7.2.1511 
Target VersionFixed in Version 
Summary0012857: BUG: unable to handle kernel paging request at <address>
DescriptionHello,

Time to time our CentOS 7 hypervisors (with several KVM VMs), which are used as compute nodes for openstack clouds, crash with next error "BUG: unable to handle kernel paging request at <address>". These servers are also used as ceph storage nodes (with ceph-osd processes)

Kernel - 3.10.0-327.36.3.el7.x86_64

Examples of stack for crashes - in the attachments.
 
BUG: unable to handle kernel paging request at 00000000fc047000
IP: [<ffffffffa064cad4>] kvm_zap_rmapp+0x34/0x60 [kvm]

BUG: unable to handle kernel paging request at 00000000fc022010
IP: [<ffffffff81190948>] anon_vma_interval_tree_remove+0x28/0x250
Tagscentos 7
abrt_hash
URL

Activities

oleksandr.mykhalskyi

oleksandr.mykhalskyi

2017-02-21 16:06

reporter  

vmcore-dmesg-1.txt (1,034,410 bytes)
oleksandr.mykhalskyi

oleksandr.mykhalskyi

2017-02-21 16:17

reporter  

vmcore-dmesg-2.txt (728,313 bytes)
crrzhao

crrzhao

2017-09-04 12:01

reporter   ~0029993

Hello

I met the same bug (kvm_zap_rmapp) on 5 machines. They happened on different time. The kernel is 3.10.0-327.36.3.el7.x86_64

---------------------------------
call trace:
[12641660.871619] BUG: unable to handle kernel paging request at 00000000fd00f000
[12641660.871990] IP: [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm]
[12641660.872210] PGD 0
[12641660.872392] Oops: 0000 [#1] SMP
[12641660.872582] Modules linked in: xt_u32 fuse 8021q garp mrp binfmt_misc vport_vxlan(OE) vhost_net vhost macvtap macvlan tun openvswitch(OE) nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_nat nf_conntrack gre ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc dm_mirror dm_region_hash dm_log dm_mod coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel iTCO_wdt iTCO_vendor_support lrw gf128mul glue_helper ablk_helper cryptd mei_me mei i2c_i801 sg lpc_ich mfd_core i2c_core pcspkr sb_edac edac_core shpchp acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ixgbe crc32c_intel mpt3sas
[12641660.874710] ahci mdio libahci ptp raid_class libata pps_core scsi_transport_sas dca [last unloaded: nbd]
[12641660.875115] CPU: 8 PID: 52423 Comm: qemu-kvm Tainted: G OE ------------ 3.10.0-327.36.3.el7.x86_64 #1
[12641660.875483] Hardware name: Huawei RH1288 V3/BC11HGSC0, BIOS 3.50 11/23/2016
[12641660.875847] task: ffff887f06d04500 ti: ffff887ae3204000 task.ti: ffff887ae3204000
[12641660.876205] RIP: 0010:[<ffffffffa052fad4>] [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm]
[12641660.876594] RSP: 0018:ffff887ae3207c08 EFLAGS: 00010206
[12641660.876781] RAX: 0000000000000000 RBX: ffffc9005b716f80 RCX: 00000000000194f0
[12641660.877147] RDX: 00000000fd00f000 RSI: 00000000fd00f000 RDI: ffff887ae32b0000
[12641660.877516] RBP: ffff887ae3207c18 R08: 0000000000000001 R09: 0000000000000000
[12641660.877877] R10: ffffea00c5cbca00 R11: ffffffff812f2a89 R12: ffff887ae32b0000
[12641660.878240] R13: ffffffffa052fb00 R14: 0000000000000000 R15: ffffc9004727c198
[12641660.878606] FS: 00007f9d38624ac0(0000) GS:ffff883f7fc00000(0000) knlGS:0000000000000000
[12641660.878972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641660.879160] CR2: 00000000fd00f000 CR3: 0000006ccd888000 CR4: 00000000003427e0
[12641660.879518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[12641660.879874] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[12641660.880227] Stack:
[12641660.880410] 0000000000000000 ffff887ae32b0000 ffff887ae3207c28 ffffffffa052fb0e
[12641660.880788] ffff887ae3207cc8 ffffffffa052bfa4 00000000e3207c98 00007f9cc1c00000
[12641660.881166] 00007f9cc0000000 ffffc90047286008 ffffc9004727c198 0000000000018c00
[12641660.881541] Call Trace:
[12641660.881751] [<ffffffffa052fb0e>] kvm_unmap_rmapp+0xe/0x20 [kvm]
[12641660.881958] [<ffffffffa052bfa4>] kvm_handle_hva_range+0x134/0x1a0 [kvm]
[12641660.882169] [<ffffffffa0537e07>] kvm_unmap_hva_range+0x17/0x20 [kvm]
[12641660.882373] [<ffffffffa050eb73>] kvm_mmu_notifier_invalidate_range_start+0x53/0x90 [kvm]
[12641660.882720] [<ffffffff811b9ba4>] __mmu_notifier_invalidate_range_start+0x64/0xc0
[12641660.883031] [<ffffffff8119fdb1>] change_protection_range+0x811/0x820
[12641660.883197] [<ffffffff8119fe25>] change_protection+0x65/0xa0
[12641660.883361] [<ffffffff811b68db>] change_prot_numa+0x1b/0x40
[12641660.883526] [<ffffffff810bd4b6>] task_numa_work+0x1f6/0x320
[12641660.883692] [<ffffffff810a2377>] task_work_run+0xa7/0xe0
[12641660.883857] [<ffffffff81014b12>] do_notify_resume+0x92/0xb0
[12641660.884022] [<ffffffff81646dfd>] int_signal+0x12/0x17
[12641660.884182] Code: 41 54 53 48 8b 16 48 89 f3 48 85 d2 74 3c 49 89 fc 31 c0 0f 1f 40 00 f6 c2 01 48 89 d6 74 07 48 83 e2 fe 48 8b 32 48 85 f6 74 1a <f6> 06 01 74 1e 4c 89 e7 e8 2f ff ff ff 48 8b 13 b8 01 00 00 00
[12641660.885002] RIP [<ffffffffa052fad4>] kvm_zap_rmapp+0x34/0x60 [kvm]
[12641660.885207] RSP <ffff887ae3207c08>
[12641660.885394] CR2: 00000000fd00f000

some crash info:

crash> kmem -i
                 PAGES TOTAL PERCENTAGE
    TOTAL MEM 131947033 503.3 GB ----
         FREE 107747967 411 GB 81% of TOTAL MEM
         USED 24199066 92.3 GB 18% of TOTAL MEM
       SHARED 2696712 10.3 GB 2% of TOTAL MEM
      BUFFERS 794 3.1 MB 0% of TOTAL MEM
       CACHED 2929837 11.2 GB 2% of TOTAL MEM
         SLAB 245167 957.7 MB 0% of TOTAL MEM

   TOTAL SWAP 4194303 16 GB ----
    SWAP USED 0 0 0% of TOTAL SWAP
    SWAP FREE 4194303 16 GB 100% of TOTAL SWAP

 COMMIT LIMIT 70167819 267.7 GB ----
    COMMITTED 27778182 106 GB 39% of TOTAL LIMIT
crash>
crash> dis -l kvm_zap_rmapp+52
0xffffffffa052fad4 <kvm_zap_rmapp+52>: testb $0x1,(%rsi)
crash> search fd00f000
ffff887925b6df80: fd00f000
ffff887925b6dff8: fd00f000
ffff887ae3207830: fd00f000
ffff887ae3207838: fd00f000
ffff887ae3207898: fd00f000
ffff887ae3207990: fd00f000
ffff887ae3207998: fd00f000
ffff887ae3207a50: fd00f000
ffff887ae3207a90: fd00f000
ffff887ae3207bb8: fd00f000
ffff887ae3207bc0: fd00f000
ffff887f027e0f98: fd00f000
ffff887f06d04c10: fd00f000
ffff887f097e4f80: fd00f000
ffffc9005b716f80: fd00f000
ffffc9005b716ff8: fd00f000
crash>
crash> rd ffffc9005b716f80 20
ffffc9005b716f80: 00000000fd00f000 00000000fd00e000 ................
ffffc9005b716f90: 00000000fd00d000 00000000fd00c000 ................
ffffc9005b716fa0: 00000000fd00b000 00000000fd00a000 ................
ffffc9005b716fb0: 00000000fd009000 00000000fd008000 ................
ffffc9005b716fc0: 00000000fd007000 00000000fd006000 .p.......`......
ffffc9005b716fd0: 00000000fd005000 00000000fd004000 .P.......@......
ffffc9005b716fe0: 00000000fd003000 00000000fd002000 .0....... ......
ffffc9005b716ff0: 00000000fd001000 00000000fd00f000 ................
ffffc9005b717000: 0000000000000000 0000000000000000 ................
ffffc9005b717010: 0000000000000000 0000000000000000 ................
crash>
crash> kmem ffffc9005b716f80
   VMAP_AREA VM_STRUCT ADDRESS RANGE SIZE
ffff887f2359f200 ffff886d81445a40 ffffc9005b64d000 - ffffc9005ba4e000 4198400

      PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea01e496db40 7925b6d000 0 0 1 6fffff00000000
crash> vm_struct ffff886d81445a40
struct vm_struct {
  next = 0x0,
  addr = 0xffffc9005b64d000,
  size = 4198400,
  flags = 18,
  pages = 0xffffc9004717f000,
  nr_pages = 1024,
  phys_addr = 0,
  caller = 0xffffffffa050ed25 <kvm_kvzalloc+37>
}
crash>
hamidane

hamidane

2020-03-10 04:29

reporter   ~0036484

no solutions yet ?

Issue History

Date Modified Username Field Change
2017-02-21 16:06 oleksandr.mykhalskyi New Issue
2017-02-21 16:06 oleksandr.mykhalskyi File Added: vmcore-dmesg-1.txt
2017-02-21 16:17 oleksandr.mykhalskyi File Added: vmcore-dmesg-2.txt
2017-02-21 16:39 oleksandr.mykhalskyi Tag Attached: centos 7
2017-09-04 12:01 crrzhao Note Added: 0029993
2020-03-10 04:29 hamidane Note Added: 0036484