2017-08-18 16:27 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013063CentOS-7kernelpublic2017-05-17 09:22
Reporterbvdheuvel 
PrioritynormalSeverityminorReproducibilitysometimes
StatusassignedResolutionopen 
Platformx86_64OSCentosOS Version7.3
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0013063: Kernel panic at nvme queuing
DescriptionAt medium load the kernel crashed and the machine rebooted
Steps To ReproduceNot sure, happened while moving storage
Additional Information400MB vmcore file is available
TagsNo tags attached.
abrt_hash
URL
Attached Files
  • txt file icon vmcore-dmesg.txt (147,093 bytes) 2017-03-30 13:33 -
    [    0.000000] microcode: microcode updated early to revision 0xb00001d, date = 2016-06-06
    [    0.000000] Initializing cgroup subsys cpuset
    [    0.000000] Initializing cgroup subsys cpu
    [    0.000000] Initializing cgroup subsys cpuacct
    [    0.000000] Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 3 00:04:05 UTC 2017
    [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/vg_hdd-root ro crashkernel=auto rd.md.uuid=a2e91d96:16d23ae4:225ceebf:ec171e89 rd.lvm.lv=vg_hdd/root rd.lvm.lv=vg_hdd/swap rd.lvm.lv=vg_hdd/usr rhgb quiet LANG=en_US.UTF-8
    [    0.000000] e820: BIOS-provided physical RAM map:
    [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000003efff] usable
    [    0.000000] BIOS-e820: [mem 0x000000000003f000-0x000000000003ffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000000040000-0x000000000009ffff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000078e4dfff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000078e4e000-0x0000000079669fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000007966a000-0x0000000079a89fff] usable
    [    0.000000] BIOS-e820: [mem 0x0000000079a8a000-0x0000000079c8afff] ACPI NVS
    [    0.000000] BIOS-e820: [mem 0x0000000079c8b000-0x000000007bcf9fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000007bcfa000-0x000000007bcfafff] usable
    [    0.000000] BIOS-e820: [mem 0x000000007bcfb000-0x000000007bd80fff] reserved
    [    0.000000] BIOS-e820: [mem 0x000000007bd81000-0x000000007bffffff] usable
    [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved
    [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
    [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000087fffffff] usable
    [    0.000000] NX (Execute Disable) protection: active
    [    0.000000] e820: update [mem 0x7549d018-0x754ae057] usable ==> usable
    [    0.000000] e820: update [mem 0x7548b018-0x7549c057] usable ==> usable
    [    0.000000] e820: update [mem 0x75479018-0x7548a057] usable ==> usable
    [    0.000000] e820: update [mem 0x75467018-0x75478057] usable ==> usable
    [    0.000000] e820: update [mem 0x7545f018-0x75466857] usable ==> usable
    [    0.000000] e820: update [mem 0x75457018-0x7545e857] usable ==> usable
    [    0.000000] e820: update [mem 0x7544e018-0x75456057] usable ==> usable
    [    0.000000] extended physical RAM map:
    [    0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000003efff] usable
    [    0.000000] reserve setup_data: [mem 0x000000000003f000-0x000000000003ffff] reserved
    [    0.000000] reserve setup_data: [mem 0x0000000000040000-0x000000000009ffff] usable
    [    0.000000] reserve setup_data: [mem 0x0000000000100000-0x000000007544e017] usable
    [    0.000000] reserve setup_data: [mem 0x000000007544e018-0x0000000075456057] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075456058-0x0000000075457017] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075457018-0x000000007545e857] usable
    [    0.000000] reserve setup_data: [mem 0x000000007545e858-0x000000007545f017] usable
    [    0.000000] reserve setup_data: [mem 0x000000007545f018-0x0000000075466857] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075466858-0x0000000075467017] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075467018-0x0000000075478057] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075478058-0x0000000075479017] usable
    [    0.000000] reserve setup_data: [mem 0x0000000075479018-0x000000007548a057] usable
    [    0.000000] reserve setup_data: [mem 0x000000007548a058-0x000000007548b017] usable
    [    0.000000] reserve setup_data: [mem 0x000000007548b018-0x000000007549c057] usable
    [    0.000000] reserve setup_data: [mem 0x000000007549c058-0x000000007549d017] usable
    [    0.000000] reserve setup_data: [mem 0x000000007549d018-0x00000000754ae057] usable
    [    0.000000] reserve setup_data: [mem 0x00000000754ae058-0x0000000078e4dfff] usable
    [    0.000000] reserve setup_data: [mem 0x0000000078e4e000-0x0000000079669fff] reserved
    [    0.000000] reserve setup_data: [mem 0x000000007966a000-0x0000000079a89fff] usable
    [    0.000000] reserve setup_data: [mem 0x0000000079a8a000-0x0000000079c8afff] ACPI NVS
    [    0.000000] reserve setup_data: [mem 0x0000000079c8b000-0x000000007bcf9fff] reserved
    [    0.000000] reserve setup_data: [mem 0x000000007bcfa000-0x000000007bcfafff] usable
    [    0.000000] reserve setup_data: [mem 0x000000007bcfb000-0x000000007bd80fff] reserved
    [    0.000000] reserve setup_data: [mem 0x000000007bd81000-0x000000007bffffff] usable
    [    0.000000] reserve setup_data: [mem 0x000000007c000000-0x000000008fffffff] reserved
    [    0.000000] reserve setup_data: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved
    [    0.000000] reserve setup_data: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
    [    0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000087fffffff] usable
    [    0.000000] efi: EFI v2.40 by American Megatrends
    [    0.000000] efi:  ACPI=0x79ada000  ACPI 2.0=0x79ada000  SMBIOS=0x7ba5b000  SMBIOS 3.0=0x7ba5a000 
    [    0.000000] efi: mem00: type=7, attr=0xf, range=[0x0000000000000000-0x000000000003f000) (0MB)
    [    0.000000] efi: mem01: type=6, attr=0x800000000000000f, range=[0x000000000003f000-0x0000000000040000) (0MB)
    [    0.000000] efi: mem02: type=2, attr=0xf, range=[0x0000000000040000-0x0000000000041000) (0MB)
    [    0.000000] efi: mem03: type=7, attr=0xf, range=[0x0000000000041000-0x00000000000a0000) (0MB)
    [    0.000000] efi: mem04: type=7, attr=0xf, range=[0x0000000000100000-0x0000000001000000) (15MB)
    [    0.000000] efi: mem05: type=2, attr=0xf, range=[0x0000000001000000-0x00000000023c0000) (19MB)
    [    0.000000] efi: mem06: type=7, attr=0xf, range=[0x00000000023c0000-0x000000003ead7000) (967MB)
    [    0.000000] efi: mem07: type=2, attr=0xf, range=[0x000000003ead7000-0x0000000040000000) (21MB)
    [    0.000000] efi: mem08: type=7, attr=0xf, range=[0x0000000040000000-0x0000000048919000) (137MB)
    [    0.000000] efi: mem09: type=2, attr=0xf, range=[0x0000000048919000-0x0000000065f39000) (470MB)
    [    0.000000] efi: mem10: type=4, attr=0xf, range=[0x0000000065f39000-0x0000000065fb9000) (0MB)
    [    0.000000] efi: mem11: type=7, attr=0xf, range=[0x0000000065fb9000-0x000000007544e000) (244MB)
    [    0.000000] efi: mem12: type=2, attr=0xf, range=[0x000000007544e000-0x00000000756a8000) (2MB)
    [    0.000000] efi: mem13: type=1, attr=0xf, range=[0x00000000756a8000-0x00000000757cf000) (1MB)
    [    0.000000] efi: mem14: type=4, attr=0xf, range=[0x00000000757cf000-0x0000000078833000) (48MB)
    [    0.000000] efi: mem15: type=7, attr=0xf, range=[0x0000000078833000-0x000000007896b000) (1MB)
    [    0.000000] efi: mem16: type=3, attr=0xf, range=[0x000000007896b000-0x0000000078e4e000) (4MB)
    [    0.000000] efi: mem17: type=0, attr=0xf, range=[0x0000000078e4e000-0x000000007966a000) (8MB)
    [    0.000000] efi: mem18: type=7, attr=0xf, range=[0x000000007966a000-0x0000000079a8a000) (4MB)
    [    0.000000] efi: mem19: type=10, attr=0xf, range=[0x0000000079a8a000-0x0000000079c8b000) (2MB)
    [    0.000000] efi: mem20: type=6, attr=0x800000000000000f, range=[0x0000000079c8b000-0x000000007bc98000) (32MB)
    [    0.000000] efi: mem21: type=5, attr=0x800000000000000f, range=[0x000000007bc98000-0x000000007bcfa000) (0MB)
    [    0.000000] efi: mem22: type=4, attr=0xf, range=[0x000000007bcfa000-0x000000007bcfb000) (0MB)
    [    0.000000] efi: mem23: type=6, attr=0x800000000000000f, range=[0x000000007bcfb000-0x000000007bd81000) (0MB)
    [    0.000000] efi: mem24: type=4, attr=0xf, range=[0x000000007bd81000-0x000000007c000000) (2MB)
    [    0.000000] efi: mem25: type=7, attr=0xf, range=[0x0000000100000000-0x0000000880000000) (30720MB)
    [    0.000000] efi: mem26: type=0, attr=0x0, range=[0x000000007c000000-0x0000000080000000) (64MB)
    [    0.000000] efi: mem27: type=11, attr=0x8000000000000001, range=[0x0000000080000000-0x0000000090000000) (256MB)
    [    0.000000] efi: mem28: type=11, attr=0x8000000000000001, range=[0x00000000fed1c000-0x00000000fed45000) (0MB)
    [    0.000000] efi: mem29: type=11, attr=0x8000000000000001, range=[0x00000000ff000000-0x0000000100000000) (16MB)
    [    0.000000] SMBIOS 3.0.0 present.
    [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
    [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
    [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
    [    0.000000] MTRR default type: write-back
    [    0.000000] MTRR fixed ranges enabled:
    [    0.000000]   00000-9FFFF write-back
    [    0.000000]   A0000-BFFFF uncachable
    [    0.000000]   C0000-DFFFF write-protect
    [    0.000000]   E0000-FFFFF uncachable
    [    0.000000] MTRR variable ranges enabled:
    [    0.000000]   0 base 000080000000 mask 3FFF80000000 uncachable
    [    0.000000]   1 base 380000000000 mask 3F8000000000 uncachable
    [    0.000000]   2 base 0000C7400000 mask 3FFFFFC00000 write-through
    [    0.000000]   3 base 0000C7800000 mask 3FFFFF800000 write-through
    [    0.000000]   4 base 0000C7F00000 mask 3FFFFFF00000 uncachable
    [    0.000000]   5 disabled
    [    0.000000]   6 disabled
    [    0.000000]   7 disabled
    [    0.000000]   8 disabled
    [    0.000000]   9 disabled
    [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
    [    0.000000] x2apic enabled by BIOS, switching to x2apic ops
    [    0.000000] e820: last_pfn = 0x7c000 max_arch_pfn = 0x400000000
    [    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
    [    0.000000] Using GB pages for direct mapping
    [    0.000000] BRK [0x01f9e000, 0x01f9efff] PGTABLE
    [    0.000000] BRK [0x01f9f000, 0x01f9ffff] PGTABLE
    [    0.000000] BRK [0x01fa0000, 0x01fa0fff] PGTABLE
    [    0.000000] BRK [0x01fa1000, 0x01fa1fff] PGTABLE
    [    0.000000] BRK [0x01fa2000, 0x01fa2fff] PGTABLE
    [    0.000000] BRK [0x01fa3000, 0x01fa3fff] PGTABLE
    [    0.000000] RAMDISK: [mem 0x3ead7000-0x3fffafff]
    [    0.000000] ACPI: RSDP 0000000079ada000 00024 (v02 SUPERM)
    [    0.000000] ACPI: XSDT 0000000079ada0b0 000DC (v01                 01072009 AMI  00010013)
    [    0.000000] ACPI: FACP 0000000079b0b228 0010C (v05 SUPERM SMCI--MB 01072009 AMI  00010013)
    [    0.000000] ACPI: DSDT 0000000079ada220 31005 (v02 SUPERM SMCI--MB 01072009 INTL 20091013)
    [    0.000000] ACPI: FACS 0000000079c89f80 00040
    [    0.000000] ACPI: APIC 0000000079b0b338 00144 (v03 SUPERM SMCI--MB 01072009 AMI  00010013)
    [    0.000000] ACPI: FPDT 0000000079b0b480 00044 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
    [    0.000000] ACPI: FIDT 0000000079b0b4c8 0009C (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
    [    0.000000] ACPI: SPMI 0000000079b0b568 00040 (v05 SUPERM SMCI--MB 00000000 AMI. 00000000)
    [    0.000000] ACPI: MCFG 0000000079b0b5a8 0003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
    [    0.000000] ACPI: UEFI 0000000079b0b5e8 00042 (v01 SUPERM SMCI--MB 01072009      00000000)
    [    0.000000] ACPI: HPET 0000000079b0b630 00038 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: MSCT 0000000079b0b668 00090 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: SLIT 0000000079b0b6f8 00030 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: SRAT 0000000079b0b728 01158 (v03 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: WDDT 0000000079b0c880 00040 (v01 SUPERM SMCI--MB 00000000 INTL 20091013)
    [    0.000000] ACPI: SSDT 0000000079b0c8c0 16DE7 (v02 SUPERM    PmMgt 00000001 INTL 20120913)
    [    0.000000] ACPI: NITR 0000000079b236a8 00071 (v02 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: SSDT 0000000079b23720 0264C (v02 SUPERM SpsNm    00000002 INTL 20120913)
    [    0.000000] ACPI: SSDT 0000000079b25d70 00064 (v02 SUPERM SpsNvs   00000002 INTL 20120913)
    [    0.000000] ACPI: PRAD 0000000079b25dd8 00102 (v02 SUPERM SMCI--MB 00000002 INTL 20120913)
    [    0.000000] ACPI: DMAR 0000000079b25ee0 001A8 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
    [    0.000000] ACPI: HEST 0000000079b26088 0027C (v01 SUPERM SMCI--MB 00000001 INTL 00000001)
    [    0.000000] ACPI: BERT 0000000079b26308 00030 (v01 SUPERM SMCI--MB 00000001 INTL 00000001)
    [    0.000000] ACPI: ERST 0000000079b26338 00230 (v01 SUPERM SMCI--MB 00000001 INTL 00000001)
    [    0.000000] ACPI: EINJ 0000000079b26568 00130 (v01 SUPERM SMCI--MB 00000001 INTL 00000001)
    [    0.000000] ACPI: NFIT 0000000079b26698 00028 (v01 SUPERM SMCI--MB 00000001    ? 00000001)
    [    0.000000] ACPI: Local APIC address 0xfee00000
    [    0.000000] Setting APIC routing to cluster x2apic.
    [    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0
    [    0.000000] SRAT: PXM 0 -> APIC 0x0e -> Node 0
    [    0.000000] SRAT: PXM 1 -> APIC 0x10 -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x12 -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x14 -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x16 -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x18 -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x1a -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x1c -> Node 1
    [    0.000000] SRAT: PXM 1 -> APIC 0x1e -> Node 1
    [    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
    [    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x47fffffff]
    [    0.000000] SRAT: Node 1 PXM 1 [mem 0x480000000-0x87fffffff]
    [    0.000000] NUMA: Initialized distance table, cnt=2
    [    0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x47fffffff] -> [mem 0x00000000-0x47fffffff]
    [    0.000000] Initmem setup node 0 [mem 0x00000000-0x47fffffff]
    [    0.000000]   NODE_DATA [mem 0x47ffd9000-0x47fffffff]
    [    0.000000] Initmem setup node 1 [mem 0x480000000-0x87fffffff]
    [    0.000000]   NODE_DATA [mem 0x87ffd6000-0x87fffcfff]
    [    0.000000] Reserving 162MB of memory at 720MB for crashkernel (System RAM: 32660MB)
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
    [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
    [    0.000000]   Normal   [mem 0x100000000-0x87fffffff]
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x00001000-0x0003efff]
    [    0.000000]   node   0: [mem 0x00040000-0x0009ffff]
    [    0.000000]   node   0: [mem 0x00100000-0x78e4dfff]
    [    0.000000]   node   0: [mem 0x7966a000-0x79a89fff]
    [    0.000000]   node   0: [mem 0x7bcfa000-0x7bcfafff]
    [    0.000000]   node   0: [mem 0x7bd81000-0x7bffffff]
    [    0.000000]   node   0: [mem 0x100000000-0x47fffffff]
    [    0.000000]   node   1: [mem 0x480000000-0x87fffffff]
    [    0.000000] On node 0 totalpages: 4166796
    [    0.000000]   DMA zone: 64 pages used for memmap
    [    0.000000]   DMA zone: 22 pages reserved
    [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
    [    0.000000]   DMA32 zone: 7700 pages used for memmap
    [    0.000000]   DMA32 zone: 492782 pages, LIFO batch:31
    [    0.000000]   Normal zone: 57344 pages used for memmap
    [    0.000000]   Normal zone: 3670016 pages, LIFO batch:31
    [    0.000000] On node 1 totalpages: 4194304
    [    0.000000]   Normal zone: 65536 pages used for memmap
    [    0.000000]   Normal zone: 4194304 pages, LIFO batch:31
    [    0.000000] tboot: non-0 tboot_addr but it is not of type E820_RESERVED
    [    0.000000] ACPI: PM-Timer IO Port: 0x408
    [    0.000000] ACPI: Local APIC address 0xfee00000
    [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x10] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x12] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x14] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x16] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x18] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x1a] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x1c] enabled)
    [    0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x1e] enabled)
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x18] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high edge lint[0x1])
    [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high edge lint[0x1])
    [    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
    [    0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
    [    0.000000] ACPI: IOAPIC (id[0x02] address[0xfec01000] gsi_base[24])
    [    0.000000] IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
    [    0.000000] ACPI: IOAPIC (id[0x03] address[0xfec40000] gsi_base[48])
    [    0.000000] IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
    [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
    [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
    [    0.000000] ACPI: IRQ0 used by override.
    [    0.000000] ACPI: IRQ9 used by override.
    [    0.000000] Using ACPI (MADT) for SMP configuration information
    [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
    [    0.000000] smpboot: Allowing 16 CPUs, 0 hotplug CPUs
    [    0.000000] PM: Registered nosave memory: [mem 0x0003f000-0x0003ffff]
    [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7544e000-0x7544efff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75456000-0x75456fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75457000-0x75457fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7545e000-0x7545efff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7545f000-0x7545ffff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75466000-0x75466fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75467000-0x75467fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75478000-0x75478fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x75479000-0x75479fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7548a000-0x7548afff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7548b000-0x7548bfff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7549c000-0x7549cfff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7549d000-0x7549dfff]
    [    0.000000] PM: Registered nosave memory: [mem 0x754ae000-0x754aefff]
    [    0.000000] PM: Registered nosave memory: [mem 0x78e4e000-0x79669fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x79a8a000-0x79c8afff]
    [    0.000000] PM: Registered nosave memory: [mem 0x79c8b000-0x7bcf9fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7bcfb000-0x7bd80fff]
    [    0.000000] PM: Registered nosave memory: [mem 0x7c000000-0x8fffffff]
    [    0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfed1bfff]
    [    0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed44fff]
    [    0.000000] PM: Registered nosave memory: [mem 0xfed45000-0xfeffffff]
    [    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
    [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices
    [    0.000000] Booting paravirtualized kernel on bare hardware
    [    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:16 nr_cpu_ids:16 nr_node_ids:2
    [    0.000000] PERCPU: Embedded 33 pages/cpu @ffff88046fc00000 s96728 r8192 d30248 u262144
    [    0.000000] pcpu-alloc: s96728 r8192 d30248 u262144 alloc=1*2097152
    [    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [1] 08 09 10 11 12 13 14 15 
    [    0.000000] Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 8230434
    [    0.000000] Policy zone: Normal
    [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/vg_hdd-root ro crashkernel=auto rd.md.uuid=a2e91d96:16d23ae4:225ceebf:ec171e89 rd.lvm.lv=vg_hdd/root rd.lvm.lv=vg_hdd/swap rd.lvm.lv=vg_hdd/usr rhgb quiet LANG=en_US.UTF-8
    [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
    [    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
    [    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form
    [    0.000000] Memory: 5854196k/35651584k available (6765k kernel code, 2207184k absent, 854804k reserved, 4432k data, 1680k init)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=2
    [    0.000000] Hierarchical RCU implementation.
    [    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=16.
    [    0.000000] NR_IRQS:327936 nr_irqs:1368 0
    [    0.000000] Console: colour dummy device 80x25
    [    0.000000] console [tty0] enabled
    [    0.000000] allocated 134217728 bytes of page_cgroup
    [    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
    [    0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
    [    0.000000] hpet clockevent registered
    [    0.000000] tsc: Fast TSC calibration using PIT
    [    0.000000] tsc: Detected 1699.999 MHz processor
    [    0.000046] Calibrating delay loop (skipped), value calculated using timer frequency.. 3399.99 BogoMIPS (lpj=1699999)
    [    0.000049] pid_max: default: 32768 minimum: 301
    [    0.001452] Security Framework initialized
    [    0.001459] SELinux:  Initializing.
    [    0.001472] SELinux:  Starting in permissive mode
    [    0.004510] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
    [    0.012805] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
    [    0.016185] Mount-cache hash table entries: 4096
    [    0.016457] Initializing cgroup subsys memory
    [    0.016472] Initializing cgroup subsys devices
    [    0.016473] Initializing cgroup subsys freezer
    [    0.016475] Initializing cgroup subsys net_cls
    [    0.016476] Initializing cgroup subsys blkio
    [    0.016478] Initializing cgroup subsys perf_event
    [    0.016480] Initializing cgroup subsys hugetlb
    [    0.016482] Initializing cgroup subsys pids
    [    0.016483] Initializing cgroup subsys net_prio
    [    0.016524] CPU: Physical Processor ID: 0
    [    0.016525] CPU: Processor Core ID: 0
    [    0.017361] mce: CPU supports 22 MCE banks
    [    0.017400] CPU0: Thermal monitoring enabled (TM1)
    [    0.017432] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
    [    0.017433] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0
    [    0.017434] tlb_flushall_shift: 6
    [    0.017897] Freeing SMP alternatives: 28k freed
    [    0.019174] ACPI: Core revision 20130517
    [    0.052374] ACPI: All ACPI Tables successfully acquired
    [    0.055712] ftrace: allocating 25810 entries in 101 pages
    [    0.069800] smpboot: APIC(0) Converting physical 0 to logical package 0
    [    0.069803] smpboot: APIC(10) Converting physical 1 to logical package 1
    [    0.069806] smpboot: Max logical packages: 2
    [    0.069842] DMAR: Host address width 46
    [    0.069844] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
    [    0.069853] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
    [    0.069855] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
    [    0.069861] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
    [    0.069863] DMAR: RMRR base: 0x0000007ba6d000 end: 0x0000007ba7cfff
    [    0.069864] DMAR: ATSR flags: 0x0
    [    0.069866] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
    [    0.069868] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
    [    0.069870] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
    [    0.069872] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
    [    0.069873] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
    [    0.069875] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
    [    0.069876] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
    [    0.070600] DMAR-IR: Enabled IRQ remapping in x2apic mode
    [    0.071290] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
    [    0.081293] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz (fam: 06, model: 4f, stepping: 01)
    [    0.081300] TSC deadline timer enabled
    [    0.081311] Performance Events: PEBS fmt2+, 16-deep LBR, Broadwell events, full-width counters, Intel PMU driver.
    [    0.081337] ... version:                3
    [    0.081338] ... bit width:              48
    [    0.081339] ... generic registers:      8
    [    0.081340] ... value mask:             0000ffffffffffff
    [    0.081341] ... max period:             0000ffffffffffff
    [    0.081342] ... fixed-purpose events:   3
    [    0.081344] ... event mask:             00000007000000ff
    [    0.089423] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
    [    0.083271] smpboot: Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 OK
    [    0.124858] smpboot: Booting Node   1, Processors  #8 #9 #10 #11 #12 #13 #14 #15 OK
    [    0.270281] Brought up 16 CPUs
    [    0.270285] smpboot: Total of 16 processors activated (54444.68 BogoMIPS)
    [    0.392411] node 0 initialised, 3079811 pages in 72ms
    [    0.484559] node 1 initialised, 3604039 pages in 164ms
    [    0.485103] devtmpfs: initialized
    [    0.489782] EVM: security.selinux
    [    0.489784] EVM: security.ima
    [    0.489785] EVM: security.capability
    [    0.489882] PM: Registering ACPI NVS region [mem 0x79a8a000-0x79c8afff] (2101248 bytes)
    [    0.490998] atomic64 test passed for x86-64 platform with CX8 and with SSE
    [    0.491002] pinctrl core: initialized pinctrl subsystem
    [    0.491204] NET: Registered protocol family 16
    [    0.491648] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
    [    0.491650] ACPI: bus type PCI registered
    [    0.491652] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
    [    0.491723] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
    [    0.491725] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
    [    0.491740] PCI: Using configuration type 1 for base access
    [    0.498295] ACPI: Added _OSI(Module Device)
    [    0.498298] ACPI: Added _OSI(Processor Device)
    [    0.498299] ACPI: Added _OSI(3.0 _SCP Extensions)
    [    0.498301] ACPI: Added _OSI(Processor Aggregator Device)
    [    0.507266] ACPI: EC: Look up EC in DSDT
    [    0.531957] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored
    [    0.643472] ACPI: Dynamic OEM Table Load:
    [    0.643476] ACPI: PRAD           (null) 00102 (v02 SUPERM SMCI--MB 00000002 INTL 20120913)
    [    0.681134] ACPI: Interpreter enabled
    [    0.681151] ACPI: (supports S0 S4 S5)
    [    0.681153] ACPI: Using IOAPIC for interrupt routing
    [    0.681179] HEST: Enabling Firmware First mode for corrected errors.
    [    0.681433] HEST: Table parsing has been initialized.
    [    0.681436] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
    [    0.719518] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
    [    0.719524] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
    [    0.722142] acpi PNP0A03:02: _OSC: platform does not support [AER]
    [    0.723048] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
    [    0.723050] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration
    [    0.723086] PCI host bridge to bus 0000:ff
    [    0.723088] pci_bus 0000:ff: root bus resource [bus ff]
    [    0.723100] pci 0000:ff:08.0: [8086:6f80] type 00 class 0x088000
    [    0.723188] pci 0000:ff:08.2: [8086:6f32] type 00 class 0x110100
    [    0.723260] pci 0000:ff:08.3: [8086:6f83] type 00 class 0x088000
    [    0.723345] pci 0000:ff:09.0: [8086:6f90] type 00 class 0x088000
    [    0.723413] pci 0000:ff:09.2: [8086:6f33] type 00 class 0x110100
    [    0.723480] pci 0000:ff:09.3: [8086:6f93] type 00 class 0x088000
    [    0.723559] pci 0000:ff:0b.0: [8086:6f81] type 00 class 0x088000
    [    0.723626] pci 0000:ff:0b.1: [8086:6f36] type 00 class 0x110100
    [    0.723691] pci 0000:ff:0b.2: [8086:6f37] type 00 class 0x110100
    [    0.723755] pci 0000:ff:0b.3: [8086:6f76] type 00 class 0x088000
    [    0.723825] pci 0000:ff:0c.0: [8086:6fe0] type 00 class 0x088000
    [    0.723890] pci 0000:ff:0c.1: [8086:6fe1] type 00 class 0x088000
    [    0.723953] pci 0000:ff:0c.2: [8086:6fe2] type 00 class 0x088000
    [    0.724021] pci 0000:ff:0c.3: [8086:6fe3] type 00 class 0x088000
    [    0.724086] pci 0000:ff:0c.4: [8086:6fe4] type 00 class 0x088000
    [    0.724159] pci 0000:ff:0c.5: [8086:6fe5] type 00 class 0x088000
    [    0.724228] pci 0000:ff:0c.6: [8086:6fe6] type 00 class 0x088000
    [    0.724292] pci 0000:ff:0c.7: [8086:6fe7] type 00 class 0x088000
    [    0.724355] pci 0000:ff:0f.0: [8086:6ff8] type 00 class 0x088000
    [    0.724423] pci 0000:ff:0f.1: [8086:6ff9] type 00 class 0x088000
    [    0.724489] pci 0000:ff:0f.4: [8086:6ffc] type 00 class 0x088000
    [    0.724552] pci 0000:ff:0f.5: [8086:6ffd] type 00 class 0x088000
    [    0.724617] pci 0000:ff:0f.6: [8086:6ffe] type 00 class 0x088000
    [    0.724683] pci 0000:ff:10.0: [8086:6f1d] type 00 class 0x088000
    [    0.724748] pci 0000:ff:10.1: [8086:6f34] type 00 class 0x110100
    [    0.724813] pci 0000:ff:10.5: [8086:6f1e] type 00 class 0x088000
    [    0.724876] pci 0000:ff:10.6: [8086:6f7d] type 00 class 0x110100
    [    0.724938] pci 0000:ff:10.7: [8086:6f1f] type 00 class 0x088000
    [    0.725005] pci 0000:ff:12.0: [8086:6fa0] type 00 class 0x088000
    [    0.725056] pci 0000:ff:12.1: [8086:6f30] type 00 class 0x110100
    [    0.725127] pci 0000:ff:13.0: [8086:6fa8] type 00 class 0x088000
    [    0.725254] pci 0000:ff:13.1: [8086:6f71] type 00 class 0x088000
    [    0.725384] pci 0000:ff:13.2: [8086:6faa] type 00 class 0x088000
    [    0.725511] pci 0000:ff:13.3: [8086:6fab] type 00 class 0x088000
    [    0.725620] pci 0000:ff:13.4: [8086:6fac] type 00 class 0x088000
    [    0.725697] pci 0000:ff:13.5: [8086:6fad] type 00 class 0x088000
    [    0.725768] pci 0000:ff:13.6: [8086:6fae] type 00 class 0x088000
    [    0.725834] pci 0000:ff:13.7: [8086:6faf] type 00 class 0x088000
    [    0.725906] pci 0000:ff:14.0: [8086:6fb0] type 00 class 0x088000
    [    0.725987] pci 0000:ff:14.1: [8086:6fb1] type 00 class 0x088000
    [    0.726064] pci 0000:ff:14.2: [8086:6fb2] type 00 class 0x088000
    [    0.726141] pci 0000:ff:14.3: [8086:6fb3] type 00 class 0x088000
    [    0.726216] pci 0000:ff:14.4: [8086:6fbc] type 00 class 0x088000
    [    0.726283] pci 0000:ff:14.5: [8086:6fbd] type 00 class 0x088000
    [    0.726350] pci 0000:ff:14.6: [8086:6fbe] type 00 class 0x088000
    [    0.726416] pci 0000:ff:14.7: [8086:6fbf] type 00 class 0x088000
    [    0.726485] pci 0000:ff:15.0: [8086:6fb4] type 00 class 0x088000
    [    0.726570] pci 0000:ff:15.1: [8086:6fb5] type 00 class 0x088000
    [    0.726645] pci 0000:ff:15.2: [8086:6fb6] type 00 class 0x088000
    [    0.726721] pci 0000:ff:15.3: [8086:6fb7] type 00 class 0x088000
    [    0.726802] pci 0000:ff:16.0: [8086:6f68] type 00 class 0x088000
    [    0.726881] pci 0000:ff:16.6: [8086:6f6e] type 00 class 0x088000
    [    0.726949] pci 0000:ff:16.7: [8086:6f6f] type 00 class 0x088000
    [    0.727017] pci 0000:ff:17.0: [8086:6fd0] type 00 class 0x088000
    [    0.727089] pci 0000:ff:17.4: [8086:6fb8] type 00 class 0x088000
    [    0.727157] pci 0000:ff:17.5: [8086:6fb9] type 00 class 0x088000
    [    0.727230] pci 0000:ff:17.6: [8086:6fba] type 00 class 0x088000
    [    0.727297] pci 0000:ff:17.7: [8086:6fbb] type 00 class 0x088000
    [    0.727372] pci 0000:ff:1e.0: [8086:6f98] type 00 class 0x088000
    [    0.727438] pci 0000:ff:1e.1: [8086:6f99] type 00 class 0x088000
    [    0.727504] pci 0000:ff:1e.2: [8086:6f9a] type 00 class 0x088000
    [    0.727571] pci 0000:ff:1e.3: [8086:6fc0] type 00 class 0x088000
    [    0.727620] pci 0000:ff:1e.4: [8086:6f9c] type 00 class 0x088000
    [    0.727692] pci 0000:ff:1f.0: [8086:6f88] type 00 class 0x088000
    [    0.727760] pci 0000:ff:1f.2: [8086:6f8a] type 00 class 0x088000
    [    0.727889] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f])
    [    0.727893] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
    [    0.728448] acpi PNP0A03:03: _OSC: platform does not support [AER]
    [    0.729361] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
    [    0.729363] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration
    [    0.729396] PCI host bridge to bus 0000:7f
    [    0.729399] pci_bus 0000:7f: root bus resource [bus 7f]
    [    0.729407] pci 0000:7f:08.0: [8086:6f80] type 00 class 0x088000
    [    0.729482] pci 0000:7f:08.2: [8086:6f32] type 00 class 0x110100
    [    0.729549] pci 0000:7f:08.3: [8086:6f83] type 00 class 0x088000
    [    0.729623] pci 0000:7f:09.0: [8086:6f90] type 00 class 0x088000
    [    0.729687] pci 0000:7f:09.2: [8086:6f33] type 00 class 0x110100
    [    0.729751] pci 0000:7f:09.3: [8086:6f93] type 00 class 0x088000
    [    0.729827] pci 0000:7f:0b.0: [8086:6f81] type 00 class 0x088000
    [    0.729887] pci 0000:7f:0b.1: [8086:6f36] type 00 class 0x110100
    [    0.729948] pci 0000:7f:0b.2: [8086:6f37] type 00 class 0x110100
    [    0.730023] pci 0000:7f:0b.3: [8086:6f76] type 00 class 0x088000
    [    0.730090] pci 0000:7f:0c.0: [8086:6fe0] type 00 class 0x088000
    [    0.730147] pci 0000:7f:0c.1: [8086:6fe1] type 00 class 0x088000
    [    0.730208] pci 0000:7f:0c.2: [8086:6fe2] type 00 class 0x088000
    [    0.730269] pci 0000:7f:0c.3: [8086:6fe3] type 00 class 0x088000
    [    0.730340] pci 0000:7f:0c.4: [8086:6fe4] type 00 class 0x088000
    [    0.730401] pci 0000:7f:0c.5: [8086:6fe5] type 00 class 0x088000
    [    0.730462] pci 0000:7f:0c.6: [8086:6fe6] type 00 class 0x088000
    [    0.730519] pci 0000:7f:0c.7: [8086:6fe7] type 00 class 0x088000
    [    0.730584] pci 0000:7f:0f.0: [8086:6ff8] type 00 class 0x088000
    [    0.730644] pci 0000:7f:0f.1: [8086:6ff9] type 00 class 0x088000
    [    0.730707] pci 0000:7f:0f.4: [8086:6ffc] type 00 class 0x088000
    [    0.730765] pci 0000:7f:0f.5: [8086:6ffd] type 00 class 0x088000
    [    0.730829] pci 0000:7f:0f.6: [8086:6ffe] type 00 class 0x088000
    [    0.730891] pci 0000:7f:10.0: [8086:6f1d] type 00 class 0x088000
    [    0.730953] pci 0000:7f:10.1: [8086:6f34] type 00 class 0x110100
    [    0.731019] pci 0000:7f:10.5: [8086:6f1e] type 00 class 0x088000
    [    0.731079] pci 0000:7f:10.6: [8086:6f7d] type 00 class 0x110100
    [    0.731145] pci 0000:7f:10.7: [8086:6f1f] type 00 class 0x088000
    [    0.731207] pci 0000:7f:12.0: [8086:6fa0] type 00 class 0x088000
    [    0.731251] pci 0000:7f:12.1: [8086:6f30] type 00 class 0x110100
    [    0.731322] pci 0000:7f:13.0: [8086:6fa8] type 00 class 0x088000
    [    0.731442] pci 0000:7f:13.1: [8086:6f71] type 00 class 0x088000
    [    0.731570] pci 0000:7f:13.2: [8086:6faa] type 00 class 0x088000
    [    0.731691] pci 0000:7f:13.3: [8086:6fab] type 00 class 0x088000
    [    0.731810] pci 0000:7f:13.4: [8086:6fac] type 00 class 0x088000
    [    0.731927] pci 0000:7f:13.5: [8086:6fad] type 00 class 0x088000
    [    0.732020] pci 0000:7f:13.6: [8086:6fae] type 00 class 0x088000
    [    0.732084] pci 0000:7f:13.7: [8086:6faf] type 00 class 0x088000
    [    0.732151] pci 0000:7f:14.0: [8086:6fb0] type 00 class 0x088000
    [    0.732223] pci 0000:7f:14.1: [8086:6fb1] type 00 class 0x088000
    [    0.732295] pci 0000:7f:14.2: [8086:6fb2] type 00 class 0x088000
    [    0.732368] pci 0000:7f:14.3: [8086:6fb3] type 00 class 0x088000
    [    0.732440] pci 0000:7f:14.4: [8086:6fbc] type 00 class 0x088000
    [    0.732503] pci 0000:7f:14.5: [8086:6fbd] type 00 class 0x088000
    [    0.732566] pci 0000:7f:14.6: [8086:6fbe] type 00 class 0x088000
    [    0.732635] pci 0000:7f:14.7: [8086:6fbf] type 00 class 0x088000
    [    0.732703] pci 0000:7f:15.0: [8086:6fb4] type 00 class 0x088000
    [    0.732773] pci 0000:7f:15.1: [8086:6fb5] type 00 class 0x088000
    [    0.732844] pci 0000:7f:15.2: [8086:6fb6] type 00 class 0x088000
    [    0.732916] pci 0000:7f:15.3: [8086:6fb7] type 00 class 0x088000
    [    0.732997] pci 0000:7f:16.0: [8086:6f68] type 00 class 0x088000
    [    0.733077] pci 0000:7f:16.6: [8086:6f6e] type 00 class 0x088000
    [    0.733140] pci 0000:7f:16.7: [8086:6f6f] type 00 class 0x088000
    [    0.733201] pci 0000:7f:17.0: [8086:6fd0] type 00 class 0x088000
    [    0.733274] pci 0000:7f:17.4: [8086:6fb8] type 00 class 0x088000
    [    0.733341] pci 0000:7f:17.5: [8086:6fb9] type 00 class 0x088000
    [    0.733404] pci 0000:7f:17.6: [8086:6fba] type 00 class 0x088000
    [    0.733468] pci 0000:7f:17.7: [8086:6fbb] type 00 class 0x088000
    [    0.733538] pci 0000:7f:1e.0: [8086:6f98] type 00 class 0x088000
    [    0.733601] pci 0000:7f:1e.1: [8086:6f99] type 00 class 0x088000
    [    0.733663] pci 0000:7f:1e.2: [8086:6f9a] type 00 class 0x088000
    [    0.733730] pci 0000:7f:1e.3: [8086:6fc0] type 00 class 0x088000
    [    0.733779] pci 0000:7f:1e.4: [8086:6f9c] type 00 class 0x088000
    [    0.733845] pci 0000:7f:1f.0: [8086:6f88] type 00 class 0x088000
    [    0.733915] pci 0000:7f:1f.2: [8086:6f8a] type 00 class 0x088000
    [    0.742083] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
    [    0.742087] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
    [    0.742512] acpi PNP0A08:00: _OSC: platform does not support [AER]
    [    0.743142] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
    [    0.743144] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
    [    0.743469] PCI host bridge to bus 0000:00
    [    0.743472] pci_bus 0000:00: root bus resource [bus 00-7e]
    [    0.743474] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
    [    0.743476] pci_bus 0000:00: root bus resource [io  0x1000-0x7fff window]
    [    0.743478] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
    [    0.743480] pci_bus 0000:00: root bus resource [mem 0xfedb0000-0xfedb000f window]
    [    0.743482] pci_bus 0000:00: root bus resource [mem 0xfedc0000-0xfedc000f window]
    [    0.743484] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window]
    [    0.743492] pci 0000:00:00.0: [8086:6f00] type 00 class 0x060000
    [    0.743647] pci 0000:00:01.0: [8086:6f02] type 01 class 0x060400
    [    0.743694] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
    [    0.743786] pci 0000:00:01.0: System wakeup disabled by ACPI
    [    0.743816] pci 0000:00:02.0: [8086:6f04] type 01 class 0x060400
    [    0.743862] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
    [    0.743956] pci 0000:00:02.0: System wakeup disabled by ACPI
    [    0.743985] pci 0000:00:02.2: [8086:6f06] type 01 class 0x060400
    [    0.744034] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
    [    0.744128] pci 0000:00:02.2: System wakeup disabled by ACPI
    [    0.744160] pci 0000:00:03.0: [8086:6f08] type 01 class 0x060400
    [    0.744205] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
    [    0.744295] pci 0000:00:03.0: System wakeup disabled by ACPI
    [    0.744326] pci 0000:00:03.1: [8086:6f09] type 01 class 0x060400
    [    0.744371] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold
    [    0.744465] pci 0000:00:03.1: System wakeup disabled by ACPI
    [    0.744494] pci 0000:00:03.2: [8086:6f0a] type 01 class 0x060400
    [    0.744539] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold
    [    0.744628] pci 0000:00:03.2: System wakeup disabled by ACPI
    [    0.744660] pci 0000:00:03.3: [8086:6f0b] type 01 class 0x060400
    [    0.744705] pci 0000:00:03.3: PME# supported from D0 D3hot D3cold
    [    0.744803] pci 0000:00:03.3: System wakeup disabled by ACPI
    [    0.744834] pci 0000:00:04.0: [8086:6f20] type 00 class 0x088000
    [    0.744846] pci 0000:00:04.0: reg 0x10: [mem 0xc732c000-0xc732ffff 64bit]
    [    0.744985] pci 0000:00:04.1: [8086:6f21] type 00 class 0x088000
    [    0.744996] pci 0000:00:04.1: reg 0x10: [mem 0xc7328000-0xc732bfff 64bit]
    [    0.745141] pci 0000:00:04.2: [8086:6f22] type 00 class 0x088000
    [    0.745151] pci 0000:00:04.2: reg 0x10: [mem 0xc7324000-0xc7327fff 64bit]
    [    0.745293] pci 0000:00:04.3: [8086:6f23] type 00 class 0x088000
    [    0.745303] pci 0000:00:04.3: reg 0x10: [mem 0xc7320000-0xc7323fff 64bit]
    [    0.745448] pci 0000:00:04.4: [8086:6f24] type 00 class 0x088000
    [    0.745459] pci 0000:00:04.4: reg 0x10: [mem 0xc731c000-0xc731ffff 64bit]
    [    0.745603] pci 0000:00:04.5: [8086:6f25] type 00 class 0x088000
    [    0.745613] pci 0000:00:04.5: reg 0x10: [mem 0xc7318000-0xc731bfff 64bit]
    [    0.745753] pci 0000:00:04.6: [8086:6f26] type 00 class 0x088000
    [    0.745764] pci 0000:00:04.6: reg 0x10: [mem 0xc7314000-0xc7317fff 64bit]
    [    0.745904] pci 0000:00:04.7: [8086:6f27] type 00 class 0x088000
    [    0.745915] pci 0000:00:04.7: reg 0x10: [mem 0xc7310000-0xc7313fff 64bit]
    [    0.746062] pci 0000:00:05.0: [8086:6f28] type 00 class 0x088000
    [    0.746201] pci 0000:00:05.1: [8086:6f29] type 00 class 0x088000
    [    0.746344] pci 0000:00:05.2: [8086:6f2a] type 00 class 0x088000
    [    0.746479] pci 0000:00:05.4: [8086:6f2c] type 00 class 0x080020
    [    0.746487] pci 0000:00:05.4: reg 0x10: [mem 0xc7339000-0xc7339fff]
    [    0.746640] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000
    [    0.746836] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601
    [    0.746853] pci 0000:00:11.4: reg 0x10: [io  0x7110-0x7117]
    [    0.746862] pci 0000:00:11.4: reg 0x14: [io  0x7100-0x7103]
    [    0.746871] pci 0000:00:11.4: reg 0x18: [io  0x70f0-0x70f7]
    [    0.746880] pci 0000:00:11.4: reg 0x1c: [io  0x70e0-0x70e3]
    [    0.746889] pci 0000:00:11.4: reg 0x20: [io  0x7020-0x703f]
    [    0.746898] pci 0000:00:11.4: reg 0x24: [mem 0xc7338000-0xc73387ff]
    [    0.746938] pci 0000:00:11.4: PME# supported from D3hot
    [    0.747057] pci 0000:00:14.0: [8086:8d31] type 00 class 0x0c0330
    [    0.747075] pci 0000:00:14.0: reg 0x10: [mem 0xc7300000-0xc730ffff 64bit]
    [    0.747136] pci 0000:00:14.0: PME# supported from D3hot D3cold
    [    0.747233] pci 0000:00:14.0: System wakeup disabled by ACPI
    [    0.747266] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000
    [    0.747284] pci 0000:00:16.0: reg 0x10: [mem 0xc7337000-0xc733700f 64bit]
    [    0.747346] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
    [    0.747465] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000
    [    0.747483] pci 0000:00:16.1: reg 0x10: [mem 0xc7336000-0xc733600f 64bit]
    [    0.747545] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
    [    0.747673] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320
    [    0.747691] pci 0000:00:1a.0: reg 0x10: [mem 0xc7334000-0xc73343ff]
    [    0.747772] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
    [    0.747875] pci 0000:00:1a.0: System wakeup disabled by ACPI
    [    0.747908] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400
    [    0.747971] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
    [    0.748063] pci 0000:00:1c.0: System wakeup disabled by ACPI
    [    0.748096] pci 0000:00:1c.2: [8086:8d14] type 01 class 0x060400
    [    0.748160] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
    [    0.748248] pci 0000:00:1c.2: System wakeup disabled by ACPI
    [    0.748287] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320
    [    0.748305] pci 0000:00:1d.0: reg 0x10: [mem 0xc7333000-0xc73333ff]
    [    0.748387] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
    [    0.748488] pci 0000:00:1d.0: System wakeup disabled by ACPI
    [    0.748524] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100
    [    0.748732] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601
    [    0.748746] pci 0000:00:1f.2: reg 0x10: [io  0x7070-0x7077]
    [    0.748754] pci 0000:00:1f.2: reg 0x14: [io  0x7060-0x7063]
    [    0.748762] pci 0000:00:1f.2: reg 0x18: [io  0x7050-0x7057]
    [    0.748770] pci 0000:00:1f.2: reg 0x1c: [io  0x7040-0x7043]
    [    0.748777] pci 0000:00:1f.2: reg 0x20: [io  0x7000-0x701f]
    [    0.748785] pci 0000:00:1f.2: reg 0x24: [mem 0xc7332000-0xc73327ff]
    [    0.748820] pci 0000:00:1f.2: PME# supported from D3hot
    [    0.748932] pci 0000:00:1f.3: [8086:8d22] type 00 class 0x0c0500
    [    0.748946] pci 0000:00:1f.3: reg 0x10: [mem 0xc7331000-0xc73310ff 64bit]
    [    0.748962] pci 0000:00:1f.3: reg 0x20: [io  0x0580-0x059f]
    [    0.749241] acpiphp: Slot [0] registered
    [    0.749277] pci 0000:01:00.0: [8086:1528] type 00 class 0x020000
    [    0.749293] pci 0000:01:00.0: reg 0x10: [mem 0xc7c00000-0xc7dfffff 64bit pref]
    [    0.749300] pci 0000:01:00.0: reg 0x18: [io  0x6020-0x603f]
    [    0.749317] pci 0000:01:00.0: reg 0x20: [mem 0xc7e04000-0xc7e07fff 64bit pref]
    [    0.749370] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
    [    0.749395] pci 0000:01:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.749398] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
    [    0.749411] pci 0000:01:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.749413] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
    [    0.749601] pci 0000:01:00.1: [8086:1528] type 00 class 0x020000
    [    0.749617] pci 0000:01:00.1: reg 0x10: [mem 0xc7a00000-0xc7bfffff 64bit pref]
    [    0.749625] pci 0000:01:00.1: reg 0x18: [io  0x6000-0x601f]
    [    0.749642] pci 0000:01:00.1: reg 0x20: [mem 0xc7e00000-0xc7e03fff 64bit pref]
    [    0.749692] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold
    [    0.749714] pci 0000:01:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.749716] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
    [    0.749729] pci 0000:01:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.749731] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
    [    0.752029] pci 0000:00:01.0: PCI bridge to [bus 01]
    [    0.752034] pci 0000:00:01.0:   bridge window [io  0x6000-0x6fff]
    [    0.752041] pci 0000:00:01.0:   bridge window [mem 0xc7a00000-0xc7efffff 64bit pref]
    [    0.752291] pci 0000:02:00.0: [8086:1528] type 00 class 0x020000
    [    0.752310] pci 0000:02:00.0: reg 0x10: [mem 0xc7600000-0xc77fffff 64bit pref]
    [    0.752319] pci 0000:02:00.0: reg 0x18: [io  0x5020-0x503f]
    [    0.752340] pci 0000:02:00.0: reg 0x20: [mem 0xc7804000-0xc7807fff 64bit pref]
    [    0.752400] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
    [    0.752427] pci 0000:02:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.752430] pci 0000:02:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
    [    0.752445] pci 0000:02:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.752448] pci 0000:02:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
    [    0.752663] pci 0000:02:00.1: [8086:1528] type 00 class 0x020000
    [    0.752679] pci 0000:02:00.1: reg 0x10: [mem 0xc7400000-0xc75fffff 64bit pref]
    [    0.752686] pci 0000:02:00.1: reg 0x18: [io  0x5000-0x501f]
    [    0.752703] pci 0000:02:00.1: reg 0x20: [mem 0xc7800000-0xc7803fff 64bit pref]
    [    0.752752] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
    [    0.752772] pci 0000:02:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.752774] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
    [    0.752788] pci 0000:02:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
    [    0.752790] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
    [    0.815403] pci 0000:00:02.0: PCI bridge to [bus 02]
    [    0.815406] pci 0000:00:02.0:   bridge window [io  0x5000-0x5fff]
    [    0.815411] pci 0000:00:02.0:   bridge window [mem 0xc7400000-0xc78fffff 64bit pref]
    [    0.815565] pci 0000:00:02.2: PCI bridge to [bus 03]
    [    0.815743] pci 0000:04:00.0: [8086:0a53] type 00 class 0x010802
    [    0.815755] pci 0000:04:00.0: reg 0x10: [mem 0xc7210000-0xc7213fff 64bit]
    [    0.815778] pci 0000:04:00.0: reg 0x30: [mem 0xc7200000-0xc720ffff pref]
    [    0.817409] pci 0000:00:03.0: PCI bridge to [bus 04]
    [    0.817414] pci 0000:00:03.0:   bridge window [mem 0xc7200000-0xc72fffff]
    [    0.817640] pci 0000:05:00.0: [8086:0a53] type 00 class 0x010802
    [    0.817669] pci 0000:05:00.0: reg 0x10: [mem 0xc7110000-0xc7113fff 64bit]
    [    0.817693] pci 0000:05:00.0: reg 0x30: [mem 0xc7100000-0xc710ffff pref]
    [    0.819414] pci 0000:00:03.1: PCI bridge to [bus 05]
    [    0.819420] pci 0000:00:03.1:   bridge window [mem 0xc7100000-0xc71fffff]
    [    0.819639] pci 0000:00:03.2: PCI bridge to [bus 06]
    [    0.819848] pci 0000:00:03.3: PCI bridge to [bus 07]
    [    0.819892] pci 0000:00:1c.0: PCI bridge to [bus 08]
    [    0.819962] pci 0000:09:00.0: [1a03:1150] type 01 class 0x060400
    [    0.820083] pci 0000:09:00.0: supports D1 D2
    [    0.820085] pci 0000:09:00.0: PME# supported from D0 D1 D2 D3hot D3cold
    [    0.821439] pci 0000:00:1c.2: PCI bridge to [bus 09-0a]
    [    0.821443] pci 0000:00:1c.2:   bridge window [io  0x4000-0x4fff]
    [    0.821447] pci 0000:00:1c.2:   bridge window [mem 0xc6000000-0xc70fffff]
    [    0.821526] pci 0000:0a:00.0: [1a03:2000] type 00 class 0x030000
    [    0.821552] pci 0000:0a:00.0: reg 0x10: [mem 0xc6000000-0xc6ffffff]
    [    0.821567] pci 0000:0a:00.0: reg 0x14: [mem 0xc7000000-0xc701ffff]
    [    0.821581] pci 0000:0a:00.0: reg 0x18: [io  0x4000-0x407f]
    [    0.821684] pci 0000:0a:00.0: supports D1 D2
    [    0.821686] pci 0000:0a:00.0: PME# supported from D0 D1 D2 D3hot D3cold
    [    0.821802] pci 0000:09:00.0: PCI bridge to [bus 0a]
    [    0.821811] pci 0000:09:00.0:   bridge window [io  0x4000-0x4fff]
    [    0.821816] pci 0000:09:00.0:   bridge window [mem 0xc6000000-0xc70fffff]
    [    0.821873] pci_bus 0000:00: on NUMA node 0
    [    0.822718] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
    [    0.822722] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
    [    0.823143] acpi PNP0A08:01: _OSC: platform does not support [AER]
    [    0.823762] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
    [    0.823764] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
    [    0.823944] PCI host bridge to bus 0000:80
    [    0.823947] pci_bus 0000:80: root bus resource [bus 80-fe]
    [    0.823949] pci_bus 0000:80: root bus resource [io  0x8000-0xffff window]
    [    0.823952] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window]
    [    0.823962] pci 0000:80:00.0: [8086:6f01] type 01 class 0x060400
    [    0.824013] pci 0000:80:00.0: PME# supported from D0 D3hot D3cold
    [    0.824084] pci 0000:80:00.0: System wakeup disabled by ACPI
    [    0.824121] pci 0000:80:01.0: [8086:6f02] type 01 class 0x060400
    [    0.824172] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
    [    0.824234] pci 0000:80:01.0: System wakeup disabled by ACPI
    [    0.824265] pci 0000:80:02.0: [8086:6f04] type 01 class 0x060400
    [    0.824315] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
    [    0.824380] pci 0000:80:02.0: System wakeup disabled by ACPI
    [    0.824414] pci 0000:80:02.2: [8086:6f06] type 01 class 0x060400
    [    0.824468] pci 0000:80:02.2: PME# supported from D0 D3hot D3cold
    [    0.824530] pci 0000:80:02.2: System wakeup disabled by ACPI
    [    0.824563] pci 0000:80:03.0: [8086:6f08] type 01 class 0x060400
    [    0.824613] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
    [    0.824675] pci 0000:80:03.0: System wakeup disabled by ACPI
    [    0.824707] pci 0000:80:03.2: [8086:6f0a] type 01 class 0x060400
    [    0.824756] pci 0000:80:03.2: PME# supported from D0 D3hot D3cold
    [    0.824818] pci 0000:80:03.2: System wakeup disabled by ACPI
    [    0.824851] pci 0000:80:04.0: [8086:6f20] type 00 class 0x088000
    [    0.824863] pci 0000:80:04.0: reg 0x10: [mem 0xfbf1c000-0xfbf1ffff 64bit]
    [    0.824984] pci 0000:80:04.1: [8086:6f21] type 00 class 0x088000
    [    0.824996] pci 0000:80:04.1: reg 0x10: [mem 0xfbf18000-0xfbf1bfff 64bit]
    [    0.825117] pci 0000:80:04.2: [8086:6f22] type 00 class 0x088000
    [    0.825128] pci 0000:80:04.2: reg 0x10: [mem 0xfbf14000-0xfbf17fff 64bit]
    [    0.825245] pci 0000:80:04.3: [8086:6f23] type 00 class 0x088000
    [    0.825256] pci 0000:80:04.3: reg 0x10: [mem 0xfbf10000-0xfbf13fff 64bit]
    [    0.825372] pci 0000:80:04.4: [8086:6f24] type 00 class 0x088000
    [    0.825384] pci 0000:80:04.4: reg 0x10: [mem 0xfbf0c000-0xfbf0ffff 64bit]
    [    0.825503] pci 0000:80:04.5: [8086:6f25] type 00 class 0x088000
    [    0.825515] pci 0000:80:04.5: reg 0x10: [mem 0xfbf08000-0xfbf0bfff 64bit]
    [    0.825639] pci 0000:80:04.6: [8086:6f26] type 00 class 0x088000
    [    0.825650] pci 0000:80:04.6: reg 0x10: [mem 0xfbf04000-0xfbf07fff 64bit]
    [    0.825768] pci 0000:80:04.7: [8086:6f27] type 00 class 0x088000
    [    0.825779] pci 0000:80:04.7: reg 0x10: [mem 0xfbf00000-0xfbf03fff 64bit]
    [    0.825895] pci 0000:80:05.0: [8086:6f28] type 00 class 0x088000
    [    0.826003] pci 0000:80:05.1: [8086:6f29] type 00 class 0x088000
    [    0.826127] pci 0000:80:05.2: [8086:6f2a] type 00 class 0x088000
    [    0.826236] pci 0000:80:05.4: [8086:6f2c] type 00 class 0x080020
    [    0.826245] pci 0000:80:05.4: reg 0x10: [mem 0xfbf20000-0xfbf20fff]
    [    0.826535] pci 0000:80:00.0: PCI bridge to [bus 81]
    [    0.826699] pci 0000:80:01.0: PCI bridge to [bus 82]
    [    0.826858] pci 0000:80:02.0: PCI bridge to [bus 83]
    [    0.827019] pci 0000:80:02.2: PCI bridge to [bus 84]
    [    0.827181] pci 0000:80:03.0: PCI bridge to [bus 85]
    [    0.827342] pci 0000:80:03.2: PCI bridge to [bus 86]
    [    0.827375] pci_bus 0000:80: on NUMA node 1
    [    0.827683] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.827739] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.827793] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
    [    0.827849] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled.
    [    0.827902] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.827955] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.828009] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.828062] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
    [    0.828171] ACPI: Enabled 2 GPEs in block 00 to 3F
    [    0.828355] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none
    [    0.828360] vgaarb: loaded
    [    0.828361] vgaarb: bridge control possible 0000:0a:00.0
    [    0.828456] SCSI subsystem initialized
    [    0.828487] ACPI: bus type USB registered
    [    0.828508] usbcore: registered new interface driver usbfs
    [    0.828516] usbcore: registered new interface driver hub
    [    0.828569] usbcore: registered new device driver usb
    [    0.828735] PCI: Using ACPI for IRQ routing
    [    0.833532] PCI: pci_cache_line_size set to 64 bytes
    [    0.833769] e820: reserve RAM buffer [mem 0x0003f000-0x0003ffff]
    [    0.833771] e820: reserve RAM buffer [mem 0x7544e018-0x77ffffff]
    [    0.833773] e820: reserve RAM buffer [mem 0x75457018-0x77ffffff]
    [    0.833775] e820: reserve RAM buffer [mem 0x7545f018-0x77ffffff]
    [    0.833777] e820: reserve RAM buffer [mem 0x75467018-0x77ffffff]
    [    0.833779] e820: reserve RAM buffer [mem 0x75479018-0x77ffffff]
    [    0.833780] e820: reserve RAM buffer [mem 0x7548b018-0x77ffffff]
    [    0.833782] e820: reserve RAM buffer [mem 0x7549d018-0x77ffffff]
    [    0.833783] e820: reserve RAM buffer [mem 0x78e4e000-0x7bffffff]
    [    0.833785] e820: reserve RAM buffer [mem 0x79a8a000-0x7bffffff]
    [    0.833787] e820: reserve RAM buffer [mem 0x7bcfb000-0x7bffffff]
    [    0.833908] NetLabel: Initializing
    [    0.833910] NetLabel:  domain hash size = 128
    [    0.833911] NetLabel:  protocols = UNLABELED CIPSOv4
    [    0.833925] NetLabel:  unlabeled traffic allowed by default
    [    0.833968] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
    [    0.833975] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
    [    0.836061] Switched to clocksource hpet
    [    0.842376] pnp: PnP ACPI init
    [    0.842388] ACPI: bus type PNP registered
    [    0.842734] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
    [    0.842853] system 00:01: [io  0x0500-0x057f] has been reserved
    [    0.842856] system 00:01: [io  0x0400-0x047f] could not be reserved
    [    0.842858] system 00:01: [io  0x0580-0x059f] has been reserved
    [    0.842860] system 00:01: [io  0x0600-0x061f] has been reserved
    [    0.842862] system 00:01: [io  0x0880-0x0883] has been reserved
    [    0.842864] system 00:01: [io  0x0800-0x081f] has been reserved
    [    0.842867] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved
    [    0.842870] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved
    [    0.842872] system 00:01: [mem 0xff000000-0xffffffff] has been reserved
    [    0.842874] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved
    [    0.842876] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved
    [    0.842878] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved
    [    0.842880] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved
    [    0.842883] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
    [    0.843134] system 00:02: [io  0x0a00-0x0a0f] has been reserved
    [    0.843137] system 00:02: [io  0x0a10-0x0a1f] has been reserved
    [    0.843138] system 00:02: [io  0x0a20-0x0a2f] has been reserved
    [    0.843140] system 00:02: [io  0x0a30-0x0a3f] has been reserved
    [    0.843142] system 00:02: [io  0x0a40-0x0a4f] has been reserved
    [    0.843145] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
    [    0.843360] pnp 00:03: [dma 0 disabled]
    [    0.843412] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
    [    0.843605] pnp 00:04: [dma 0 disabled]
    [    0.843658] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
    [    0.844350] pnp: PnP ACPI: found 5 devices
    [    0.844352] ACPI: bus type PNP unregistered
    [    0.851373] pci 0000:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 400000 add_align 100000
    [    0.851380] pci 0000:00:02.0: bridge window [mem 0x00100000-0x000fffff] to [bus 02] add_size 400000 add_align 100000
    [    0.851390] pci 0000:00:03.0: bridge window [io  0x1000-0x0fff] to [bus 04] add_size 1000
    [    0.851393] pci 0000:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000
    [    0.851398] pci 0000:00:03.1: bridge window [io  0x1000-0x0fff] to [bus 05] add_size 1000
    [    0.851401] pci 0000:00:03.1: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
    [    0.851406] pci 0000:00:03.2: bridge window [io  0x1000-0x0fff] to [bus 06] add_size 1000
    [    0.851409] pci 0000:00:03.2: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 06] add_size 200000 add_align 100000
    [    0.851411] pci 0000:00:03.2: bridge window [mem 0x00100000-0x000fffff] to [bus 06] add_size 200000 add_align 100000
    [    0.851416] pci 0000:00:03.3: bridge window [io  0x1000-0x0fff] to [bus 07] add_size 1000
    [    0.851418] pci 0000:00:03.3: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 07] add_size 200000 add_align 100000
    [    0.851420] pci 0000:00:03.3: bridge window [mem 0x00100000-0x000fffff] to [bus 07] add_size 200000 add_align 100000
    [    0.851456] pci 0000:00:01.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 400000 min_align 100000
    [    0.851459] pci 0000:00:01.0: res[14]=[mem 0x00100000-0x004fffff] res_to_dev_res add_size 400000 min_align 100000
    [    0.851460] pci 0000:00:02.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 400000 min_align 100000
    [    0.851462] pci 0000:00:02.0: res[14]=[mem 0x00100000-0x004fffff] res_to_dev_res add_size 400000 min_align 100000
    [    0.851465] pci 0000:00:03.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851467] pci 0000:00:03.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851469] pci 0000:00:03.1: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851471] pci 0000:00:03.1: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851473] pci 0000:00:03.2: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000
    [    0.851475] pci 0000:00:03.2: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000
    [    0.851477] pci 0000:00:03.2: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851479] pci 0000:00:03.2: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851481] pci 0000:00:03.3: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000
    [    0.851483] pci 0000:00:03.3: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000
    [    0.851485] pci 0000:00:03.3: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851487] pci 0000:00:03.3: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
    [    0.851490] pci 0000:00:03.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851492] pci 0000:00:03.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851494] pci 0000:00:03.1: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851496] pci 0000:00:03.1: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851497] pci 0000:00:03.2: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851499] pci 0000:00:03.2: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851501] pci 0000:00:03.3: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851503] pci 0000:00:03.3: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
    [    0.851508] pci 0000:00:01.0: BAR 14: assigned [mem 0x90000000-0x903fffff]
    [    0.851512] pci 0000:00:02.0: BAR 14: assigned [mem 0x90400000-0x907fffff]
    [    0.851518] pci 0000:00:03.0: BAR 15: assigned [mem 0x90800000-0x909fffff 64bit pref]
    [    0.851523] pci 0000:00:03.1: BAR 15: assigned [mem 0x90a00000-0x90bfffff 64bit pref]
    [    0.851526] pci 0000:00:03.2: BAR 14: assigned [mem 0x90c00000-0x90dfffff]
    [    0.851531] pci 0000:00:03.2: BAR 15: assigned [mem 0x90e00000-0x90ffffff 64bit pref]
    [    0.851534] pci 0000:00:03.3: BAR 14: assigned [mem 0x91000000-0x911fffff]
    [    0.851538] pci 0000:00:03.3: BAR 15: assigned [mem 0x91200000-0x913fffff 64bit pref]
    [    0.851542] pci 0000:00:03.0: BAR 13: assigned [io  0x1000-0x1fff]
    [    0.851544] pci 0000:00:03.1: BAR 13: assigned [io  0x2000-0x2fff]
    [    0.851546] pci 0000:00:03.2: BAR 13: assigned [io  0x3000-0x3fff]
    [    0.851549] pci 0000:00:03.3: BAR 13: no space for [io  size 0x1000]
    [    0.851551] pci 0000:00:03.3: BAR 13: failed to assign [io  size 0x1000]
    [    0.851557] pci 0000:00:03.3: BAR 13: assigned [io  0x1000-0x1fff]
    [    0.851560] pci 0000:00:03.2: BAR 13: assigned [io  0x2000-0x2fff]
    [    0.851562] pci 0000:00:03.1: BAR 13: assigned [io  0x3000-0x3fff]
    [    0.851565] pci 0000:00:03.0: BAR 13: no space for [io  size 0x1000]
    [    0.851566] pci 0000:00:03.0: BAR 13: failed to assign [io  size 0x1000]
    [    0.851571] pci 0000:01:00.0: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851573] pci 0000:01:00.0: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851575] pci 0000:01:00.1: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851577] pci 0000:01:00.1: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851582] pci 0000:01:00.0: BAR 7: assigned [mem 0x90000000-0x900fffff 64bit pref]
    [    0.851591] pci 0000:01:00.0: BAR 10: assigned [mem 0x90100000-0x901fffff 64bit pref]
    [    0.851599] pci 0000:01:00.1: BAR 7: assigned [mem 0x90200000-0x902fffff 64bit pref]
    [    0.851607] pci 0000:01:00.1: BAR 10: assigned [mem 0x90300000-0x903fffff 64bit pref]
    [    0.851615] pci 0000:00:01.0: PCI bridge to [bus 01]
    [    0.851617] pci 0000:00:01.0:   bridge window [io  0x6000-0x6fff]
    [    0.851621] pci 0000:00:01.0:   bridge window [mem 0x90000000-0x903fffff]
    [    0.851624] pci 0000:00:01.0:   bridge window [mem 0xc7a00000-0xc7efffff 64bit pref]
    [    0.851629] pci 0000:02:00.0: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851631] pci 0000:02:00.0: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851633] pci 0000:02:00.1: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851636] pci 0000:02:00.1: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 100000 min_align 0
    [    0.851639] pci 0000:02:00.0: BAR 7: assigned [mem 0x90400000-0x904fffff 64bit pref]
    [    0.851647] pci 0000:02:00.0: BAR 10: assigned [mem 0x90500000-0x905fffff 64bit pref]
    [    0.851655] pci 0000:02:00.1: BAR 7: assigned [mem 0x90600000-0x906fffff 64bit pref]
    [    0.851663] pci 0000:02:00.1: BAR 10: assigned [mem 0x90700000-0x907fffff 64bit pref]
    [    0.851670] pci 0000:00:02.0: PCI bridge to [bus 02]
    [    0.851672] pci 0000:00:02.0:   bridge window [io  0x5000-0x5fff]
    [    0.851675] pci 0000:00:02.0:   bridge window [mem 0x90400000-0x907fffff]
    [    0.851678] pci 0000:00:02.0:   bridge window [mem 0xc7400000-0xc78fffff 64bit pref]
    [    0.851682] pci 0000:00:02.2: PCI bridge to [bus 03]
    [    0.851689] pci 0000:00:03.0: PCI bridge to [bus 04]
    [    0.851693] pci 0000:00:03.0:   bridge window [mem 0xc7200000-0xc72fffff]
    [    0.851696] pci 0000:00:03.0:   bridge window [mem 0x90800000-0x909fffff 64bit pref]
    [    0.851699] pci 0000:00:03.1: PCI bridge to [bus 05]
    [    0.851701] pci 0000:00:03.1:   bridge window [io  0x3000-0x3fff]
    [    0.851705] pci 0000:00:03.1:   bridge window [mem 0xc7100000-0xc71fffff]
    [    0.851708] pci 0000:00:03.1:   bridge window [mem 0x90a00000-0x90bfffff 64bit pref]
    [    0.851711] pci 0000:00:03.2: PCI bridge to [bus 06]
    [    0.851714] pci 0000:00:03.2:   bridge window [io  0x2000-0x2fff]
    [    0.851717] pci 0000:00:03.2:   bridge window [mem 0x90c00000-0x90dfffff]
    [    0.851720] pci 0000:00:03.2:   bridge window [mem 0x90e00000-0x90ffffff 64bit pref]
    [    0.851723] pci 0000:00:03.3: PCI bridge to [bus 07]
    [    0.851726] pci 0000:00:03.3:   bridge window [io  0x1000-0x1fff]
    [    0.851729] pci 0000:00:03.3:   bridge window [mem 0x91000000-0x911fffff]
    [    0.851732] pci 0000:00:03.3:   bridge window [mem 0x91200000-0x913fffff 64bit pref]
    [    0.851736] pci 0000:00:1c.0: PCI bridge to [bus 08]
    [    0.851746] pci 0000:09:00.0: PCI bridge to [bus 0a]
    [    0.851750] pci 0000:09:00.0:   bridge window [io  0x4000-0x4fff]
    [    0.851756] pci 0000:09:00.0:   bridge window [mem 0xc6000000-0xc70fffff]
    [    0.851768] pci 0000:00:1c.2: PCI bridge to [bus 09-0a]
    [    0.851771] pci 0000:00:1c.2:   bridge window [io  0x4000-0x4fff]
    [    0.851775] pci 0000:00:1c.2:   bridge window [mem 0xc6000000-0xc70fffff]
    [    0.851783] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
    [    0.851785] pci_bus 0000:00: resource 5 [io  0x1000-0x7fff window]
    [    0.851787] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
    [    0.851789] pci_bus 0000:00: resource 7 [mem 0xfedb0000-0xfedb000f window]
    [    0.851791] pci_bus 0000:00: resource 8 [mem 0xfedc0000-0xfedc000f window]
    [    0.851793] pci_bus 0000:00: resource 9 [mem 0x90000000-0xc7ffbfff window]
    [    0.851795] pci_bus 0000:01: resource 0 [io  0x6000-0x6fff]
    [    0.851797] pci_bus 0000:01: resource 1 [mem 0x90000000-0x903fffff]
    [    0.851798] pci_bus 0000:01: resource 2 [mem 0xc7a00000-0xc7efffff 64bit pref]
    [    0.851800] pci_bus 0000:02: resource 0 [io  0x5000-0x5fff]
    [    0.851802] pci_bus 0000:02: resource 1 [mem 0x90400000-0x907fffff]
    [    0.851804] pci_bus 0000:02: resource 2 [mem 0xc7400000-0xc78fffff 64bit pref]
    [    0.851806] pci_bus 0000:04: resource 1 [mem 0xc7200000-0xc72fffff]
    [    0.851808] pci_bus 0000:04: resource 2 [mem 0x90800000-0x909fffff 64bit pref]
    [    0.851809] pci_bus 0000:05: resource 0 [io  0x3000-0x3fff]
    [    0.851811] pci_bus 0000:05: resource 1 [mem 0xc7100000-0xc71fffff]
    [    0.851813] pci_bus 0000:05: resource 2 [mem 0x90a00000-0x90bfffff 64bit pref]
    [    0.851815] pci_bus 0000:06: resource 0 [io  0x2000-0x2fff]
    [    0.851816] pci_bus 0000:06: resource 1 [mem 0x90c00000-0x90dfffff]
    [    0.851818] pci_bus 0000:06: resource 2 [mem 0x90e00000-0x90ffffff 64bit pref]
    [    0.851820] pci_bus 0000:07: resource 0 [io  0x1000-0x1fff]
    [    0.851822] pci_bus 0000:07: resource 1 [mem 0x91000000-0x911fffff]
    [    0.851823] pci_bus 0000:07: resource 2 [mem 0x91200000-0x913fffff 64bit pref]
    [    0.851825] pci_bus 0000:09: resource 0 [io  0x4000-0x4fff]
    [    0.851827] pci_bus 0000:09: resource 1 [mem 0xc6000000-0xc70fffff]
    [    0.851829] pci_bus 0000:0a: resource 0 [io  0x4000-0x4fff]
    [    0.851831] pci_bus 0000:0a: resource 1 [mem 0xc6000000-0xc70fffff]
    [    0.851859] pci 0000:80:00.0: PCI bridge to [bus 81]
    [    0.851867] pci 0000:80:01.0: PCI bridge to [bus 82]
    [    0.851874] pci 0000:80:02.0: PCI bridge to [bus 83]
    [    0.851881] pci 0000:80:02.2: PCI bridge to [bus 84]
    [    0.851888] pci 0000:80:03.0: PCI bridge to [bus 85]
    [    0.851896] pci 0000:80:03.2: PCI bridge to [bus 86]
    [    0.851904] pci_bus 0000:80: resource 4 [io  0x8000-0xffff window]
    [    0.851905] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window]
    [    0.851949] NET: Registered protocol family 2
    [    0.852449] TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
    [    0.852922] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
    [    0.853113] TCP: Hash tables configured (established 262144 bind 65536)
    [    0.853160] TCP: reno registered
    [    0.853209] UDP hash table entries: 16384 (order: 7, 524288 bytes)
    [    0.853345] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
    [    0.853531] NET: Registered protocol family 1
    [    0.885178] PCI: CLS mismatch (64 != 32), using 64 bytes
    [    0.885191] pci 0000:0a:00.0: Boot video device
    [    0.885256] Unpacking initramfs...
    [    1.329683] Freeing initrd memory: 21648k freed
    [    1.335897] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
    [    1.335901] software IO TLB [mem 0x7144e000-0x7544e000] (64MB) mapped at [ffff88007144e000-ffff88007544dfff]
    [    1.335968] Intel CQM monitoring enabled
    [    1.335969] Intel MBM enabled
    [    1.335992] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer
    [    1.335993] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
    [    1.335994] RAPL PMU: hw unit of domain package 2^-14 Joules
    [    1.335995] RAPL PMU: hw unit of domain dram 2^-16 Joules
    [    1.338425] sha1_ssse3: Using AVX2 optimized SHA-1 implementation
    [    1.338542] sha256_ssse3: Using AVX2 optimized SHA-256 implementation
    [    1.339000] futex hash table entries: 4096 (order: 6, 262144 bytes)
    [    1.339053] Initialise system trusted keyring
    [    1.339112] audit: initializing netlink socket (disabled)
    [    1.339124] type=2000 audit(1488910608.073:1): initialized
    [    1.370739] HugeTLB registered 1 GB page size, pre-allocated 0 pages
    [    1.370742] HugeTLB registered 2 MB page size, pre-allocated 0 pages
    [    1.371942] zpool: loaded
    [    1.371946] zbud: loaded
    [    1.372237] VFS: Disk quotas dquot_6.5.2
    [    1.372273] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    [    1.372482] msgmni has been set to 32768
    [    1.372548] Key type big_key registered
    [    1.372551] SELinux:  Registering netfilter hooks
    [    1.373699] NET: Registered protocol family 38
    [    1.373707] Key type asymmetric registered
    [    1.373710] Asymmetric key parser 'x509' registered
    [    1.373747] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
    [    1.373810] io scheduler noop registered
    [    1.373814] io scheduler deadline registered (default)
    [    1.373839] io scheduler cfq registered
    [    1.374364] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X
    [    1.374508] pcieport 0000:00:02.0: irq 27 for MSI/MSI-X
    [    1.374638] pcieport 0000:00:02.2: irq 28 for MSI/MSI-X
    [    1.374782] pcieport 0000:00:03.0: irq 30 for MSI/MSI-X
    [    1.374924] pcieport 0000:00:03.1: irq 31 for MSI/MSI-X
    [    1.375065] pcieport 0000:00:03.2: irq 32 for MSI/MSI-X
    [    1.375203] pcieport 0000:00:03.3: irq 33 for MSI/MSI-X
    [    1.375349] pcieport 0000:00:1c.0: irq 34 for MSI/MSI-X
    [    1.375500] pcieport 0000:00:1c.2: irq 35 for MSI/MSI-X
    [    1.375852] pcieport 0000:80:00.0: irq 37 for MSI/MSI-X
    [    1.376124] pcieport 0000:80:01.0: irq 39 for MSI/MSI-X
    [    1.376334] pcieport 0000:80:02.0: irq 41 for MSI/MSI-X
    [    1.376527] pcieport 0000:80:02.2: irq 42 for MSI/MSI-X
    [    1.376790] pcieport 0000:80:03.0: irq 44 for MSI/MSI-X
    [    1.377108] pcieport 0000:80:03.2: irq 45 for MSI/MSI-X
    [    1.377267] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
    [    1.377270] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
    [    1.377273] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt
    [    1.377277] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
    [    1.377297] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt
    [    1.377299] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt
    [    1.377302] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt
    [    1.377305] pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded
    [    1.377323] pcieport 0000:00:02.2: Signaling PME through PCIe PME interrupt
    [    1.377327] pcie_pme 0000:00:02.2:pcie01: service driver pcie_pme loaded
    [    1.377347] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
    [    1.377349] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt
    [    1.377353] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded
    [    1.377371] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt
    [    1.377374] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt
    [    1.377377] pcie_pme 0000:00:03.1:pcie01: service driver pcie_pme loaded
    [    1.377404] pcieport 0000:00:03.2: Signaling PME through PCIe PME interrupt
    [    1.377408] pcie_pme 0000:00:03.2:pcie01: service driver pcie_pme loaded
    [    1.377435] pcieport 0000:00:03.3: Signaling PME through PCIe PME interrupt
    [    1.377439] pcie_pme 0000:00:03.3:pcie01: service driver pcie_pme loaded
    [    1.377457] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt
    [    1.377462] pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded
    [    1.377485] pcieport 0000:00:1c.2: Signaling PME through PCIe PME interrupt
    [    1.377487] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt
    [    1.377490] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt
    [    1.377494] pcie_pme 0000:00:1c.2:pcie01: service driver pcie_pme loaded
    [    1.377520] pcieport 0000:80:00.0: Signaling PME through PCIe PME interrupt
    [    1.377524] pcie_pme 0000:80:00.0:pcie01: service driver pcie_pme loaded
    [    1.377558] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt
    [    1.377561] pcie_pme 0000:80:01.0:pcie01: service driver pcie_pme loaded
    [    1.377584] pcieport 0000:80:02.0: Signaling PME through PCIe PME interrupt
    [    1.377587] pcie_pme 0000:80:02.0:pcie01: service driver pcie_pme loaded
    [    1.377604] pcieport 0000:80:02.2: Signaling PME through PCIe PME interrupt
    [    1.377607] pcie_pme 0000:80:02.2:pcie01: service driver pcie_pme loaded
    [    1.377624] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt
    [    1.377627] pcie_pme 0000:80:03.0:pcie01: service driver pcie_pme loaded
    [    1.377643] pcieport 0000:80:03.2: Signaling PME through PCIe PME interrupt
    [    1.377646] pcie_pme 0000:80:03.2:pcie01: service driver pcie_pme loaded
    [    1.377674] ioapic: probe of 0000:00:05.4 failed with error -22
    [    1.377707] ioapic: probe of 0000:80:05.4 failed with error -22
    [    1.377716] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
    [    1.377729] pciehp 0000:00:03.0:pcie04: Slot #7 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
    [    1.377772] pciehp 0000:00:03.0:pcie04: service driver pciehp loaded
    [    1.377779] pciehp 0000:00:03.1:pcie04: Slot #8 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
    [    1.377813] pciehp 0000:00:03.1:pcie04: service driver pciehp loaded
    [    1.377823] pciehp 0000:00:03.2:pcie04: Slot #9 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
    [    1.377857] pciehp 0000:00:03.2:pcie04: service driver pciehp loaded
    [    1.377864] pciehp 0000:00:03.3:pcie04: Slot #10 AttnBtn+ PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
    [    1.377899] pciehp 0000:00:03.3:pcie04: service driver pciehp loaded
    [    1.377904] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
    [    1.378034] efifb: probing for efifb
    [    1.378051] efifb: framebuffer at 0xc6000000, mapped to 0xffffc90003200000, using 1876k, total 1875k
    [    1.378053] efifb: mode is 800x600x32, linelength=3200, pages=1
    [    1.378054] efifb: scrolling: redraw
    [    1.378065] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
    [    1.388447] Console: switching to colour frame buffer device 100x37
    [    1.398526] fb0: EFI VGA frame buffer device
    [    1.398536] intel_idle: MWAIT substates: 0x2120
    [    1.398537] intel_idle: v0.4.1 model 0x4F
    [    1.398538] intel_idle: lapic_timer_reliable_states 0xffffffff
    [    1.398835] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
    [    1.398839] ACPI: Power Button [PWRB]
    [    1.398872] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
    [    1.398874] ACPI: Power Button [PWRF]
    [    1.398922] ACPI: Requesting acpi_cpufreq
    [    1.404073] ERST: Error Record Serialization Table (ERST) support is initialized.
    [    1.404080] pstore: Registered erst as persistent store backend
    [    1.404419] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
    [    1.404587] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
    [    1.425343] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
    [    1.446116] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
    [    1.447090] Non-volatile memory driver v1.3
    [    1.447098] Linux agpgart interface v0.103
    [    1.447634] crash memory driver: version 1.1
    [    1.447919] rdac: device handler registered
    [    1.447970] hp_sw: device handler registered
    [    1.447979] emc: device handler registered
    [    1.447986] alua: device handler registered
    [    1.448052] libphy: Fixed MDIO Bus: probed
    [    1.448137] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
    [    1.448148] ehci-pci: EHCI PCI platform driver
    [    1.448367] ehci-pci 0000:00:1a.0: EHCI Host Controller
    [    1.448434] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1
    [    1.448446] ehci-pci 0000:00:1a.0: debug port 2
    [    1.452359] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported
    [    1.452387] ehci-pci 0000:00:1a.0: irq 18, io mem 0xc7334000
    [    1.458070] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
    [    1.458122] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
    [    1.458124] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    1.458126] usb usb1: Product: EHCI Host Controller
    [    1.458128] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7.x86_64 ehci_hcd
    [    1.458129] usb usb1: SerialNumber: 0000:00:1a.0
    [    1.458247] hub 1-0:1.0: USB hub found
    [    1.458253] hub 1-0:1.0: 2 ports detected
    [    1.458543] ehci-pci 0000:00:1d.0: EHCI Host Controller
    [    1.458588] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2
    [    1.458598] ehci-pci 0000:00:1d.0: debug port 2
    [    1.462490] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported
    [    1.462495] ehci-pci 0000:00:1d.0: irq 18, io mem 0xc7333000
    [    1.468071] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00
    [    1.468112] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
    [    1.468114] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    1.468116] usb usb2: Product: EHCI Host Controller
    [    1.468117] usb usb2: Manufacturer: Linux 3.10.0-514.10.2.el7.x86_64 ehci_hcd
    [    1.468119] usb usb2: SerialNumber: 0000:00:1d.0
    [    1.468220] hub 2-0:1.0: USB hub found
    [    1.468225] hub 2-0:1.0: 2 ports detected
    [    1.468372] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
    [    1.468383] ohci-pci: OHCI PCI platform driver
    [    1.468433] uhci_hcd: USB Universal Host Controller Interface driver
    [    1.468676] xhci_hcd 0000:00:14.0: xHCI Host Controller
    [    1.468726] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3
    [    1.470832] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00009810
    [    1.470837] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
    [    1.470862] xhci_hcd 0000:00:14.0: irq 46 for MSI/MSI-X
    [    1.470931] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
    [    1.470933] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    1.470934] usb usb3: Product: xHCI Host Controller
    [    1.470936] usb usb3: Manufacturer: Linux 3.10.0-514.10.2.el7.x86_64 xhci-hcd
    [    1.470938] usb usb3: SerialNumber: 0000:00:14.0
    [    1.471036] hub 3-0:1.0: USB hub found
    [    1.471054] hub 3-0:1.0: 15 ports detected
    [    1.472044] xhci_hcd 0000:00:14.0: xHCI Host Controller
    [    1.472125] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4
    [    1.472165] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
    [    1.472167] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    1.472168] usb usb4: Product: xHCI Host Controller
    [    1.472170] usb usb4: Manufacturer: Linux 3.10.0-514.10.2.el7.x86_64 xhci-hcd
    [    1.472172] usb usb4: SerialNumber: 0000:00:14.0
    [    1.472277] hub 4-0:1.0: USB hub found
    [    1.472289] hub 4-0:1.0: 6 ports detected
    [    1.472895] usbcore: registered new interface driver usbserial
    [    1.472907] usbcore: registered new interface driver usbserial_generic
    [    1.472918] usbserial: USB Serial support registered for generic
    [    1.472965] i8042: PNP: No PS/2 controller found. Probing ports directly.
    [    1.760071] usb 1-1: new high-speed USB device number 2 using ehci-pci
    [    1.770066] usb 2-1: new high-speed USB device number 2 using ehci-pci
    [    1.875426] usb 1-1: New USB device found, idVendor=8087, idProduct=800a
    [    1.875429] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [    1.875571] hub 1-1:1.0: USB hub found
    [    1.875675] hub 1-1:1.0: 6 ports detected
    [    1.885428] usb 2-1: New USB device found, idVendor=8087, idProduct=8002
    [    1.885430] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [    1.885571] hub 2-1:1.0: USB hub found
    [    1.885679] hub 2-1:1.0: 8 ports detected
    [    1.927069] usb 3-14: new high-speed USB device number 3 using xhci_hcd
    [    1.942083] usb 4-6: new SuperSpeed USB device number 2 using xhci_hcd
    [    1.953840] usb 4-6: New USB device found, idVendor=26bd, idProduct=9917
    [    1.953842] usb 4-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [    1.953843] usb 4-6: Product: USB DISK 3.0
    [    1.953845] usb 4-6: Manufacturer:         
    [    1.953846] usb 4-6: SerialNumber: 07016329B32A0C24
    [    2.091257] usb 3-14: New USB device found, idVendor=0557, idProduct=7000
    [    2.091260] usb 3-14: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [    2.091434] hub 3-14:1.0: USB hub found
    [    2.091514] hub 3-14:1.0: 4 ports detected
    [    2.339068] tsc: Refined TSC clocksource calibration: 1699.998 MHz
    [    2.514081] i8042: No controller found
    [    2.514219] usb 3-14.1: new low-speed USB device number 4 using xhci_hcd
    [    2.514230] Switched to clocksource tsc
    [    2.514281] mousedev: PS/2 mouse device common for all mice
    [    2.514484] rtc_cmos 00:00: RTC can wake from S4
    [    2.514644] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
    [    2.514685] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
    [    2.514773] Intel P-state driver initializing.
    [    2.515738] cpuidle: using governor menu
    [    2.516321] EFI Variables Facility v0.08 2004-May-17
    [    2.519140] hidraw: raw HID events driver (C) Jiri Kosina
    [    2.519341] usbcore: registered new interface driver usbhid
    [    2.519343] usbhid: USB HID core driver
    [    2.519521] drop_monitor: Initializing network drop monitor service
    [    2.519742] TCP: cubic registered
    [    2.519755] Initializing XFRM netlink socket
    [    2.520232] NET: Registered protocol family 10
    [    2.520717] NET: Registered protocol family 17
    [    2.521507] microcode: CPU0 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521518] microcode: CPU1 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521529] microcode: CPU2 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521545] microcode: CPU3 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521554] microcode: CPU4 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521567] microcode: CPU5 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521579] microcode: CPU6 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521591] microcode: CPU7 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521604] microcode: CPU8 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521644] microcode: CPU9 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521686] microcode: CPU10 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521699] microcode: CPU11 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521713] microcode: CPU12 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521727] microcode: CPU13 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521741] microcode: CPU14 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521754] microcode: CPU15 sig=0x406f1, pf=0x1, revision=0xb00001d
    [    2.521807] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
    [    2.521983] Loading compiled-in X.509 certificates
    [    2.522024] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
    [    2.522050] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
    [    2.523230] Loaded X.509 cert 'CentOS Linux kernel signing key: 27f20485ebeb3b2d54add61e57b308fae070f41f'
    [    2.523267] registered taskstats version 1
    [    2.527308] Key type trusted registered
    [    2.531305] Key type encrypted registered
    [    2.535085] IMA: No TPM chip found, activating TPM-bypass!
    [    2.536703] rtc_cmos 00:00: setting system clock to 2017-03-07 18:16:49 UTC (1488910609)
    [    2.538336] Freeing unused kernel memory: 1680k freed
    [    2.550094] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
    [    2.550674] systemd[1]: Detected architecture x86-64.
    [    2.550679] systemd[1]: Running in initial RAM disk.
    [    2.550706] systemd[1]: Set hostname to <storage-02.maxfurn.com>.
    [    2.601657] systemd[1]: Reached target Swap.
    [    2.601673] systemd[1]: Starting Swap.
    [    2.601686] systemd[1]: Reached target Local File Systems.
    [    2.601694] systemd[1]: Starting Local File Systems.
    [    2.601865] systemd[1]: Created slice Root Slice.
    [    2.601875] systemd[1]: Starting Root Slice.
    [    2.601926] systemd[1]: Listening on udev Control Socket.
    [    2.601936] systemd[1]: Starting udev Control Socket.
    [    2.602030] systemd[1]: Created slice System Slice.
    [    2.602041] systemd[1]: Starting System Slice.
    [    2.602114] systemd[1]: Listening on Journal Socket.
    [    2.602126] systemd[1]: Starting Journal Socket.
    [    2.602738] systemd[1]: Starting dracut cmdline hook...
    [    2.603200] systemd[1]: Starting Journal Service...
    [    2.603795] systemd[1]: Starting Create list of required static device nodes for the current kernel...
    [    2.603827] systemd[1]: Reached target Timers.
    [    2.603842] systemd[1]: Starting Timers.
    [    2.604352] systemd[1]: Starting Setup Virtual Console...
    [    2.604373] systemd[1]: Reached target Slices.
    [    2.604385] systemd[1]: Starting Slices.
    [    2.604432] systemd[1]: Listening on udev Kernel Socket.
    [    2.604441] systemd[1]: Starting udev Kernel Socket.
    [    2.604458] systemd[1]: Reached target Sockets.
    [    2.604466] systemd[1]: Starting Sockets.
    [    2.605147] systemd[1]: Starting Apply Kernel Variables...
    [    2.606404] systemd[1]: Started Create list of required static device nodes for the current kernel.
    [    2.607155] systemd[1]: Starting Create Static Device Nodes in /dev...
    [    2.607624] systemd[1]: Started Apply Kernel Variables.
    [    2.610109] systemd[1]: Started Create Static Device Nodes in /dev.
    [    2.615549] usb 3-14.1: New USB device found, idVendor=0557, idProduct=2419
    [    2.615551] usb 3-14.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [    2.618020] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-14/3-14.1/3-14.1:1.0/input/input2
    [    2.653518] systemd[1]: Started Journal Service.
    [    2.668230] hid-generic 0003:0557:2419.0001: input,hidraw0: USB HID v1.00 Keyboard [HID 0557:2419] on usb-0000:00:14.0-14.1/input0
    [    2.669171] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-14/3-14.1/3-14.1:1.1/input/input3
    [    2.669330] hid-generic 0003:0557:2419.0002: input,hidraw1: USB HID v1.00 Mouse [HID 0557:2419] on usb-0000:00:14.0-14.1/input1
    [    2.809100] device-mapper: uevent: version 1.0.3
    [    2.809199] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
    [    3.010703] FUJITSU Extended Socket Network Device Driver - version 1.1 - Copyright (c) 2015 FUJITSU LIMITED
    [    3.056616] dca service started, version 1.12.1
    [    3.057717] pps_core: LinuxPPS API ver. 1 registered
    [    3.057719] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    3.058762] nvme 0000:04:00.0: irq 47 for MSI/MSI-X
    [    3.058806] PTP clock support registered
    [    3.058852] nvme 0000:05:00.0: irq 49 for MSI/MSI-X
    [    3.066450] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.4.0-k-rh7.3
    [    3.066453] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
    [    3.066677] pmd_set_huge: Cannot satisfy [mem 0xc7c00000-0xc7e00000] with a huge-page mapping due to MTRR override.
    [    3.067194] libata version 3.00 loaded.
    [    3.067479] usb-storage 4-6:1.0: USB Mass Storage device detected
    [    3.067557] scsi host0: usb-storage 4-6:1.0
    [    3.068235] usbcore: registered new interface driver usb-storage
    [    3.069432] usbcore: registered new interface driver uas
    [    3.069973] ahci 0000:00:11.4: version 3.0
    [    3.070155] ahci 0000:00:11.4: irq 50 for MSI/MSI-X
    [    3.070197] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode
    [    3.070200] ahci 0000:00:11.4: flags: 64bit ncq pm led clo pio slum part ems apst 
    [    3.078236] scsi host1: ahci
    [    3.078564] scsi host2: ahci
    [    3.078680] [drm] Initialized drm 1.1.0 20060810
    [    3.078872] scsi host3: ahci
    [    3.079325] scsi host4: ahci
    [    3.079590] ata1: SATA max UDMA/133 abar m2048@0xc7338000 port 0xc7338100 irq 50
    [    3.079594] ata2: SATA max UDMA/133 abar m2048@0xc7338000 port 0xc7338180 irq 50
    [    3.079598] ata3: SATA max UDMA/133 abar m2048@0xc7338000 port 0xc7338200 irq 50
    [    3.079601] ata4: SATA max UDMA/133 abar m2048@0xc7338000 port 0xc7338280 irq 50
    [    3.079824] ahci 0000:00:1f.2: irq 51 for MSI/MSI-X
    [    3.079878] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
    [    3.079881] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst 
    [    3.092676] scsi host5: ahci
    [    3.092871] scsi host6: ahci
    [    3.093565] scsi host7: ahci
    [    3.093832] scsi host8: ahci
    [    3.094039] scsi host9: ahci
    [    3.094152] scsi host10: ahci
    [    3.094210] ata5: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332100 irq 51
    [    3.094213] ata6: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332180 irq 51
    [    3.094215] ata7: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332200 irq 51
    [    3.094217] ata8: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332280 irq 51
    [    3.094219] ata9: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332300 irq 51
    [    3.094221] ata10: SATA max UDMA/133 abar m2048@0xc7332000 port 0xc7332380 irq 51
    [    3.100325] [drm] AST 2400 detected
    [    3.100344] [drm] Analog VGA only
    [    3.100362] [drm] dram 1632000000 1 16 01000000
    [    3.100424] [TTM] Zone  kernel: Available graphics memory: 16335290 kiB
    [    3.100426] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
    [    3.100427] [TTM] Initializing pool allocator
    [    3.100432] [TTM] Initializing DMA pool allocator
    [    3.103863] checking generic (c6000000 1d5000) vs hw (c6000000 1000000)
    [    3.103866] fb: conflicting fb hw usage astdrmfb vs EFI VGA - removing generic driver
    [    3.103880] Console: switching to colour dummy device 80x25
    [    3.104183] fbcon: astdrmfb (fb0) is primary device
    [    3.121177] Console: switching to colour frame buffer device 128x48
    [    3.146411] ast 0000:0a:00.0: fb0: astdrmfb frame buffer device
    [    3.176110] [drm] Initialized ast 0.1.0 20120228 for 0000:0a:00.0 on minor 0
    [    3.307560] ixgbe 0000:01:00.0: irq 52 for MSI/MSI-X
    [    3.307573] ixgbe 0000:01:00.0: irq 53 for MSI/MSI-X
    [    3.307585] ixgbe 0000:01:00.0: irq 54 for MSI/MSI-X
    [    3.307594] ixgbe 0000:01:00.0: irq 55 for MSI/MSI-X
    [    3.307603] ixgbe 0000:01:00.0: irq 56 for MSI/MSI-X
    [    3.307623] ixgbe 0000:01:00.0: irq 57 for MSI/MSI-X
    [    3.307632] ixgbe 0000:01:00.0: irq 58 for MSI/MSI-X
    [    3.307642] ixgbe 0000:01:00.0: irq 59 for MSI/MSI-X
    [    3.307652] ixgbe 0000:01:00.0: irq 60 for MSI/MSI-X
    [    3.307662] ixgbe 0000:01:00.0: irq 61 for MSI/MSI-X
    [    3.307672] ixgbe 0000:01:00.0: irq 62 for MSI/MSI-X
    [    3.307684] ixgbe 0000:01:00.0: irq 63 for MSI/MSI-X
    [    3.307694] ixgbe 0000:01:00.0: irq 64 for MSI/MSI-X
    [    3.307712] ixgbe 0000:01:00.0: irq 65 for MSI/MSI-X
    [    3.307722] ixgbe 0000:01:00.0: irq 66 for MSI/MSI-X
    [    3.307731] ixgbe 0000:01:00.0: irq 67 for MSI/MSI-X
    [    3.307741] ixgbe 0000:01:00.0: irq 68 for MSI/MSI-X
    [    3.307801] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
    [    3.343102] ixgbe 0000:01:00.0: PCI Express bandwidth of 32GT/s available
    [    3.343107] ixgbe 0000:01:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
    [    3.367152] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 010A00-000
    [    3.367158] ixgbe 0000:01:00.0: 0c:c4:7a:d2:93:12
    [    3.384125] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    3.384159] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    3.386126] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    3.386156] ata4: SATA link down (SStatus 0 SControl 300)
    [    3.391097] ata2.00: ATA-9: HGST HDN726060ALE614, APGNW7JH, max UDMA/133
    [    3.391103] ata2.00: 11721045168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
    [    3.395984] ata2.00: configured for UDMA/133
    [    3.399119] ata8: SATA link down (SStatus 0 SControl 300)
    [    3.399150] ata9: SATA link down (SStatus 0 SControl 300)
    [    3.400113] ata7: SATA link down (SStatus 0 SControl 300)
    [    3.400140] ata6: SATA link down (SStatus 0 SControl 300)
    [    3.400171] ata10: SATA link down (SStatus 0 SControl 300)
    [    3.400453] ata3.00: ATA-9: HGST HDN726060ALE614, APGNW7JH, max UDMA/133
    [    3.400459] ata3.00: 11721045168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
    [    3.401123] ata5: SATA link down (SStatus 0 SControl 300)
    [    3.408735] ata1.00: ATA-9: HGST HDN726060ALE614, APGNW7JH, max UDMA/133
    [    3.408741] ata1.00: 11721045168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
    [    3.413601] ata1.00: configured for UDMA/133
    [    3.413945] scsi 1:0:0:0: Direct-Access     ATA      HGST HDN726060AL W7JH PQ: 0 ANSI: 5
    [    3.421608] ata3.00: configured for UDMA/133
    [    3.428543] scsi 2:0:0:0: Direct-Access     ATA      HGST HDN726060AL W7JH PQ: 0 ANSI: 5
    [    3.437498] sd 1:0:0:0: [sda] 11721045168 512-byte logical blocks: (6.00 TB/5.45 TiB)
    [    3.437503] sd 1:0:0:0: [sda] 4096-byte physical blocks
    [    3.437533] sd 1:0:0:0: [sda] Write Protect is off
    [    3.437536] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
    [    3.437550] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    3.449953] sd 1:0:0:0: [sda] Attached SCSI disk
    [    3.453406] sd 2:0:0:0: [sdb] 11721045168 512-byte logical blocks: (6.00 TB/5.45 TiB)
    [    3.453414] sd 2:0:0:0: [sdb] 4096-byte physical blocks
    [    3.453508] scsi 3:0:0:0: Direct-Access     ATA      HGST HDN726060AL W7JH PQ: 0 ANSI: 5
    [    3.453596] sd 2:0:0:0: [sdb] Write Protect is off
    [    3.453605] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
    [    3.453651] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    3.470405] sd 3:0:0:0: [sdc] 11721045168 512-byte logical blocks: (6.00 TB/5.45 TiB)
    [    3.470413] sd 3:0:0:0: [sdc] 4096-byte physical blocks
    [    3.470659] sd 3:0:0:0: [sdc] Write Protect is off
    [    3.470669] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
    [    3.470722] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    3.484933] sd 2:0:0:0: [sdb] Attached SCSI disk
    [    3.496981] sd 3:0:0:0: [sdc] Attached SCSI disk
    [    3.516232] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
    [    3.556934] md: bind<sdb>
    [    3.568982] md: bind<sdc>
    [    3.574216] md: bind<sda>
    [    3.596066] raid6: sse2x1   gen()  4687 MB/s
    [    3.613057] raid6: sse2x2   gen()  6441 MB/s
    [    3.630058] raid6: sse2x4   gen()  7941 MB/s
    [    3.647060] raid6: avx2x1   gen() 10777 MB/s
    [    3.664058] raid6: avx2x2   gen() 12667 MB/s
    [    3.681059] raid6: avx2x4   gen() 14347 MB/s
    [    3.681060] raid6: using algorithm avx2x4 gen() (14347 MB/s)
    [    3.681062] raid6: using avx2x2 recovery algorithm
    [    3.682022] async_tx: api initialized (async)
    [    3.682941] xor: automatically using best checksumming function:
    [    3.692057]    avx       : 17180.000 MB/sec
    [    3.697970] md: raid6 personality registered for level 6
    [    3.697973] md: raid5 personality registered for level 5
    [    3.697974] md: raid4 personality registered for level 4
    [    3.698358] md/raid:md0: device sda operational as raid disk 0
    [    3.698361] md/raid:md0: device sdc operational as raid disk 2
    [    3.698362] md/raid:md0: device sdb operational as raid disk 1
    [    3.698782] md/raid:md0: allocated 3316kB
    [    3.698807] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
    [    3.698808] RAID conf printout:
    [    3.698810]  --- level:5 rd:3 wd:3
    [    3.698811]  disk 0, o:1, dev:sda
    [    3.698812]  disk 1, o:1, dev:sdb
    [    3.698813]  disk 2, o:1, dev:sdc
    [    3.698998] created bitmap (44 pages) for device md0
    [    3.700422] md0: bitmap initialized from disk: read 3 pages, set 0 of 89423 bits
    [    3.704259] md0: detected capacity change from 0 to 12002081636352
    [    3.757844] ixgbe 0000:01:00.1: irq 70 for MSI/MSI-X
    [    3.757858] ixgbe 0000:01:00.1: irq 71 for MSI/MSI-X
    [    3.757872] ixgbe 0000:01:00.1: irq 74 for MSI/MSI-X
    [    3.757892] ixgbe 0000:01:00.1: irq 75 for MSI/MSI-X
    [    3.757902] ixgbe 0000:01:00.1: irq 76 for MSI/MSI-X
    [    3.757912] ixgbe 0000:01:00.1: irq 77 for MSI/MSI-X
    [    3.757922] ixgbe 0000:01:00.1: irq 78 for MSI/MSI-X
    [    3.757932] ixgbe 0000:01:00.1: irq 79 for MSI/MSI-X
    [    3.757944] ixgbe 0000:01:00.1: irq 80 for MSI/MSI-X
    [    3.757954] ixgbe 0000:01:00.1: irq 81 for MSI/MSI-X
    [    3.757964] ixgbe 0000:01:00.1: irq 82 for MSI/MSI-X
    [    3.757984] ixgbe 0000:01:00.1: irq 83 for MSI/MSI-X
    [    3.757994] ixgbe 0000:01:00.1: irq 84 for MSI/MSI-X
    [    3.758004] ixgbe 0000:01:00.1: irq 85 for MSI/MSI-X
    [    3.758014] ixgbe 0000:01:00.1: irq 86 for MSI/MSI-X
    [    3.758024] ixgbe 0000:01:00.1: irq 87 for MSI/MSI-X
    [    3.758034] ixgbe 0000:01:00.1: irq 88 for MSI/MSI-X
    [    3.758114] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
    [    3.794402] ixgbe 0000:01:00.1: PCI Express bandwidth of 32GT/s available
    [    3.794407] ixgbe 0000:01:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
    [    3.818241] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 010A00-000
    [    3.818247] ixgbe 0000:01:00.1: 0c:c4:7a:d2:93:13
    [    3.966887] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
    [    4.069929] scsi 0:0:0:0: Direct-Access              USB DISK 3.0     PMAP PQ: 0 ANSI: 6
    [    4.071166] sd 0:0:0:0: [sdd] 15126528 512-byte logical blocks: (7.74 GB/7.21 GiB)
    [    4.071713] sd 0:0:0:0: [sdd] Write Protect is off
    [    4.071717] sd 0:0:0:0: [sdd] Mode Sense: 45 00 00 00
    [    4.072270] sd 0:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
    [    4.124825]  sdd: sdd1 sdd2
    [    4.127964] sd 0:0:0:0: [sdd] Attached SCSI removable disk
    [    4.209141] ixgbe 0000:02:00.0: irq 89 for MSI/MSI-X
    [    4.209169] ixgbe 0000:02:00.0: irq 90 for MSI/MSI-X
    [    4.209180] ixgbe 0000:02:00.0: irq 91 for MSI/MSI-X
    [    4.209192] ixgbe 0000:02:00.0: irq 92 for MSI/MSI-X
    [    4.209204] ixgbe 0000:02:00.0: irq 93 for MSI/MSI-X
    [    4.209215] ixgbe 0000:02:00.0: irq 94 for MSI/MSI-X
    [    4.209226] ixgbe 0000:02:00.0: irq 95 for MSI/MSI-X
    [    4.209237] ixgbe 0000:02:00.0: irq 96 for MSI/MSI-X
    [    4.209249] ixgbe 0000:02:00.0: irq 97 for MSI/MSI-X
    [    4.209272] ixgbe 0000:02:00.0: irq 98 for MSI/MSI-X
    [    4.209283] ixgbe 0000:02:00.0: irq 99 for MSI/MSI-X
    [    4.209332] ixgbe 0000:02:00.0: irq 100 for MSI/MSI-X
    [    4.209344] ixgbe 0000:02:00.0: irq 101 for MSI/MSI-X
    [    4.209355] ixgbe 0000:02:00.0: irq 102 for MSI/MSI-X
    [    4.209366] ixgbe 0000:02:00.0: irq 103 for MSI/MSI-X
    [    4.209377] ixgbe 0000:02:00.0: irq 104 for MSI/MSI-X
    [    4.209389] ixgbe 0000:02:00.0: irq 105 for MSI/MSI-X
    [    4.209455] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
    [    4.245308] ixgbe 0000:02:00.0: PCI Express bandwidth of 32GT/s available
    [    4.245314] ixgbe 0000:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
    [    4.269253] ixgbe 0000:02:00.0: MAC: 3, PHY: 0, PBA No: 010A00-000
    [    4.269259] ixgbe 0000:02:00.0: 0c:c4:7a:d2:93:14
    [    6.391462] nvme 0000:04:00.0: irq 47 for MSI/MSI-X
    [    6.391479] nvme 0000:04:00.0: irq 106 for MSI/MSI-X
    [    6.391492] nvme 0000:04:00.0: irq 107 for MSI/MSI-X
    [    6.391512] nvme 0000:04:00.0: irq 108 for MSI/MSI-X
    [    6.391524] nvme 0000:04:00.0: irq 109 for MSI/MSI-X
    [    6.391535] nvme 0000:04:00.0: irq 110 for MSI/MSI-X
    [    6.391546] nvme 0000:04:00.0: irq 111 for MSI/MSI-X
    [    6.391559] nvme 0000:04:00.0: irq 112 for MSI/MSI-X
    [    6.391571] nvme 0000:04:00.0: irq 113 for MSI/MSI-X
    [    6.391582] nvme 0000:04:00.0: irq 114 for MSI/MSI-X
    [    6.391593] nvme 0000:04:00.0: irq 115 for MSI/MSI-X
    [    6.391613] nvme 0000:04:00.0: irq 116 for MSI/MSI-X
    [    6.391625] nvme 0000:04:00.0: irq 117 for MSI/MSI-X
    [    6.391636] nvme 0000:04:00.0: irq 118 for MSI/MSI-X
    [    6.391647] nvme 0000:04:00.0: irq 119 for MSI/MSI-X
    [    6.391658] nvme 0000:04:00.0: irq 120 for MSI/MSI-X
    [    6.694495] nvme 0000:05:00.0: irq 49 for MSI/MSI-X
    [    6.694512] nvme 0000:05:00.0: irq 121 for MSI/MSI-X
    [    6.694524] nvme 0000:05:00.0: irq 122 for MSI/MSI-X
    [    6.694535] nvme 0000:05:00.0: irq 123 for MSI/MSI-X
    [    6.694547] nvme 0000:05:00.0: irq 124 for MSI/MSI-X
    [    6.694567] nvme 0000:05:00.0: irq 125 for MSI/MSI-X
    [    6.694578] nvme 0000:05:00.0: irq 126 for MSI/MSI-X
    [    6.694589] nvme 0000:05:00.0: irq 127 for MSI/MSI-X
    [    6.694601] nvme 0000:05:00.0: irq 128 for MSI/MSI-X
    [    6.694612] nvme 0000:05:00.0: irq 129 for MSI/MSI-X
    [    6.694623] nvme 0000:05:00.0: irq 130 for MSI/MSI-X
    [    6.694637] nvme 0000:05:00.0: irq 131 for MSI/MSI-X
    [    6.694649] nvme 0000:05:00.0: irq 132 for MSI/MSI-X
    [    6.694668] nvme 0000:05:00.0: irq 133 for MSI/MSI-X
    [    6.694679] nvme 0000:05:00.0: irq 134 for MSI/MSI-X
    [    6.694690] nvme 0000:05:00.0: irq 135 for MSI/MSI-X
    [    8.620708] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
    [    8.863369] ixgbe 0000:02:00.1: irq 137 for MSI/MSI-X
    [    8.863384] ixgbe 0000:02:00.1: irq 138 for MSI/MSI-X
    [    8.863397] ixgbe 0000:02:00.1: irq 139 for MSI/MSI-X
    [    8.863416] ixgbe 0000:02:00.1: irq 140 for MSI/MSI-X
    [    8.863428] ixgbe 0000:02:00.1: irq 141 for MSI/MSI-X
    [    8.863439] ixgbe 0000:02:00.1: irq 142 for MSI/MSI-X
    [    8.863450] ixgbe 0000:02:00.1: irq 143 for MSI/MSI-X
    [    8.863461] ixgbe 0000:02:00.1: irq 144 for MSI/MSI-X
    [    8.863473] ixgbe 0000:02:00.1: irq 145 for MSI/MSI-X
    [    8.863484] ixgbe 0000:02:00.1: irq 146 for MSI/MSI-X
    [    8.863496] ixgbe 0000:02:00.1: irq 147 for MSI/MSI-X
    [    8.863514] ixgbe 0000:02:00.1: irq 148 for MSI/MSI-X
    [    8.863525] ixgbe 0000:02:00.1: irq 149 for MSI/MSI-X
    [    8.863537] ixgbe 0000:02:00.1: irq 150 for MSI/MSI-X
    [    8.863551] ixgbe 0000:02:00.1: irq 151 for MSI/MSI-X
    [    8.863562] ixgbe 0000:02:00.1: irq 152 for MSI/MSI-X
    [    8.863573] ixgbe 0000:02:00.1: irq 153 for MSI/MSI-X
    [    8.863643] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
    [    8.899423] ixgbe 0000:02:00.1: PCI Express bandwidth of 32GT/s available
    [    8.899429] ixgbe 0000:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
    [    8.923272] ixgbe 0000:02:00.1: MAC: 3, PHY: 0, PBA No: 010A00-000
    [    8.923275] ixgbe 0000:02:00.1: 0c:c4:7a:d2:93:15
    [   13.274751] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
    [   14.282166] SGI XFS with ACLs, security attributes, no debug enabled
    [   14.286837] XFS (dm-0): Mounting V5 Filesystem
    [   14.392334] XFS (dm-0): Ending clean mount
    [   14.477632] XFS (dm-2): Mounting V5 Filesystem
    [   14.601758] XFS (dm-2): Ending clean mount
    [   14.827101] systemd-journald[178]: Received SIGTERM from PID 1 (systemd).
    [   15.140538] SELinux:  Disabled at runtime.
    [   15.140590] SELinux:  Unregistering netfilter hooks
    [   15.180111] type=1404 audit(1488910622.142:2): selinux=0 auid=4294967295 ses=4294967295
    [   15.307595] ip_tables: (C) 2000-2006 Netfilter Core Team
    [   15.307702] systemd[1]: Inserted module 'ip_tables'
    [   16.438180] RPC: Registered named UNIX socket transport module.
    [   16.438185] RPC: Registered udp transport module.
    [   16.438187] RPC: Registered tcp transport module.
    [   16.438189] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [   16.501227] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    [   16.913391] ipmi message handler version 39.2
    [   16.925533] ipmi_si IPI0001:00: ipmi_si: probing via ACPI
    [   16.925563] ipmi_si IPI0001:00: [io  0x0ca2] regsize 1 spacing 1 irq 0
    [   16.925565] ipmi_si: Adding ACPI-specified kcs state machine
    [   16.925628] IPMI System Interface driver.
    [   16.925771] ipmi_si: probing via SMBIOS
    [   16.925773] ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
    [   16.925774] ipmi_si: Adding SMBIOS-specified kcs state machine duplicate interface
    [   16.925777] ipmi_si: probing via SPMI
    [   16.925778] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0
    [   16.925779] ipmi_si: Adding SPMI-specified kcs state machine duplicate interface
    [   16.925782] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
    [   16.959061] ipmi_si IPI0001:00: The BMC does not support clearing the recv irq bit, compensating, but the BMC needs to be fixed.
    [   16.972261] ipmi_si IPI0001:00: Found new BMC (man_id: 0x002a7c, prod_id: 0x0844, dev_id: 0x20)
    [   16.972285] ipmi_si IPI0001:00: IPMI kcs interface initialized
    [   16.975142] ipmi device interface
    [   17.015604] power_meter ACPI000D:00: Found ACPI power meter.
    [   17.015839] power_meter ACPI000D:00: Ignoring unsafe software power cap!
    [   17.017563] wmi: Mapper loaded
    [   17.025689] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
    [   17.076344] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
    [   17.081867] EDAC MC: Ver: 3.0.0
    [   17.131641] sd 1:0:0:0: Attached scsi generic sg0 type 0
    [   17.131683] sd 2:0:0:0: Attached scsi generic sg1 type 0
    [   17.131746] sd 3:0:0:0: Attached scsi generic sg2 type 0
    [   17.131827] sd 0:0:0:0: Attached scsi generic sg3 type 0
    [   17.171711] i801_smbus 0000:00:1f.3: enabling device (0000 -> 0003)
    [   17.171870] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
    [   17.184325] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
    [   17.184333] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
    [   17.184342] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
    [   17.184352] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    [   17.184356] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    [   17.184362] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
    [   17.184368] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    [   17.184372] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    [   17.184378] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
    [   17.184383] EDAC sbridge: Seeking for: PCI ID 8086:6f60
    [   17.184393] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
    [   17.184397] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
    [   17.184403] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
    [   17.184409] EDAC sbridge: Seeking for: PCI ID 8086:6f71
    [   17.184413] EDAC sbridge: Seeking for: PCI ID 8086:6f71
    [   17.184419] EDAC sbridge: Seeking for: PCI ID 8086:6f71
    [   17.184424] EDAC sbridge: Seeking for: PCI ID 8086:6faa
    [   17.184428] EDAC sbridge: Seeking for: PCI ID 8086:6faa
    [   17.184434] EDAC sbridge: Seeking for: PCI ID 8086:6faa
    [   17.184439] EDAC sbridge: Seeking for: PCI ID 8086:6fab
    [   17.184444] EDAC sbridge: Seeking for: PCI ID 8086:6fab
    [   17.184450] EDAC sbridge: Seeking for: PCI ID 8086:6fab
    [   17.184455] EDAC sbridge: Seeking for: PCI ID 8086:6fac
    [   17.184459] EDAC sbridge: Seeking for: PCI ID 8086:6fac
    [   17.184465] EDAC sbridge: Seeking for: PCI ID 8086:6fac
    [   17.184470] EDAC sbridge: Seeking for: PCI ID 8086:6fad
    [   17.184475] EDAC sbridge: Seeking for: PCI ID 8086:6fad
    [   17.184480] EDAC sbridge: Seeking for: PCI ID 8086:6fad
    [   17.184485] EDAC sbridge: Seeking for: PCI ID 8086:6faf
    [   17.184490] EDAC sbridge: Seeking for: PCI ID 8086:6faf
    [   17.184496] EDAC sbridge: Seeking for: PCI ID 8086:6faf
    [   17.184501] EDAC sbridge: Seeking for: PCI ID 8086:6f68
    [   17.184506] EDAC sbridge: Seeking for: PCI ID 8086:6f68
    [   17.184512] EDAC sbridge: Seeking for: PCI ID 8086:6f68
    [   17.184516] EDAC sbridge: Seeking for: PCI ID 8086:6f79
    [   17.184526] EDAC sbridge: Seeking for: PCI ID 8086:6f6a
    [   17.184536] EDAC sbridge: Seeking for: PCI ID 8086:6f6b
    [   17.184545] EDAC sbridge: Seeking for: PCI ID 8086:6f6c
    [   17.184555] EDAC sbridge: Seeking for: PCI ID 8086:6f6d
    [   17.184711] EDAC MC0: Giving out device to 'sbridge_edac.c' 'Broadwell Socket#0': DEV 0000:ff:12.0
    [   17.184861] EDAC MC1: Giving out device to 'sbridge_edac.c' 'Broadwell Socket#1': DEV 0000:7f:12.0
    [   17.184862] EDAC sbridge:  Ver: 1.1.1 
    [   17.198275] input: PC Speaker as /devices/platform/pcspkr/input/input4
    [   17.203303] iTCO_vendor_support: vendor-support=0
    [   17.318356] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
    [   17.318434] iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460)
    [   17.318565] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
    [   17.550956] AES CTR mode by8 optimization enabled
    [   17.554444] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
    [   17.564533] Adding 1048572k swap on /dev/mapper/vg_hdd-swap.  Priority:-1 extents:1 across:1048572k FS
    [   17.588228] alg: No test for crc32 (crc32-pclmul)
    [   17.680426] XFS (sdd2): Mounting V5 Filesystem
    [   17.762732] intel_rapl: Found RAPL domain package
    [   17.762752] intel_rapl: Found RAPL domain dram
    [   17.762754] intel_rapl: DRAM domain energy unit 15300pj
    [   17.762772] intel_rapl: Found RAPL domain package
    [   17.762787] intel_rapl: Found RAPL domain dram
    [   17.762790] intel_rapl: DRAM domain energy unit 15300pj
    [   17.953819] XFS (sdd2): Ending clean mount
    [   17.994452] FAT-fs (sdd1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
    [   18.331816] device-mapper: raid: Loading target version 1.9.1
    [   18.346224] md: raid1 personality registered for level 1
    [   18.346625] md/raid1:mdX: active with 2 out of 2 mirrors
    [   18.346777] created bitmap (250 pages) for device mdX
    [   18.358314] mdX: bitmap initialized from disk: read 16 pages, set 1 of 512000 bits
    [   18.433215] md/raid1:mdX: active with 2 out of 2 mirrors
    [   18.433284] created bitmap (250 pages) for device mdX
    [   18.458107] mdX: bitmap initialized from disk: read 16 pages, set 1 of 512000 bits
    [   18.467910] md/raid1:mdX: active with 2 out of 2 mirrors
    [   18.468100] created bitmap (256 pages) for device mdX
    [   18.490487] mdX: bitmap initialized from disk: read 17 pages, set 1 of 524288 bits
    [   18.497888] md/raid1:mdX: active with 2 out of 2 mirrors
    [   18.497969] created bitmap (256 pages) for device mdX
    [   18.515182] mdX: bitmap initialized from disk: read 17 pages, set 1 of 524288 bits
    [   18.690436] XFS (dm-24): Mounting V5 Filesystem
    [   18.743314] XFS (dm-23): Mounting V5 Filesystem
    [   18.968874] XFS (dm-24): Ending clean mount
    [   19.005203] systemd-journald[822]: Received request to flush runtime journal from PID 1
    [   19.167898] XFS (dm-23): Ending clean mount
    [   19.639303] type=1305 audit(1488910626.601:3): audit_pid=1242 old=0 auid=4294967295 ses=4294967295 res=1
    [   20.974162] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
    [   21.050918] pps pps0: new PPS source ptp0
    [   21.050927] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
    [   21.184555] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
    [   21.189865] IPv6: ADDRCONF(NETDEV_UP): enp1s0f1: link is not ready
    [   21.267106] pps pps1: new PPS source ptp1
    [   21.267114] ixgbe 0000:01:00.1: registered PHC device on enp1s0f1
    [   21.400948] IPv6: ADDRCONF(NETDEV_UP): enp1s0f1: link is not ready
    [   21.405545] IPv6: ADDRCONF(NETDEV_UP): enp2s0f0: link is not ready
    [   21.480367] pps pps2: new PPS source ptp2
    [   21.480374] ixgbe 0000:02:00.0: registered PHC device on enp2s0f0
    [   21.613899] IPv6: ADDRCONF(NETDEV_UP): enp2s0f0: link is not ready
    [   21.617916] IPv6: ADDRCONF(NETDEV_UP): enp2s0f1: link is not ready
    [   21.692756] pps pps3: new PPS source ptp3
    [   21.692762] ixgbe 0000:02:00.1: registered PHC device on enp2s0f1
    [   21.826215] IPv6: ADDRCONF(NETDEV_UP): enp2s0f1: link is not ready
    [   21.859215] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
    [   21.876044] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
    [   21.876820] IPv6: ADDRCONF(NETDEV_UP): bond1: link is not ready
    [   21.884843] bond0: Setting MII monitoring interval to 100
    [   21.884898] bond0: Setting up delay to 0
    [   21.884944] bond0: Setting down delay to 0
    [   21.885038] bond0: Setting ad_actor_sys_prio to 65535
    [   21.885110] bond0: Setting ad_select to stable (0)
    [   21.885162] bond0: Setting ad_user_port_key to 0
    [   21.885255] bond0: Setting arp_all_targets to any (0)
    [   21.885302] bond0: Setting fail_over_mac to none (0)
    [   21.885349] bond0: Setting LACP rate to fast (1)
    [   21.885533] bond0: Setting min links value to 0
    [   21.885609] bond0: Setting primary_reselect to always (0)
    [   21.885657] bond0: Setting resend_igmp to 1
    [   21.885708] bond0: Setting use_carrier to 1
    [   21.885755] bond0: Setting xmit hash policy to layer2 (0)
    [   21.885832] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
    [   21.887200] bond1: Setting ARP monitoring interval to 0
    [   21.887250] bond1: Setting MII monitoring interval to 100
    [   21.887295] bond1: Setting up delay to 0
    [   21.887359] bond1: Setting down delay to 0
    [   21.887402] bond1: Setting arp_validate to none (0)
    [   21.887480] bond1: Setting ad_select to stable (0)
    [   21.887574] bond1: Setting arp_all_targets to any (0)
    [   21.887620] bond1: Setting fail_over_mac to none (0)
    [   21.887798] bond1: Setting min links value to 0
    [   21.887891] bond1: Setting primary_reselect to always (0)
    [   21.887936] bond1: Setting resend_igmp to 1
    [   21.887983] bond1: Setting use_carrier to 1
    [   21.888029] bond1: Setting xmit hash policy to layer2 (0)
    [   21.888108] IPv6: ADDRCONF(NETDEV_UP): bond1: link is not ready
    [   26.544304] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   26.544460] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f0: link becomes ready
    [   26.552069] ixgbe 0000:01:00.0: removed PHC on enp1s0f0
    [   26.701969] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   26.866718] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f1: link becomes ready
    [   26.867125] ixgbe 0000:01:00.0 enp1s0f0: changing MTU from 1500 to 9000
    [   26.939009] pps pps0: new PPS source ptp0
    [   26.939017] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
    [   27.062634] bond0: Enslaving enp1s0f0 as a backup interface with a down link
    [   27.157395] ixgbe 0000:01:00.1: removed PHC on enp1s0f1
    [   27.480550] ixgbe 0000:01:00.1 enp1s0f1: changing MTU from 1500 to 9000
    [   27.552795] pps pps1: new PPS source ptp1
    [   27.552803] ixgbe 0000:01:00.1: registered PHC device on enp1s0f1
    [   27.675500] bond0: Enslaving enp1s0f1 as a backup interface with a down link
    [   27.789164] ixgbe 0000:02:00.0 enp2s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   27.789534] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0: link becomes ready
    [   27.823833] ixgbe 0000:02:00.0: removed PHC on enp2s0f0
    [   27.976197] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   28.145554] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f1: link becomes ready
    [   28.145866] ixgbe 0000:02:00.0 enp2s0f0: changing MTU from 1500 to 9000
    [   28.217900] pps pps2: new PPS source ptp2
    [   28.217908] ixgbe 0000:02:00.0: registered PHC device on enp2s0f0
    [   28.340131] bond1: Enslaving enp2s0f0 as an active interface with a down link
    [   28.497746] ixgbe 0000:02:00.1: removed PHC on enp2s0f1
    [   28.819898] ixgbe 0000:02:00.1 enp2s0f1: changing MTU from 1500 to 9000
    [   28.892012] pps pps3: new PPS source ptp3
    [   28.892025] ixgbe 0000:02:00.1: registered PHC device on enp2s0f1
    [   29.015182] bond1: Enslaving enp2s0f1 as an active interface with a down link
    [   32.342176] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   32.385123] bond0: link status definitely up for interface enp1s0f0, 10000 Mbps full duplex
    [   32.385132] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
    [   32.385144] bond0: first active interface up!
    [   32.385221] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
    [   32.842273] ixgbe 0000:02:00.0 enp2s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   32.891089] bond1: link status definitely up for interface enp2s0f0, 10000 Mbps full duplex
    [   32.891101] bond1: first active interface up!
    [   32.891120] IPv6: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
    [   33.018971] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   33.085127] bond0: link status definitely up for interface enp1s0f1, 10000 Mbps full duplex
    [   33.370288] ixgbe 0000:02:00.1 enp2s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   33.391106] bond1: link status definitely up for interface enp2s0f1, 10000 Mbps full duplex
    [  719.094198] Request for unknown module key 'The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b' err -11
    [  719.094213] drbd: loading out-of-tree module taints kernel.
    [  719.094467] drbd: module verification failed: signature and/or required key missing - tainting kernel
    [  719.096062] Request for unknown module key 'The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b' err -11
    [  719.096243] Request for unknown module key 'The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b' err -11
    [  719.097539] Request for unknown module key 'The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b' err -11
    [  719.100449] drbd: initialized. Version: 8.4.9-1 (api:1/proto:86-101)
    [  719.100452] drbd: GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48
    [  719.100454] drbd: registered as block device major 147
    [  719.492756] drbd r2: Starting worker thread (from drbdsetup-84 [17365])
    [  719.493093] block drbd2: disk( Diskless -> Attaching ) 
    [  719.493473] drbd r2: Method to ensure write ordering: drain
    [  719.493478] block drbd2: max BIO size = 131072
    [  719.493484] block drbd2: drbd_bm_resize called with capacity == 536854456
    [  719.497445] block drbd2: resync bitmap: bits=67106807 words=1048544 pages=2048
    [  719.497450] block drbd2: size = 256 GB (268427228 KB)
    [  719.512327] block drbd2: recounting of set bits took additional 2 jiffies
    [  719.512332] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
    [  719.512339] block drbd2: disk( Attaching -> Consistent ) 
    [  719.512356] block drbd2: attached to UUIDs 6933DB6D9FA88774:0000000000000000:EA721AF9F7F939EC:0000000000000004
    [  719.512661] drbd r3: Starting worker thread (from drbdsetup-84 [17366])
    [  719.513013] block drbd3: disk( Diskless -> Attaching ) 
    [  719.513341] drbd r3: Method to ensure write ordering: drain
    [  719.513345] block drbd3: max BIO size = 131072
    [  719.513350] block drbd3: drbd_bm_resize called with capacity == 536854456
    [  719.517480] block drbd3: resync bitmap: bits=67106807 words=1048544 pages=2048
    [  719.517485] block drbd3: size = 256 GB (268427228 KB)
    [  719.532376] block drbd3: recounting of set bits took additional 2 jiffies
    [  719.532380] block drbd3: 13 GB (3470336 bits) marked out-of-sync by on disk bit-map.
    [  719.532385] block drbd3: disk( Attaching -> Consistent ) 
    [  719.532401] block drbd3: attached to UUIDs 9F491B7BD89C8229:EA25D3F5295C355B:91E8451A364669BB:91E7451A364669BB
    [  719.541530] drbd r2: conn( StandAlone -> Unconnected ) 
    [  719.541559] drbd r2: Starting receiver thread (from drbd_w_r2 [17369])
    [  719.541642] drbd r2: receiver (re)started
    [  719.541677] drbd r2: conn( Unconnected -> WFConnection ) 
    [  719.545698] drbd r3: conn( StandAlone -> Unconnected ) 
    [  719.545732] drbd r3: Starting receiver thread (from drbd_w_r3 [17370])
    [  719.545770] drbd r3: receiver (re)started
    [  719.545801] drbd r3: conn( Unconnected -> WFConnection ) 
    [  719.620365] drbd r1: Starting worker thread (from drbdsetup-84 [17400])
    [  719.621704] block drbd1: disk( Diskless -> Attaching ) 
    [  719.621965] drbd r1: Method to ensure write ordering: drain
    [  719.621969] block drbd1: max BIO size = 131072
    [  719.621974] block drbd1: Adjusting my ra_pages to backing device's (32 -> 512)
    [  719.621977] block drbd1: drbd_bm_resize called with capacity == 8589672376
    [  719.669637] block drbd1: resync bitmap: bits=1073709047 words=16776704 pages=32767
    [  719.669642] block drbd1: size = 4096 GB (4294836188 KB)
    [  719.931702] block drbd1: recounting of set bits took additional 32 jiffies
    [  719.931707] block drbd1: 13 GB (3470336 bits) marked out-of-sync by on disk bit-map.
    [  719.931715] block drbd1: disk( Attaching -> Consistent ) 
    [  719.931731] block drbd1: attached to UUIDs 13FEE1D0B6C0C3B5:8ADAED18520DD8CD:92F0E1E0FF621304:92EFE1E0FF621304
    [  719.932205] drbd nfs: Starting worker thread (from drbdsetup-84 [17458])
    [  719.933082] block drbd4: disk( Diskless -> Attaching ) 
    [  719.933357] drbd nfs: Method to ensure write ordering: drain
    [  719.933360] block drbd4: max BIO size = 1048576
    [  719.933365] block drbd4: Adjusting my ra_pages to backing device's (32 -> 512)
    [  719.933368] block drbd4: drbd_bm_resize called with capacity == 83883448
    [  719.933769] block drbd4: resync bitmap: bits=10485431 words=163835 pages=320
    [  719.933772] block drbd4: size = 40 GB (41941724 KB)
    [  719.960606] block drbd4: recounting of set bits took additional 0 jiffies
    [  719.960610] block drbd4: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
    [  719.960616] block drbd4: disk( Attaching -> Consistent ) 
    [  719.960630] block drbd4: attached to UUIDs 3F17D18242CE41D0:0000000000000000:8A8DC62709B1DD06:8A8CC62709B1DD07
    [  719.973468] drbd r1: conn( StandAlone -> Unconnected ) 
    [  719.973495] drbd r1: Starting receiver thread (from drbd_w_r1 [17402])
    [  719.973557] drbd r1: receiver (re)started
    [  719.973594] drbd r1: conn( Unconnected -> WFConnection ) 
    [  719.977043] drbd nfs: conn( StandAlone -> Unconnected ) 
    [  719.977066] drbd nfs: Starting receiver thread (from drbd_w_nfs [17463])
    [  719.977126] drbd nfs: receiver (re)started
    [  719.977157] drbd nfs: conn( Unconnected -> WFConnection ) 
    [  720.054692] drbd r3: Handshake successful: Agreed network protocol version 101
    [  720.054697] drbd r3: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  720.054743] drbd r3: conn( WFConnection -> WFReportParams ) 
    [  720.054749] drbd r3: Starting ack_recv thread (from drbd_r_r3 [17389])
    [  720.070697] block drbd3: drbd_sync_handshake:
    [  720.070702] block drbd3: self 9F491B7BD89C8228:EA25D3F5295C355B:91E8451A364669BB:91E7451A364669BB bits:3470336 flags:0
    [  720.070706] block drbd3: peer EA25D3F5295C355A:0000000000000000:91E8451A364669BA:91E7451A364669BB bits:0 flags:0
    [  720.070708] block drbd3: uuid_compare()=1 by rule 70
    [  720.070714] block drbd3: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> Consistent ) 
    [  720.072314] block drbd3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 1339(1), total 1339; compression: 100.0%
    [  720.074172] block drbd3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 1339(1), total 1339; compression: 100.0%
    [  720.074179] block drbd3: helper command: /sbin/drbdadm before-resync-source minor-3
    [  720.076822] block drbd3: helper command: /sbin/drbdadm before-resync-source minor-3 exit code 0 (0x0)
    [  720.076840] block drbd3: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent ) 
    [  720.076867] block drbd3: Began resync as SyncSource (will sync 13881344 KB [3470336 bits set]).
    [  720.076900] block drbd3: updated sync UUID 9F491B7BD89C8228:EA26D3F5295C355B:EA25D3F5295C355B:91E8451A364669BB
    [  720.092479] drbd r2: Handshake successful: Agreed network protocol version 101
    [  720.092487] drbd r2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  720.092544] drbd r2: conn( WFConnection -> WFReportParams ) 
    [  720.092553] drbd r2: Starting ack_recv thread (from drbd_r_r2 [17385])
    [  720.116701] block drbd2: drbd_sync_handshake:
    [  720.116706] block drbd2: self 6933DB6D9FA88774:0000000000000000:EA721AF9F7F939EC:0000000000000004 bits:0 flags:0
    [  720.116710] block drbd2: peer 5876079376FF9EBC:6933DB6D9FA88775:EA721AF9F7F939ED:0000000000000004 bits:3470336 flags:2
    [  720.116713] block drbd2: uuid_compare()=-1 by rule 50
    [  720.116719] block drbd2: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate ) 
    [  720.117080] block drbd2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 1304(1), total 1304; compression: 100.0%
    [  720.118390] block drbd2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 1304(1), total 1304; compression: 100.0%
    [  720.118394] block drbd2: conn( WFBitMapT -> WFSyncUUID ) 
    [  720.121072] block drbd2: updated sync uuid 6934DB6D9FA88774:0000000000000000:EA721AF9F7F939EC:0000000000000004
    [  720.121234] block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2
    [  720.123427] block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
    [  720.123445] block drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) 
    [  720.123472] block drbd2: Began resync as SyncTarget (will sync 13881344 KB [3470336 bits set]).
    [  720.237889] block drbd2: peer( Secondary -> Primary ) 
    [  720.245582] block drbd3: role( Secondary -> Primary ) 
    [  720.417487] drbd r0: Starting worker thread (from drbdsetup-84 [17776])
    [  720.418907] block drbd0: disk( Diskless -> Attaching ) 
    [  720.419263] drbd r0: Method to ensure write ordering: drain
    [  720.419267] block drbd0: max BIO size = 131072
    [  720.419271] block drbd0: Adjusting my ra_pages to backing device's (32 -> 512)
    [  720.419275] block drbd0: drbd_bm_resize called with capacity == 8589672376
    [  720.469333] block drbd0: resync bitmap: bits=1073709047 words=16776704 pages=32767
    [  720.469339] block drbd0: size = 4096 GB (4294836188 KB)
    [  720.477695] drbd nfs: Handshake successful: Agreed network protocol version 101
    [  720.477699] drbd nfs: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  720.477757] drbd nfs: conn( WFConnection -> WFReportParams ) 
    [  720.477764] drbd nfs: Starting ack_recv thread (from drbd_r_nfs [17495])
    [  720.666669] block drbd4: drbd_sync_handshake:
    [  720.666674] block drbd4: self 3F17D18242CE41D0:0000000000000000:8A8DC62709B1DD06:8A8CC62709B1DD07 bits:0 flags:0
    [  720.666678] block drbd4: peer 4CF39433A70FC49A:3F17D18242CE41D1:8A8DC62709B1DD07:8A8CC62709B1DD07 bits:2393088 flags:2
    [  720.666680] block drbd4: uuid_compare()=-1 by rule 50
    [  720.666686] block drbd4: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate ) 
    [  720.667436] block drbd4: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 517(1), total 517; compression: 100.0%
    [  720.667800] block drbd4: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 517(1), total 517; compression: 100.0%
    [  720.667803] block drbd4: conn( WFBitMapT -> WFSyncUUID ) 
    [  720.671060] block drbd4: updated sync uuid 3F18D18242CE41D0:0000000000000000:8A8DC62709B1DD06:8A8CC62709B1DD07
    [  720.671343] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4
    [  720.675411] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4 exit code 0 (0x0)
    [  720.675425] block drbd4: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) 
    [  720.675449] block drbd4: Began resync as SyncTarget (will sync 9572352 KB [2393088 bits set]).
    [  720.712245] block drbd0: recounting of set bits took additional 31 jiffies
    [  720.712250] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
    [  720.712257] block drbd0: disk( Attaching -> Consistent ) 
    [  720.712272] block drbd0: attached to UUIDs 850F2703222A805E:0000000000000000:690EC328768FC354:690DC328768FC355
    [  720.725343] drbd r0: conn( StandAlone -> Unconnected ) 
    [  720.725374] drbd r0: Starting receiver thread (from drbd_w_r0 [17778])
    [  720.725438] drbd r0: receiver (re)started
    [  720.725472] drbd r0: conn( Unconnected -> WFConnection ) 
    [  721.166542] drbd r1: Handshake successful: Agreed network protocol version 101
    [  721.166548] drbd r1: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  721.166616] drbd r1: conn( WFConnection -> WFReportParams ) 
    [  721.166677] drbd r1: Starting ack_recv thread (from drbd_r_r1 [17491])
    [  721.181715] block drbd1: drbd_sync_handshake:
    [  721.181722] block drbd1: self 13FEE1D0B6C0C3B4:8ADAED18520DD8CD:92F0E1E0FF621304:92EFE1E0FF621304 bits:3470336 flags:0
    [  721.181726] block drbd1: peer 8ADAED18520DD8CC:0000000000000000:92F0E1E0FF621304:92EFE1E0FF621304 bits:0 flags:0
    [  721.181729] block drbd1: uuid_compare()=1 by rule 70
    [  721.181736] block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> Consistent ) 
    [  721.208655] block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 1307(1), total 1307; compression: 100.0%
    [  721.225692] drbd r0: Handshake successful: Agreed network protocol version 101
    [  721.225708] drbd r0: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  721.225765] drbd r0: conn( WFConnection -> WFReportParams ) 
    [  721.225794] drbd r0: Starting ack_recv thread (from drbd_r_r0 [17794])
    [  721.234923] block drbd1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 1307(1), total 1307; compression: 100.0%
    [  721.234931] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1
    [  721.238195] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
    [  721.238215] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent ) 
    [  721.238245] block drbd1: Began resync as SyncSource (will sync 13881344 KB [3470336 bits set]).
    [  721.238291] block drbd1: updated sync UUID 13FEE1D0B6C0C3B4:8ADBED18520DD8CD:8ADAED18520DD8CD:92F0E1E0FF621304
    [  721.246715] block drbd0: drbd_sync_handshake:
    [  721.246722] block drbd0: self 850F2703222A805E:0000000000000000:690EC328768FC354:690DC328768FC355 bits:0 flags:0
    [  721.246726] block drbd0: peer F6CE093812A307F8:850F2703222A805F:690EC328768FC355:690DC328768FC355 bits:3470336 flags:2
    [  721.246729] block drbd0: uuid_compare()=-1 by rule 50
    [  721.246736] block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate ) 
    [  721.269065] block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 733(1), total 733; compression: 100.0%
    [  721.290674] block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 733(1), total 733; compression: 100.0%
    [  721.290682] block drbd0: conn( WFBitMapT -> WFSyncUUID ) 
    [  721.293655] block drbd0: updated sync uuid 85102703222A805E:0000000000000000:690EC328768FC354:690DC328768FC355
    [  721.293826] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
    [  721.296840] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
    [  721.296859] block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) 
    [  721.296885] block drbd0: Began resync as SyncTarget (will sync 13881344 KB [3470336 bits set]).
    [  735.928031] block drbd1: role( Secondary -> Primary ) 
    [  736.033222] block drbd0: peer( Secondary -> Primary ) 
    [  736.571276] Rounding down aligned max_sectors from 4294967295 to 4294967288
    [  737.265274] Unable to locate Target IQN: iqn.2016-12.com.maxfurn.storage:disk2 in Storage Node
    [  737.265364] iSCSI Login negotiation failed.
    [  737.265446] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-11 is not authorized to access iSCSI target portal group: 1.
    [  737.265548] iSCSI Login negotiation failed.
    [  737.309466] Unable to locate Target IQN: iqn.2016-12.com.maxfurn.storage:disk2 in Storage Node
    [  737.309535] iSCSI Login negotiation failed.
    [  738.119559] Unable to locate Target IQN: iqn.2016-12.com.maxfurn.storage:disk2 in Storage Node
    [  738.119655] iSCSI Login negotiation failed.
    [  738.310275] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-12 is not authorized to access iSCSI target portal group: 1.
    [  738.310359] iSCSI Login negotiation failed.
    [  739.266889] Unable to locate Target IQN: iqn.2016-12.com.maxfurn.storage:disk2 in Storage Node
    [  739.266945] iSCSI Login negotiation failed.
    [  739.291844] drbd nfs: peer( Secondary -> Unknown ) conn( SyncTarget -> TearDown ) pdsk( UpToDate -> DUnknown ) 
    [  739.291865] drbd nfs: ack_receiver terminated
    [  739.291867] drbd nfs: Terminating drbd_a_nfs
    [  739.311598] Unable to locate Target IQN: iqn.2016-12.com.maxfurn.storage:disk2 in Storage Node
    [  739.311663] iSCSI Login negotiation failed.
    [  739.426590] drbd nfs: Connection closed
    [  739.426625] drbd nfs: conn( TearDown -> Unconnected ) 
    [  739.426630] drbd nfs: receiver terminated
    [  739.426632] drbd nfs: Restarting receiver thread
    [  739.426633] drbd nfs: receiver (re)started
    [  739.426656] drbd nfs: conn( Unconnected -> WFConnection ) 
    [  739.474033] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-09 is not authorized to access iSCSI target portal group: 1.
    [  739.474121] iSCSI Login negotiation failed.
    [  740.122861] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-10 is not authorized to access iSCSI target portal group: 1.
    [  740.122965] iSCSI Login negotiation failed.
    [  741.267969] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-11 is not authorized to access iSCSI target portal group: 1.
    [  741.268073] iSCSI Login negotiation failed.
    [  741.313460] iSCSI Initiator Node: iqn.2016-12.com.maxfurn.storage:virtualix-12 is not authorized to access iSCSI target portal group: 1.
    [  741.313548] iSCSI Login negotiation failed.
    [  743.954683] drbd nfs: Handshake successful: Agreed network protocol version 101
    [  743.954689] drbd nfs: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
    [  743.954751] drbd nfs: conn( WFConnection -> WFReportParams ) 
    [  743.954781] drbd nfs: Starting ack_recv thread (from drbd_r_nfs [17495])
    [  743.968711] block drbd4: drbd_sync_handshake:
    [  743.968719] block drbd4: self 3F18D18242CE41D0:0000000000000000:8A8DC62709B1DD06:8A8CC62709B1DD07 bits:2372184 flags:0
    [  743.968723] block drbd4: peer 4CF39433A70FC49B:3F18D18242CE41D1:3F17D18242CE41D1:8A8DC62709B1DD07 bits:2372386 flags:2
    [  743.968726] block drbd4: uuid_compare()=-1 by rule 50
    [  743.968729] block drbd4: Becoming sync target due to disk states.
    [  743.968736] block drbd4: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) 
    [  744.002724] block drbd4: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 524(1), total 524; compression: 100.0%
    [  744.003071] block drbd4: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 524(1), total 524; compression: 100.0%
    [  744.003078] block drbd4: conn( WFBitMapT -> WFSyncUUID ) 
    [  744.119907] block drbd4: updated sync uuid 3F19D18242CE41D0:0000000000000000:8A8DC62709B1DD06:8A8CC62709B1DD07
    [  744.122679] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4
    [  744.125675] block drbd4: helper command: /sbin/drbdadm before-resync-target minor-4 exit code 0 (0x0)
    [  744.125697] block drbd4: conn( WFSyncUUID -> SyncTarget ) 
    [  744.125723] block drbd4: Began resync as SyncTarget (will sync 9489544 KB [2372386 bits set]).
    [ 1084.771323] block drbd3: Resync done (total 364 sec; paused 0 sec; 38132 K/sec)
    [ 1084.771332] block drbd3: updated UUIDs 9F491B7BD89C8229:0000000000000000:EA26D3F5295C355B:EA25D3F5295C355B
    [ 1084.771343] block drbd3: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) 
    [ 1084.771540] drbd r3: helper command: /sbin/drbdadm unfence-peer r3
    [ 1084.774686] block drbd2: Resync done (total 364 sec; paused 0 sec; 38132 K/sec)
    [ 1084.774696] block drbd2: updated UUIDs 5876079376FF9EBC:0000000000000000:6934DB6D9FA88774:6933DB6D9FA88775
    [ 1084.774702] block drbd2: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) 
    [ 1084.774804] block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2
    [ 1084.774928] drbd r3: helper command: /sbin/drbdadm unfence-peer r3 exit code 0 (0x0)
    [ 1084.832189] block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
    [ 1437.164826] block drbd1: Resync done (total 715 sec; paused 0 sec; 19412 K/sec)
    [ 1437.164836] block drbd1: updated UUIDs 13FEE1D0B6C0C3B5:0000000000000000:8ADBED18520DD8CD:8ADAED18520DD8CD
    [ 1437.164848] block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) 
    [ 1437.165427] drbd r1: helper command: /sbin/drbdadm unfence-peer r1
    [ 1437.168855] drbd r1: helper command: /sbin/drbdadm unfence-peer r1 exit code 0 (0x0)
    [ 1493.893487] block drbd4: Resync done (total 749 sec; paused 0 sec; 12668 K/sec)
    [ 1493.893498] block drbd4: updated UUIDs 4CF39433A70FC49A:0000000000000000:3F19D18242CE41D0:3F18D18242CE41D1
    [ 1493.893507] block drbd4: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) 
    [ 1493.894051] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4
    [ 1493.948089] block drbd4: helper command: /sbin/drbdadm after-resync-target minor-4 exit code 0 (0x0)
    [ 1521.722455] block drbd0: Resync done (total 800 sec; paused 0 sec; 17348 K/sec)
    [ 1521.722465] block drbd0: updated UUIDs F6CE093812A307F8:0000000000000000:85102703222A805E:850F2703222A805F
    [ 1521.722475] block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) 
    [ 1521.722649] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0
    [ 1521.775314] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
    [ 2590.875695] perf: interrupt took too long (2523 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
    [ 3378.775251] perf: interrupt took too long (3167 > 3153), lowering kernel.perf_event_max_sample_rate to 63000
    [ 8871.075077] perf: interrupt took too long (3966 > 3958), lowering kernel.perf_event_max_sample_rate to 50000
    [38366.871324] perf: interrupt took too long (4968 > 4957), lowering kernel.perf_event_max_sample_rate to 40000
    [84898.487321] perf: interrupt took too long (6211 > 6210), lowering kernel.perf_event_max_sample_rate to 32000
    [366194.554190] md: data-check of RAID array md0
    [366194.554196] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
    [366194.554199] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
    [366194.554207] md: using 128k window, over a total of 5860391424k.
    [403153.626418] md: md0: data-check done.
    [970994.258998] md: data-check of RAID array md0
    [970994.259004] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
    [970994.259007] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
    [970994.259014] md: using 128k window, over a total of 5860391424k.
    [1010509.452051] md: md0: data-check done.
    [1575794.166186] md: data-check of RAID array md0
    [1575794.166192] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
    [1575794.166194] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
    [1575794.166201] md: using 128k window, over a total of 5860391424k.
    [1616029.452122] md: md0: data-check done.
    [1817589.635500] ------------[ cut here ]------------
    [1817589.635544] kernel BUG at drivers/nvme/host/pci.c:467!
    [1817589.635584] invalid opcode: 0000 [#1] SMP 
    [1817589.635621] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod drbd(OE) bonding dm_cache_smq dm_cache dm_persistent_data dm_bio_prison dm_bufio raid1 dm_raid vfat fat intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac i2c_i801 sg edac_core mei_me mei lpc_ich shpchp wmi ipmi_devintf ipmi_si acpi_power_meter ipmi_msghandler acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt
    [1817589.636241]  fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci uas libahci ixgbe libata usb_storage i2c_core mdio ptp nvme pps_core dca fjes dm_mirror dm_region_hash dm_log dm_mod
    [1817589.636399] CPU: 5 PID: 22123 Comm: kworker/u32:1 Tainted: G           OE  ------------   3.10.0-514.10.2.el7.x86_64 #1
    [1817589.636473] Hardware name: Supermicro SYS-6028U-TNR4T+/X10DRU-i+, BIOS 2.0b 08/09/2016
    [1817589.636542] Workqueue: drbd1_submit do_submit [drbd]
    [1817589.636581] task: ffff88046ba23ec0 ti: ffff88036bfa8000 task.ti: ffff88036bfa8000
    [1817589.636634] RIP: 0010:[<ffffffffa0062fd8>]  [<ffffffffa0062fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
    [1817589.639123] RSP: 0018:ffff88036bfaba10  EFLAGS: 00010286
    [1817589.641528] RAX: 0000000000000018 RBX: 00000000fffffc00 RCX: 0000000000003c00
    [1817589.643953] RDX: ffffea000dbecec0 RSI: ffff88003f650100 RDI: ffff880077aae460
    [1817589.646382] RBP: ffff88036bfabae8 R08: 0000000000001000 R09: 000000036fb3c000
    [1817589.648796] R10: 0000000000001000 R11: ffff880077aae480 R12: ffff88003f650100
    [1817589.651164] R13: 0000000000000004 R14: 0000000000001000 R15: 0000000000000c00
    [1817589.653499] FS:  0000000000000000(0000) GS:ffff88046fd40000(0000) knlGS:0000000000000000
    [1817589.655803] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [1817589.658106] CR2: 000000000230e230 CR3: 00000000019ba000 CR4: 00000000003407e0
    [1817589.660420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [1817589.662691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [1817589.664923] Stack:
    [1817589.667115]  ffff880179c4b800 ffff88046bf6cb80 ffff88046ba77f00 ffff880077aae4c0
    [1817589.669346]  ffff88046a43b400 ffff88046c641180 0000000376413000 ffff880800004c00
    [1817589.671515]  ffff880071368850 0000000000001000 ffff880000000200 ffffffff00000001
    [1817589.673692] Call Trace:
    [1817589.675829]  [<ffffffff812f8a5a>] __blk_mq_run_hw_queue+0x1fa/0x3c0
    [1817589.677972]  [<ffffffff812f8835>] blk_mq_run_hw_queue+0xa5/0xd0
    [1817589.680099]  [<ffffffff812f9b3b>] blk_mq_insert_requests+0xcb/0x160
    [1817589.682190]  [<ffffffff812fa81d>] blk_mq_flush_plug_list+0xbd/0x160
    [1817589.684237]  [<ffffffff812f0059>] blk_flush_plug_list+0xc9/0x230
    [1817589.686236]  [<ffffffff812f0574>] blk_finish_plug+0x14/0x40
    [1817589.688230]  [<ffffffffa06c3de3>] send_and_submit_pending+0xa3/0xd0 [drbd]
    [1817589.690220]  [<ffffffffa06c4917>] do_submit+0x397/0x4c0 [drbd]
    [1817589.692143]  [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30
    [1817589.694059]  [<ffffffff810a810b>] process_one_work+0x17b/0x470
    [1817589.695937]  [<ffffffff810a8f46>] worker_thread+0x126/0x410
    [1817589.697767]  [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
    [1817589.699592]  [<ffffffff810b06ff>] kthread+0xcf/0xe0
    [1817589.701378]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
    [1817589.703130]  [<ffffffff81696a58>] ret_from_fork+0x58/0x90
    [1817589.704874]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
    [1817589.706623] Code: 1f 44 00 00 e9 32 f9 ff ff 8b 73 28 f0 66 83 43 28 02 f6 43 2a 01 74 e5 48 8b 7d 80 e8 d5 41 00 00 eb da 41 bd 02 00 00 00 eb c8 <0f> 0b 4c 8b 0d 7f 18 97 e1 e9 33 ff ff ff 66 2e 0f 1f 84 00 00 
    [1817589.710404] RIP  [<ffffffffa0062fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
    [1817589.712217]  RSP <ffff88036bfaba10>
    
    txt file icon vmcore-dmesg.txt (147,093 bytes) 2017-03-30 13:33 +
  • txt file icon vmcore-dmesg-2.txt (4,365 bytes) 2017-03-31 15:05 -
    [  390.030908] ------------[ cut here ]------------
    [  390.030938] kernel BUG at drivers/nvme/host/pci.c:467!
    [  390.030961] invalid opcode: 0000 [#1] SMP 
    [  390.030982] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun ebtable_filter ebtables mpt3sas mpt2sas raid_class mptctl mptbase 8021q garp mrp bridge stp llc bonding ipt_REJECT ip6t_REJECT nf_reject_ipv4 nf_reject_ipv6 nf_conntrack_ipv4 nf_conntrack_ipv6 nf_defrag_ipv4 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure scsi_transport_sas sb_edac edac_core sg mei_me i2c_i801 mei ioatdma shpchp lpc_ich ipmi_ssif ipmi_si ipmi_msghandler wmi dm_multipath acpi_power_meter acpi_pad dm_mod nfsd
    [  390.031364]  drbd(OE) auth_rpcgss nfs_acl lockd binfmt_misc libcrc32c grace sunrpc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm drm megaraid_sas nvme ahci libahci ixgbe i40e i2c_core libata mdio ptp dca pps_core fjes
    [  390.031550] CPU: 4 PID: 4105 Comm: drbd_r_seafile- Tainted: G           OE  ------------   3.10.0-514.10.2.el7.x86_64 #1
    [  390.031591] Hardware name: Supermicro X10DRH/X10DRH-IT, BIOS 2.0a 06/30/2016
    [  390.031619] task: ffff883fdb66af10 ti: ffff881fe4e40000 task.ti: ffff881fe4e40000
    [  390.031649] RIP: 0010:[<ffffffffa0370fd8>]  [<ffffffffa0370fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
    [  390.031693] RSP: 0018:ffff881fe4e43b38  EFLAGS: 00010286
    [  390.031715] RAX: 0000000000000000 RBX: 00000000ffffe800 RCX: 0000000000002600
    [  390.031743] RDX: 0000003ff3faa200 RSI: ffff883ff3faa200 RDI: 0000000000000246
    [  390.031770] RBP: ffff881fe4e43c10 R08: 0000000000001000 R09: 0000001ffdc33000
    [  390.031798] R10: 00000000fffff800 R11: ffff881ff17ed980 R12: ffff883ff3faa200
    [  390.031826] R13: 0000000000000001 R14: 0000000000001000 R15: 0000000000002600
    [  390.031854] FS:  0000000000000000(0000) GS:ffff881fffb00000(0000) knlGS:0000000000000000
    [  390.031885] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [  390.031908] CR2: 00007f5c8ae99e00 CR3: 00000000019ba000 CR4: 00000000001407e0
    [  390.031936] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [  390.031964] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [  390.031991] Stack:
    [  390.032002]  ffff883ffd423c00 ffff883ffa9d6b80 ffff88407ff02300 ffff881ff17eda20
    [  390.032037]  ffff883ff8821d40 ffff88407feb2100 0000001ffdc32000 ffff882000003600
    [  390.032072]  ffff883ffa8e63c0 0000000000001000 ffff881f00000200 ffff884000000001
    [  390.032107] Call Trace:
    [  390.032125]  [<ffffffff812f8a5a>] __blk_mq_run_hw_queue+0x1fa/0x3c0
    [  390.032153]  [<ffffffff812f8835>] blk_mq_run_hw_queue+0xa5/0xd0
    [  390.032178]  [<ffffffff812f9b3b>] blk_mq_insert_requests+0xcb/0x160
    [  390.032203]  [<ffffffff812fa89b>] blk_mq_flush_plug_list+0x13b/0x160
    [  390.032230]  [<ffffffff812f0059>] blk_flush_plug_list+0xc9/0x230
    [  390.032255]  [<ffffffff812f0574>] blk_finish_plug+0x14/0x40
    [  390.032288]  [<ffffffffa02dc158>] drbd_unplug_all_devices+0x38/0x50 [drbd]
    [  390.032320]  [<ffffffffa02dc58f>] receive_UnplugRemote+0x4f/0x70 [drbd]
    [  390.032352]  [<ffffffffa02eb210>] drbd_receiver+0x150/0x350 [drbd]
    [  390.032384]  [<ffffffffa02f6500>] ? drbd_destroy_connection+0x160/0x160 [drbd]
    [  390.032417]  [<ffffffffa02f651d>] drbd_thread_setup+0x1d/0x110 [drbd]
    [  390.032448]  [<ffffffffa02f6500>] ? drbd_destroy_connection+0x160/0x160 [drbd]
    [  390.032479]  [<ffffffff810b06ff>] kthread+0xcf/0xe0
    [  390.032501]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
    [  390.032530]  [<ffffffff81696a58>] ret_from_fork+0x58/0x90
    [  390.032553]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
    [  390.032579] Code: 1f 44 00 00 e9 32 f9 ff ff 8b 73 28 f0 66 83 43 28 02 f6 43 2a 01 74 e5 48 8b 7d 80 e8 d5 41 00 00 eb da 41 bd 02 00 00 00 eb c8 <0f> 0b 4c 8b 0d 7f 38 66 e1 e9 33 ff ff ff 66 2e 0f 1f 84 00 00 
    [  390.032746] RIP  [<ffffffffa0370fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
    [  390.032779]  RSP <ffff881fe4e43b38>
    
    txt file icon vmcore-dmesg-2.txt (4,365 bytes) 2017-03-31 15:05 +

-Relationships
+Relationships

-Notes

~0028980

bvdheuvel (reporter)

[1817589.635500] ------------[ cut here ]------------
[1817589.635544] kernel BUG at drivers/nvme/host/pci.c:467!
[1817589.635584] invalid opcode: 0000 [#1] SMP
[1817589.635621] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod drbd(OE) bonding dm_cache_smq dm_cache dm_persistent_data dm_bio_prison dm_bufio raid1 dm_raid vfat fat intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac i2c_i801 sg edac_core mei_me mei lpc_ich shpchp wmi ipmi_devintf ipmi_si acpi_power_meter ipmi_msghandler acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt
[1817589.636241] fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci uas libahci ixgbe libata usb_storage i2c_core mdio ptp nvme pps_core dca fjes dm_mirror dm_region_hash dm_log dm_mod
[1817589.636399] CPU: 5 PID: 22123 Comm: kworker/u32:1 Tainted: G OE ------------ 3.10.0-514.10.2.el7.x86_64 #1
[1817589.636473] Hardware name: Supermicro SYS-6028U-TNR4T+/X10DRU-i+, BIOS 2.0b 08/09/2016
[1817589.636542] Workqueue: drbd1_submit do_submit [drbd]
[1817589.636581] task: ffff88046ba23ec0 ti: ffff88036bfa8000 task.ti: ffff88036bfa8000
[1817589.636634] RIP: 0010:[<ffffffffa0062fd8>] [<ffffffffa0062fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
[1817589.639123] RSP: 0018:ffff88036bfaba10 EFLAGS: 00010286
[1817589.641528] RAX: 0000000000000018 RBX: 00000000fffffc00 RCX: 0000000000003c00
[1817589.643953] RDX: ffffea000dbecec0 RSI: ffff88003f650100 RDI: ffff880077aae460
[1817589.646382] RBP: ffff88036bfabae8 R08: 0000000000001000 R09: 000000036fb3c000
[1817589.648796] R10: 0000000000001000 R11: ffff880077aae480 R12: ffff88003f650100
[1817589.651164] R13: 0000000000000004 R14: 0000000000001000 R15: 0000000000000c00
[1817589.653499] FS: 0000000000000000(0000) GS:ffff88046fd40000(0000) knlGS:0000000000000000
[1817589.655803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1817589.658106] CR2: 000000000230e230 CR3: 00000000019ba000 CR4: 00000000003407e0
[1817589.660420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1817589.662691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1817589.664923] Stack:
[1817589.667115] ffff880179c4b800 ffff88046bf6cb80 ffff88046ba77f00 ffff880077aae4c0
[1817589.669346] ffff88046a43b400 ffff88046c641180 0000000376413000 ffff880800004c00
[1817589.671515] ffff880071368850 0000000000001000 ffff880000000200 ffffffff00000001
[1817589.673692] Call Trace:
[1817589.675829] [<ffffffff812f8a5a>] __blk_mq_run_hw_queue+0x1fa/0x3c0
[1817589.677972] [<ffffffff812f8835>] blk_mq_run_hw_queue+0xa5/0xd0
[1817589.680099] [<ffffffff812f9b3b>] blk_mq_insert_requests+0xcb/0x160
[1817589.682190] [<ffffffff812fa81d>] blk_mq_flush_plug_list+0xbd/0x160
[1817589.684237] [<ffffffff812f0059>] blk_flush_plug_list+0xc9/0x230
[1817589.686236] [<ffffffff812f0574>] blk_finish_plug+0x14/0x40
[1817589.688230] [<ffffffffa06c3de3>] send_and_submit_pending+0xa3/0xd0 [drbd]
[1817589.690220] [<ffffffffa06c4917>] do_submit+0x397/0x4c0 [drbd]
[1817589.692143] [<ffffffff810b17d0>] ? wake_up_atomic_t+0x30/0x30
[1817589.694059] [<ffffffff810a810b>] process_one_work+0x17b/0x470
[1817589.695937] [<ffffffff810a8f46>] worker_thread+0x126/0x410
[1817589.697767] [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
[1817589.699592] [<ffffffff810b06ff>] kthread+0xcf/0xe0
[1817589.701378] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1817589.703130] [<ffffffff81696a58>] ret_from_fork+0x58/0x90
[1817589.704874] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1817589.706623] Code: 1f 44 00 00 e9 32 f9 ff ff 8b 73 28 f0 66 83 43 28 02 f6 43 2a 01 74 e5 48 8b 7d 80 e8 d5 41 00 00 eb da 41 bd 02 00 00 00 eb c8 <0f> 0b 4c 8b 0d 7f 18 97 e1 e9 33 ff ff ff 66 2e 0f 1f 84 00 00
[1817589.710404] RIP [<ffffffffa0062fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
[1817589.712217] RSP <ffff88036bfaba10>

~0028995

geraldh (reporter)

Hello,
same problem here - we can reproduce the crash - but the setup is somewhat complicated.

We mirror a LVM volume on NVME devices (intel p3700) with drbd (8.4 and 9 - same effect) from ServerA to ServerB

On ServerA we start a KVM centos7 instance and give the drbd-mirrored volume to this VM. When we start a fio random write test inside this virtual centos7 machine ServerB always chrashes within minutes.

This chrash does not occur, if we place the mirrored volume on spinnig disks (LSI megaraid)

We tested with drdb version 8.4.8 and 9.0.6 - no difference - but no problem on LSI megaraid controller.

We should have put this setup into production weeks ago - so if anybody has some sugestions - please tell us.

Thanks

~0028996

bvdheuvel (reporter)

Hello,

We have a storage cluster of 2 nodes with a raid5 set of spinning disks, and a raid-1 set of nvme disks (2 per node, also intel p3700 (maybe a hint I guess)).
We use the mirrored nvme set with dm-cache for the raid5 spinning disk set.
We did a lot of testing with kvm virtuals (and on the kvm nodes) and found no problems then.

This is info from the nvme list command:

[root@storage-01 ~]# nvme list
Node SN Model Version Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- -------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 CVPF6341008Q1P2NGN INTEL SSDPE2MX012T7 1.0 1 1,20 TB / 1,20 TB 512 B + 0 B MDV10253
/dev/nvme1n1 CVPF635100DK1P2NGN INTEL SSDPE2MX012T7 1.0 1 1,20 TB / 1,20 TB 512 B + 0 B MDV10253

~0028997

bvdheuvel (reporter)

Forgot to mention that we also use drbd between the two nodes:

version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48

~0028998

TrevorH (developer)

No idea if this might fix it but do you run with

options drbd disable_sendpage=1

?

~0029001

geraldh (reporter)

@0028996,@0028997

we had a similar setup but used our two p3700 cards (400GB) in striped mode. While testing the io performance with fio inside the vm we encountered the kernel crashes.

To ease to testing we removed our p3700 SSDs from dm-cache and created a separate NVME volume group.

~0029002

geraldh (reporter)

@0028998

options drbd disable_sendpage=1

does not help unfortunately.

~0029003

TrevorH (developer)

geraldh: in 0029001 you say you recreated with the 2 P3700s in striped mode and got the crash testing with fio. Was that with or without DRBD in the picture?

~0029005

geraldh (reporter)

Status after some more tests:

* when using zfs inside the centos 7 VM as basis for the fio tests - the kernel on ServerB crashes within 1-2 minutes

* xfs takes longer to trigger a kernel crash on ServerB

* using virtio-disk or virtio-scsi as driver for the VM disk makes not difference

* ServerB (and ServerA) did not crash if drbd on ServerB is disconnected and later reconnected

* ServerB did not crash if I was running the fio tests inside the VM on the raw device

* ServerB did not crash if I was running the fio tests outside the VM on the raw drbd device

Somehow the combination of filesystem, kvm and drbd triggers the crash

~0029009

geraldh (reporter)

Retested today with DRBD 9.0.6

* using the DRBD NVME volume inside the CentOS7 VM formatted with ZFS for the fio test always crashes the remote Server (ServerB)

* using the NVME volume without DRDB inside the CentOS7 VM formatted with ZFS - works

All other tests (xfs, ext4, raw) did not crash any Server today.

So DRBD (over NVME) is definitely involved in this crash.

I'll post to the DRDB mailing list.

~0029010

geraldh (reporter)

fio test used:

fio --filename=/ztest/50G --size=50G --ioengine=libaio --direct=0 --norandommap --randrepeat=0 -runtime=300 --blocksize=4K --rw=randwrite --iodepth=32 --numjobs=8 --group_reporting --name=write_rnd

~0029024

ChristiaanB (reporter)

Hello,

Same problem here. My production file-server just crashed with kernel BUG at drivers/nvme/host/pci.c:467!
Similar setup:
- Two P3700 800Gb in software raid 1.
- DRBD between serverA and serverB
- LVM on top.

DRBD version: 8.4.9-1 (api:1/proto:86-101)

Any solution to fix this bug. Please help.
Thanks

~0029202

ChristiaanB (reporter)

@TrevorH, Any progress yet?
@geraldh, in 0029009 you created a setup that always crashed the remote ServerB.
I tried a similar setup, but I am unable to reproduce the kernel panic.
Can you tell me if the bug still exists in the latest kernel 3.10.0-514.16.1.el7.x86_64?
Did you try the fio test with a newer kernel. You could try it with the latest Oracle's Unbreakable Enterprise Kernel (with 4k device page size fix) or latest ELRepo kernel-ml?
I would appreciate it if you could test this, so that I can add the second server again.
Thanks.

~0029210

szebel (reporter)

Hello,

Same bug appears here too. Its a Dell OptiPlex 5040 with a single Hynix M.2 NVME. DRBD is not in use!

The bug triggered after the workstation was updated to 3.10.0-514.16.1.el7.x86_64 this morning - five times in 9 hours.

Sebastian

--- cut here ---
:kernel BUG at drivers/nvme/host/pci.c:467!
:invalid opcode: 0000 [#1] SMP
:Modules linked in: fuse ext4 mbcache jbd2 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) snd_hda_codec_hdmi dell_led snd_hda_codec_realtek intel_powerclamp coretemp snd_hda_codec_generic intel_rapl iosf_mbi dell_wmi sparse_keymap kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel dcdbas snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul snd_hwdep glue_helper ablk_helper snd_seq cryptd snd_seq_device snd_pcm snd_timer snd i2c_i801 soundcore pcspkr shpchp mei_me mei wmi tpm_crb acpi_pad nfsd auth_rpcgss nfs_acl lockd binfmt_misc grace sunrpc ip_tables xfs libcrc32c crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i915 i2c_algo_bit drm_kms_helper ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e
: libata drm ptp pps_core nvme video i2c_hid i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
:CPU: 0 PID: 10818 Comm: AioMgr0-N Tainted: G OE ------------ 3.10.0-514.16.1.el7.x86_64 #1
:Hardware name: Dell Inc. OptiPlex 5040/0T7D40, BIOS 1.5.9 02/06/2017
:task: ffff88040513ce70 ti: ffff8803e26dc000 task.ti: ffff8803e26dc000
:RIP: 0010:[<ffffffffa0061fd8>] [<ffffffffa0061fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
:RSP: 0018:ffff8803e26df7d8 EFLAGS: 00010286
:RAX: 0000000000000000 RBX: 00000000ffffe400 RCX: 0000000000002400
:RDX: 00000001cbe7c400 RSI: ffff8801cbe7c400 RDI: 0000000000000246
:RBP: ffff8803e26df8b0 R08: 0000000000001000 R09: 000000025b614000
:R10: 00000000fffff400 R11: ffff8803f8b810c0 R12: ffff8801cbe7c400
:R13: 0000000000000001 R14: 0000000000001000 R15: 0000000000002400
:FS: 00007fe4167b2700(0000) GS:ffff88041dc00000(0000) knlGS:0000000000000000
:CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
:CR2: fffff8a00a896818 CR3: 00000000d0f4a000 CR4: 00000000003427f0
:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
:Stack:
: ffff880179d76400 ffff880408314d80 ffff880408ba5700 ffff8803f8b81160
: ffff88040777f780 ffff880408aec080 000000025b613000 ffff880400003400
: ffff880408bf8000 0000000000001000 ffff880300000200 ffff880300000001
:Call Trace:
: [<ffffffff812fa8f2>] blk_mq_make_request+0x222/0x440
: [<ffffffff812eeb89>] generic_make_request+0x109/0x1e0
: [<ffffffff812eecd1>] submit_bio+0x71/0x150
: [<ffffffff8123ef12>] do_blockdev_direct_IO+0x1dc2/0x2020
: [<ffffffffa091bc30>] ? ext4_block_zero_page_range+0x3e0/0x3e0 [ext4]
: [<ffffffff8123f1c5>] __blockdev_direct_IO+0x55/0x60
: [<ffffffffa091bc30>] ? ext4_block_zero_page_range+0x3e0/0x3e0 [ext4]
: [<ffffffffa0918c40>] ? do_journal_get_write_access+0xa0/0xa0 [ext4]
: [<ffffffffa091ad60>] ext4_direct_IO+0x520/0x5d0 [ext4]
: [<ffffffffa091bc30>] ? ext4_block_zero_page_range+0x3e0/0x3e0 [ext4]
: [<ffffffffa0918c40>] ? do_journal_get_write_access+0xa0/0xa0 [ext4]
: [<ffffffff81182e6d>] generic_file_direct_write+0xcd/0x190
: [<ffffffff81183167>] __generic_file_aio_write+0x237/0x400
: [<ffffffffa0915055>] ext4_file_write+0x225/0x470 [ext4]
: [<ffffffffa0914e30>] ? ext4_dax_fault+0x160/0x160 [ext4]
: [<ffffffff8124fe58>] do_io_submit+0x3b8/0x870
: [<ffffffff81250320>] SyS_io_submit+0x10/0x20
: [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
:Code: 1f 44 00 00 e9 32 f9 ff ff 8b 73 28 f0 66 83 43 28 02 f6 43 2a 01 74 e5 48 8b 7d 80 e8 d5 41 00 00 eb da 41 bd 02 00 00 00 eb c8 <0f> 0b 4c 8b 0d bf 28 97 e1 e9 33 ff ff ff 66 2e 0f 1f 84 00 00
:RIP [<ffffffffa0061fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
: RSP <ffff8803e26df7d8>

~0029213

geraldh (reporter)

@ChristiaanB

for me it was reproducable as soon as I created a ZFS pool (inside a VM) on the striped p3700 DRBD resource and used this ZFS pool for fio tests.

But I do not have access to my test servers at the moment - they are in production
already (we mirror inside the VMs instead of DRBD in the backend)

The "4k device page size fix" could help - Lars Ellenberg from DRBD mentioned this fix also (http://lists.linbit.com/pipermail/drbd-user/2017-April/023202.html)

I will try again with differnet kernels in a maintenance window.

~0029215

ChristiaanB (reporter)

@geraldh

I have two test servers available with two P3700 800Gb each.
I use a 20Gb bonding link between the servers for DRBD synchronization.

Did you use the default 512K or other chunk size in creating the sriped p3700 resource?

Can you post your DRBD resource configuration?

Did you create the ZFS pool inside the VM on the raw DRBD resource?
(-drive file=/dev/drbd0,format=raw,if=none,id=drive-virtio-disk1)

Can you tell me the commands used in creating the ZFS pool.

Thanks.

~0029216

mdslimited (reporter)

I also have this issue. Kernel 3.10.0-514.16.1.el7.x86_64. I am using ZFS on linux (ZFS: Loaded module v0.6.5.9-1, ZFS pool version 5000, ZFS filesystem version 5). I use DRBD 9 for replication (kmod-drbd-9.0.5_3.10.0_514.2.2). There is a single pool with 4 drives and two NVME drives acting as the ZIL and LOG devices. I have created DRBD resources on the ZFS file system and can provide the configuration details if it would be helpful. I have documented how everything was set up and this is also available if it would help.

~0029217

ChristiaanB (reporter)

@mdslimited

Did you test with newer kernels?
Can you reproduce the issue?
If easily reproducible, can you post or upload your configuration details?

~0029218

mdslimited (reporter)

The kernel mentioned is the only one in use. Is there a specific kernel version that I should try?

The issue occurred once only after three days' uptime and this is a production server so it is not desirable to stress test it in an attempt to repeat the issue.

If it will help with diagnosis, however, I am happy to make the system available for testing as I have alternative storage servers available to use while assisting.

~0029219

ChristiaanB (reporter)

@mdslimited

If it isn't easily reproducible, I cannot test other kernels.
I hope I can setup the same configuration as setup by geraldh.

You could try the latest Oracle's Unbreakable Enterprise Kernel(with 4k device page size fix).

~0029225

geraldh (reporter)

@ChristiaanB

my test setup:

2x supermicro X10 server with 2x p3700 each connected with 40gbit (Intel x710) DRBD link

I've created a striped 512GB LV on the two p3700 on each Server
vgcreate -s 64m NVMEVG /dev/nvme0n1 /dev/nvme1n1
lvcreate -n cache -L 512g -I 16 -i2 NVMEVG

DRBD is used to mirror this LV to server B

This striped replicated DRBD device is given to a VM as scsi disk (<target dev='sdb' bus='scsi'/>)

Inside the VM (centos 7) I install zfsonlinox and create a zfs pool on the DRBD device

Runnig fio in this zfs pool has always crashed server B within minutes

fio --filename=/ztest/50G --size=50G --ioengine=libaio --direct=0 --norandommap --randrepeat=0 -runtime=300 --blocksize=4K --rw=randwrite --iodepth=32 --numjobs=8 --group_reporting --name=write_rnd

Hope this helps.

~0029228

ChristiaanB (reporter)

I was able to reproduce the BUG with a similar setup as geraldh.

Two Supermicro X10 servers with two P3700 800Gb connected with 20gbit DRBD link
- File-system on first partition RAID0 /dev/nvme0n1p1 /dev/nvme1n1p1
- Created a striped 512GB LV on the two P3700 on each Server
  vgcreate -s 64m NVMEVG /dev/nvme0n1p3 /dev/nvme1n1p3
  lvcreate -n cache -L 512g -I 16 -i2 NVMEVG
- DRBD used to mirror LV to server B

On serverA:
# drbdadm primary --force drbd0
# cat /proc/drbd
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by root@serverA, 2017-05-02 13:50:56
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:15699932 nr:0 dw:0 dr:15700572 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
# zpool create -f storage /dev/drbd0
# zfs create storage/test
# cd /storage/test
# fio --filename=50G --size=50G --ioengine=libaio --direct=0 --norandommap --randrepeat=0 -runtime=300 --blocksize=4K --rw=randwrite --iodepth=32 --numjobs=8 --group_reporting --name=write_rnd

With kernel 3.10.0-514.10.2.el7 crash with kernel BUG at drivers/nvme/host/pci.c:467! within a minute, first serverA and then serverB

With kernel 4.1.12-94.2.1.el7uek.x86_64 no kernel crash.

I will change to the Oracle Unbreakable Enterprise Kernel on my servers.
Hope this solves the issue.

~0029230

TrevorH (developer)

While this discussion may be useful to debug and track down where the problem resides, so far, as far as I can see, we've narrowed it down to only happening if the distro kernel is in use and DRBD is loaded. That means that if it is a kernel bug, since CentOS is a rebuild of RHEL, any actual *fix* has to come from Redhat and has to be made to the upstream kernel. OTOH, if it's a DRBD bug then the fix has to come from Linbit. If it works on CentOS in the same way that it works on RHEL of the same version then it's not a bug in CentOS. CentOS bugs are only if the behaviour differs from RHEL.

Has anyone raised a bugzilla entry for this and is this discussion being repeated/reported there?

~0029234

mdslimited (reporter)

ChristiaanB: I would be interested if using the Oracle Unbreakable Enterprise Kernel does not exhibit the problem. Would you mind updating this thread with your findings? I imagine it would provide useful feedback to anyone else finding this thread.

TrevorH: If the problem does not exhibit itself on the Oracle kernel, would it make it more difficult for upstream kernel developers to argue that the issue is with DRBD? In my use-case, though the DRBD kernel modules are present, DRBD itself is interacting with ZFS on Linux (though I appreciate that the kernel module could interact with the kernel in an undesirable way).

Is there anything (as users) that we can be doing that would assist in determining the root cause?

~0029236

szebel (reporter)

I could reproduce the error at another workstation. Again, without using DRBD. So either this bug is not related to DRBD or it is present at multiple parts of the kernel.

Up to now the Bug was triggered only if a VirtualBox VM is running. Doing heavy IO without the VM running the workstation works just fine.

:[ 4372.199180] ------------[ cut here ]------------
:[ 4372.199194] kernel BUG at drivers/nvme/host/pci.c:467!
:[ 4372.199204] invalid opcode: 0000 [#1] SMP
:[ 4372.199214] Modules linked in: fuse vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache snd_hda_codec_hdmi dell_led snd_hda_codec_realtek snd_hda_codec_generic intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass crc32_pclmul iTCO_vendor_support dell_wmi sparse_keymap snd_hda_intel ghash_clmulni_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device aesni_intel dcdbas lrw gf128mul glue_helper ablk_helper cryptd snd_pcm pcspkr snd_timer snd soundcore mei_me shpchp mei i2c_i801 wmi acpi_pad tpm_crb binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c i915 i2c_algo_bit drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common e1000e crc32c_intel serio_raw ahci sysfillrect sysimgblt libahci
:[ 4372.199394] fb_sys_fops ptp pps_core drm libata nvme video i2c_hid i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
:[ 4372.199422] CPU: 3 PID: 26379 Comm: AioMgr0-N Tainted: G OE ------------ 3.10.0-514.16.1.el7.x86_64 #1
:[ 4372.199442] Hardware name: Dell Inc. Precision Tower 3420/02K9CR, BIOS 2.1.9 01/24/2017
:[ 4372.199457] task: ffff880417bcde20 ti: ffff8803fdfec000 task.ti: ffff8803fdfec000
:[ 4372.199470] RIP: 0010:[<ffffffffa0061fd8>] [<ffffffffa0061fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
:[ 4372.199492] RSP: 0018:ffff8803fdfef888 EFLAGS: 00010286
:[ 4372.199502] RAX: 0000000000000008 RBX: 00000000fffffc00 RCX: 0000000000002c00
:[ 4372.199515] RDX: ffffea0010900000 RSI: ffff88040eadbe00 RDI: ffff880432607240
:[ 4372.199529] RBP: ffff8803fdfef960 R08: 0000000000001000 R09: 0000000424001400
:[ 4372.199542] R10: 0000000000001000 R11: ffff880432607260 R12: ffff88040eadbe00
:[ 4372.199555] R13: 0000000000000002 R14: 0000000000001000 R15: 0000000000001c00
:[ 4372.199568] FS: 00007f00927fc700(0000) GS:ffff880455d80000(0000) knlGS:0000000000000000
:[ 4372.199584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
:[ 4372.199595] CR2: 000007feee50d224 CR3: 00000004559b6000 CR4: 00000000003427e0
:[ 4372.199608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:[ 4372.199621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
:[ 4372.199634] Stack:
:[ 4372.199638] ffff880440bbf800 ffff8800358ab380 ffff8800358ab800 ffff8804326072e0
:[ 4372.199655] ffff880035aa0d80 ffff880455f18500 00000003eb7e9000 ffff880400003c00
:[ 4372.199671] ffff880035a30850 0000000000001000 ffff880400000200 ffff880000000002
:[ 4372.199687] Call Trace:
:[ 4372.199695] [<ffffffff812fa8f2>] blk_mq_make_request+0x222/0x440
:[ 4372.199708] [<ffffffff812eeb89>] generic_make_request+0x109/0x1e0
:[ 4372.199720] [<ffffffff812eecd1>] submit_bio+0x71/0x150
:[ 4372.199731] [<ffffffff8118c092>] ? set_page_dirty_lock+0x32/0x50
:[ 4372.199744] [<ffffffff8123ef12>] do_blockdev_direct_IO+0x1dc2/0x2020
:[ 4372.199768] [<ffffffffa0370d90>] ? xfs_get_blocks+0x20/0x20 [xfs]
:[ 4372.199780] [<ffffffff8123f1c5>] __blockdev_direct_IO+0x55/0x60
:[ 4372.199800] [<ffffffffa0370d90>] ? xfs_get_blocks+0x20/0x20 [xfs]
:[ 4372.199819] [<ffffffffa0370e21>] xfs_vm_direct_IO+0x71/0xf0 [xfs]
:[ 4372.199838] [<ffffffffa0370d90>] ? xfs_get_blocks+0x20/0x20 [xfs]
:[ 4372.199852] [<ffffffff81182d1c>] generic_file_aio_read+0x70c/0x790
:[ 4372.199864] [<ffffffff8124fc21>] ? do_io_submit+0x181/0x870
:[ 4372.199877] [<ffffffff810295ec>] ? __switch_to+0x15c/0x4c0
:[ 4372.199889] [<ffffffff8168b2e2>] ? down_read+0x12/0x30
:[ 4372.199908] [<ffffffffa037bcad>] xfs_file_aio_read+0x18d/0x340 [xfs]
:[ 4372.199928] [<ffffffffa037bb20>] ? xfs_file_splice_read+0x1b0/0x1b0 [xfs]
:[ 4372.199942] [<ffffffff8124fe58>] do_io_submit+0x3b8/0x870
:[ 4372.199953] [<ffffffff81250320>] SyS_io_submit+0x10/0x20
:[ 4372.199964] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
:[ 4372.199975] Code: 1f 44 00 00 e9 32 f9 ff ff 8b 73 28 f0 66 83 43 28 02 f6 43 2a 01 74 e5 48 8b 7d 80 e8 d5 41 00 00 eb da 41 bd 02 00 00 00 eb c8 <0f> 0b 4c 8b 0d bf 28 97 e1 e9 33 ff ff ff 66 2e 0f 1f 84 00 00
:[ 4372.200051] RIP [<ffffffffa0061fd8>] nvme_queue_rq+0xa58/0xa70 [nvme]
:[ 4372.200067] RSP <ffff8803fdfef888>

~0029289

dritz (reporter)

Hi Guys, we have this same problem without using DRDB or ZFS.
Our stack :

On top : bcache -> 2 x nvme in software raid 1 and mpathc from disk array

Kernel version : 3.10.0-514.16.1.el7.x86_64

Test params : fio --name="xxxx" --numjobs=30 --runtime=10 --rw=randwrite --ioengine=libaio --direct=0 --filename=/tmp/io_test/bcache/fio_test --size=1024M --group_reporting

Result : PANIC: "kernel BUG at drivers/nvme/host/pci.c:467!"

~0029290

TrevorH (developer)

If you can recreate without DRBD in the mix then definitely report this on bugzilla.redhat.com
+Notes

-Issue History
Date Modified Username Field Change
2017-03-30 13:33 bvdheuvel New Issue
2017-03-30 13:33 bvdheuvel File Added: vmcore-dmesg.txt
2017-03-30 13:41 bvdheuvel Note Added: 0028980
2017-03-31 15:05 geraldh File Added: vmcore-dmesg-2.txt
2017-03-31 15:05 geraldh Note Added: 0028995
2017-03-31 15:17 bvdheuvel Note Added: 0028996
2017-03-31 15:18 bvdheuvel Note Added: 0028997
2017-03-31 15:28 TrevorH Note Added: 0028998
2017-03-31 21:09 geraldh Note Added: 0029001
2017-03-31 21:11 geraldh Note Added: 0029002
2017-03-31 22:32 TrevorH Note Added: 0029003
2017-04-01 00:03 geraldh Note Added: 0029005
2017-04-01 16:42 toracat Status new => assigned
2017-04-01 21:03 geraldh Note Added: 0029009
2017-04-01 21:04 geraldh Note Added: 0029010
2017-04-05 15:30 ChristiaanB Note Added: 0029024
2017-05-02 09:47 ChristiaanB Note Added: 0029202
2017-05-02 12:56 szebel Note Added: 0029210
2017-05-02 20:47 geraldh Note Added: 0029213
2017-05-03 08:23 ChristiaanB Note Added: 0029215
2017-05-03 08:34 mdslimited Note Added: 0029216
2017-05-03 08:48 ChristiaanB Note Added: 0029217
2017-05-03 09:19 mdslimited Note Added: 0029218
2017-05-03 09:48 ChristiaanB Note Added: 0029219
2017-05-03 20:11 geraldh Note Added: 0029225
2017-05-04 16:58 ChristiaanB Note Added: 0029228
2017-05-04 19:59 TrevorH Note Added: 0029230
2017-05-05 09:24 mdslimited Note Added: 0029234
2017-05-05 09:50 szebel Note Added: 0029236
2017-05-17 08:40 dritz Note Added: 0029289
2017-05-17 09:22 TrevorH Note Added: 0029290
+Issue History