View Issue Details

IDProjectCategoryView StatusLast Update
0015115CentOS-7kernelpublic2018-08-16 05:19
Reporterkirill.bogachev 
PriorityurgentSeveritycrashReproducibilitysometimes
Status assignedResolutionopen 
Platformx86_64OSCentOSOS Version7.5
Product Version7.5.1804 
Target VersionFixed in Version 
Summary0015115: kernel 3.10.0-862.3.3.el7.x86_64 panic under heavy load: kernel BUG at mm/mempolicy.c:1759
Description3 cluster nodes are crashed during first 12 hours of load after re-imaging from CentOS 6.5 to CentOS 7.5

Jul 29 09:36:46 E5-2680v2-71 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 29 09:36:46 E5-2680v2-71 kernel: invalid opcode: 0000 [#1] SMP
Jul 29 09:36:46 E5-2680v2-71 kernel: Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE)
 mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel iTCO_wdt kvm iTCO
_vendor_support irqbypass crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper aesni_intel lrw syscopyarea gf128mul sysfillrect glue_helper sysimgblt a
blk_helper fb_sys_fops cryptd drm mei_me i2c_i801 ipmi_si ioatdma joydev pcspkr mei lpc_ich ipmi_devintf shpchp ipmi_msghandler wmi knem(OE) sch_fq_co
del ip_tables nfsv3 nfs_acl nfs lockd grace fscache mlx4_ib(OE) ib_core(OE) igb mlx4_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel dca ptp p
ps_core i2c_algo_bit devlink i2c_core mlx_compat(OE) sunrpc

Jul 29 09:46:01 E5-2680v2-16 kernel: ------------[ cut here ]------------
Jul 29 09:46:01 E5-2680v2-16 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 29 09:46:01 E5-2680v2-16 kernel: invalid opcode: 0000 [#1] SMP

Jul 29 03:37:18 E5-2680v2-36 kernel: ------------[ cut here ]------------
Jul 29 03:37:18 E5-2680v2-36 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 29 03:37:18 E5-2680v2-36 kernel: invalid opcode: 0000 [#1] SMP

Screen photo with call stack is attached. Systems are disk-less with PXE boot so it is difficult to save information for kernel crash except screen photo.
Steps To ReproduceRe-image stable worked from 2014 systems from CentOS 6.5 to CentOS 7.5 and start load.
Tags3.10.0-862
abrt_hash
URL

Activities

kirill.bogachev

kirill.bogachev

2018-07-29 14:22

reporter  

20180729_crash.jpg (927,852 bytes)
kirill.bogachev

kirill.bogachev

2018-07-29 18:14

reporter   ~0032387

One more node dead with the same diagnostic:
Jul 29 18:34:51 E5-2680v2-25 kernel: ------------[ cut here ]------------
Jul 29 18:34:51 E5-2680v2-25 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 29 18:34:51 E5-2680v2-25 kernel: invalid opcode: 0000 [#1] SMP

There are no MCE reports for these nodes. It is definitely kernel bug.
tru

tru

2018-07-29 18:44

administrator   ~0032388

can you try under kernel-3.10.0-862.9.1.el7.x86_64 ?
kirill.bogachev

kirill.bogachev

2018-07-29 18:50

reporter   ~0032389

All file systems are mounted via Infiniband. Is Mellanox OFED supported by this kernel?
kirill.bogachev

kirill.bogachev

2018-07-30 08:29

reporter   ~0032393

Two more nodes dead
Jul 30 04:47:56 E5-2680v2-96 kernel: ------------[ cut here ]------------
Jul 30 04:47:56 E5-2680v2-96 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 30 11:03:50 E5-2680v2-13 kernel: ------------[ cut here ]------------
kirill.bogachev

kirill.bogachev

2018-07-30 09:00

reporter   ~0032394

Two more nodes with different hardware (V4 processors and Tesla P100) crashed:
Jul 30 11:21:50 E5-2680v4-P100-3 kernel: ------------[ cut here ]------------
Jul 30 11:21:50 E5-2680v4-P100-3 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 30 11:21:50 E5-2680v4-P100-3 kernel: invalid opcode: 0000 [#1] SMP
Jul 30 11:46:40 E5-2680v4-P100-7 kernel: ------------[ cut here ]------------
Jul 30 11:46:40 E5-2680v4-P100-7 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 30 11:46:40 E5-2680v4-P100-7 kernel: invalid opcode: 0000 [#1] SMP

CentOS 7.5 looks unusable.
pgreco

pgreco

2018-07-30 10:50

developer   ~0032395

@toracat, it seems like kernel upstream changed this BUG() to a WARN(), can you check if this applies?
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/mm/mempolicy.c?h=next-20180730&id=6d8409580bee356ce418dcb94260b24dda639934

Pablo.
kirill.bogachev

kirill.bogachev

2018-07-30 11:05

reporter   ~0032396

In 3.10.0 BUG() is triggered in "unsigned slab_node(void)" function. In 4.17.11 this function renamed to "unsigned int mempolicy_slab_node(void)" and BUG() is in the same place. Function "policy_zonelist" is different.
toracat

toracat

2018-07-30 17:12

manager   ~0032398

@kirill.bogachev

There is a way to demonstrate what you mentioned. You can test-install kernel-ml [1] from ELRepo. Because the 'BUG' in "unsigned int mempolicy_slab_node(void)" is still in the mainline kernel, this kernel should have the same problem. Can you run this test?

[1] http://elrepo.org/tiki/kernel-ml
kirill.bogachev

kirill.bogachev

2018-07-30 21:40

reporter   ~0032402

One more node with different hardware (v4 CPUs) crashed:
Jul 30 17:36:58 E5-2630v4-1 kernel: ------------[ cut here ]------------
Jul 30 17:36:58 E5-2630v4-1 kernel: kernel BUG at mm/mempolicy.c:1759!
Jul 30 17:36:58 E5-2630v4-1 kernel: invalid opcode: 0000 [#1] SMP
Jul 30 17:36:58 E5-2630v4-1 kernel: Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(
OE) ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) sd_mod crc_t10dif crct
10dif_generic sg sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul iTCO_wdt gha
sh_clmulni_intel iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ahci libahci pcspkr mei_me
joydev libata mei lpc_ich ioatdma i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad sch
_fq_codel knem(OE) ip_tables nfsv3 nfs_acl nfs lockd grace fscache mlx4_ib(OE) ib_core(OE) drm_kms_helper syscopyarea sy
sfillrect sysimgblt fb_sys_fops ttm drm igb mlx4_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel dca ptp

I will prepare test system to test mainline stable kernel ASAP.
pgreco

pgreco

2018-07-31 00:00

developer   ~0032406

@toracat built a kernel with the patch from upstream included https://people.centos.org/toracat/kernel/7/plus/bug15115/
Please try to test that one also.
kirill.bogachev

kirill.bogachev

2018-07-31 19:06

reporter   ~0032409

Latest Mellanox drivers MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.5-x86_64 do not support kernel 4.17. There are errors during kernel module compilation:

In file included from /tmp/mlnx_iso.20349/OFED_topdir/BUILD/mlnx-ofa_kernel-4.4/obj/default/drivers/infiniband/core/cma.c:54:0:
include/net/ip6_route.h:130:18: note: expected 'const struct sk_buff *' but argument is of type 'int'
 struct rt6_info *rt6_lookup(struct net *net, const struct in6_addr *daddr,
                  ^
/tmp/mlnx_iso.20349/OFED_topdir/BUILD/mlnx-ofa_kernel-4.4/obj/default/drivers/infiniband/core/cma.c:1394:7: error: too few arguments to function 'rt6_lookup'
       strict);
       ^
In file included from /tmp/mlnx_iso.20349/OFED_topdir/BUILD/mlnx-ofa_kernel-4.4/obj/default/drivers/infiniband/core/cma.c:54:0:
include/net/ip6_route.h:130:18: note: declared here
 struct rt6_info *rt6_lookup(struct net *net, const struct in6_addr *daddr,

Thus will try kernel-plus-3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64 next.
pgreco

pgreco

2018-07-31 20:20

developer   ~0032410

If I understand correctly the download page, you should use the version for fedora27 with kernel 4.17, not the rhel/centos 7.5
kirill.bogachev

kirill.bogachev

2018-07-31 21:29

reporter   ~0032411

Thank you for suggestion. I will try it next. For now I re-imaged 100 nodes with kernel 3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64 which share OFED with stock kernel. I just rebuilt drivers for Tesla boards. I have started load with this kernel on 100 nodes. Lets wait for results. With stock kernel crash rate was about 5 nodes per 24 hours.
kirill.bogachev

kirill.bogachev

2018-08-01 23:29

reporter   ~0032415

There are not crashes during first 24 hours of load for 100 nodes with 3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64 kernel. It looks very promising.
pgreco

pgreco

2018-08-02 10:08

developer   ~0032417

That really looks promising.
Since the BUG changed to WARN_ON_ONCE (apart from some logic change), can you check the logs for that WARN?
kirill.bogachev

kirill.bogachev

2018-08-02 20:54

reporter   ~0032423

There are no more crashes on nodes re-imaged with 3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64 for 48 hours of load for 100 nodes.
There are 3 more crashes on on nodes with stock 3.10.0-862.3.3.el7.x86_64 kernel for 48 hours of load for 8 nodes.

There are no WARN messages from kernel 3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64 on 100 nodes for last 48 hours.

The message
Aug 2 20:37:39 E5-2680v4-P100-3 kernel: kernel BUG at mm/mempolicy.c:1759!
is triggered in "unsigned slab_node(void)" function. in 3.10.0. In 4.17.11 this function renamed to "unsigned int mempolicy_slab_node(void)" and BUG() is in the same place, at mm/mempolicy.c:1791.

The message
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/mm/mempolicy.c?h=next-20180730&id=6d8409580bee356ce418dcb94260b24dda639934
applies to function policy_zonelist (in 3.10.0) renamed to policy_node in 4.17.11 where BUG() in 3.10.0 at mm/mempolicy.c:1688 was replaced in 4.17.11 by WARN_ON_ONCE at mm/mempolicy.c:1731.
The BUG in slab_node (in 3.10.0) / mempolicy_slab_node (in 4.17.11) is on place. So it is not surprise that there are no WARNs.

Thus I think in kernel 3.10.0-862.9.1 the bug which triggered BUG at mm/mempolicy.c:1759 is fixed already. Where I can get it to try?
pgreco

pgreco

2018-08-02 21:26

developer   ~0032425

I don't know how you are checking the line numbers, but mm/mempolicy.c:1759 matches exactly with the line that was changed by the patch @toracat applied.
You can check by downloading and decompressing the source rpm from here http://vault.centos.org/7.5.1804/updates/Source/SPackages/kernel-3.10.0-862.9.1.el7.src.rpm
kirill.bogachev

kirill.bogachev

2018-08-02 23:20

reporter   ~0032430

I have re-imaged 108 nodes with kernel 3.10.0-862.9.1.el7.x86_64 and provided load. Lets wait for results.
kirill.bogachev

kirill.bogachev

2018-08-06 21:05

reporter   ~0032444

We have the same crashes with kernel 3.10.0-862.9.1.el7.x86_64 with a bit lower rate: 7 nodes crashed per 72 hours. You were wright. The patch applied to kernel (forming 3.10.0-862.9.1.bug15115.el7.centos.plus.x86_64) fix the problem. Last experiment definitely proved that. Thank you.

When we may expect the patch in the stock kernel update for CentOS 7.5?
toracat

toracat

2018-08-06 22:24

manager   ~0032445

CentOS can only apply the patch to the centosplus kernel (which will be done in the next update). Any fix to the stock kernel must come from upstream (RHEL kernel).
kirill.bogachev

kirill.bogachev

2018-08-06 23:07

reporter   ~0032446

Do you know the release date or/and kernel rpm version number?
toracat

toracat

2018-08-06 23:37

manager   ~0032447

The bug has to be reported upstream at http://bugzilla.redhat.com . I believe @pgreco is planning to do that. :)

Once it is fixed by RH, it will be inherited by the CentOS stock kernel. This, however, might not happen quickly, depending on certain factors. In the meantime, you may want to keep using the centosplus kernel.
pgreco

pgreco

2018-08-07 00:22

developer   ~0032448

reported upstream as https://bugzilla.redhat.com/show_bug.cgi?id=1613081
@toracat I've added you to cc
toracat

toracat

2018-08-07 02:17

manager   ~0032449

@pgreco
Thanks a bunch.
toracat

toracat

2018-08-16 05:19

manager   ~0032500

kernel-plus-3.10.0-862.11.6.el7.centos.plus now has the patch.

Issue History

Date Modified Username Field Change
2018-07-29 14:22 kirill.bogachev New Issue
2018-07-29 14:22 kirill.bogachev File Added: 20180729_crash.jpg
2018-07-29 14:22 kirill.bogachev Tag Attached: 3.10.0-862
2018-07-29 18:14 kirill.bogachev Note Added: 0032387
2018-07-29 18:44 tru Note Added: 0032388
2018-07-29 18:50 kirill.bogachev Note Added: 0032389
2018-07-30 08:29 kirill.bogachev Note Added: 0032393
2018-07-30 09:00 kirill.bogachev Note Added: 0032394
2018-07-30 10:50 pgreco Note Added: 0032395
2018-07-30 11:05 kirill.bogachev Note Added: 0032396
2018-07-30 17:12 toracat Note Added: 0032398
2018-07-30 21:40 kirill.bogachev Note Added: 0032402
2018-07-31 00:00 pgreco Status new => assigned
2018-07-31 00:00 pgreco Note Added: 0032406
2018-07-31 19:06 kirill.bogachev Note Added: 0032409
2018-07-31 20:20 pgreco Note Added: 0032410
2018-07-31 21:29 kirill.bogachev Note Added: 0032411
2018-08-01 23:29 kirill.bogachev Note Added: 0032415
2018-08-02 10:08 pgreco Note Added: 0032417
2018-08-02 20:54 kirill.bogachev Note Added: 0032423
2018-08-02 21:26 pgreco Note Added: 0032425
2018-08-02 23:20 kirill.bogachev Note Added: 0032430
2018-08-06 21:05 kirill.bogachev Note Added: 0032444
2018-08-06 22:24 toracat Note Added: 0032445
2018-08-06 23:07 kirill.bogachev Note Added: 0032446
2018-08-06 23:37 toracat Note Added: 0032447
2018-08-07 00:22 pgreco Note Added: 0032448
2018-08-07 02:17 toracat Note Added: 0032449
2018-08-16 05:19 toracat Note Added: 0032500