0018336: Kmalloc-32 consuming servers whole memory

ID	Project	Category	View Status	Date Submitted	Last Update

0018336	CentOS-7	kernel	public	2021-10-25 13:13	2022-05-23 06:13

Reporter	naman.jain	Assigned To
Priority	urgent	Severity	major	Reproducibility	random
Status	new	Resolution	open
Platform	Production	OS	centos	OS Version	centos 7.6.1810
Product Version	7.6.1810

Summary	0018336: Kmalloc-32 consuming servers whole memory
Description	Hi Team, I am using the centos 7.6.1810 version with the kernel as 3.10.0-957.el7.x86_64. And this centos version I am facing the issue of the kmalloc-32 which is consuming my whole server RAM. Due to which we are unable to perform other actions on the server which causes my Production application to down. I tried by running: sync; echo 2 > /proc/sys/vm/drop_caches But that doesn't help me out. I am not able to understand what is causing the kmalloc-32 to consume the server's whole memory. I the attached screenshot of slabtop we can see the kmalloc-32 is having 100% usage. I search on this I lot but didn't get much on this just to reboot the server. After Rebooting the server the kmalloc-32 memory usage also decreases and my server starts running fine but in the next month, I again face the same issue so I again reboot the server. Is there any proper solution to remove this problem completely instead of restarting my server again and again? Note: I am running the Kubernetes application on this machine.
Tags	#Centos7, 3.10.0-957.21.3.el7, hyper-v, kernel, kernel bug

abrt_hash
URL

naman.jain 2021-10-25 13:13 reporter	kmalloc-32.png (26,598 bytes) kmalloc-32.png (26,598 bytes)

TrevorH 2021-10-25 13:21 manager ~0038676	> I am using the centos 7.6.1810 version with the kernel as 3.10.0-957.el7.x86_64 ONLY the CURRENT version is supported so you need to update to 7.9.2009 with the latest kernel kernel-3.10.0-1160.45.1.el7.x86_64 All older versions of CentOS 7 (i.e. 7.8 and earlier are explicitly unsupported.

naman.jain 2021-10-25 18:28 reporter ~0038677	Thanks for the quick response.

naman.jain 2022-05-23 06:09 reporter ~0038930	Hi Team, After updating all my worker's nodes to kernel version: kernel-3.10.0-1160.45.1.el7.x86_64. Again, my kmalloc-32 is still consuming high memory and due to this, we have to reboot the nodes. And After Rebooting the server, the kmalloc-32 memory usage also decreases, and my server starts running fine, but in the next month, I again face the same issue, so I reboot the server again. So is there any permanent solution to remove this problem entirely instead of restarting my server again?. Screenshot from 2022-05-12 13-12-23.png (191,776 bytes) Screenshot from 2022-05-12 13-12-23.png (191,776 bytes)

tigalch 2022-05-23 06:13 manager ~0038931	Please update your kernel to the current version - 3.10.0-1160.66.1

Date Modified	Username	Field
2021-10-25 13:13	naman.jain	New Issue
2021-10-25 13:13	naman.jain	Tag Attached: #Centos7
2021-10-25 13:13	naman.jain	Tag Attached: 3.10.0-957.21.3.el7
2021-10-25 13:13	naman.jain	Tag Attached: hyper-v
2021-10-25 13:13	naman.jain	Tag Attached: kernel
2021-10-25 13:13	naman.jain	Tag Attached: kernel bug
2021-10-25 13:13	naman.jain	File Added: kmalloc-32.png
2021-10-25 13:21	TrevorH	Note Added: 0038676
2021-10-25 18:28	naman.jain	Note Added: 0038677
2022-05-23 06:09	naman.jain	Note Added: 0038930
2022-05-23 06:09	naman.jain	File Added: Screenshot from 2022-05-12 13-12-23.png
2022-05-23 06:13	tigalch	Note Added: 0038931