2017-12-16 20:34 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0008954CentOS-6kernelpublic2017-11-14 21:44
ReporterSimonHF 
PriorityurgentSeveritymajorReproducibilityalways
StatusnewResolutionopen 
Product Version6.6 
Target VersionFixed in Version 
Summary0008954: mlock() on CentOS is 100 to 520 times slower than the next slowest OS tested. Why?
DescriptionWhile debugging why libmcrypt is so slow on CentOS, discovered that in its xmemory.c, every malloc() has an associated mlock(). For some reason on CentOS these mlock() calls are between 100 to 520 times slower than the next slowest OS tested. I attached a simple C program which reproduces the problem without libmcrypt, together with some benchmark info showing the times on OSX, Ubuntu, RedHat, Amazon Linux, and CentOS.
Steps To ReproduceCompile and run the attached mlock-test.c:

// Testing on Mid-2014 MacBook Pro:

// OSX : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// OSX : - mlock_status 0
// OSX : - elapsed_without_mlock 0.003493 seconds
// OSX : - elapsed_with____mlock 0.027737 seconds (7.9 times slower), average per mlock() 0.000000 seconds

// Testing OS on VMware Workstation on OSX on Mid-2014 MacBook Pro:

// Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// Ubuntu 14.04 : - mlock_status -99039
// Ubuntu 14.04 : - elapsed_without_mlock 0.003451 seconds
// Ubuntu 14.04 : - elapsed_with____mlock 0.015507 seconds (4.5 times slower), average per mlock() 0.000000 seconds

// CentOS 6.6 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// CentOS 6.6 : - mlock_status -99039
// CentOS 6.6 : - elapsed_without_mlock 0.003735 seconds
// CentOS 6.6 : - elapsed_with____mlock 11.424610 seconds (3058.9 times slower), average per mlock() 0.000114 seconds

// Testing OS on EC2 c3.8xlarge instance via "AWS Marketplace" Launch:

// "CentOS 6 (x86_64) - with Updates HVM": # sudo yum install glibc gcc
// "CentOS 6 (x86_64) - with Updates HVM": # gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// "CentOS 6 (x86_64) - with Updates HVM": - mlock_status 0
// "CentOS 6 (x86_64) - with Updates HVM": - elapsed_without_mlock 0.004739 seconds
// "CentOS 6 (x86_64) - with Updates HVM": - elapsed_with____mlock 73.994046 seconds (15614.5 times slower), average per mlock() 0.000740 seconds
// "CentOS 6 (x86_64) - with Updates HVM": [root@ip-10-130-32-44 ~]# gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test

// "CentOS 7 (x86_64) with Updates HVM" : $ sudo yum install glibc gcc
// "CentOS 7 (x86_64) with Updates HVM" : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// "CentOS 7 (x86_64) with Updates HVM" : - mlock_status -99039
// "CentOS 7 (x86_64) with Updates HVM" : - elapsed_without_mlock 0.003984 seconds
// "CentOS 7 (x86_64) with Updates HVM" : - elapsed_with____mlock 15.466390 seconds (3882.2 times slower), average per mlock() 0.000155 seconds

// Testing OS on EC2 c3.8xlarge instance via "Quick Start" Launch:

// ami-1ecae776: Amazon Linux 2015.03 : $ sudo yum install glibc gcc
// ami-1ecae776: Amazon Linux 2015.03 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// ami-1ecae776: Amazon Linux 2015.03 : - mlock_status -99039
// ami-1ecae776: Amazon Linux 2015.03 : - elapsed_without_mlock 0.003959 seconds
// ami-1ecae776: Amazon Linux 2015.03 : - elapsed_with____mlock 0.120816 seconds (30.5 times slower), average per mlock() 0.000001 seconds

// ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get update
// ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get install build-essential
// ami-d05e75b8: Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// ami-d05e75b8: Ubuntu 14.04 : - mlock_status -99039
// ami-d05e75b8: Ubuntu 14.04 : - elapsed_without_mlock 0.004230 seconds
// ami-d05e75b8: Ubuntu 14.04 : - elapsed_with____mlock 0.049550 seconds (11.7 times slower), average per mlock() 0.000000 seconds

// ami-12663b7a: RedHat 7.1 : $ sudo yum install glibc gcc
// ami-12663b7a: RedHat 7.1 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test
// ami-12663b7a: RedHat 7.1 : - mlock_status -99039
// ami-12663b7a: RedHat 7.1 : - elapsed_without_mlock 0.003817 seconds
// ami-12663b7a: RedHat 7.1 : - elapsed_with____mlock 0.079827 seconds (20.9 times slower), average per mlock() 0.000001 seconds
TagsNo tags attached.
Attached Files

-Relationships
+Relationships

-Notes

~0030574

kylethielk (reporter)

I just wanted to confirm I'm also seeing this on Centos 7. We suffered from significant performance issues while using mcrypt on Centos that we did not see when we were on Ubuntu.
+Notes

-Issue History
Date Modified Username Field Change
2015-06-23 23:28 SimonHF New Issue
2015-06-23 23:28 SimonHF File Added: mlock-test.c
2017-11-14 21:44 kylethielk Note Added: 0030574
+Issue History