View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0008954 | CentOS-6 | kernel | public | 2015-06-23 23:28 | 2017-11-14 21:44 |
Reporter | SimonHF | Assigned To | |||
Priority | urgent | Severity | major | Reproducibility | always |
Status | new | Resolution | open | ||
Product Version | 6.6 | ||||
Summary | 0008954: mlock() on CentOS is 100 to 520 times slower than the next slowest OS tested. Why? | ||||
Description | While debugging why libmcrypt is so slow on CentOS, discovered that in its xmemory.c, every malloc() has an associated mlock(). For some reason on CentOS these mlock() calls are between 100 to 520 times slower than the next slowest OS tested. I attached a simple C program which reproduces the problem without libmcrypt, together with some benchmark info showing the times on OSX, Ubuntu, RedHat, Amazon Linux, and CentOS. | ||||
Steps To Reproduce | Compile and run the attached mlock-test.c: // Testing on Mid-2014 MacBook Pro: // OSX : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // OSX : - mlock_status 0 // OSX : - elapsed_without_mlock 0.003493 seconds // OSX : - elapsed_with____mlock 0.027737 seconds (7.9 times slower), average per mlock() 0.000000 seconds // Testing OS on VMware Workstation on OSX on Mid-2014 MacBook Pro: // Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // Ubuntu 14.04 : - mlock_status -99039 // Ubuntu 14.04 : - elapsed_without_mlock 0.003451 seconds // Ubuntu 14.04 : - elapsed_with____mlock 0.015507 seconds (4.5 times slower), average per mlock() 0.000000 seconds // CentOS 6.6 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // CentOS 6.6 : - mlock_status -99039 // CentOS 6.6 : - elapsed_without_mlock 0.003735 seconds // CentOS 6.6 : - elapsed_with____mlock 11.424610 seconds (3058.9 times slower), average per mlock() 0.000114 seconds // Testing OS on EC2 c3.8xlarge instance via "AWS Marketplace" Launch: // "CentOS 6 (x86_64) - with Updates HVM": # sudo yum install glibc gcc // "CentOS 6 (x86_64) - with Updates HVM": # gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 6 (x86_64) - with Updates HVM": - mlock_status 0 // "CentOS 6 (x86_64) - with Updates HVM": - elapsed_without_mlock 0.004739 seconds // "CentOS 6 (x86_64) - with Updates HVM": - elapsed_with____mlock 73.994046 seconds (15614.5 times slower), average per mlock() 0.000740 seconds // "CentOS 6 (x86_64) - with Updates HVM": [root@ip-10-130-32-44 ~]# gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 7 (x86_64) with Updates HVM" : $ sudo yum install glibc gcc // "CentOS 7 (x86_64) with Updates HVM" : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 7 (x86_64) with Updates HVM" : - mlock_status -99039 // "CentOS 7 (x86_64) with Updates HVM" : - elapsed_without_mlock 0.003984 seconds // "CentOS 7 (x86_64) with Updates HVM" : - elapsed_with____mlock 15.466390 seconds (3882.2 times slower), average per mlock() 0.000155 seconds // Testing OS on EC2 c3.8xlarge instance via "Quick Start" Launch: // ami-1ecae776: Amazon Linux 2015.03 : $ sudo yum install glibc gcc // ami-1ecae776: Amazon Linux 2015.03 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-1ecae776: Amazon Linux 2015.03 : - mlock_status -99039 // ami-1ecae776: Amazon Linux 2015.03 : - elapsed_without_mlock 0.003959 seconds // ami-1ecae776: Amazon Linux 2015.03 : - elapsed_with____mlock 0.120816 seconds (30.5 times slower), average per mlock() 0.000001 seconds // ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get update // ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get install build-essential // ami-d05e75b8: Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-d05e75b8: Ubuntu 14.04 : - mlock_status -99039 // ami-d05e75b8: Ubuntu 14.04 : - elapsed_without_mlock 0.004230 seconds // ami-d05e75b8: Ubuntu 14.04 : - elapsed_with____mlock 0.049550 seconds (11.7 times slower), average per mlock() 0.000000 seconds // ami-12663b7a: RedHat 7.1 : $ sudo yum install glibc gcc // ami-12663b7a: RedHat 7.1 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-12663b7a: RedHat 7.1 : - mlock_status -99039 // ami-12663b7a: RedHat 7.1 : - elapsed_without_mlock 0.003817 seconds // ami-12663b7a: RedHat 7.1 : - elapsed_with____mlock 0.079827 seconds (20.9 times slower), average per mlock() 0.000001 seconds | ||||
Tags | No tags attached. | ||||
mlock-test.c (5,314 bytes)
#include <stdio.h> #include <sys/time.h> #include <stdlib.h> #include <sys/mman.h> // Testing on Mid-2014 MacBook Pro: // OSX : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // OSX : - mlock_status 0 // OSX : - elapsed_without_mlock 0.003493 seconds // OSX : - elapsed_with____mlock 0.027737 seconds (7.9 times slower), average per mlock() 0.000000 seconds // Testing OS on VMware Workstation on OSX on Mid-2014 MacBook Pro: // Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // Ubuntu 14.04 : - mlock_status -99039 // Ubuntu 14.04 : - elapsed_without_mlock 0.003451 seconds // Ubuntu 14.04 : - elapsed_with____mlock 0.015507 seconds (4.5 times slower), average per mlock() 0.000000 seconds // CentOS 6.6 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // CentOS 6.6 : - mlock_status -99039 // CentOS 6.6 : - elapsed_without_mlock 0.003735 seconds // CentOS 6.6 : - elapsed_with____mlock 11.424610 seconds (3058.9 times slower), average per mlock() 0.000114 seconds // Testing OS on EC2 c3.8xlarge instance via "AWS Marketplace" Launch: // "CentOS 6 (x86_64) - with Updates HVM": # sudo yum install glibc gcc // "CentOS 6 (x86_64) - with Updates HVM": # gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 6 (x86_64) - with Updates HVM": - mlock_status 0 // "CentOS 6 (x86_64) - with Updates HVM": - elapsed_without_mlock 0.004739 seconds // "CentOS 6 (x86_64) - with Updates HVM": - elapsed_with____mlock 73.994046 seconds (15614.5 times slower), average per mlock() 0.000740 seconds // "CentOS 6 (x86_64) - with Updates HVM": [root@ip-10-130-32-44 ~]# gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 7 (x86_64) with Updates HVM" : $ sudo yum install glibc gcc // "CentOS 7 (x86_64) with Updates HVM" : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // "CentOS 7 (x86_64) with Updates HVM" : - mlock_status -99039 // "CentOS 7 (x86_64) with Updates HVM" : - elapsed_without_mlock 0.003984 seconds // "CentOS 7 (x86_64) with Updates HVM" : - elapsed_with____mlock 15.466390 seconds (3882.2 times slower), average per mlock() 0.000155 seconds // Testing OS on EC2 c3.8xlarge instance via "Quick Start" Launch: // ami-1ecae776: Amazon Linux 2015.03 : $ sudo yum install glibc gcc // ami-1ecae776: Amazon Linux 2015.03 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-1ecae776: Amazon Linux 2015.03 : - mlock_status -99039 // ami-1ecae776: Amazon Linux 2015.03 : - elapsed_without_mlock 0.003959 seconds // ami-1ecae776: Amazon Linux 2015.03 : - elapsed_with____mlock 0.120816 seconds (30.5 times slower), average per mlock() 0.000001 seconds // ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get update // ami-d05e75b8: Ubuntu 14.04 : $ sudo apt-get install build-essential // ami-d05e75b8: Ubuntu 14.04 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-d05e75b8: Ubuntu 14.04 : - mlock_status -99039 // ami-d05e75b8: Ubuntu 14.04 : - elapsed_without_mlock 0.004230 seconds // ami-d05e75b8: Ubuntu 14.04 : - elapsed_with____mlock 0.049550 seconds (11.7 times slower), average per mlock() 0.000000 seconds // ami-12663b7a: RedHat 7.1 : $ sudo yum install glibc gcc // ami-12663b7a: RedHat 7.1 : $ gcc -O0 -std=c99 -o mlock-test mlock-test.c && ./mlock-test // ami-12663b7a: RedHat 7.1 : - mlock_status -99039 // ami-12663b7a: RedHat 7.1 : - elapsed_without_mlock 0.003817 seconds // ami-12663b7a: RedHat 7.1 : - elapsed_with____mlock 0.079827 seconds (20.9 times slower), average per mlock() 0.000001 seconds static double get_time_in_seconds(void) { struct timeval tv; gettimeofday(&tv, NULL); return (double)tv.tv_sec + 1.e-6 * (double)tv.tv_usec; } int main() { double elapsed_without_mlock; double elapsed_with____mlock; size_t len = 42; int iterations = 100000; int mlock_status = 0; { double start_time = get_time_in_seconds(); for (int i = 0; i < iterations; i++) { char * p = malloc(len); mlock_status += mlock(p, len); } double end_time = get_time_in_seconds(); elapsed_with____mlock = end_time - start_time; } { double start_time = get_time_in_seconds(); for (int i = 0; i < iterations; i++) { char * p = malloc(len); } double end_time = get_time_in_seconds(); elapsed_without_mlock = end_time - start_time; } printf("- mlock_status %d\n", mlock_status); printf("- elapsed_without_mlock %f seconds\n", elapsed_without_mlock); printf("- elapsed_with____mlock %f seconds (%.1f times slower), average per mlock() %f seconds\n", elapsed_with____mlock, elapsed_with____mlock / elapsed_without_mlock, (elapsed_with____mlock - elapsed_without_mlock) / iterations); } |
|
I just wanted to confirm I'm also seeing this on Centos 7. We suffered from significant performance issues while using mcrypt on Centos that we did not see when we were on Ubuntu. | |
Date Modified | Username | Field | Change |
---|---|---|---|
2015-06-23 23:28 | SimonHF | New Issue | |
2015-06-23 23:28 | SimonHF | File Added: mlock-test.c | |
2017-11-14 21:44 | kylethielk | Note Added: 0030574 |