View Issue Details

IDProjectCategoryView StatusLast Update
0006226CentOS-6kernelpublic2013-01-29 19:29
Reporterand-g 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version6.3 
Target VersionFixed in Version 
Summary0006226: nanosleep is too CPU-hungry
DescriptionIt appears that nanosleep function in v6.3 consumes a lot of CPU power, comparing with v5.8. Implications are serious - idle tool, only waiting for incoming requests, consumes like 15-20% CPU.
Steps To ReproduceSample program:
int main(int argc, const char* argv[])
{
    for (int i=0; i<100000; ++i) {
      struct timespec delay;
      delay.tv_sec = 0;
      delay.tv_nsec = 10000;
      nanosleep(&delay, NULL);
    }
    return 0;
}
I compile it with GCC 4.4.2 and run on CentOS 5.8 and CentOS 6.3, hardware is identical.
Additional Informationstrace results on v5.8:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.62 0.001998 0 100000 nanosleep
  0.79 0.000016 1 30 23 open
  0.59 0.000012 1 22 mmap
  0.00 0.000000 0 6 read
  0.00 0.000000 0 7 close
  0.00 0.000000 0 16 14 stat
  0.00 0.000000 0 7 fstat
  0.00 0.000000 0 12 mprotect
  0.00 0.000000 0 1 munmap
  0.00 0.000000 0 1 brk
  0.00 0.000000 0 2 rt_sigaction
  0.00 0.000000 0 1 rt_sigprocmask
  0.00 0.000000 0 1 1 access
  0.00 0.000000 0 1 execve
  0.00 0.000000 0 1 getrlimit
  0.00 0.000000 0 1 arch_prctl
  0.00 0.000000 0 2 futex
  0.00 0.000000 0 1 set_tid_address
  0.00 0.000000 0 1 set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00 0.002026 100113 38 total


strace results on v6.3:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.071253 1 100000 1 nanosleep
  0.00 0.000000 0 6 read
  0.00 0.000000 0 30 23 open
  0.00 0.000000 0 7 close
  0.00 0.000000 0 16 14 stat
  0.00 0.000000 0 7 fstat
  0.00 0.000000 0 21 mmap
  0.00 0.000000 0 12 mprotect
  0.00 0.000000 0 1 munmap
  0.00 0.000000 0 1 brk
  0.00 0.000000 0 2 rt_sigaction
  0.00 0.000000 0 1 rt_sigprocmask
  0.00 0.000000 0 1 1 access
  0.00 0.000000 0 1 execve
  0.00 0.000000 0 1 getrlimit
  0.00 0.000000 0 1 arch_prctl
  0.00 0.000000 0 3 1 futex
  0.00 0.000000 0 1 set_tid_address
  0.00 0.000000 0 1 restart_syscall
  0.00 0.000000 0 1 set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00 0.071253 100114 40 total
TagsNo tags attached.

Activities

toracat

toracat

2013-01-29 18:16

manager   ~0016359

Probably related :

http://kerneltrap.org/node/17240

There are 2 answers :

"IIRC the scheduler was changed to more accurately measure the time (via the sched_clock), now time is measured in cpu clock ticks instead of timer ticks. processes giving up the cpu very fast can't 'fall between the cracks' any more.

also 2.6.25 and 2.6.27 contain the high resolution timers code. before everything would be rounded up to timer ticks. you should benchmark how many loops/s you get.

in 1000ns your cpu has about 3000 clock cycles (assuming 3GHz). a syscall can take several 100 clock cycles due to the context switches and handler code. 40% looks plausible."

"My guess is that your new kernel is doing exactly what you asked, and waking your process 1 million times a second. This is expected to chew up a fair amount of CPU.
Your old kernel probably rounded up to the nearest scheduling interval, and actually gave a delay three or four orders of magnitude larger than you requested."
and-g

and-g

2013-01-29 19:29

reporter   ~0016360

That is all good, but it does not help.

Well, my server calls epoll_wait with timeout of 10ms, it never calls any sleep function directly. 100 calls per second is probably not too much. When tested on CentOS v6.3, it consumes 15% CPU just doing nothing. Strace shows epoll_wait is the problem.

I have changed epoll_wait to use zero timeout and added call to nanosleep for the same 10ms. Behaviour of my app did not change much. But, now strace blamed nanosleep.

I believe, the real problem is in time measurement, not epoll. Accuracy is great, but the price is also important. So far, I find the price unacceptable. And, in fact, I have no idea what I can do to reduce it.

Issue History

Date Modified Username Field Change
2013-01-29 16:11 and-g New Issue
2013-01-29 18:16 toracat Note Added: 0016359
2013-01-29 19:29 and-g Note Added: 0016360