 |
@tylerni7 You can test more recent kernels on CentOS by using ELRepo's kernel-lt (4.4) or kernel-ml (4.18). http://elrepo.org/linux/kernel/el7/x86_64/RPMS/ Mind giving them a try? |
 |
And if you want to get it fixed in the CentOS/RHEL kernels then you'll also need to report this on bugzilla.redhat.com. CentOS only rebuilds what is used to build RHEL so fixes have to come from upstream. |
 |
Sounds good. I'll try to test with those kernels and get back to you. Is it worth keeping this report open and also filing a report on bugzilla.redhat.com? Given that I only have CentOS to test, I wasn't sure if there were things I wouldn't be able to test that I would be asked about if I filed there. |
|
|
@tylerni7 If we can identify the patch(es)/commits that fix the issue and if they can be backported to the current kernel, CentOS will offer the centosplus kernel that includes the patch(es). Then you can provide the info upstream (RH) for them to consider. |
|
|
Using the ElRepo kernel-lt (specifically 4.4.161-1.el7.elrepo.x86_64 ) the issue seems to be gone. $ time sudo bash -c 'for i in $(seq 100); do unshare -n echo -n ; done' real 0m5.456s Not sure what the best way to track down the patch(es)/commit(s), but I guess that's a start. |
 |
@tylerni7, I've been trying to follow the patches, but there are too many changes related to netfilter. If you could test with older versions of 4.4, it would be a good way to limit the amount of changes. You can get older elrepo 4.4.x kernels from here http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/, so you don't have to rebuild. Thanks! |
|
|
Also, I've been trying to replicate this bug locally and I couldn't, 3.10.0-862.14.4 is even faster than 4.4.161-1.el7.elrepo.x86_64. Please tell us more about your system to help us replicate the problem. Thanks. |
|
|
Hm gross. I'll try to see if I can find a good way to reproduce the issue on a standard environment. I can verify that what I was doing does not reproduce on fresh machine. I don't know if it only manifests itself after the machine gets into a particular state first or something else. The machines I was testing on are in use, which may be triggering the issue. I know there have been a number of previous(?) kernel issues related to this when I was researching the issue which might be relevant. One is a race condition with clone(CLONE_NEWNET) which once triggered prevents clone(CLONE_NEWNET) from succeeding until the machine reboots https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1021471 (I know that's the Ubuntu issue tracker, but seems like just a kernel issue). The other seems to be likely the same issue, http://lkml.iu.edu/hypermail/linux/kernel/1704.2/04491.html which has some stuff about reproducing the issue and more, so I'll try to look further into that and report back. |
|
|
Just a quick update: still having some issues reproducing this nicely. When we hit issues locally the machine is running Kubernetes which has set up a bunch of things that might somehow affect the performance. On newer kernels (under slightly different circumstances, and also still need to figure out what "newer" means, but at least 4.15) the issue can still happen where unshare calls take several seconds to complete, but by setting RCU_EXPEDITED=1 the calls go back to taking milliseconds. I will try to come up with a better way to reproduce the issue and post an update when that happens. |
|
|
@tylerni7 I'm doing some cleanup here. Are you still seeing this, even without a specific way to replicate it? |
|
|
We were still seeing this. Eventually we moved to using a kernel >= 4.17 from ELRepo , which has https://github.com/torvalds/linux/commit/1a57feb847c56d6193f67d0e892c24e71f9e3ab1 in it, which seems to help quite a bit. It seems difficult to reproduce easily because something needs to start causing contention before everything starts slowing down, so a fresh system may not hit it. Anyway, since we've worked around this, feel free to do whatever you think is appropriate for the ticket. Cheers! |
- Anonymous
