View Issue Details

IDProjectCategoryView StatusLast Update
0016728CentOS-8kernelpublic2019-11-13 19:01
Reportermnelson 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version8.0.1905 
Target VersionFixed in Version 
Summary0016728: Transparent Huge Pages set to [always] is sub-optimal for many applications
DescriptionTransparent Huge Pages provides real benefit to certain applications by potentially reducing TLB misses and improving performance. For other applications, it can bloat memory usage and cause performance regressions. By default, the kernel enables THP for applications that explicitly ask for it via MADV_HUGEPAGE:

> "madvise" will enter direct reclaim like "always" but only for regions
> that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.

https://www.kernel.org/doc/Documentation/vm/transhuge.txt

RHEL, CentOS, and CoreOS (but not Fedora) all appear to override this behavior and set THP to [always]. This unfortunately causes issues with a large variety of software including, but not limited to:

splunk: https://docs.splunk.com/Documentation/Splunk/7.3.2/ReleaseNotes/SplunkandTHP
mongodb: https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
couchbase: https://docs.couchbase.com/server/current/install/thp-disable.html
oracle: https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp
nuodb: http://doc.nuodb.com/4.0/Content/OpenShift-disable-THP.htm
Go runtime: https://github.com/golang/go/issues/8832
jemalloc: https://blog.digitalocean.com/transparent-huge-pages-and-alternative-memory-allocators/
node.js: https://github.com/nodejs/node/issues/11077
tcmalloc: https://github.com/gperftools/gperftools/issues/1073

More recently, we've also seen memory usage bloat in Ceph (using tcmalloc) when THP is set to always potentially resulting in OOM when running inside containers. There are various ways to potentially work around this at the application level including using MADV_NOHUGEPAGE or a prctl flag. Requiring these workarounds to disable THP for a given application is counter-intuitive for several reasons:

1) It deviates from the default kernel behavior without a strong justification as to why.

2) It puts the onus on developers to explicitly stop the kernel from engaging in sub-optimal behavior.

3) It's incredibly confusing to have a system-wide default that claims to "always" enable a setting that many applications may or may not silently disable through workarounds.

Finally, when another prominent distribution was faced with a similar choice, they ran stream and malloc tests showing improvement at various allocation sizes when THP was disabled. Ultimately that lead them to switching back to the kernel default (ie madvise) with no apparent performance regressions:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1703742
Steps To ReproduceThis is a well known issue that can be reproduced via a variety of software. Steps to reproduce in ceph are listed below.

Steps to Reproduce:
1. Install a single OSD ceph cluster.
2. Run a background write workload using hsbench or fio sufficient to fill the ceph-osd caches.
3. compare memory usage of the OSD process when THP is set to [always] vs [madvise]

Results:

https://docs.google.com/spreadsheets/d/1Xl3nWapi7ZKEmpnsSHHWO96iopEG0hK6GeDWhWKSfDo/edit?usp=sharing
Additional Informationhttps://unix.stackexchange.com/questions/495816/which-distributions-enable-transparent-huge-pages-for-all-applications
https://www.percona.com/blog/2019/03/06/settling-the-myth-of-transparent-hugepages-for-databases/
https://blog.nelhage.com/post/transparent-hugepages/
https://alexandrnikitin.github.io/blog/transparent-hugepages-measuring-the-performance-impact/
https://dl.acm.org/citation.cfm?id=3359640
TagsNo tags attached.

Activities

mnelson

mnelson

2019-11-13 19:01

reporter   ~0035687

Update:

While the kernel documentation claims that madvise is the default, the actual code in mm/Kconfig shows that "always" is the default choice, so I retract the statement about differing from the kernel. See:

https://github.com/torvalds/linux/blob/master/mm/Kconfig#L385-L407

Issue History

Date Modified Username Field Change
2019-11-13 18:06 mnelson New Issue
2019-11-13 19:01 mnelson Note Added: 0035687