View Issue Details

IDProjectCategoryView StatusLast Update
0005732CentOS-6nss-pam-ldapdpublic2012-05-17 15:21
Reporterarno3t 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
Platformx86_64OSOS Version
Product Version6.2 
Target VersionFixed in Version 
Summary0005732: extensively used nslcd consumes all RAM (with LDAP in Active Directory)
DescriptionWe use Centos 6.2 as NFS and Samba fileserver in MS Windows 2003 Active Directory environment (AD) with AD4Unix plugin. We use AD as LDAP server and we authenticate using Kerberos in AD. We obtain user and group information by means of nslcd daemon from package nss-pam-ldapd-0.7.5-14.el6_2.1.x86_64. Whenever nslcd daemon is extensively used (by ACL backup via getfacl for example) the RAM and SWAP consumption is rapidly increasing up to the kernel kill the nslcd daemon to prevent RAM and SWAP exhausting. Sometimes the daemon takes hundrets MB RAM per hour during standard fileserver operation. I have not found a reason yet.

Now cron starts every five minutes a script which checks a memory usage by nslcd. Whenever nslcd uses more then 200MB RAM the script restarts the daemon. In attached file you can see frequency of restarting.
Steps To ReproduceI can reproduce it on our living system but I have not tried to reproduce it in a lab.
Additional InformationPart of /var/log/messages and /etc/nslcd.conf are attached in file.
TagsNo tags attached.

Activities

arno3t

arno3t

2012-05-17 15:21

reporter  

reportnslcd.txt (18,564 bytes)
At 00:20 cron starts command 'getfacl -R -P --skip-base /DATA > DATA.acl'
and after more then an hour we can see in /var/log/messages
---------------------------------
May 10 01:28:06 atlas smbd[4700]: [2012/05/10 01:28:06.165395,  0] smbd/server.c:281(remove_child_pid)
May 10 01:28:06 atlas smbd[4700]:   Could not find child 10009 -- ignoring
May 10 01:41:06 atlas smbd[4700]: [2012/05/10 01:41:06.678035,  0] smbd/server.c:281(remove_child_pid)
May 10 01:41:06 atlas smbd[4700]:   Could not find child 10134 -- ignoring
May 10 01:45:04 atlas kernel: mrtg invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
May 10 01:45:04 atlas kernel: mrtg cpuset=/ mems_allowed=0
May 10 01:45:04 atlas kernel: Pid: 10159, comm: mrtg Not tainted 2.6.32-220.7.1.el6.x86_64 #1
May 10 01:45:04 atlas kernel: Call Trace:
May 10 01:45:04 atlas kernel: [<ffffffff810c2c61>] ? cpuset_print_task_mems_allowed+0x91/0xb0
May 10 01:45:04 atlas kernel: [<ffffffff811139e0>] ? dump_header+0x90/0x1b0
May 10 01:45:04 atlas kernel: [<ffffffff8120d7ac>] ? security_real_capable_noaudit+0x3c/0x70
May 10 01:45:04 atlas kernel: [<ffffffff81113e6a>] ? oom_kill_process+0x8a/0x2c0
May 10 01:45:04 atlas kernel: [<ffffffff81113da1>] ? select_bad_process+0xe1/0x120
May 10 01:45:04 atlas kernel: [<ffffffff811142c0>] ? out_of_memory+0x220/0x3c0
May 10 01:45:04 atlas kernel: [<ffffffff81132449>] ? next_zones_zonelist+0x29/0x60
May 10 01:45:04 atlas kernel: [<ffffffff81123fde>] ? __alloc_pages_nodemask+0x89e/0x940
May 10 01:45:04 atlas kernel: [<ffffffff81158b2a>] ? alloc_pages_current+0xaa/0x110
May 10 01:45:04 atlas kernel: [<ffffffff81110e07>] ? __page_cache_alloc+0x87/0x90
May 10 01:45:04 atlas kernel: [<ffffffff81126a2b>] ? __do_page_cache_readahead+0xdb/0x210
May 10 01:45:04 atlas kernel: [<ffffffff81126b81>] ? ra_submit+0x21/0x30
May 10 01:45:04 atlas kernel: [<ffffffff811120d3>] ? filemap_fault+0x4c3/0x500
May 10 01:45:04 atlas kernel: [<ffffffff8113b274>] ? __do_fault+0x54/0x510
May 10 01:45:04 atlas kernel: [<ffffffff8113b827>] ? handle_pte_fault+0xf7/0xb50
May 10 01:45:04 atlas kernel: [<ffffffff81112550>] ? generic_file_aio_read+0x380/0x700
May 10 01:45:04 atlas kernel: [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
May 10 01:45:04 atlas kernel: [<ffffffff8113c464>] ? handle_mm_fault+0x1e4/0x2b0
May 10 01:45:04 atlas kernel: [<ffffffff81042b79>] ? __do_page_fault+0x139/0x480
May 10 01:45:04 atlas kernel: [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
May 10 01:45:04 atlas kernel: [<ffffffff814ecb0e>] ? thread_return+0x4e/0x760
May 10 01:45:04 atlas kernel: [<ffffffff814f253e>] ? do_page_fault+0x3e/0xa0
May 10 01:45:04 atlas kernel: [<ffffffff814ef8f5>] ? page_fault+0x25/0x30
May 10 01:45:04 atlas kernel: Mem-Info:
May 10 01:45:04 atlas kernel: Node 0 DMA per-cpu:
May 10 01:45:04 atlas kernel: CPU    0: hi:    0, btch:   1 usd:   0
May 10 01:45:04 atlas kernel: CPU    1: hi:    0, btch:   1 usd:   0
May 10 01:45:04 atlas kernel: CPU    2: hi:    0, btch:   1 usd:   0
May 10 01:45:04 atlas kernel: CPU    3: hi:    0, btch:   1 usd:   0
May 10 01:45:04 atlas kernel: Node 0 DMA32 per-cpu:
May 10 01:45:04 atlas kernel: CPU    0: hi:  186, btch:  31 usd:   0
May 10 01:45:04 atlas kernel: CPU    1: hi:  186, btch:  31 usd:  30
May 10 01:45:04 atlas kernel: CPU    2: hi:  186, btch:  31 usd:   0
May 10 01:45:04 atlas kernel: CPU    3: hi:  186, btch:  31 usd:   0
May 10 01:45:04 atlas kernel: active_anon:350438 inactive_anon:117426 isolated_anon:0
May 10 01:45:04 atlas kernel: active_file:12 inactive_file:25 isolated_file:0
May 10 01:45:04 atlas kernel: unevictable:0 dirty:0 writeback:0 unstable:0
May 10 01:45:04 atlas kernel: free:13267 slab_reclaimable:3241 slab_unreclaimable:11449
May 10 01:45:04 atlas kernel: mapped:75 shmem:2 pagetables:6196 bounce:0
May 10 01:45:04 atlas kernel: Node 0 DMA free:8340kB min:332kB low:412kB high:496kB active_anon:3340kB inactive_anon:3548kB active_file:12kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:196kB kernel_stack:184kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? yes
May 10 01:45:04 atlas kernel: lowmem_reserve[]: 0 2004 2004 2004
May 10 01:45:04 atlas kernel: Node 0 DMA32 free:44728kB min:44720kB low:55900kB high:67080kB active_anon:1398412kB inactive_anon:466156kB active_file:36kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052256kB mlocked:0kB dirty:0kB writeback:0kB mapped:268kB shmem:8kB slab_reclaimable:12936kB slab_unreclaimable:45600kB kernel_stack:1936kB pagetables:24772kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
May 10 01:45:04 atlas kernel: lowmem_reserve[]: 0 0 0 0
May 10 01:45:04 atlas kernel: Node 0 DMA: 25*4kB 14*8kB 18*16kB 11*32kB 11*64kB 5*128kB 4*256kB 2*512kB 2*1024kB 1*2048kB 0*4096kB = 8340kB
May 10 01:45:04 atlas kernel: Node 0 DMA32: 10210*4kB 2*8kB 2*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 44920kB
May 10 01:45:04 atlas kernel: 20338 total pagecache pages
May 10 01:45:04 atlas kernel: 20126 pages in swap cache
May 10 01:45:04 atlas kernel: Swap cache stats: add 6184721, delete 6164595, find 3690780/3999697
May 10 01:45:04 atlas kernel: Free swap  = 0kB
May 10 01:45:04 atlas kernel: Total swap = 4194296kB
May 10 01:45:04 atlas kernel: 524272 pages RAM
May 10 01:45:04 atlas kernel: 10690 pages reserved
May 10 01:45:04 atlas kernel: 608 pages shared
May 10 01:45:04 atlas kernel: 496074 pages non-shared
May 10 01:45:04 atlas kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
May 10 01:45:04 atlas kernel: [  531]     0   531     2747        0   0     -17         -1000 udevd
May 10 01:45:04 atlas kernel: [ 1625]     0  1625    62170       49   0       0             0 rsyslogd
May 10 01:45:04 atlas kernel: [ 1654]     0  1654     2284       26   3       0             0 irqbalance
May 10 01:45:04 atlas kernel: [ 1668]    32  1668     4739       15   0       0             0 rpcbind
May 10 01:45:04 atlas kernel: [ 1686]    29  1686     6839        1   0       0             0 rpc.statd
May 10 01:45:04 atlas kernel: [ 1721]     0  1721     6842        1   3       0             0 rpc.idmapd
May 10 01:45:04 atlas kernel: [ 1751]    81  1751     5865        1   2       0             0 dbus-daemon
May 10 01:45:04 atlas kernel: [ 1762]    70  1762     7574      123   2       0             0 avahi-daemon
May 10 01:45:04 atlas kernel: [ 1763]    70  1763     7436        2   0       0             0 avahi-daemon
May 10 01:45:04 atlas kernel: [ 1773]     0  1773    47310       65   0       0             0 cupsd
May 10 01:45:04 atlas kernel: [ 1799]     0  1799     1016        0   0       0             0 acpid
May 10 01:45:04 atlas kernel: [ 1808]    68  1808     6843      115   0       0             0 hald
May 10 01:45:04 atlas kernel: [ 1809]     0  1809     4523        1   3       0             0 hald-runner
May 10 01:45:04 atlas kernel: [ 1837]     0  1837     5052        1   2       0             0 hald-addon-inpu
May 10 01:45:04 atlas kernel: [ 1850]    68  1850     4448        1   0       0             0 hald-addon-acpi
May 10 01:45:04 atlas kernel: [ 1878]     0  1878   113063       35   0       0             0 automount
May 10 01:45:04 atlas kernel: [ 1917]     0  1917    26823        0   2       0             0 rpc.rquotad
May 10 01:45:04 atlas kernel: [ 1933]     0  1933     6951       41   2       0             0 rpc.mountd
May 10 01:45:04 atlas kernel: [ 1956]     0  1956     1561        0   2       0             0 mcelog
May 10 01:45:04 atlas kernel: [ 2067]     0  2067    29704        1   1       0             0 abrtd
May 10 01:45:04 atlas kernel: [ 2075]     0  2075     2323       17   0       0             0 abrt-dump-oops
May 10 01:45:04 atlas kernel: [ 2086]   498  2086   121025       90   0       0             0 qpidd
May 10 01:45:04 atlas kernel: [ 2121]     0  2121    29303       23   0       0             0 crond
May 10 01:45:04 atlas kernel: [ 2150]     0  2150     5356        0   3       0             0 atd
May 10 01:45:04 atlas kernel: [ 2165]     0  2165     1012        1   0       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2167]     0  2167     1012        1   1       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2169]     0  2169     1012        1   1       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2171]     0  2171     1012        1   3       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2173]     0  2173     1012        1   1       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2175]     0  2175     1012        1   0       0             0 mingetty
May 10 01:45:04 atlas kernel: [ 2177]     0  2177     2747        0   1     -17         -1000 udevd
May 10 01:45:04 atlas kernel: [ 2178]     0  2178     2747        0   0     -17         -1000 udevd
May 10 01:45:04 atlas kernel: [ 2199]     0  2199    23289       31   2     -17         -1000 auditd
May 10 01:45:04 atlas kernel: [ 4014]     0  4014    18219        0   2       0             0 squid
May 10 01:45:04 atlas kernel: [ 4016]     0  4016    22810      689   3       0             0 squid
May 10 01:45:04 atlas kernel: [ 4018]    23  4018     5017        1   2       0             0 unlinkd
May 10 01:45:04 atlas kernel: [ 5136]     0  5136    19663       22   2       0             0 master
May 10 01:45:04 atlas kernel: [ 5139]    89  5139    20247       20   0       0             0 qmgr
May 10 01:45:04 atlas kernel: [12968]     0 12968    43396       12   1       0             0 httpd
May 10 01:45:04 atlas kernel: [10408]    95 10408   120525        1   1       0             0 radiusd
May 10 01:45:04 atlas kernel: [17663]    55 17663   129508       13   2       0             0 slapd
May 10 01:45:04 atlas kernel: [ 4700]     0  4700    54636       70   3       0             0 smbd
May 10 01:45:04 atlas kernel: [ 4712]     0  4712    46302       49   3       0             0 winbindd
May 10 01:45:04 atlas kernel: [ 4714]     0  4714    46889       59   3       0             0 winbindd
May 10 01:45:04 atlas kernel: [ 4715]     0  4715    45182        1   1       0             0 winbindd
May 10 01:45:04 atlas kernel: [ 4716]     0  4716    54633        7   3       0             0 smbd
May 10 01:45:04 atlas kernel: [ 4736]     0  4736    46354        1   2       0             0 winbindd
May 10 01:45:04 atlas kernel: [ 4753]     0  4753    46858       44   3       0             0 winbindd
May 10 01:45:04 atlas kernel: [ 4951]     0  4951    44130       46   3       0             0 nmbd
May 10 01:45:04 atlas kernel: [20540]     0 20540    55602       71   3       0             0 smbd
May 10 01:45:04 atlas kernel: [20628]     0 20628    55887      149   2       0             0 smbd
May 10 01:45:04 atlas kernel: [20733]     0 20733    55784      160   1       0             0 smbd
May 10 01:45:04 atlas kernel: [22230]     0 22230    55154       93   1       0             0 smbd
May 10 01:45:04 atlas kernel: [25534]     0 25534    16008       25   1       0             0 sshd
May 10 01:45:04 atlas kernel: [32062]     0 32062    55619       93   0       0             0 smbd
May 10 01:45:04 atlas kernel: [  356]    48   356    43919        4   3       0             0 httpd
May 10 01:45:04 atlas kernel: [  357]    48   357    43919      108   1       0             0 httpd
May 10 01:45:04 atlas kernel: [  358]    48   358    43919        4   3       0             0 httpd
May 10 01:45:04 atlas kernel: [  359]    48   359    43919       10   1       0             0 httpd
May 10 01:45:04 atlas kernel: [  360]    48   360    43919        4   1       0             0 httpd
May 10 01:45:04 atlas kernel: [  361]    48   361    43919      152   2       0             0 httpd
May 10 01:45:04 atlas kernel: [  362]    48   362    43919       12   3       0             0 httpd
May 10 01:45:04 atlas kernel: [  363]    48   363    43919       39   2       0             0 httpd
May 10 01:45:04 atlas kernel: [ 1814]     0  1814    55006       70   3       0             0 smbd
May 10 01:45:04 atlas kernel: [ 1112]     0  1112    55588       96   2       0             0 smbd
May 10 01:45:04 atlas kernel: [ 2871]     0  2871    98312        1   1       0             0 dsmcad
May 10 01:45:04 atlas kernel: [ 2895]     0  2895   118631       10   3       0             0 dsmc
May 10 01:45:04 atlas kernel: [ 3353]     0  3353    55603       76   3       0             0 smbd
May 10 01:45:04 atlas kernel: [ 4850]    65  4850  1470266   421823   0       0             0 nslcd
May 10 01:45:04 atlas kernel: [ 5396]     0  5396    28992        1   3       0             0 sshd
May 10 01:45:04 atlas kernel: [ 5399]     0  5399    27114        1   3       0             0 bash
May 10 01:45:04 atlas kernel: [ 5935]   303  5935    10229       11   0       0             0 nrpe
May 10 01:45:04 atlas kernel: [ 9367]     0  9367    35548        5   1       0             0 crond
May 10 01:45:04 atlas kernel: [ 9368]     0  9368     2304        1   3       0             0 sh
May 10 01:45:04 atlas kernel: [ 9371]     0  9371    45311      880   3       0             0 getfacl
May 10 01:45:04 atlas kernel: [ 9388]     0  9388    19678        1   1       0             0 sendmail
May 10 01:45:04 atlas kernel: [ 9389]     0  9389    19669        1   1       0             0 postdrop
May 10 01:45:04 atlas kernel: [ 9436]    89  9436    20206       18   1       0             0 pickup
May 10 01:45:04 atlas kernel: [10109]     0 10109    35547       87   1       0             0 crond
May 10 01:45:04 atlas kernel: [10113]     0 10113     2304       39   0       0             0 sh
May 10 01:45:04 atlas kernel: [10116]     0 10116    25575    19877   3       0             0 rsync
May 10 01:45:04 atlas kernel: [10117]     0 10117    14972      216   3       0             0 ssh
May 10 01:45:04 atlas kernel: [10158]     0 10158    35547       88   1       0             0 crond
May 10 01:45:04 atlas kernel: [10159]     0 10159     9602     2328   1       0             0 mrtg
May 10 01:45:04 atlas kernel: Out of memory: Kill process 4850 (nslcd) score 894 or sacrifice child
May 10 01:45:04 atlas kernel: Killed process 4850, UID 65, (nslcd) total-vm:5881064kB, anon-rss:1687276kB, file-rss:16kB
--------------------------------------------------

Now cron starts every five minutes a script which checks a memory usage by nslcd. Whenever nslcd uses more then 200MB RAM the script restarts the daemon. Bellow you can see frequency of the restarting.

---------------------------------

May 17 00:25:05 atlas nslcd[30469]: version 0.7.5 starting
May 17 00:30:05 atlas nslcd[30571]: version 0.7.5 starting
May 17 00:35:06 atlas nslcd[30655]: version 0.7.5 starting
May 17 00:40:05 atlas nslcd[30735]: version 0.7.5 starting
May 17 00:45:05 atlas nslcd[30822]: version 0.7.5 starting
May 17 00:50:06 atlas nslcd[30903]: version 0.7.5 starting
May 17 00:55:05 atlas nslcd[30991]: version 0.7.5 starting
May 17 01:00:05 atlas nslcd[31072]: version 0.7.5 starting
May 17 01:05:06 atlas nslcd[31175]: version 0.7.5 starting
May 17 01:10:05 atlas nslcd[31264]: version 0.7.5 starting
May 17 01:15:05 atlas nslcd[31344]: version 0.7.5 starting
May 17 01:20:06 atlas nslcd[31429]: version 0.7.5 starting
May 17 01:25:05 atlas nslcd[31511]: version 0.7.5 starting
May 17 01:30:05 atlas nslcd[31591]: version 0.7.5 starting
May 17 01:35:06 atlas nslcd[31680]: version 0.7.5 starting
May 17 01:40:05 atlas nslcd[31765]: version 0.7.5 starting
May 17 01:45:06 atlas nslcd[31846]: version 0.7.5 starting
May 17 01:50:01 atlas nslcd[31922]: version 0.7.5 starting

Sometimes it happens without getfacl ???

May 17 11:25:05 atlas nslcd[6786]: version 0.7.5 starting
May 17 11:55:01 atlas nslcd[7215]: version 0.7.5 starting
May 17 12:35:01 atlas nslcd[7775]: version 0.7.5 starting
May 17 13:15:04 atlas nslcd[8360]: version 0.7.5 starting
May 17 13:55:02 atlas nslcd[8907]: version 0.7.5 starting
May 17 14:35:01 atlas nslcd[9498]: version 0.7.5 starting
May 17 15:15:01 atlas nslcd[10070]: version 0.7.5 starting
May 17 15:55:01 atlas nslcd[10713]: version 0.7.5 starting
--------------------------------------------------------------


/etc/nslcd.conf

--------------------------------
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.


# Mappings for Services for UNIX 3.5
#filter passwd (objectClass=User)
#map    passwd uid              msSFU30Name
#map    passwd userPassword     msSFU30Password
#map    passwd homeDirectory    msSFU30HomeDirectory
#map    passwd homeDirectory    msSFUHomeDirectory
#filter shadow (objectClass=User)
#map    shadow uid              msSFU30Name
#map    shadow userPassword     msSFU30Password
#filter group  (objectClass=Group)
#map    group  uniqueMember     msSFU30PosixMember

# Mappings for Active Directory
#pagesize 1000
#referrals off
#filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map    passwd uid              sAMAccountName
#map    passwd homeDirectory    unixHomeDirectory
#map    passwd gecos            displayName
#filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map    shadow uid              sAMAccountName
#map    shadow shadowLastChange pwdLastSet
#filter group  (objectClass=group)
#map    group  uniqueMember     member

uid nslcd
gid ldap
# This comment prevents repeated auto-migration of settings.
uri ldap://10.0.8.80 10.0.8.181/
base dc=our_domain,dc=ourcompany,dc=cz
binddn cn=AD Reader,cn=users,dc=our_domain,dc=ourcompany,dc=cz
bindpw wiped
scope sub
ssl no
base dc=our_domain,dc=ourcompany,dc=cz
ldap_version 3

base passwd         DC=our_domain,DC=ourcompany,DC=cz
base shadow         CN=Users,DC=our_domain,DC=ourcompany,DC=cz
base group          CN=Users,DC=our_domain,DC=ourcompany,DC=cz

filter passwd (&(ObjectClass=person)(ObjectCategory=person)(uidNumber=*))
map passwd uid sAMAccountName
# map passwd uniqueMember Member
map passwd userPassword msSFUPassword
map passwd homeDirectory msSFUHomeDirectory
filter group  (&(objectClass=Group)(gidNumber=*))
map group cn msSFUName
map    group  uniqueMember     member

reportnslcd.txt (18,564 bytes)

Issue History

Date Modified Username Field Change
2012-05-17 15:21 arno3t New Issue
2012-05-17 15:21 arno3t File Added: reportnslcd.txt