View Issue Details

IDProjectCategoryView StatusLast Update
0018084CentOS-8cifs-utilspublic2021-03-24 21:55
Reporterthenomad Assigned To 
PrioritynormalSeveritymajorReproducibilitysometimes
Status newResolutionopen 
Platformamd64OSCentOS 8OS Version8.3.2011
Product Version8.3.2011 
Summary0018084: automount of CIFS share works for some users but not others
DescriptionThis *might* be related to a problem report from May, 2020:
https://serverfault.com/questions/1025734/cifs-automounts-suddenly-stopped-working

This problem does not happen in CentOS 7 with exactly the same auto.cifs
file. It is happening in CentOS 8 (see below for version information)

Two users with exactly the same group memberships. One can access
automounted CIFS mount, the other can't. User 'nomad' was created by
cloning user 'lvd' in AD, then assigning a new UID and password. All
group memberships were retained. Both accounts are in Active Directory,
using sssd to access. The problem always happens for user 'nomad' but
never happens for user 'lvd'.

/cifs/footest was unmounted and autofs was reloaded between each of the
following tests to make sure everything was clean.

user 'lvd' can see the contents:
  : || lvd@nomaddev ~ [1251] ; ls /cifs/footest
  galf

user 'nomad' can not:
  [nomad@nomaddev ~]$ ls /cifs/footest
  ls: cannot open directory '/cifs/footest': No such file or directory

Additionally, user lvd can access /cifs/footest even with nomad's kerberos credentials:
  : || lvd@nomaddev ~ [1254] ; kinit nomad
  Password for nomad@[REDACTED]
  : || lvd@nomaddev ~ [1255] ; ls /cifs/footest
  galf

As expected, user lvd can not access the mount without credentials:
  : || lvd@nomaddev ~ [1261] ; kdestroy -A
  : || lvd@nomaddev ~ [1262] ; ls /cifs/footest
  ls: cannot open directory '/cifs/footest': No such file or directory

I note 'cifs.upcall[23271]: get_tgt_time: unable to get principal' logged during the failed attempt (from journactl, see below)

The following were logged with /proc/fs/cifs/cifsFYI set to '1'.

from journalctl, successful access as lvd:
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: key description: cifs.spnego;0;0;39010000;ver=0x2;host=testfs1;ip4=140.142.220.125;sec=krb5;uid=0x4b80;creduid=0x4b80;user=lvd;pid=0x5b88
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: ver=2
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: host=testfs1
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: ip=140.142.220.125
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: sec=1
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: uid=19328
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: creduid=19328
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: user=lvd
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: pid=23432
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: get_cachename_from_process_env: pathname=/proc/23432/environ
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: get_existing_cc: default ccache is KCM:19328
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: handle_krb5_mech: getting service ticket for testfs1
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: handle_krb5_mech: obtained service ticket
  Feb 25 11:51:18 nomaddev.[redacted] cifs.upcall[23435]: Exit status 0

from journactl, unsuccessful access as nomad:
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: key description: cifs.spnego;0;0;39010000;ver=0x2;host=testfs1;ip4=140.142.220.125;sec=krb5;uid=0xf428c;creduid=0xf428c;user=nomad;pid=0x5ae4
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: ver=2
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: host=testfs1
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: ip=140.142.220.125
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: sec=1
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: uid=1000076
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: creduid=1000076
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: user=nomad
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: pid=23268
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: get_cachename_from_process_env: pathname=/proc/23268/environ
  Feb 25 11:47:23 nomaddev.[redacted] systemd[1]: Starting SSSD Kerberos Cache Manager...
  Feb 25 11:47:23 nomaddev.[redacted] systemd[1]: Started SSSD Kerberos Cache Manager.
  Feb 25 11:47:23 nomaddev.[redacted] kcm[23274]: Starting up
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: get_existing_cc: default ccache is KCM:1000076
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: get_tgt_time: unable to get principal
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: krb5_get_init_creds_keytab: -1765328174
  Feb 25 11:47:23 nomaddev.[redacted] cifs.upcall[23271]: Exit status 1
  Feb 25 11:47:23 nomaddev.[redacted] kernel: CIFS VFS: \\testfs1 Send error in SessSetup = -126
  Feb 25 11:47:23 nomaddev.[redacted] kernel: CIFS VFS: cifs_mount failed w/return code = -2

from journactl, unsuccessful access as lvd with no credentials:
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: key description: cifs.spnego;0;0;39010000;ver=0x2;host=testfs1;ip4=140.142.220.125;sec=krb5;uid=0x4b80;creduid=0x4b80;user=lvd;pid=0x6290
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: ver=2
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: host=testfs1
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: ip=140.142.220.125
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: sec=1
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: uid=19328
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: creduid=19328
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: user=lvd
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: pid=25232
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: get_cachename_from_process_env: pathname=/proc/25232/environ
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: get_existing_cc: default ccache is KCM:19328
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: get_tgt_time: unable to get principal
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: krb5_get_init_creds_keytab: -1765328174
  Feb 25 13:20:49 nomaddev.[redacted] cifs.upcall[25235]: Exit status 1
  Feb 25 13:20:49 nomaddev.[redacted] kernel: CIFS VFS: \\testfs1 Send error in SessSetup = -126
  Feb 25 13:20:49 nomaddev.[redacted] kernel: CIFS VFS: cifs_mount failed w/return code = -2


Client:
  CentOS Linux release 8.3.2011
  Linux nomaddev.[redacted] 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19 17:20:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

relevant map entry:
  footest -fstype=cifs,vers=2.1,sec=krb5,user=$USER,uid=$UID,gid=$GID,cruid=$UID,file_mode=0600,dir_mode=0700,nounix,noserverino ://testfs1/footest

file server: OmniOS 151030 (same problem with 151036)

I do not have the resources to test this from any other file server.
Steps To ReproduceAn an AD environment, smbshare a filesystem with relevant permissions

on client, add a map entry to auto.cifs (or equivalent) autofs map that points to it.

attempt to ls the share. May have to try with multiple users.

See description for more details.
Tagsautofs, cifs

Activities

thenomad

thenomad

2021-03-17 19:59

reporter   ~0038303

For the record, I just tested this with a CentOS 8 Stream box and had the same result.

4.18.0-277.el8.x86_64

nomad
thenomad

thenomad

2021-03-22 16:31

reporter   ~0038311

It appears to be a problem with KRB5CCNAME.
  The user who could access the CIFS mount did not have it set.
  The user who could not access the CIFS mount did have it set.

Both users' caches were in /tmp/krb5cc_... files.

When KRB5CCNAME is set it seems to be correct, pointing to the same value as the cache reported by klist:

[nomad@gertrude ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000076_Xq1K7p
Default principal: nomad@[REDACTED]

Valid starting Expires Service principal
03/22/2021 09:12:08 03/22/2021 19:12:08 krbtgt/[REDACTED]@[REDACTED]
    renew until 03/22/2021 19:12:08
[nomad@gertrude ~]$ echo $KRB5CCNAME
KRB5CCNAME=FILE:/tmp/krb5cc_1000076_Xq1K7p

However, journalctl -xe says it's not entirely happy with the mapping in some way

  get_existing_cc: default ccache is KCM:1000076
  get_tgt_time: unable to get principal

As a test I unset it, kinit, then could successfully access the cifs filesystem I was denied previously.
  get_existing_cc: default ccache is KCM:1000076
  handle_krb5_mech: getting service ticket for [redacted]

Oddly, it looks like I may have had to only do this once, as additional access - even after a reboot and the variable being set again through a new login - has continued to work. (I should note that is only about 30 minutes so far, I don't know if it will survive a ticket expiration and re-issue yet.)
thenomad

thenomad

2021-03-24 15:15

reporter   ~0038323

More notes for future people who find this ticket...

This is about ssh, not console (I haven't had a chance to test console because of the WFH thing).

I've narrowed down the 'why is it different for different users' question. It isn't the 'different users' it's 'different kerberos starting points'.
 - If you have an existing, valid kerberos ticket and can ssh in without entering a password the system does not set $KRB5CCNAME and things like CIFS mount access work. (klist shows the ticket cache as KCM:$UID:#####)
 - If you do not have an existing kerberos ticket so ssh prompts you for a password the system sets $KRB5CCNAME to point to FILE:/tmp/krb5cc_$UID_noise and CIFS mount access does not work.

The reason CIFS mounts don't work is because it really, Really, REALLY wants a KCM:$UID instead of a FILE:/ cache. It appears either autofs or something deeper in the chain is ignoring/not-aware-of $KRB5CCNAME.
  : || lvd@gertrude ~ [1003] ; klist
  Ticket cache: FILE:/tmp/krb5cc_19328_U92QF5
  ...

  Mar 24 08:10:49 [redacted] cifs.upcall[18113]: get_existing_cc: default ccache is KCM:19328
  Mar 24 08:10:49 [redacted] cifs.upcall[18113]: get_tgt_time: unable to get principal
thenomad

thenomad

2021-03-24 21:55

reporter   ~0038327

Basically, the bug is that autofs ignores the setting/existence of the $KRB5CCNAME environment variable.

I have a work-around of removing all references to 'krb5_cc*" from /etc/sssd/sssd.conf but that's not the right fix.

The right fix is to have autofs pick up the correct kerberos reference.

Issue History

Date Modified Username Field Change
2021-02-25 22:17 thenomad New Issue
2021-02-25 22:17 thenomad Tag Attached: autofs
2021-02-25 22:17 thenomad Tag Attached: cifs
2021-03-17 19:59 thenomad Note Added: 0038303
2021-03-22 16:31 thenomad Note Added: 0038311
2021-03-24 15:15 thenomad Note Added: 0038323
2021-03-24 21:55 thenomad Note Added: 0038327