View Issue Details

IDProjectCategoryView StatusLast Update
0016107CentOS-7kernelpublic2019-06-12 01:42
Reporterino1988 
PriorityhighSeveritycrashReproducibilitysometimes
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0016107: nfs41_callback_svc related kernel panic
Descriptionsystem info:
[root@su02 ~]# uname -a
Linux su02 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

panic info:
[105457.823563] BUG: unable to handle kernel paging request at 0000000000001020
[105457.823616] IP: [<ffffffffc09db8c7>] svc_process_common+0x67/0x760 [sunrpc]
[105457.823677] PGD 8000001cbee9c067 PUD e17bc3067 PMD 0
[105457.823712] Oops: 0000 [#1] SMP
[105457.823735] Modules linked in: nfsv3 mxfs(OE) fuse ext4 mbcache jbd2 macvlan rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nfsd auth_rpcgss nfs_acl loc
kd grace ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_f
ilter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop binfmt_misc tcp_diag inet_dia
g overlay ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma sunrpc rdma_ucm ib_uverbs ib_iser ib_umad ib_i
poib rdma_cm iw_cm vfat ib_cm fat libiscsi scsi_transport_iscsi iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi k
vm_intel kvm irqbypass crc32_pclmul
[105457.824263] ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 ses enclosure mlx4_ib sg ib_core lpc_ich
 mei_me mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx4_en a
st crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core drm mpt3sas drm_panel_orientati
on_quirks igb devlink raid_class scsi_transport_sas ptp pps_core dca i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mxfs]
[105457.824657] CPU: 5 PID: 568639 Comm: nfsv4.1-svc Kdump: loaded Tainted: G OE ------------ 3.10.0-957.10.1.el7.x86_64 #1
[105457.824723] Hardware name: Inspur SA5212M4/YZMB-00370-102, BIOS 4.1.5 03/03/2016
[105457.824765] task: ffff8ad76bb530c0 ti: ffff8ad9bd700000 task.ti: ffff8ad9bd700000
[105457.824807] RIP: 0010:[<ffffffffc09db8c7>] [<ffffffffc09db8c7>] svc_process_common+0x67/0x760 [sunrpc]
[105457.824875] RSP: 0000:ffff8ad9bd703db0 EFLAGS: 00010202
[105457.824906] RAX: 0000000000001000 RBX: ffff8ad9f2bfa1e0 RCX: 00000000000000bc
[105457.824946] RDX: 00000000000000b4 RSI: ffff8ad9f2bfa1a0 RDI: ffff8ad9f2bfa000
[105457.824985] RBP: ffff8ad9bd703e08 R08: ffff8ad9bd703e80 R09: ffff8ada79ddce00
[105457.825024] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8ad9f2bfa000
[105457.825063] R13: ffff8ada1ee80408 R14: ffff8ad9f2bfa1a0 R15: ffff8ad9f2bfa1a0
[105457.825104] FS: 0000000000000000(0000) GS:ffff8adaffd40000(0000) knlGS:0000000000000000
[105457.825148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[105457.825180] CR2: 0000000000001020 CR3: 0000001092f32000 CR4: 00000000001607e0
[105457.825220] Call Trace:
[105457.825273] [<ffffffffc0693cf4>] ? __xfs_filemap_fault+0x64/0x1d0 [xfs]
[105457.825315] [<ffffffffa696878f>] ? __schedule+0x3ff/0x890
[105457.825360] [<ffffffffc09dc35d>] bc_svc_process+0x20d/0x390 [sunrpc]
[105457.825410] [<ffffffffc0d2b345>] nfs41_callback_svc+0x105/0x1c0 [nfsv4]
[105457.825451] [<ffffffffa62c2d40>] ? wake_up_atomic_t+0x30/0x30
[105457.825494] [<ffffffffc0d2b240>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
[105457.825532] [<ffffffffa62c1c71>] kthread+0xd1/0xe0
[105457.825562] [<ffffffffa62c1ba0>] ? insert_kthread_work+0x40/0x40
[105457.825599] [<ffffffffa6975c37>] ret_from_fork_nospec_begin+0x21/0x21
[105457.825636] [<ffffffffa62c1ba0>] ? insert_kthread_work+0x40/0x40
[105457.825670] Code: 17 0f 86 f5 00 00 00 4c 89 4d c0 f0 80 8f 98 1a 00 00 10 f0 80 8f 98 1a 00 00 04 f0 80 a7 98 1a 00 00 f7 48 8b 47 20 48 8b 40 08 <48>
8b 40 20 e8 70 b5 ba e5 41 8b 8c 24 80 1a 00 00 48 8b 43 08
[105457.827542] RIP [<ffffffffc09db8c7>] svc_process_common+0x67/0x760 [sunrpc]
[105457.829276] RSP <ffff8ad9bd703db0>
[105457.830987] CR2: 0000000000001020


This panic occur 2 times in 2 days.

When disabled nfsv41 via menuconfig and compile nfs, an error raised, and need the follow patch:
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index 0b9f20b..8d5badf 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -293,6 +293,8 @@ const struct file_operations_extend nfs4_file_operations = {
        .llseek = nfs_file_llseek,
 #endif
    },
+#ifdef CONFIG_NFS_V4_2
    .copy_file_range = nfs4_copy_file_range,
    .clone_file_range = nfs42_clone_file_range,
+#endif
 };
Steps To Reproduce1. system A and B .
2. install nfs server and client on A, install nfs client on B.
3. export a nfs shared dir on A .
4. mount the exported dis on A and B.
5. continuous write, read and filelock operation to files in the mounted nfs dir on A and B.

And then, the panic may occur.
Tags3.10.0-957.10.1.el7.x86_64, nfsv4.1
abrt_hash
URL

Activities

ino1988

ino1988

2019-05-23 01:34

reporter   ~0034500

core dump file:
yrl: https://pan.baidu.com/s/1inTbeL_UNELHc-r_ZPi90A
password: ta17
ino1988

ino1988

2019-05-23 01:41

reporter   ~0034501

may the same issue : https://access.redhat.com/solutions/3820742
ino1988

ino1988

2019-05-23 02:13

reporter   ~0034502

Is there any way to disable nfsv4.1 permanently except by modify menuconfig and recompile nfs kernel modules ?
Thanks.
gaby.kropp@gmail.com

gaby.kropp@gmail.com

2019-06-04 10:36

reporter   ~0034591

Hello,
I am facing the same issue in my company, 30 people impacted.
Happening up to 2 times / day.

Users have NIS + NFS automount with NFS4 homedirs to work on, continuous writing.
When I disable autofs, it works without crash.

Something strange is that my clients are mounting with 4.0 version, but bugtrace is talking about nfs41:
nfs:/srv/nfs/userb/guihalt on /home/guihalt type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.8.165,local_lock=none,addr=10.11.255.38)

[24423.565160] Call Trace:
[24423.565167] [<ffffffff8976896f>] ? __schedule+0x3ff/0x890
[24423.565181] [<ffffffffc0b3635d>] bc_svc_process+0x20d/0x390 [sunrpc]
[24423.565198] [<ffffffffc0eb9345>] nfs41_callback_svc+0x105/0x1c0 [nfsv4]
[24423.565204] [<ffffffff890c2d40>] ? wake_up_atomic_t+0x30/0x30
[24423.565216] [<ffffffffc0eb9240>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
[24423.565219] [<ffffffff890c1c71>] kthread+0xd1/0xe0
[24423.565223] [<ffffffff890c1ba0>] ? insert_kthread_work+0x40/0x40
[24423.565228] [<ffffffff89775bdd>] ret_from_fork_nospec_begin+0x7/0x21
[24423.565232] [<ffffffff890c1ba0>] ? insert_kthread_work+0x40/0x40

So it's affecting both 4.0 and 4.1 clients?
Thanks,
Gabriel
ino1988

ino1988

2019-06-12 01:42

reporter   ~0034644

Hello,
     NfsV4.0 has no this bug. My env is working for about two weeks after disable nfsv4.1.

Issue History

Date Modified Username Field Change
2019-05-22 10:28 ino1988 New Issue
2019-05-22 10:28 ino1988 Tag Attached: nfsv4.1 3.10.0-957.10.1.el7.x86_64
2019-05-22 10:29 ino1988 Tag Detached: nfsv4.1 3.10.0-957.10.1.el7.x86_64
2019-05-22 10:29 ino1988 Tag Attached: nfsv4.1
2019-05-22 10:29 ino1988 Tag Attached: 3.10.0-957.10.1.el7.x86_64
2019-05-23 01:34 ino1988 Note Added: 0034500
2019-05-23 01:41 ino1988 Note Added: 0034501
2019-05-23 02:13 ino1988 Note Added: 0034502
2019-05-23 06:25 TrevorH Note View State: 0034502: private
2019-05-23 06:25 TrevorH Note View State: 0034502: public
2019-06-04 10:36 gaby.kropp@gmail.com Note Added: 0034591
2019-06-12 01:42 ino1988 Note Added: 0034644