View Issue Details

IDProjectCategoryView StatusLast Update
0017740CentOS-8cifs-utilspublic2021-03-29 08:35
Reporterhurleyef Assigned To 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version8.2.2004 
Summary0017740: Cifs transfers fail after about 2MB
DescriptionCifs transfers fail after about 2MB. This issue is reproducible 100% of the time for me on both kernel 4.18.0-193.14.2.el8_2.x86_64 and 4.18.0-193.19.1.el8_2.x86_64. Whether the share is being hosted by a Windows server or Centos doesn't seem to make a difference. I've tested this with VMs on hyper-v and esxi. The only workaround I know of is to blacklist the 8.2 kernels in yum.
Steps To Reproduce1. Install cifs-utils on a centos 8.2 machine.
2. Mount a samba or smb share hosted on a remote server.
3. Copy a file larger than 3MB from the network share to a local device.
Additional Information[ 11.894350] vmxnet3 0000:0b:00.0 ens192: intr type 3, mode 0, 3 vectors allocated
[ 11.895744] vmxnet3 0000:0b:00.0 ens192: NIC Link is Up 10000 Mbps
[ 81.179155] Key type dns_resolver registered
[ 81.340491] Key type cifs.spnego registered
[ 81.340499] Key type cifs.idmap registered
[ 81.341712] CIFS: Attempting to mount //smb.int.domain.tld/share
[ 83.744379] srv rsp padded more than expected. Length 384 not 208 for cmd:5 mid:5
[ 194.582292] WARNING: CPU: 0 PID: 5394 at mm/page_alloc.c:4106 __alloc_pages_slowpath+0xbc8/0xd60
[ 194.582295] Modules linked in: arc4 libarc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs dns_resolver nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nft_chain_route_ipv6 nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nft_chain_route_ipv4 ip6_tables nft_compat ip_set nf_tables nfnetlink vmw_vsock_vmci_transport vsock vfat fat ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel vmw_balloon intel_rapl_perf joydev pcspkr vmw_vmci i2c_piix4 ip_tables xfs libcrc32c ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm sg ata_piix drm crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
[ 194.582462] CPU: 0 PID: 5394 Comm: cifsd Not tainted 4.18.0-193.19.1.el8_2.x86_64 #1
[ 194.582464] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.13989454.B64.1906190538 06/19/2019
[ 194.582469] RIP: 0010:__alloc_pages_slowpath+0xbc8/0xd60
[ 194.582483] Code: 24 0c 00 00 00 00 c7 44 24 50 00 00 00 00 c7 44 24 3c 01 00 00 00 e9 71 fd ff ff 31 c0 41 81 e6 00 02 00 00 0f 85 14 fa ff ff <0f> 0b e9 0d fa ff ff 65 48 8b 04 25 80 5c 01 00 8b 74 24 58 3b b0
[ 194.582485] RSP: 0018:ffffa42582efb928 EFLAGS: 00010246
[ 194.582488] RAX: 0000000000000000 RBX: 0000000000484020 RCX: ffff8c777ffd7cd0
[ 194.582490] RDX: ffffa42582efba20 RSI: 000000000000000b RDI: 0000000000484020
[ 194.582491] RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000000701
[ 194.582492] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000484020
[ 194.582493] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8c775f42b800
[ 194.582496] FS: 0000000000000000(0000) GS:ffff8c777fc00000(0000) knlGS:0000000000000000
[ 194.582498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 194.582499] CR2: 000056230a080000 CR3: 0000000022a0a005 CR4: 00000000000606f0
[ 194.582571] Call Trace:
[ 194.582655] ? skb_copy_bits+0x70/0x250
[ 194.582667] ? vmxnet3_tq_xmit.isra.63+0x5c2/0xe00 [vmxnet3]
[ 194.582716] ? __switch_to_asm+0x35/0x70
[ 194.582720] ? __switch_to_asm+0x35/0x70
[ 194.582722] ? __switch_to_asm+0x35/0x70
[ 194.582725] ? __switch_to_asm+0x35/0x70
[ 194.582727] ? __switch_to_asm+0x41/0x70
[ 194.582731] __alloc_pages_nodemask+0x245/0x280
[ 194.582756] kmalloc_order+0x14/0x30
[ 194.582761] kmalloc_order_trace+0x1d/0xa0
[ 194.582812] gcmaes_decrypt.constprop.15+0xd4/0x380
[ 194.582853] ? __crypto_alloc_tfm+0x14f/0x180
[ 194.582857] ? crypto_alloc_base+0x3c/0xd0
[ 194.582861] ? __alloc_pages_nodemask+0xef/0x280
[ 194.582866] ? generic_gcmaes_decrypt+0x56/0x70
[ 194.582921] ? crypt_message+0x5cc/0x8d0 [cifs]
[ 194.582925] ? generic_gcmaes_decrypt+0x56/0x70
[ 194.582965] ? crypt_message+0x7ce/0x8d0 [cifs]
[ 194.583006] ? decrypt_raw_data+0xb2/0x140 [cifs]
[ 194.583047] ? smb3_receive_transform+0x61a/0x7b0 [cifs]
[ 194.583080] ? cifs_demultiplex_thread+0xaf0/0xbd0 [cifs]
[ 194.583113] ? cifs_handle_standard+0x1a0/0x1a0 [cifs]
[ 194.583132] ? kthread+0x112/0x130
[ 194.583137] ? kthread_flush_work_fn+0x10/0x10
[ 194.583140] ? ret_from_fork+0x35/0x40
[ 194.583144] ---[ end trace 8f600f3e7b33aa6f ]---
TagsNo tags attached.

Activities

yrro

yrro

2021-03-26 13:13

reporter   ~0038336

I'm seeing this too, on RHEL CoreOS 46.82.202103050041-0 (Ootpa) with kernel 4.18.0-193.41.1.el8_2.x86_64.

The server is running samba-4.12.3-12.el8.3.x86_64 on CentOS 8.

One difference is that I don't see this when mounting a share from a Windows server--only when the Samba server.

Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: WARNING: CPU: 1 PID: 4192061 at mm/page_alloc.c:4106 __alloc_pages_slowpath+0xbc8/0xd60
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: Modules linked in: arc4 libarc4 md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs dns_resolver veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_REDIRECT tcp_diag inet_diag nft_chain_route_ipv4 ipt_MASQUERADE xt_conntrack xt_comment nft_counter xt_mark nft_compat nft_chain_nat_ipv6 nft_chain_nat_ipv4 nf_tables overlay vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_conncount nf_nat nf_conntrack rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_umad vfat fat iw_cxgb4 ext4 ib_uverbs rdma_cm iw_cm ib_cm mbcache ib_core jbd2 cirrus drm_kms_helper syscopyarea sysfillrect intel_rapl_msr sysimgblt intel_rapl_common fb_sys_fops drm joydev pcspkr virtio_balloon i2c_piix4 ip_tables xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: virtio_net ata_piix net_failover failover libata ghash_clmulni_intel virtio_scsi virtio_blk serio_raw dm_multipath dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: CPU: 1 PID: 4192061 Comm: dd Not tainted 4.18.0-193.41.1.el8_2.x86_64 #1
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: Hardware name: Red Hat OpenStack Compute, BIOS 1.11.0-2.el7 04/01/2014
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: RIP: 0010:__alloc_pages_slowpath+0xbc8/0xd60
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: Code: 24 0c 00 00 00 00 c7 44 24 50 00 00 00 00 c7 44 24 3c 01 00 00 00 e9 71 fd ff ff 31 c0 41 81 e6 00 02 00 00 0f 85 14 fa ff ff <0f> 0b e9 0d fa ff ff 65 48 8b 04 25 80 5c 01 00 8b 74 24 58 3b b0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: RSP: 0018:ffffafc2060d75f8 EFLAGS: 00010246
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: RAX: 0000000000000000 RBX: 0000000000484020 RCX: 0000000000000000
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: RDX: ffffafc2060d76f0 RSI: 000000000000000b RDI: 0000000000484020
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 000000000000078a
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000484020
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff9183b3a8fc50
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: FS: 00007fad60929580(0000) GS:ffff91872fa80000(0000) knlGS:0000000000000000
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: CR2: 00007fad60925000 CR3: 0000000021c6a005 CR4: 00000000003606e0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: Call Trace:
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x41/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x41/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x35/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x41/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x35/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __switch_to_asm+0x41/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? finish_task_switch+0x76/0x2b0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __schedule+0x257/0x650
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: __alloc_pages_nodemask+0x245/0x280
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: kmalloc_order+0x14/0x30
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: kmalloc_order_trace+0x1d/0xa0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: gcmaes_encrypt.constprop.16+0x1df/0x310
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? _cond_resched+0x15/0x30
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __crypto_alloc_tfm+0x14f/0x180
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? crypto_alloc_base+0x3c/0xd0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? reschedule_interrupt+0xa/0x20
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? generic_gcmaes_encrypt+0x56/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? crypt_message+0x5cc/0x8d0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? generic_gcmaes_encrypt+0x56/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? crypt_message+0x666/0x8d0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? smb3_init_transform_rq+0x26f/0x2f0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? smb_send_rqst+0xd4/0x180 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_call_async+0x12b/0x290 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? smb2_async_writev+0x241/0x4a0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_echo_callback+0x60/0x60 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __test_set_page_writeback+0x1f8/0x3f0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_writepages+0x804/0xaf0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_writepages+0x804/0xaf0 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? do_writepages+0x41/0xd0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_strict_writev+0x142/0x250 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __filemap_fdatawrite_range+0xbe/0xf0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? filemap_write_and_wait+0x2c/0x70
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? cifs_flush+0x73/0x80 [cifs]
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? filp_close+0x31/0x60
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? __x64_sys_close+0x1e/0x50
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? do_syscall_64+0x5b/0x1a0
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ? entry_SYSCALL_64_after_hwframe+0x65/0xca
Mar 26 11:02:02 ocp-por-rfjsq-master-10 kernel: ---[ end trace 5da0a87028646ec9 ]---
hurleyef

hurleyef

2021-03-27 01:35

reporter   ~0038337

I filed a bug report for this with redhat a while back after I got no response here. They verified it and there is a fix in the pipeline.

There is a knowledge base article here:
https://access.redhat.com/solutions/5762091


I believe this is fixed in kernel-4.18.0-246.el8. The patch has already been live on Centos Stream for some time now.
yrro

yrro

2021-03-29 08:35

reporter   ~0038340

Thanks for that - downgrading to vers=3 does indeed work around the problem.

Issue History

Date Modified Username Field Change
2020-09-19 23:57 hurleyef New Issue
2021-03-26 13:13 yrro Note Added: 0038336
2021-03-27 01:35 hurleyef Note Added: 0038337
2021-03-29 08:35 yrro Note Added: 0038340