View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0017245CentOS-7CentOS-7-Pluspublic2020-04-13 04:102021-11-19 02:51
Reporterzhanghaozz Assigned To 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.6.1810 
Summary0017245: When the network fails and then recovers, NFS cannot reestablish a connection
DescriptionClient error message：
xs_tcp_state_change client ffff8b6696f85000...
state 7 conn 0 dead 0 zapped 1 sk_shutdown 3
.....
call_connect xprt ffff8b6696513000 is connected

my analyse:
1.CentOS 7.4 and CentOS 7.6 were tested respectively, and CentOS 7.4 did not have this problem
2.Analyze CentOS 7.4 and 7.6 kernel code,
in xs_tcp_state change processing :
When case TCP_Close, The call of centos7.6 is xs_tcp_force_close, and it must meet (test_and_set_bit(XPRT_LOCKED, &xprt->state) == 0) before it can be cleaned up. When there is a problem, the condition is not satisfied, so that it cannot be cleaned up, so that the connection cannot be destroyed and the new connection can be re established.
Steps To ReproduceReiteration steps
1. Use centos7.6 as the NFS client and mount it on the NFS server of centos7.4
2. If the network card fails on the NFS server, restart it in 1 minute
3. NFS client cannot establish connection, and access stuck
TagsNo tags attached.
abrt_hash
URL

Activities

TrevorH

TrevorH

2020-04-13 10:59

manager   ~0036682

Neither 7.4 nor 7.6 are supported versions. The current version is 7.7 and 7.8 is in the CR repo waiting. Both of these versions have fixes to NFS code. Test with the CR repo enabled using `yum --enablerepo=cr update` then reboot into the 3.10.0-1127 kernel and retest.
zhanghaozz

zhanghaozz

2020-04-14 08:19

reporter   ~0036684

Thank you for your reply.
I checked the kernel code corresponding to 7.7, and the processing logic of xs_tcp_state_change has been modified, just like that of 7.4. But no corresponding modification point was found on the release note of 7.7
At the same time, the 7.8 version and the corresponding kernel source code were not found, only 8.0 was found.
Another question is whether the kernel version of CentOS release belt does not correspond to the version of kernel official website, because no corresponding version is found on kernel official website.
zhanghaozz

zhanghaozz

2020-04-14 08:23

reporter   ~0036685

7.6---->kernel version 3.10.0-957
case TCP_CLOSE:
        if (test_and_clear_bit(XPRT_SOCK_CONNECTING,
                    &transport->sock_state))
            xprt_clear_connecting(xprt);
        clear_bit(XPRT_CLOSING, &xprt->state);
        if (sk->sk_err)
            xprt_wake_pending_tasks(xprt, -sk->sk_err);
        /* Trigger the socket release */
        xs_tcp_force_close(xprt);

And 7.7 ----->kernel version 3.10.0-1062
ase TCP_CLOSE:
        if (test_and_clear_bit(XPRT_SOCK_CONNECTING,
                    &transport->sock_state))
            xprt_clear_connecting(xprt);
        if (sk->sk_err)
            xprt_wake_pending_tasks(xprt, -sk->sk_err);
        xs_sock_mark_closed(xprt);
ManuelWolfshant

ManuelWolfshant

2020-04-14 09:08

manager   ~0036687

Last edited: 2020-06-20 09:05

 All sources are at git.centos.org.
Please note that we do not and will not support other versions than the last released ones. If you need support for older releases, you will have to purchase Extended Update Support from RHEL. Mind that 7.4 is not supported even by RH since 31 August 2019 and 7.6 will be supported only until 31 Oct 2020.

On top of that, if you are confident that the last released kernel has a bug, you are encouraged to file a bug at bugzilla.redhat.com . Once RH fixes the issue, the fix will automatically propagate to CentOS
zhanghaozz

zhanghaozz

2020-04-15 05:54

reporter   ~0036690

ok，thanks.
tingyin.duan@gmail.com

tingyin.duan@gmail.com

2021-11-19 02:51

reporter   ~0038742

3.10.0-957.27.2 fix this bug.
957vs957.27.2.png (764,560 bytes)

Issue History

Date Modified Username Field Change
2020-04-13 04:10 zhanghaozz New Issue
2020-04-13 10:59 TrevorH Note Added: 0036682
2020-04-14 08:19 zhanghaozz Note Added: 0036684
2020-04-14 08:23 zhanghaozz Note Added: 0036685
2020-04-14 09:08 ManuelWolfshant Note Added: 0036687
2020-04-15 05:54 zhanghaozz Note Added: 0036690
2020-06-20 09:05 ManuelWolfshant Note Edited: 0036687
2021-11-19 02:51 tingyin.duan@gmail.com Note Added: 0038742
2021-11-19 02:51 tingyin.duan@gmail.com File Added: 957vs957.27.2.png