View Issue Details

IDProjectCategoryView StatusLast Update
0013569CentOS-7kernelpublic2017-07-19 10:12
Reporterscottcai 
PriorityimmediateSeveritymajorReproducibilitysometimes
Status newResolutionopen 
Product Version7.2.1511 
Target VersionFixed in Version 
Summary0013569: High Send-Q, and empty Recv-Q. TCP retransmission did not work?
DescriptionClient handshake with server, and connection established. At begining, it works well. But 1s later, the server kept sending dup ack (ackno = 179521), and the client seems to ignore the mistake, and still send packets with increasing ackno > 179521. At the fallowing 15min, the client didn't send any packets, then client timeout and reset this connection.
If lookup with netstat -an, show the connection status: client's Send-Q is 300736, server's Recv-Q is 0. But client and server transmits datas via local loopback, it's looks like the network or kernel didn't work.

A) netstat output (59256 is client, 3260 is server, high Send-Q, 0 Recv-Q):

    tcp 0 0 10.154.56.37:3260 10.154.56.37:59256 ESTABLISHED
    tcp 0 300736 10.154.56.37:59256 10.154.56.37:3260 ESTABLISHED

B) cat /proc/net/tcp output (cwnd = 1, RTO = 12ms, RTT = 30us)

    8: 8A3B9A0A:0CBC 8A3B9A0A:C04C 01 00000000:00000000 02:0001CB79 00000000 996 0 354327 2 ffff8800d7452d00 20 4 30 10 -1
    15: 8A3B9A0A:C04C 8A3B9A0A:0CBC 01 0002FF70:00000000 01:0000188D 0000000D 0 0 353609 2 ffff8800d7454380 12000 4 1 1 12

C) tcpdump output:

1) normal handshake

    20:48:48.566160 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [S], seq 3304205286, win 65535, options [mss 65495,sackOK,TS val 4167260 ecr 0,nop,wscale 2], length 0
    11:44:05.586221 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [S.], seq 1361916887, ack 3304205287, win 43690, options [mss 65495,sackOK,TS val 4167260 ecr 4167260,nop,wscale 7], length 0
    20:48:48.566197 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [.], ack 1, win 32748, options [nop,nop,TS val 4167260 ecr 4167260], length 0


2) normal send & recv

    20:48:48.566439 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 1:49, ack 1, win 32748, options [nop,nop,TS val 4167260 ecr 4167260], length 48
    20:48:48.566453 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 49, win 342, options [nop,nop,TS val 4167260 ecr 4167260], length 0
    20:48:48.566465 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 49:513, ack 1, win 32748, options [nop,nop,TS val 4167260 ecr 4167260], length 464
    20:48:48.566471 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 513, win 350, options [nop,nop,TS val 4167260 ecr 4167260], length 0
    20:48:48.835560 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [P.], seq 1:349, ack 513, win 350, options [nop,nop,TS val 4167529 ecr 4167260], length 348
    20:48:48.835583 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [.], ack 349, win 33016, options [nop,nop,TS val 4167529 ecr 4167529], length 0
    20:48:48.841985 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 513:561, ack 349, win 33016, options [nop,nop,TS val 4167536 ecr 4167529], length 48
    20:48:48.841995 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 561, win 350, options [nop,nop,TS val 4167536 ecr 4167536], length 0
    20:48:48.842141 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 561:609, ack 349, win 33016, options [nop,nop,TS val 4167536 ecr 4167536], length 48

3) **server kept ack 179521 and client still don't retransmission (TCP Dup Ack for 82 times).** last for a few seconds

    20:48:48.920088 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [P.], seq 1569:1617, ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:345121}], length 48
    20:48:48.920126 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 345121:345169, ack 1617, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 48
    20:48:48.920136 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:345169}], length 0
    20:48:48.920150 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 345169:345217, ack 1617, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 48
    20:48:48.920154 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:345217}], length 0
    20:48:48.920170 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 345217:349313, ack 1617, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 4096
    20:48:48.920173 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:349313}], length 0
    20:48:48.920390 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [P.], seq 1617:1665, ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:349313}], length 48
    20:48:48.920430 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 349313:349361, ack 1665, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 48
    20:48:48.920438 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:349361}], length 0
    20:48:48.920458 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 349361:349409, ack 1665, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 48
    20:48:48.920462 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:349409}], length 0
    20:48:48.920501 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [P.], seq 349409:353505, ack 1665, win 33016, options [nop,nop,TS val 4167614 ecr 4167614], length 4096
    20:48:48.920505 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4167614 ecr 4167536,nop,nop,sack 1 {200337:353505}], length 0

4) no packets at least 11 min

5) **client timeout, send rst**

    20:59:50.496428 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [R.], seq 353505, ack 1665, win 33016, options [nop,nop,TS val 0 ecr 4167614], length 0
    20:59:50.496447 IP 10.154.59.138.iscsi-target > 10.154.59.138.54795: Flags [.], ack 179521, win 3308, options [nop,nop,TS val 4829190 ecr 4167536,nop,nop,sack 1 {200337:353505}], length 0
    20:59:50.496460 IP 10.154.59.138.54795 > 10.154.59.138.iscsi-target: Flags [R], seq 3304384807, win 0, length 0
Steps To ReproduceRun iscsi-initiator, connect iscsi-target address, initiator and target is at the same machine, they transfer data via lo.

We put a lot pressure on the connection, and the connection broken.

For not a long time after handshake, the initiator lost a packet, and target send dup-ack (for 82 times), and the retransmission seems did not work. The initiator timeout at 11min later, and reset the connection.
Additional Informationthe tcpdump output is in the attachment.

sendQ high, recvQ zero.

RTO is 12000, cwnd is 1.
TagsNo tags attached.
abrt_hash
URL

Relationships

duplicate of 0013565 new High Send-Q, and empty Recv-Q. TCP retransmission did not work? 

Activities

scottcai

scottcai

2017-07-19 02:48

reporter  

kbug.pcap (260,458 bytes)

Issue History

Date Modified Username Field Change
2017-07-19 02:48 scottcai New Issue
2017-07-19 02:48 scottcai File Added: kbug.pcap
2017-07-19 10:12 tigalch Relationship added duplicate of 0013565