View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0007796 | CentOS-6 | kernel | public | 2014-10-29 00:30 | 2016-11-21 17:58 |
| Reporter | bwelmers | ||||
| Priority | normal | Severity | major | Reproducibility | always |
| Status | closed | Resolution | no change required | ||
| Product Version | 6.6 | ||||
| Target Version | Fixed in Version | ||||
| Summary | 0007796: ipv6 connectivity breaks down after upgrading to kernel 2.6.32-504.el6 | ||||
| Description | After I ran upgrade to CentOS 6.6, the ipv6 connectivity breaks down after a period of about 5 minutes. When I initate ping to another host in the same subnet, communication to and from that host is possible again for a few minutes and then breaks down again. Configuration: # brctl show bridge name bridge id STP enabled interfaces br0 8000.6c626dcf364b no eth0 # ifconfig br0 br0 Link encap:Ethernet HWaddr 6C:62:6D:CF:36:4B inet addr:192.168.0.66 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b/64 Scope:Global inet6 addr: fe80::6e62:6dff:fecf:364b/64 Scope:Link When I boot the machine, it obtains an ipv6 prefix from the network routing advertisor and sets up an IP address. ipv6 connectivity on this address works for several minutes, but then suddenly stops working. The ipv6 address is no longer pingable from another machine within the same network: veriton$ ping6 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b PING 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b(2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b) 56 data bytes ^C --- 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2016ms Same when I try to ping the link-local address with fe80::6e62:6dff:fecf:364b%eth0 However, when I initiate a ping6 from my CentOS machine to the remote machine within the same network: # ping6 veriton PING veriton(2001:XXXX:XXXX:XXXX::2) 56 data bytes 64 bytes from 2001:XXXX:XXXX:XXXX::2: icmp_seq=1 ttl=64 time=1.22 m it works. and by then, also reverse-ping works again (of course), but only for several minutes: veriton$ ping6 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b PING 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b(2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b) 56 data bytes 64 bytes from 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b: icmp_seq=1 ttl=255 time=1.25 ms Because the default route is advertised as a link-local address and return path will be the global ipv6 address, so also all outgoing connections will stuck within a few minutes. I have connected devices with several different physical paths and switches, so problems in network hardware can be eliminated. When I reboot back to kernel 2.6.32-431.29.2.el6 the problems are gone. | ||||
| Additional Information | Extra check: ip6tables is empty: # ip6tables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Full kernel version string: 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux network adaptor: 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03) | ||||
| Tags | No tags attached. | ||||
|
Can you show us the device ID pairing of your NIC? lspci -nn | grep -i eth |
|
|
Hello, same problem with two hosts, others running fine. All hosts have the same virtual-NIC. IPv6 stops working after some time, sometimes 2 hours. 00:03.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000] |
|
| 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03) | |
|
Does it help if you turn off multicast snooping? echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping |
|
| yes, disable multicast_snooping on all bridges on the host help solve the problem | |
|
Then this is likely missing backport of this commit: commit cc0fdd802859eaeb00e1c87dbb655594bed2844c Author: Linus Lüssing <linus.luessing@web.de> Date: Fri Aug 30 17:28:17 2013 +0200 bridge: separate querier and query timer into IGMP/IPv4 and MLD/IPv6 ones Currently we would still potentially suffer multicast packet loss if there is just either an IGMP or an MLD querier: For the former case, we would possibly drop IPv6 multicast packets, for the latter IPv4 ones. This is because we are currently assuming that if either an IGMP or MLD querier is present that the other one is present, too. You could confirm that by running tcpdump to check, that you are seeing IGMP queries on your network and at the same time there are no MLD ones. As a workaround you can: - disable CONFIG_BRIDGE_IGMP_SNOOPING=y and rebuild kernel or - echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping |
|
|
@jstancek Thanks for the detailed note with a possible patch. I had a quick look at the code but it seems like the patch cannot be applied cleanly on the current CentOS kernel's source. On the other hand BRIDGE_IGMP_SNOOPING can be disabled in the centosplus kernel. What are the consequences of disabling this option (other than working around the reported issue)? |
|
| Disabling BRIDGE_IGMP_SNOOPING should make the bridge "dumber" and make multicast traffic go to each port regardless. | |
| OK, with no adverse effect, I will go ahead and disable BRIDGE_IGMP_SNOOPING in the next plus kernel update. | |
| To get the patch into the distro kernel, the bug must be fixed in the upstream (RHEL) kernel. Could someone file a bug report at http://bugzilla.redhat.com ? | |
| RH BZ 1167003 - ipv6 connectivity breaks down after upgrading to kernel 2.6.32-504.1.3 | |
| Thanks. Like all other kernel-related BZs, 1167003 is private. Please update the status here with any progress. | |
| The centosplus kernel 2.6.32-504.3.3.el6 has been released. BRIDGE_IGMP_SNOOPING is now disabled. | |
| I run the 2.6.32-504.3.3 kernel now, but unfortunatly BRIDGE_IGMP_SNOOPING is still enabled. This follows from the /boot/config file and the fact that ipv6 is broken. However I can do 'echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping' and then the problems are over. I have put it in /etc/rc.local so far. | |
| OK I see you mention centosplus kernel, the one I am not running, so never mind | |
| Right, the distro kernel cannot be modified as it's supposed to be a bug-for-bug rebuild of the upstream (RH) kernel. We have to wait for their response in the RH BZ filed by jstancek (1167003). | |
|
This should be fixed in kernel-2.6.32-525.el6 by backport of: ef5e0d8237287db3a12d84f08fb2483d7a30a943 fe0d692bbc645786bce1a98439e548ae619269f5 cc0fdd802859eaeb00e1c87dbb655594bed2844c 6565b9eeef194afbb3beec80d6dd2447f4091f8c 9ed973cc40c588abeaa58aea0683ea665132d11d 20a599bec95a52fa72432b2376a2ce47c5bb68fb |
|
|
@jstancek Thanks for the updated information. Sounds as if the fix will be in the EL 6.7 kernel (?). |
|
|
Correct, RHEL6.6 already GA-ed with kernel 2.6.32-504: https://access.redhat.com/articles/3078 |
|
|
I suspect that disabling BRIDGE_IGMP_SNOOPING has an undesirable effect on the brctl function ( see http://bugs.centos.org/view.php?id=8862 ). I may need to re-enable this option. Those who are affected by bug 7796 can use the other workaround, that is, echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping, as suggested in comment 21814. |
|
|
*** Notice *** A new kernel update has just been released upstream. I am going to revert the config change and re-enable BRIDGE_IGMP_SNOOPING in this update. [EDIT] there was a typo which has been corrected. s/disable/re-enable/ |
|
| Closing. Kernel is at 2.6.32-642.6.2.el6. Please open a new ticket for any issue seen with this kernel. | |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2014-10-29 00:30 | bwelmers | New Issue | |
| 2014-10-29 12:29 | toracat | Note Added: 0021449 | |
| 2014-10-29 18:55 | DangerDevil | Note Added: 0021460 | |
| 2014-10-29 22:19 | bwelmers | Note Added: 0021461 | |
| 2014-11-25 09:20 | jstancek | Note Added: 0021798 | |
| 2014-11-26 18:02 | DangerDevil | Note Added: 0021813 | |
| 2014-11-26 20:48 | jstancek | Note Added: 0021814 | |
| 2014-11-26 21:45 | toracat | Note Added: 0021819 | |
| 2014-11-26 22:24 | jstancek | Note Added: 0021821 | |
| 2014-11-26 22:49 | toracat | Note Added: 0021822 | |
| 2014-11-26 22:59 | toracat | Status | new => assigned |
| 2014-11-26 23:02 | toracat | Note Added: 0021824 | |
| 2014-11-26 23:13 | jstancek | Note Added: 0021825 | |
| 2014-11-26 23:28 | toracat | Note Added: 0021826 | |
| 2014-12-17 22:06 | toracat | Note Added: 0021989 | |
| 2015-01-03 17:19 | bwelmers | Note Added: 0022069 | |
| 2015-01-03 17:22 | bwelmers | Note Added: 0022070 | |
| 2015-01-03 18:28 | toracat | Note Added: 0022071 | |
| 2015-01-21 07:40 | jstancek | Note Added: 0022197 | |
| 2015-01-25 18:48 | toracat | Note Added: 0022214 | |
| 2015-01-25 19:26 | jstancek | Note Added: 0022215 | |
| 2015-06-09 03:27 | toracat | Relationship added | related to 0008862 |
| 2015-06-09 03:35 | toracat | Note Added: 0023342 | |
| 2015-06-09 15:30 | toracat | Note Added: 0023351 | |
| 2015-06-15 16:19 | toracat | Note Edited: 0023351 | View Revisions |
| 2016-11-21 17:58 | toracat | Status | assigned => closed |
| 2016-11-21 17:58 | toracat | Resolution | open => no change required |
| 2016-11-21 17:58 | toracat | Note Added: 0027953 |