View Issue Details

IDProjectCategoryView StatusLast Update
0007796CentOS-6kernelpublic2016-11-21 17:58
Reporterbwelmers 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Product Version6.6 
Target VersionFixed in Version 
Summary0007796: ipv6 connectivity breaks down after upgrading to kernel 2.6.32-504.el6
DescriptionAfter I ran upgrade to CentOS 6.6, the ipv6 connectivity breaks down after a period of about 5 minutes. When I initate ping to another host in the same subnet, communication to and from that host is possible again for a few minutes and then breaks down again.

Configuration:
# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.6c626dcf364b no eth0

# ifconfig br0
br0 Link encap:Ethernet HWaddr 6C:62:6D:CF:36:4B
          inet addr:192.168.0.66 Bcast:192.168.0.255 Mask:255.255.255.0
          inet6 addr: 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b/64 Scope:Global
          inet6 addr: fe80::6e62:6dff:fecf:364b/64 Scope:Link

When I boot the machine, it obtains an ipv6 prefix from the network routing advertisor and sets up an IP address. ipv6 connectivity on this address works for several minutes, but then suddenly stops working. The ipv6 address is no longer pingable from another machine within the same network:

veriton$ ping6 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b
PING 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b(2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b) 56 data bytes
^C
--- 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2016ms

Same when I try to ping the link-local address with fe80::6e62:6dff:fecf:364b%eth0

However, when I initiate a ping6 from my CentOS machine to the remote machine within the same network:

# ping6 veriton
PING veriton(2001:XXXX:XXXX:XXXX::2) 56 data bytes
64 bytes from 2001:XXXX:XXXX:XXXX::2: icmp_seq=1 ttl=64 time=1.22 m

it works. and by then, also reverse-ping works again (of course), but only for several minutes:

veriton$ ping6 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b
PING 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b(2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b) 56 data bytes
64 bytes from 2001:XXXX:XXXX:XXXX:6e62:6dff:fecf:364b: icmp_seq=1 ttl=255 time=1.25 ms

Because the default route is advertised as a link-local address and return path will be the global ipv6 address, so also all outgoing connections will stuck within a few minutes.

I have connected devices with several different physical paths and switches, so problems in network hardware can be eliminated.

When I reboot back to kernel 2.6.32-431.29.2.el6 the problems are gone.
Additional InformationExtra check: ip6tables is empty:
# ip6tables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Full kernel version string: 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

network adaptor:
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)
TagsNo tags attached.

Relationships

related to 0008862 resolvedtoracat brctl with Centos Plus kernel: /sys/class/net/<bridge_name>/bridge: No such file or directory 

Activities

toracat

toracat

2014-10-29 12:29

manager   ~0021449

Can you show us the device ID pairing of your NIC?

lspci -nn | grep -i eth
DangerDevil

DangerDevil

2014-10-29 18:55

reporter   ~0021460

Hello,
same problem with two hosts, others running fine.
All hosts have the same virtual-NIC.
IPv6 stops working after some time, sometimes 2 hours.

00:03.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
bwelmers

bwelmers

2014-10-29 22:19

reporter   ~0021461

05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03)
jstancek

jstancek

2014-11-25 09:20

reporter   ~0021798

Does it help if you turn off multicast snooping?
  echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping
DangerDevil

DangerDevil

2014-11-26 18:02

reporter   ~0021813

yes, disable multicast_snooping on all bridges on the host help solve the problem
jstancek

jstancek

2014-11-26 20:48

reporter   ~0021814

Then this is likely missing backport of this commit:
  commit cc0fdd802859eaeb00e1c87dbb655594bed2844c
  Author: Linus Lüssing <linus.luessing@web.de>
  Date: Fri Aug 30 17:28:17 2013 +0200
    bridge: separate querier and query timer into IGMP/IPv4 and MLD/IPv6 ones
    
    Currently we would still potentially suffer multicast packet loss if there
    is just either an IGMP or an MLD querier: For the former case, we would
    possibly drop IPv6 multicast packets, for the latter IPv4 ones. This is
    because we are currently assuming that if either an IGMP or MLD querier
    is present that the other one is present, too.

You could confirm that by running tcpdump to check, that you are seeing IGMP queries on your network and at the same time there are no MLD ones.

As a workaround you can:
- disable CONFIG_BRIDGE_IGMP_SNOOPING=y and rebuild kernel or
- echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping
toracat

toracat

2014-11-26 21:45

manager   ~0021819

@jstancek

Thanks for the detailed note with a possible patch. I had a quick look at the code but it seems like the patch cannot be applied cleanly on the current CentOS kernel's source.

On the other hand BRIDGE_IGMP_SNOOPING can be disabled in the centosplus kernel. What are the consequences of disabling this option (other than working around the reported issue)?
jstancek

jstancek

2014-11-26 22:24

reporter   ~0021821

Disabling BRIDGE_IGMP_SNOOPING should make the bridge "dumber" and make multicast traffic go to each port regardless.
toracat

toracat

2014-11-26 22:49

manager   ~0021822

OK, with no adverse effect, I will go ahead and disable BRIDGE_IGMP_SNOOPING in the next plus kernel update.
toracat

toracat

2014-11-26 23:02

manager   ~0021824

To get the patch into the distro kernel, the bug must be fixed in the upstream (RHEL) kernel. Could someone file a bug report at http://bugzilla.redhat.com ?
jstancek

jstancek

2014-11-26 23:13

reporter   ~0021825

RH BZ 1167003 - ipv6 connectivity breaks down after upgrading to kernel 2.6.32-504.1.3
toracat

toracat

2014-11-26 23:28

manager   ~0021826

Thanks. Like all other kernel-related BZs, 1167003 is private. Please update the status here with any progress.
toracat

toracat

2014-12-17 22:06

manager   ~0021989

The centosplus kernel 2.6.32-504.3.3.el6 has been released. BRIDGE_IGMP_SNOOPING is now disabled.
bwelmers

bwelmers

2015-01-03 17:19

reporter   ~0022069

I run the 2.6.32-504.3.3 kernel now, but unfortunatly BRIDGE_IGMP_SNOOPING is still enabled. This follows from the /boot/config file and the fact that ipv6 is broken. However I can do 'echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping' and then the problems are over. I have put it in /etc/rc.local so far.
bwelmers

bwelmers

2015-01-03 17:22

reporter   ~0022070

OK I see you mention centosplus kernel, the one I am not running, so never mind
toracat

toracat

2015-01-03 18:28

manager   ~0022071

Right, the distro kernel cannot be modified as it's supposed to be a bug-for-bug rebuild of the upstream (RH) kernel. We have to wait for their response in the RH BZ filed by jstancek (1167003).
jstancek

jstancek

2015-01-21 07:40

reporter   ~0022197

This should be fixed in kernel-2.6.32-525.el6 by backport of:
ef5e0d8237287db3a12d84f08fb2483d7a30a943
fe0d692bbc645786bce1a98439e548ae619269f5
cc0fdd802859eaeb00e1c87dbb655594bed2844c
6565b9eeef194afbb3beec80d6dd2447f4091f8c
9ed973cc40c588abeaa58aea0683ea665132d11d
20a599bec95a52fa72432b2376a2ce47c5bb68fb
toracat

toracat

2015-01-25 18:48

manager   ~0022214

@jstancek

Thanks for the updated information. Sounds as if the fix will be in the EL 6.7 kernel (?).
jstancek

jstancek

2015-01-25 19:26

reporter   ~0022215

Correct, RHEL6.6 already GA-ed with kernel 2.6.32-504:
  https://access.redhat.com/articles/3078
toracat

toracat

2015-06-09 03:35

manager   ~0023342

I suspect that disabling BRIDGE_IGMP_SNOOPING has an undesirable effect on the brctl function ( see http://bugs.centos.org/view.php?id=8862 ).

I may need to re-enable this option. Those who are affected by bug 7796 can use the other workaround, that is, echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping, as suggested in comment 21814.
toracat

toracat

2015-06-09 15:30

manager   ~0023351

Last edited: 2015-06-15 16:19

View 2 revisions

*** Notice ***

A new kernel update has just been released upstream. I am going to revert the config change and re-enable BRIDGE_IGMP_SNOOPING in this update.

[EDIT] there was a typo which has been corrected. s/disable/re-enable/

toracat

toracat

2016-11-21 17:58

manager   ~0027953

Closing. Kernel is at 2.6.32-642.6.2.el6. Please open a new ticket for any issue seen with this kernel.

Issue History

Date Modified Username Field Change
2014-10-29 00:30 bwelmers New Issue
2014-10-29 12:29 toracat Note Added: 0021449
2014-10-29 18:55 DangerDevil Note Added: 0021460
2014-10-29 22:19 bwelmers Note Added: 0021461
2014-11-25 09:20 jstancek Note Added: 0021798
2014-11-26 18:02 DangerDevil Note Added: 0021813
2014-11-26 20:48 jstancek Note Added: 0021814
2014-11-26 21:45 toracat Note Added: 0021819
2014-11-26 22:24 jstancek Note Added: 0021821
2014-11-26 22:49 toracat Note Added: 0021822
2014-11-26 22:59 toracat Status new => assigned
2014-11-26 23:02 toracat Note Added: 0021824
2014-11-26 23:13 jstancek Note Added: 0021825
2014-11-26 23:28 toracat Note Added: 0021826
2014-12-17 22:06 toracat Note Added: 0021989
2015-01-03 17:19 bwelmers Note Added: 0022069
2015-01-03 17:22 bwelmers Note Added: 0022070
2015-01-03 18:28 toracat Note Added: 0022071
2015-01-21 07:40 jstancek Note Added: 0022197
2015-01-25 18:48 toracat Note Added: 0022214
2015-01-25 19:26 jstancek Note Added: 0022215
2015-06-09 03:27 toracat Relationship added related to 0008862
2015-06-09 03:35 toracat Note Added: 0023342
2015-06-09 15:30 toracat Note Added: 0023351
2015-06-15 16:19 toracat Note Edited: 0023351 View Revisions
2016-11-21 17:58 toracat Status assigned => closed
2016-11-21 17:58 toracat Resolution open => no change required
2016-11-21 17:58 toracat Note Added: 0027953