View Issue Details

IDProjectCategoryView StatusLast Update
0018310CentOS-8kernelpublic2021-09-22 16:57
Reportermgsmith Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version8.4.2105 
Summary0018310: Intel X553 NICs stopped working on Atom C3xxx with upgrade to CentOS 8.4
DescriptionIntel X553 NICs worked with earlier versions of CentOS 8 up through 8.3. After upgrading to CentOS 8.4, only 1 of 4 Intel X553 ports is available to the system, the other 3 cannot be bound to the ixgbe driver. These are the onboard NICs on Intel Atom C3xxxx SoCs. No error is logged when loading the ixgbe kernel module, but after doing so, only one of 4 X553 ports shows up on the system as an available interface.

[root@PB1 ~]# fgrep "model name" /proc/cpuinfo | uniq
model name : Intel(R) Atom(TM) CPU C3558 @ 2.20GHz
[root@PB1 ~]# lspci -nn | grep X553
06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e4] (rev 11)
06:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e4] (rev 11)
08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e5] (rev 11)
08:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e5] (rev 11)
[root@PB1 ~]# ls -l /sys/bus/pci/devices/0000:0[68]:00.[01]/driver
lrwxrwxrwx 1 root root 0 Sep 22 09:00 /sys/bus/pci/devices/0000:06:00.0/driver -> ../../../../bus/pci/drivers/ixgbe
[root@PB1 ~]# dmesg | grep ixgbe
[ 5.364610] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 5.375097] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 5.742886] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5.885014] ixgbe 0000:06:00.0: MAC: 6, PHY: 27, PBA No: 000600-000
[ 5.891306] ixgbe 0000:06:00.0: 00:90:0b:7a:83:5f
[ 5.941007] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
[ 5.947765] libphy: ixgbe-mdio: probed
[ 6.308170] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 6.444008] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 000600-000
[ 6.450277] ixgbe 0000:06:00.1: 00:90:0b:7a:83:60
[ 6.502007] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection
[ 6.867115] ixgbe 0000:08:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 7.003007] ixgbe 0000:08:00.0: MAC: 6, PHY: 27, PBA No: 000600-000
[ 7.009288] ixgbe 0000:08:00.0: 00:90:0b:7a:83:61
[ 7.061007] ixgbe 0000:08:00.0: Intel(R) 10 Gigabit Network Connection
[ 7.433111] ixgbe 0000:08:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 7.569008] ixgbe 0000:08:00.1: MAC: 6, PHY: 27, PBA No: 000600-000
[ 7.575288] ixgbe 0000:08:00.1: 00:90:0b:7a:83:62
[ 7.627007] ixgbe 0000:08:00.1: Intel(R) 10 Gigabit Network Connection
[ 7.651172] ixgbe 0000:06:00.0 enp6s0f0: renamed from eth0
[ 207.877109] ixgbe 0000:06:00.0: complete
[root@PB1 ~]# rpm -q kernel-core
kernel-core-4.18.0-240.1.1.el8_3.x86_64
kernel-core-4.18.0-305.12.1.el8_4.x86_64
kernel-core-4.18.0-305.19.1.el8_4.x86_64
[root@PB1 ~]# uname -r
4.18.0-305.19.1.el8_4.x86_64

Only one interface ends up being available - enp6s0f0, which is associated with 0000:06:00.0. Before upgrading, the system also had interfaces enp6s0f1, enp8s0f0, and enp8s0f1 associated with 0000:06:00.1, 0000:08:00.0, and 0000:08.00.1, respectively.

If I remove the ixgbe module with 'modprobe -r ixgbe' and add it again with 'modprobe ixgbe', dmesg displays additional messages identical to the ones pasted above. If I try to manually bind one of the devices, an error is displayed:
[root@PB1 ~]# echo "0000:06:00.1" > /sys/bus/pci/drivers/ixgbe/bind
bash: echo: write error: No such device

After attempting manual binding, dmesg shows messages printed by ixgbe while trying to intialize the device:
[ 811.372716] ixgbe 0000:06:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 811.509160] ixgbe 0000:06:00.1: MAC: 6, PHY: 27, PBA No: 000600-000
[ 811.515470] ixgbe 0000:06:00.1: 00:90:0b:7a:83:60
[ 811.556941] ixgbe 0000:06:00.1 enp6s0f1: renamed from eth0
[ 811.563163] ixgbe 0000:06:00.1: Intel(R) 10 Gigabit Network Connection

Interface enp6s0f1 is mentioned in those messages, but that interface does not exist and /sys still does not show the device bound to any driver:
[root@PB1 ~]# ip link show dev enp6s0f1
Device "enp6s0f1" does not exist.
[root@PB1 ~]# ls -ld /sys/bus/pci/devices/0000:06:00.1
lrwxrwxrwx 1 root root 0 Sep 22 09:35 /sys/bus/pci/devices/0000:06:00.1 -> ../../../devices/pci0000:00/0000:00:16.0/0000:06:00.1
[root@PB1 ~]# ls -l /sys/bus/pci/devices/0000:06:00.1/driver
ls: cannot access '/sys/bus/pci/devices/0000:06:00.1/driver': No such file or directory

The PCI device IDs for the affected NICs are 15e4 and 15e5. The vendor ID is 8086 (Intel).
Steps To Reproduce1. On an Atom C3xxx system, install CentOS 8.4. Or upgrade from an older release of CentOS 8.
2. Reboot the system to the 8.4 kernel
Additional InformationBy "C3xxx" I mean systems in the C3000 series of Atom devices by Intel. This series of devices is also sometimes referred to as "Denverton".

The interfaces function correctly again after building an RPM containing the latest stable ixgbe driver downloaded from https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.12.5/ixgbe-5.12.5.tar.gz/download and installing/loading the ixgbe module. That version of the driver was published in July 2021, a month or two after CentOS 8.4. the README lists "Added support for RHEL 8.4" as one of the changes included in the release.

The problem was observed on two separate C3xxx SoCs from different vendors. The one that I pasted output from is a system manufactured by Lanner. The exact same issue manifested on a different box that was manufactured by Silicom.

This was also reported on a CentOS forum post by at least one other user - https://forums.centos.org/viewtopic.php?t=77693&p=326656. I don't know the person that posted that message. From the data they included, they are also having the problem on a C3xxxx system and the symptoms appear to be identical.
TagsNo tags attached.

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2021-09-22 15:58 mgsmith New Issue