View Issue Details

IDProjectCategoryView StatusLast Update
0015250CentOS-7[All Projects] generalpublic2018-09-07 20:36
Reportererickmiller 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Platformx86OSOS Version
Product Version7.5.1804 
Target VersionFixed in Version 
Summary0015250: teamd sub-interface becomes UP at boot when ONBOOT=no
DescriptionOnly as of the last yum update run tonight, with the last yum update performed a couple months back, our team interfaces did not change to UP due to the sub-interfaces having already been turned up. We have ONBOOT=no in the sub-interface ifcfg files, since the teamd driver turns up the sub-interfaces itself.

If the sub-interfaces are turned up prior to the team interface, teamd complains that the sub-interfaces must be down prior to turning up the team interface.

Adding HOTPLUG=no to the sub-interfaces' ifcfg files made no difference.

So far, we have seen this problem across NIC vendors, so we believe this is a CentOS issue, not a NIC driver issue.
Steps To ReproduceUpgrade to the latest CentOS 7.5 version (I did this on September 7, 2018 around 1:00am Central).
Tagsifcfg onboot
abrt_hash
URL

Activities

erickmiller

erickmiller

2018-09-07 07:58

reporter   ~0032669

In case this helps - the ifcfg files are below.

ifcfg-enp1s0f0

NAME=enp1s0f0
DEVICE=enp1s0f0
BOOTPROTO=none
TEAM_MASTER=team0
DEVICETYPE=TeamPort
TEAM_PORT_CONFIG='{"prio":99}'
MTU=9200
NM_CONTROLLED=no
ONBOOT=no


ifcfg-enp1s0f1

DEVICE=enp1s0f1
NAME=enp1s0f1
BOOTPROTO=none
TEAM_MASTER=team0
DEVICETYPE=TeamPort
TEAM_PORT_CONFIG='{"prio":100}'
MTU=9200
NM_CONTROLLED=no
ONBOOT=no


ifcfg-team0

DEVICE=team0
NAME=team0
BOOTPROTO=none
DEVICETYPE=Team
TEAM_CONFIG='{"runner": {"name": "lacp", "active": true, "fast_rate": true, "tx_hash": ["l3", "l4"]}, "link_watch": {"name": "ethtool"}, "ports": {"enp1s0f0": {}, "enp1s0f1": {}}}'
MTU=9200
NM_CONTROLLED=no
ONBOOT=yes
TrevorH

TrevorH

2018-09-07 08:02

manager   ~0032670

Does it still do it if you downgrade the teamd package to the 7.4 version? e.g yum --enablerepo=C7.4\* downgrade teamd
erickmiller

erickmiller

2018-09-07 08:08

reporter   ~0032671

I don't believe this is a teamd issue, since it happens on non-team'd interfaces, but I am trying your suggestion on one machine. I got the following error, but am ignoring it and rebooting to see if the problem remains.

--> Finished Dependency Resolution
Error: Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:

         1. You have an upgrade for libteam which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of libteam of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude libteam.otherarch ... this should give you an error
            message showing the root cause of the problem.

         2. You have multiple architectures of libteam installed, but
            yum can only see an upgrade for one of those architectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.

         3. You have duplicate versions of libteam installed already.
            You can use "yum check" to get yum show these errors.

       ...you can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).

       Protected multilib versions: libteam-1.25-6.el7_4.3.i686 != libteam-1.27-4.el7.x86_64
TrevorH

TrevorH

2018-09-07 08:10

manager   ~0032672

yum downgrade is not clever and you have to specify all the packages on the same command that you want to downgrade at the same time. Unlike update, it doesn't expand the package selection.

Also, you sort of neglected to say that it affected other things than team interfaces...
erickmiller

erickmiller

2018-09-07 08:14

reporter   ~0032673

Sorry about that - I accidentally missed mentioning this (I instead mentioned it in a different ticket to our NIC vendor).

The first reboot shows no difference. I removed the HOTPLUG=no and a LINKDELAY test I was doing, and am rebooting again to see if there is any change.

It just finished rebooting - same issue - both sub-interfaces are up.

I was about to re-image these machines with the CentOS-1805 ISO installer (we're using the SuperMicro SSM to install CentOS from the Minimal ISO) to be sure we can "go back" and see that the problem is no longer. Let me know if you have any other suggestions before I perform the re-image.
erickmiller

erickmiller

2018-09-07 08:17

reporter   ~0032674

This shows the issue (partial output of "ip a") - where enp1s0f0 and enp1s0f1 are not teamed and even have NO-CARRIER, but are UP. Both have ONBOOT=no in their respective ifcfg files.

ens2 and ens2d1 are the team's sub-interfaces.

4: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:d2:f8:14 brd ff:ff:ff:ff:ff:ff
5: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:d2:f8:15 brd ff:ff:ff:ff:ff:ff
6: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9200 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::268a:7ff:fe72:b710/64 scope link
       valid_lft forever preferred_lft forever
7: ens2d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9200 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:11 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::268a:7ff:fe72:b711/64 scope link
       valid_lft forever preferred_lft forever
8: team0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 76:4b:00:4d:0d:a4 brd ff:ff:ff:ff:ff:ff
erickmiller

erickmiller

2018-09-07 09:18

reporter   ~0032675

So I have re-imaged some of the machines with a fresh copy of CentOS 7.5 1805 Minimal and am simply testing down'ing an interface using ifdown.

I'm a bit baffled since I don't see any change after running "ifdown ens2".

BEFORE:
6: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:30 brd ff:ff:ff:ff:ff:ff
7: ens2d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:31 brd ff:ff:ff:ff:ff:ff

AFTER:
6: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:30 brd ff:ff:ff:ff:ff:ff
7: ens2d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:31 brd ff:ff:ff:ff:ff:ff

Both appear to be "UP", both physically and administratively, even with ONBOOT=no and attempting to down the interface administratively with ifdown.

I can run "ip link set down ens2" and this physically disables the interface properly:

6: ens2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 24:8a:07:72:b7:30 brd ff:ff:ff:ff:ff:ff
7: ens2d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:72:b7:31 brd ff:ff:ff:ff:ff:ff

Any ideas why ifdown and ONBOOT=no don't work?

Thanks!

Eric
erickmiller

erickmiller

2018-09-07 09:49

reporter   ~0032676

I believe there is a combination of problems here. First, I had forgotten to disable NetworkManager. Second, I think Mellanox is bringing up their interface regardless of the ONBOOT=no setting.

One I have a better handle on the problem, I will report back with my results.

Eric
erickmiller

erickmiller

2018-09-07 20:36

reporter   ~0032682

Our original problem where the team interface could not be brought up due to sub-interfaces being brought up first was solved by fixing a problem we caused by not adjusting our /sbin/ifup-local script, which delays bringing up VLAN interfaces on the team interface. This doesn't answer why ONBOOT=no was no effective on a bare CentOS configuration, but our original issue is resolved.

The latest CentOS release still works after we fixed our issue with ifup-local.

Issue History

Date Modified Username Field Change
2018-09-07 07:46 erickmiller New Issue
2018-09-07 07:46 erickmiller Tag Attached: teamd onboot
2018-09-07 07:58 erickmiller Note Added: 0032669
2018-09-07 08:02 TrevorH Note Added: 0032670
2018-09-07 08:08 erickmiller Note Added: 0032671
2018-09-07 08:10 TrevorH Note Added: 0032672
2018-09-07 08:14 erickmiller Note Added: 0032673
2018-09-07 08:17 erickmiller Note Added: 0032674
2018-09-07 08:36 erickmiller Tag Renamed teamd onboot => ifcfg onboot
2018-09-07 09:18 erickmiller Note Added: 0032675
2018-09-07 09:49 erickmiller Note Added: 0032676
2018-09-07 20:36 erickmiller Note Added: 0032682