View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0016815CentOS-7kernelpublic2019-12-10 18:472021-04-17 23:59
Reporterossgeek Assigned To 
PrioritynormalSeverityminorReproducibilityrandom
Status resolvedResolutionfixed 
Product Version7.7-1908 
Summary0016815: r8169 stops working after receiving `NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out`
DescriptionI have a Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller built into the motherboard of a GIGABYTE MZGLKAP-00/MZGLKAP-00, BIOS F1 12/21/2017. The network interface works as expected without issue but after random intervals (sometimes days... sometimes only a few hours... but possibly after going `idle`) the kernel will spit out a debug message and then the network interface will become stop transmitting data. It requires a reboot to restore network operations.

I'm running the stock kernel with no external modules.
Steps To ReproduceHave a Realtek network card using the r8169 module
Let the system run 24/7 until the kernel gives a debug message
Additional InformationI found this bug and kernel patch (which the 3.10.0 does not have) that addresses this exact issue. Can we apply this patch?

https://bugzilla.kernel.org/show_bug.cgi?id=199549

This is my kernel output:

Dec 10 11:03:57 mogweb kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
Dec 10 11:03:57 mogweb kernel: NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out
Dec 10 11:03:57 mogweb kernel: Modules linked in: xt_set xt_multiport ip_set_hash_ip xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ext4 vfat fat mbcache jbd2 intel_powerclamp coretemp intel_rapl kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel snd_soc_rt298 snd_soc_rt286 snd_soc_rl6347a snd_soc_core
Dec 10 11:03:57 mogweb kernel: snd_compress snd_seq snd_seq_device aesni_intel sg lrw gf128mul snd_pcm glue_helper ablk_helper cryptd pcspkr wdat_wdt i2c_i801 snd_timer snd soundcore pcc_cpufreq tpm_crb ip_tables xfs libcrc32c i915 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit iosf_mbi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci drm libahci sdhci_pci nvme cqhci sdhci libata mmc_core crct10dif_pclmul crct10dif_common crc32c_intel nvme_core r8169 drm_panel_orientation_quirks uas i2c_hid video usb_storage dm_mirror dm_region_hash dm_log dm_mod
Dec 10 11:03:57 mogweb kernel: CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Not tainted 3.10.0-1062.4.3.el7.x86_64 #1
Dec 10 11:03:57 mogweb kernel: Hardware name: GIGABYTE MZGLKAP-00/MZGLKAP-00, BIOS F1 12/21/2017
Dec 10 11:03:57 mogweb kernel: Call Trace:
Dec 10 11:03:57 mogweb kernel: <IRQ> [<ffffffffa0779ba4>] dump_stack+0x19/0x1b
Dec 10 11:03:57 mogweb kernel: [<ffffffffa009b958>] __warn+0xd8/0x100
Dec 10 11:03:57 mogweb kernel: [<ffffffffa009b9df>] warn_slowpath_fmt+0x5f/0x80
Dec 10 11:03:57 mogweb kernel: [<ffffffffa067bf88>] dev_watchdog+0x248/0x260
Dec 10 11:03:57 mogweb kernel: [<ffffffffa067bd40>] ? dev_deactivate_queue.constprop.27+0x60/0x60
Dec 10 11:03:57 mogweb kernel: [<ffffffffa00ac358>] call_timer_fn+0x38/0x110
Dec 10 11:03:57 mogweb kernel: [<ffffffffa067bd40>] ? dev_deactivate_queue.constprop.27+0x60/0x60
Dec 10 11:03:57 mogweb kernel: [<ffffffffa00ae7bd>] run_timer_softirq+0x24d/0x300
Dec 10 11:03:57 mogweb kernel: [<ffffffffa00a5305>] __do_softirq+0xf5/0x280
Dec 10 11:03:57 mogweb kernel: [<ffffffffa079042c>] call_softirq+0x1c/0x30
Dec 10 11:03:57 mogweb kernel: [<ffffffffa002f715>] do_softirq+0x65/0xa0
Dec 10 11:03:57 mogweb kernel: [<ffffffffa00a5685>] irq_exit+0x105/0x110
Dec 10 11:03:57 mogweb kernel: [<ffffffffa07919d8>] smp_apic_timer_interrupt+0x48/0x60
Dec 10 11:03:57 mogweb kernel: [<ffffffffa078defa>] apic_timer_interrupt+0x16a/0x170
Dec 10 11:03:57 mogweb kernel: <EOI> [<ffffffffa05c10f7>] ? cpuidle_enter_state+0x57/0xd0
Dec 10 11:03:57 mogweb kernel: [<ffffffffa05c10ed>] ? cpuidle_enter_state+0x4d/0xd0
Dec 10 11:03:57 mogweb kernel: [<ffffffffa05c124e>] cpuidle_idle_call+0xde/0x230
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0037c6e>] arch_cpu_idle+0xe/0xc0
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0100d3a>] cpu_startup_entry+0x14a/0x1e0
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0768b57>] rest_init+0x77/0x80
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0d881cb>] start_kernel+0x450/0x471
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0d87b7b>] ? repair_env_string+0x5c/0x5c
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0d87120>] ? early_idt_handler_array+0x120/0x120
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0d8772f>] x86_64_start_reservations+0x24/0x26
Dec 10 11:03:57 mogweb kernel: [<ffffffffa0d87885>] x86_64_start_kernel+0x154/0x177
Dec 10 11:03:57 mogweb kernel: [<ffffffffa00000d5>] start_cpu+0x5/0x14
Dec 10 11:03:57 mogweb kernel: ---[ end trace 48f07bcd9213d5ea ]---
Tags"3.10.0-1062.9.1.el7.x85_64", "Network", realtek
abrt_hash
URL

Activities

NeK

NeK

2019-12-25 03:14

reporter   ~0035886

The afforementioned kernel issue does not seem to be the same issue. It may appear relevant but I have verified that it is not the same.

I have the same issue with the same network chip, after a while running (hours to days) the 'NETDEV WATCHDOG: enp2s0 (r8169): transmit queue 0 timed out' message is logged and after some time* the network card just stops working.

with kernel 3.10.0-957.27.2 this issue is not reproducible, but with 3.10.0-1062 up to the current 3.10.0-1062.9.1 it is. Therefore some patch is responsible for this issue that was committed in between 3.10.0-957.27.2 and 3.10.0-1062. I checked the source code and the diffs and tried to find if the kernel issue patch had been applied and it turns out that that specific patch has already been committed to 3.10.0-957.27.2, so this cannot be a fix. Also, in the kernel issue discussion, it is reported that the Runtime Power Management is responsible and that the issue is caused when the network card is set to 'auto' instead of 'on', but in my system the issue is reproducible and the power/control is set to 'on'. So the issue in my system is not related to that sysfs setting.

Furthermore the specific kernel issue description is the exact opposite of the current centos issue, it first the system has no network activity *until* this kernel warning message appears where the network card works fine afterwards. This issue is about the other way around: network card works fine until this kernel message gets logged and then after some time the card completely stops sending or receiving any packets.

So, I don't have a solution, but at least I excluded a wrong direction that could waste time and effort to continue to pursue.

* minutes to hours. I don't know why, can't figure this out. But the issue always happens *after* this log message.
toracat

toracat

2019-12-25 07:15

manager   ~0035887

Can you test-install kernel-ml from ELRepo? As of today, the current kernel version is 5.4.6.el7. This test will show if the issue you are seeing has been fixed in the latest upstream (kernel.org) kernel.
NeK

NeK

2019-12-25 22:45

reporter   ~0035893

I just found a way to reproduce the issue at will:

setup another system (HOST2) and install iperf3 in both systems and execute the following in HOST1:

iperf3 -s

and then execute in HOST2:

iperf3 -b0 -c HOST1 -u

After 10-15secs the HOST1 will immediately exhibit the issue and all its network traffic will just stop completely. So now I can do quick tests and eventually find what causes this issue. I will also test with the new 5.4.6 kernel too. I'll write up my finding ASAP.
ossgeek

ossgeek

2019-12-25 23:01

reporter   ~0035894

Haven't be able to try the v5.4.6 kernel yet but ...

I can verify that the iperf3 setup does cause the issue as described in the initial report.
NeK

NeK

2019-12-25 23:06

reporter   ~0035895

And here they are:

- 1062.9.1.el7: ISSUE EXISTS
- 957-27-2.el7: NO ISSUE
- 5.4.6-1.el7: NO ISSUE

So in 5.4.6 kernel, the issue has been fixed. That's good! Now how do we find out what patch fixed it in order to patch 1062.9.1.el7 and backport the fix? I don't want to run the latest kernel on centos 7 for various reasons (I don't trust running latest kernel on old OS env. and I also use modules like ZFS that are compiled against 1062.9.1.el7 etc.)
jprediger

jprediger

2020-02-26 20:17

reporter   ~0036391

I comment to inform that I have the same issue. The network interface stops working erratically.
I enclose statistics of our servers that are running different versions of the kernel to confirm the comment of @Nek


** server001 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B250M-DS3H-CF
Ethernet:
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 23 23:33:33 server001 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 25 15:00:15 server001 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 11 01:14:54 server001 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server002 **
Kernel: 3.10.0-957.5.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B250M-DS3H-CF
Ethernet:
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server003 **
Kernel: 3.10.0-957.5.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B250M-DS3H-CF
Ethernet:
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server004 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B250M-DS3H-CF
Ethernet:
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server005 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: ASUSTeK COMPUTER INC.
Motherboard Model: PRIME B250M-A
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15))
Kernel Events:

** server006 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: ASUSTeK COMPUTER INC.
Motherboard Model: PRIME B250M-A
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15))
Kernel Events:

** server007 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: ASUSTeK COMPUTER INC.
Motherboard Model: PRIME B250M-A
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15))
Kernel Events:

** server008 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: ASUSTeK COMPUTER INC.
Motherboard Model: PRIME B250M-A
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15))
Kernel Events:

** server009 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: ASUSTeK COMPUTER INC.
Motherboard Model: PRIME B250M-A
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15))
Kernel Events:

** server010 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server011 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server012 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server013 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server014 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server015 **
Kernel: 3.10.0-862.14.4.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server016 **
Kernel: 3.10.0-1062.4.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server017 **
Kernel: 3.10.0-1062.4.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M D3H-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server018 **
Kernel: 3.10.0-1062.4.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M AORUS Gaming 3-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server019 **
Kernel: 3.10.0-957.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M D3H-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server020 **
Kernel: 3.10.0-957.1.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M D3H-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server021 **
Kernel: 3.10.0-957.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server022 **
Kernel: 3.10.0-957.5.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M D3H-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server023 **
Kernel: 3.10.0-957.21.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server024 **
Kernel: 3.10.0-957.10.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server025 **
Kernel: 3.10.0-957.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server026 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: H370M DS3H-CF
Ethernet:
 - nic0: (Intel Corporation Ethernet Connection (7) I219-V (rev 10))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server027 **
Kernel: 3.10.0-957.21.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server028 **
Kernel: 3.10.0-957.21.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server029 **
Kernel: 3.10.0-957.21.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server030 **
Kernel: 3.10.0-1062.4.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 5 17:59:36 server030 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server031 **
Kernel: 3.10.0-957.21.3.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server032 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server033 **
Kernel: 3.10.0-1062.1.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server034 **
Kernel: 3.10.0-957.27.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server035 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 23 11:28:32 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 25 07:21:01 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 26 10:23:29 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 21 22:46:11 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 22 02:38:38 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 22 05:52:05 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 22 17:34:43 server035 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server036 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server037 **
Kernel: 3.10.0-1062.1.2.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 5 17:56:43 server037 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server038 **
Kernel: 3.10.0-1062.4.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server039 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 5 17:44:16 server039 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server040 **
Kernel: 3.10.0-1062.7.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server041 **
Kernel: 3.10.0-1062.9.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 5 17:45:57 server041 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server042 **
Kernel: 3.10.0-1062.9.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 5 19:32:02 server042 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server043 **
Kernel: 3.10.0-1062.9.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B360M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:

** server044 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B365M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 25 17:25:29 server044 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 20 18:27:11 server044 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 11 13:20:35 server044 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out

** server045 **
Kernel: 3.10.0-1062.12.1.el7.x86_64
Motherboard Manufacturer: Gigabyte Technology Co., Ltd.
Motherboard Model: B365M DS3H
Ethernet:
 - nic1: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
 - nic0: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16))
 - nic2: (Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06))
Kernel Events:
Feb 18 10:10:10 server045 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 20 20:19:38 server045 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Feb 12 17:04:23 server045 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
jprediger

jprediger

2020-03-02 14:06

reporter   ~0036415

I can see that the problem arises in all cases in the onboard network interface.

I also believe that in all cases the device is detected as "Generic Realtek PHY"


Mar 2 10:28:34 srvbkp02 kernel: ------------[ cut here ]------------
Mar 2 10:28:34 srvbkp02 kernel: WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
Mar 2 10:28:34 srvbkp02 kernel: NETDEV WATCHDOG: nic0 (r8169): transmit queue 0 timed out
Mar 2 10:28:34 srvbkp02 kernel: Modules linked in: ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter dm_mirror dm_region_hash dm_log dm_mod xfs libcrc32c eeepc_wmi asus_wmi sparse_keymap rfkill ppdev edac_mce_amd kvm_amd kvm snd_hda_codec_hdmi irqbypass crc32_pclmul crc32c_intel snd_hda_intel ghash_clmulni_intel snd_hda_codec aesni_intel snd_hda_core snd_hwdep lrw gf128mul snd_seq glue_helper ablk_helper cryptd snd_seq_device snd_pcm pcspkr r8169 k10temp realtek snd_timer snd sg soundcore wmi parport_pc parport tpm_infineon video i2c_piix4 pcc_cpufreq acpi_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm libata crct10dif_pclmul crct10dif_common
Mar 2 10:28:34 srvbkp02 kernel: serio_raw drm_panel_orientation_quirks
Mar 2 10:28:34 srvbkp02 kernel: CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.0-1062.9.1.el7.x86_64 #1
Mar 2 10:28:34 srvbkp02 kernel: Hardware name: System manufacturer System Product Name/A88XM-PLUS, BIOS 2903 03/10/2016
Mar 2 10:28:34 srvbkp02 kernel: Call Trace:
Mar 2 10:28:34 srvbkp02 kernel: <IRQ> [<ffffffffb057ac23>] dump_stack+0x19/0x1b
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe9b958>] __warn+0xd8/0x100
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe9b9df>] warn_slowpath_fmt+0x5f/0x80
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb047cf88>] dev_watchdog+0x248/0x260
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb047cd40>] ? dev_deactivate_queue.constprop.27+0x60/0x60
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafeac358>] call_timer_fn+0x38/0x110
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb047cd40>] ? dev_deactivate_queue.constprop.27+0x60/0x60
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafeae7bd>] run_timer_softirq+0x24d/0x300
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafea5305>] __do_softirq+0xf5/0x280
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb059142c>] call_softirq+0x1c/0x30
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe2f715>] do_softirq+0x65/0xa0
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafea5685>] irq_exit+0x105/0x110
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb05929d8>] smp_apic_timer_interrupt+0x48/0x60
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb058eefa>] apic_timer_interrupt+0x16a/0x170
Mar 2 10:28:34 srvbkp02 kernel: <EOI> [<ffffffffb03c1a04>] ? cpuidle_enter_state+0x54/0xd0
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffb03c1b5e>] cpuidle_idle_call+0xde/0x230
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe37c6e>] arch_cpu_idle+0xe/0xc0
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffaff0158a>] cpu_startup_entry+0x14a/0x1e0
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe5a0b7>] start_secondary+0x1f7/0x270
Mar 2 10:28:34 srvbkp02 kernel: [<ffffffffafe000d5>] start_cpu+0x5/0x14
Mar 2 10:28:34 srvbkp02 kernel: ---[ end trace 016f0553218baf85 ]---
Mar 2 10:28:34 srvbkp02 kernel: r8169 0000:03:00.0 nic0: rtl_txcfg_empty_cond == 0 (loop: 666, delay: 100).



[srvbkp02 ~] # dmesg | grep 'attached PHY driver'
[ 130.400462] RTL8211DN Gigabit Ethernet r8169-200:00: attached PHY driver [RTL8211DN Gigabit Ethernet] (mii_bus:phy_addr=r8169-200:00, irq=-2)
[ 134.864954] Generic Realtek PHY r8169-300:00: attached PHY driver [Generic Realtek PHY] (mii_bus:phy_addr=r8169-300:00, irq=-2)
[srvbkp02 ~] #
jprediger

jprediger

2020-03-02 14:24

reporter   ~0036416

Please change the severity of this incident. Many critical systems depend on the proper functioning of the network. This is similar to a system crashing because it requires a reboot to restore normal operation.
TrevorH

TrevorH

2020-03-02 15:36

manager   ~0036418

CentOS is a rebuild of the sources used to create RHEL. We do not modify anything except to remove branding and logos. You will need to submit your request to Redhat via bugzilla.redhat.com and if/when RH accepts it and incorporates it into RHEL and releases a patched version, then CentOS will pick it up and rebuild it.
toracat

toracat

2020-03-02 18:42

manager   ~0036420

First off, the patch referenced in the original report ( https://bugzilla.kernel.org/show_bug.cgi?id=199549 ) is now in the CentOS 7.7 kernel (3.10.0-1062.x.el7).

Second, I'm curious to see which modules are loaded. What is the output from:

lsmod | grep -e ^realtek -e ^r8169 -e ^libphy
jprediger

jprediger

2020-03-02 19:47

reporter   ~0036423

With kernel 3.10.0-1062

[root@srvbkp02 ~] # lsmod | grep -e ^realtek -e ^r8169 -e ^libphy
r8169 84623 0
realtek 13330 0
[root@srvbkp02 ~] #

[root@server001 ~] # lsmod | grep -e ^realtek -e ^r8169 -e ^libphy
r8169 84623 0
realtek 13330 0
[root@server001 ~] #


With kernel 3.10.0-957

[root@server029 ~] # lsmod | grep -e ^realtek -e ^r8169 -e ^libphy
r8169 89607 0
[root@server029 ~] #
toracat

toracat

2020-03-02 23:07

manager   ~0036428

Looks like, in the -1062 kernel, realtek is loaded followed by r8169, which should be the correct order. And curiously, there is no realtek module loaded in the -957 kernel.

As TrevorH suggested, please file a report upstream at RH and see what they say.
toracat

toracat

2020-03-03 01:31

manager   ~0036435

OK, I built a plus kernel using a patch from:

https://lore.kernel.org/patchwork/patch/1000208/
"r8169: fix network stalls due to missing bit TXCFG_AUTO_FIFO"

Please test the patched plus kernel available here:

https://people.centos.org/toracat/kernel/7/plus/r8168fix/

If this fixes the issue, great. If not, the search will go on.
jprediger

jprediger

2020-03-03 17:05

reporter   ~0036441

I loaded the bug into RedHat and it was marked duplicate of "https://bugzilla.redhat.com/show_bug.cgi?id=1733837"

Thanks a lot.
AlexV

AlexV

2020-05-05 21:33

reporter   ~0036873

> Please test the patched plus kernel available here:

Already month on this kernel, don't see the bug anymore.
toracat

toracat

2020-06-15 01:05

manager   ~0037099

Forgot to mention that the referenced patch has been in the plus kernel after 3.10.0-1062.12.1.el7.centos.plus. The only way to get this patch into the distro kernel is by way of the upstream (RHEL) kernel.

Perhaps, @jprediger can reopen the RHBZ bug report filed for RHEL (which was closed as a dup of the corresponding Fedora bug) and ask the RHEL kernel get fixed as well.
toracat

toracat

2021-04-17 23:59

manager   ~0038393

Closing due to inactivity. Feel free to reopen if/when there is a status change upstream.

Issue History

Date Modified Username Field Change
2019-12-10 18:47 ossgeek New Issue
2019-12-10 18:47 ossgeek Tag Attached: "Network"
2019-12-10 18:47 ossgeek Tag Attached: "3.10.0-1062.9.1.el7.x85_64"
2019-12-10 18:47 ossgeek Tag Attached: realtek
2019-12-25 03:14 NeK Note Added: 0035886
2019-12-25 07:15 toracat Note Added: 0035887
2019-12-25 22:45 NeK Note Added: 0035893
2019-12-25 23:01 ossgeek Note Added: 0035894
2019-12-25 23:06 NeK Note Added: 0035895
2020-02-26 20:17 jprediger Note Added: 0036391
2020-03-02 14:06 jprediger Note Added: 0036415
2020-03-02 14:24 jprediger Note Added: 0036416
2020-03-02 15:36 TrevorH Note Added: 0036418
2020-03-02 18:42 toracat Note Added: 0036420
2020-03-02 19:47 jprediger Note Added: 0036423
2020-03-02 23:07 toracat Note Added: 0036428
2020-03-03 01:31 toracat Status new => assigned
2020-03-03 01:31 toracat Note Added: 0036435
2020-03-03 17:05 jprediger Note Added: 0036441
2020-05-05 21:33 AlexV Note Added: 0036873
2020-06-15 01:05 toracat Note Added: 0037099
2021-04-17 23:59 toracat Status assigned => resolved
2021-04-17 23:59 toracat Resolution open => fixed
2021-04-17 23:59 toracat Note Added: 0038393