CentOS Bug Tracker
CentOS Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006810CentOS-6kernelpublic2013-12-03 07:472014-08-28 08:50
ReporterFireMaster 
PriorityimmediateSeveritycrashReproducibilityrandom
StatusassignedResolutionopen 
PlatformCentOS 6.5 x86_64OSPower Edge 1950OS Version6.5
Product Version 
Target VersionFixed in Version 
Summary0006810: LAN Driver Crash after update and reboot
DescriptionDear All,
yesterday I was update my server from 6.4 to 6.5 and two hours letter I lost connection to the server. After reboot, the machine boot correctly.
Here is the lspci output:
_________________________________________________________
00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller Hub (rev 12)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev 12)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev 12)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev 12)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev 12)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev 12)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev 12)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09)
00:1d.0 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller 0000001 (rev 09)
00:1d.1 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller 0000002 (rev 09)
00:1d.2 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09)
00:1d.7 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller (rev 09)
01:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
04:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
04:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01)
05:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01)
05:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01)
06:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
0e:0d.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
_________________________________________________________
Additional InformationDec 2 21:58:18 server kernel: ------------[ cut here ]------------
Dec 2 21:58:18 server kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted)
Dec 2 21:58:18 server kernel: Hardware name: PowerEdge 1950
Dec 2 21:58:18 server kernel: NETDEV WATCHDOG: eth0 (bnx2): transmit queue 0 timed out
Dec 2 21:58:18 server kernel: Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables p4_clockmod freq_table speedstep_lib 8021q garp stp llc ipv6 iTCO_wdt iTCO_vendor_support microcode dcdbas bnx2 serio_raw lpc_ich mfd_core i5000_edac edac_core i5k_amb sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_sas radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]
Dec 2 21:58:18 server kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 0000001
Dec 2 21:58:18 server kernel: Call Trace:
Dec 2 21:58:18 server kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Dec 2 21:58:18 server kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
Dec 2 21:58:18 server kernel: [<ffffffff8147b74b>] ? dev_watchdog+0x26b/0x280
Dec 2 21:58:18 server kernel: [<ffffffff81083e75>] ? internal_add_timer+0xb5/0x110
Dec 2 21:58:18 server kernel: [<ffffffff8147b4e0>] ? dev_watchdog+0x0/0x280
Dec 2 21:58:18 server kernel: [<ffffffff81084b07>] ? run_timer_softirq+0x197/0x340
Dec 2 21:58:18 server kernel: [<ffffffff810ac8f5>] ? tick_dev_program_event+0x65/0xc0
Dec 2 21:58:18 server kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
Dec 2 21:58:18 server kernel: [<ffffffff810ac9ca>] ? tick_program_event+0x2a/0x30
Dec 2 21:58:18 server kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Dec 2 21:58:18 server kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Dec 2 21:58:18 server kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90
Dec 2 21:58:18 server kernel: [<ffffffff815310aa>] ? smp_apic_timer_interrupt+0x4a/0x60
Dec 2 21:58:18 server kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Dec 2 21:58:18 server kernel: <EOI> [<ffffffff81016667>] ? mwait_idle+0x77/0xd0
Dec 2 21:58:18 server kernel: [<ffffffff8152d57a>] ? atomic_notifier_call_chain+0x1a/0x20
Dec 2 21:58:18 server kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Dec 2 21:58:18 server kernel: [<ffffffff81520e13>] ? start_secondary+0x2ac/0x2ef
Dec 2 21:58:18 server kernel: ---[ end trace bae82140b112a8b7 ]---
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- start FTQ dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: RV2P_PFTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: RV2P_TFTQ_CTL 00020000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: RV2P_MFTQ_CTL 00020000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TBDR_FTQ_CTL 00004000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TDMA_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TXP_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TXP_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TPAT_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: RXP_CFTQ_CTL 00008000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: RXP_FTQ_CTL 00100000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: COM_COMXQ_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: COM_COMTQ_FTQ_CTL 00020000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: COM_COMQ_FTQ_CTL 00010000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: CP_CPQ_FTQ_CTL 00008000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: CPU states:
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 045000 mode b84c state 80001000 evt_mask 500 pc 8000be8 pc 8000bc4 instr ac280068
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 085000 mode b84c state 80005000 evt_mask 500 pc 8000674 pc 80006ac instr 8ca5045c
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0c5000 mode b84c state 80001000 evt_mask 500 pc 80044c0 pc 80044bc instr 32a20001
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 105000 mode b84c state 80001000 evt_mask 500 pc 8000750 pc 80007b0 instr 8cca0000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 145000 mode b880 state 80000000 evt_mask 500 pc 8002774 pc 8004e48 instr 3e00008
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 185000 mode b84c state 80000000 evt_mask 500 pc 8000424 pc 8000740 instr e66823
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- end FTQ dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- start TBDC dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: TBDC free cnt: 32
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: LINE CID BIDX CMD VALIDS
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 00 000800 4668 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 01 000800 4668 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 02 000800 5d08 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 03 000800 fe78 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 04 000800 cc58 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 05 000800 cc68 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 06 000800 cc70 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 07 000800 cc30 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 08 000800 cc38 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 09 000800 c658 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0a 000800 c660 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0b 000800 8de0 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0c 000800 72c8 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0d 000800 1648 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0e 000800 bd58 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 0f 000800 8610 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 10 000800 bf50 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 11 000800 bf68 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 12 000800 bf70 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 13 000800 bfb8 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 14 000800 bfc0 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 15 000800 bfc8 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 16 000800 bfd0 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 17 000800 c000 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 18 000800 bc00 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 19 000800 9bf8 00 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1a 118880 0400 90 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1b 100200 e8e0 cb [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1c 02d080 0050 99 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1d 006600 0b20 40 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1e 081a00 08c8 01 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: 1f 190c00 0628 70 [0]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- end TBDC dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: intr_sem[0] PCI_CMD[02b8055e]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: PCI_PM[1d002000] PCI_MISC_CFG[81020088]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: RPM_MGMT_PKT_CTRL[00000000]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: HC_STATS_INTERRUPT_STATUS[00000000]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- start MCP states dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: MCP_STATE_P0[00000106] MCP_STATE_P1[dbfdff7f]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: MCP mode[0000b880] state[80000000] evt_mask[00000500]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: pc[08005e9c] pc[08002b14] instr[00401021]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: shmem states:
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: drv_mb[01030003] fw_mb[00000003] link_status[0000006b] drv_pulse_mb[00003e42]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: dev_info_signature[44564905] reset_type[01005254] condition[00000106]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: 000001c0: 01005254 42530000 00000106 fbffffff
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: 000003cc: 44444444 44444444 44444444 00000a28
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: 000003dc: 0004ffff 00000000 00000000 00000000
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: 000003ec: 00000000 00000000 00000000 00edcf4e
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: DEBUG: 0x3fc[0000ffff]
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: <--- end MCP states dump --->
Dec 2 21:58:18 server kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Down
TagsNo tags attached.
Attached Files

- Relationships
has duplicate 0006813closedIssue Tracker Network down 
has duplicate 0006814closedIssue Tracker Major Failure of Intel 82574L under kernel-2.6.32-431.el6 
has duplicate 0006899closedtoracat kickstart based network install of CentOS 6.5 fails with e1000 driver 

-  Notes
(0018530)
FireMaster (reporter)
2013-12-03 08:46

Now I install the lastest driver for my ethernet adapter. The package name is kmod-netxtreme2-7.8.56-1.rhel6u4.x86_64.The previous driver was installed automatically from DVD installation.
(0018531)
Alexey (reporter)
2013-12-03 10:56

Confirm this problem.

Upgrade from 6.4 to 6.5 and network broken after 2-3 uptime hours.

Dec 2 21:37:49 dd07 kernel: ------------[ cut here ]------------
Dec 2 21:37:49 dd07 kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted)
Dec 2 21:37:49 dd07 kernel: Hardware name: X8DT6
Dec 2 21:37:49 dd07 kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
Dec 2 21:37:49 dd07 kernel: Modules linked in: bnx2fc fcoe libfcoe libfc scsi_transport_fc scsi_tgt bonding 8021q garp stp llc nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables iTCO_wdt iTCO_vendor_support serio_raw sg i2c_i801 i2c_core lpc_ich mfd_core ioatdma dca i7core_edac edac_core ext4 jbd2 mbcache raid1 sd_mod crc_t10dif ahci e1000e ptp pps_core dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
Dec 2 21:37:49 dd07 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 0000001
Dec 2 21:37:49 dd07 kernel: Call Trace:
Dec 2 21:37:49 dd07 kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Dec 2 21:37:49 dd07 kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
Dec 2 21:37:49 dd07 kernel: [<ffffffff8147b74b>] ? dev_watchdog+0x26b/0x280
Dec 2 21:37:49 dd07 kernel: [<ffffffff8105ddae>] ? scheduler_tick+0x11e/0x260
Dec 2 21:37:49 dd07 kernel: [<ffffffff8147b4e0>] ? dev_watchdog+0x0/0x280
Dec 2 21:37:49 dd07 kernel: [<ffffffff81084b07>] ? run_timer_softirq+0x197/0x340
Dec 2 21:37:49 dd07 kernel: [<ffffffff810ac8f5>] ? tick_dev_program_event+0x65/0xc0
Dec 2 21:37:49 dd07 kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
Dec 2 21:37:49 dd07 kernel: [<ffffffff810ac9ca>] ? tick_program_event+0x2a/0x30
Dec 2 21:37:49 dd07 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Dec 2 21:37:49 dd07 kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Dec 2 21:37:49 dd07 kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90
Dec 2 21:37:49 dd07 kernel: [<ffffffff815310aa>] ? smp_apic_timer_interrupt+0x4a/0x60
Dec 2 21:37:49 dd07 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Dec 2 21:37:49 dd07 kernel: <EOI> [<ffffffff812e09be>] ? intel_idle+0xde/0x170
Dec 2 21:37:49 dd07 kernel: [<ffffffff812e09a1>] ? intel_idle+0xc1/0x170
Dec 2 21:37:49 dd07 kernel: [<ffffffff814266f7>] ? cpuidle_idle_call+0xa7/0x140
Dec 2 21:37:49 dd07 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Dec 2 21:37:49 dd07 kernel: [<ffffffff81520e13>] ? start_secondary+0x2ac/0x2ef
Dec 2 21:37:49 dd07 kernel: ---[ end trace f46362668a2e1929 ]---
(0018533)
Ave (reporter)
2013-12-03 16:47

Confirm this problem

http://bugs.centos.org/view.php?id=6813 [^]
(0018536)
toracat (developer)
2013-12-03 16:53

This could be either a hardware or a software issue. Please make sure the BIOS and firmware are up to date.
(0018538)
vip8439 (reporter)
2013-12-03 17:16

Experiencing the same issue with loss of connectivity using the e1000e driver on multiple machines with different hardware.
(0018539)
toracat (developer)
2013-12-03 17:31

Could some of you try the following kernel parameter(s) and see if that helps?

pcie_aspm=off
intremap=off
(0018540)
vip8439 (reporter)
2013-12-03 19:34

toracat, I saw your message after doing a yum update and have not made those changes. In my instance, the machine would drop of the network after a few seconds of being SSH'd into it. The issue occurred after a fresh install from the CentOS 6.5 x86_64 minimal iso. After performing a yum update the following packages were updated:

dracut-004-336.el6_5.2.noarch
dracut-kernel-004-336.el6_5.2.noarch
centos-release-6-5.el6.centos.11.2.x86_64

After the update I rebooted and changed the ACPI setting in the BIOS from APCI 3.0 to ACPI 2.0. Since then the machine has not dropped off the network and has been up for 1:50. Unfortunately, I am not sure which change seemed to resolve the issue. If it drops off of the network again I will report back.
(0018542)
vip8439 (reporter)
2013-12-03 22:34

I have deployed CentOS 6.5 x86_64 from the minimal iso on another machine with identical hardware and installed all available updates from the CentOS repo. This machine stayed on the network for about 10 minutes and then began dropping of the network fairly quickly after each reboot. On all machines I have now added the kernel parameters toracat suggested. I will post back after I have let the machines run for a while. So far so good.

I do get the following in /var/log/messages when the issue occurs:

Dec 3 22:06:01 edge2 kernel: ------------[ cut here ]------------
Dec 3 22:06:01 edge2 kernel: WARNING: at drivers/pci/dmar.c:588 warn_invalid_dmar+0x7a/0x90() (Not tainted)
Dec 3 22:06:01 edge2 kernel: Hardware name: X8SIL
Dec 3 22:06:01 edge2 kernel: [Firmware Warn]: Your BIOS is broken; DMAR reported at address fed93000 returns all ones!
Dec 3 22:06:01 edge2 kernel: BIOS vendor: American Megatrends Inc.; Ver: 1.1; Product Version: 0123456789
Dec 3 22:06:01 edge2 kernel: Modules linked in:
Dec 3 22:06:01 edge2 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 0000001
Dec 3 22:06:01 edge2 kernel: Call Trace:
Dec 3 22:06:01 edge2 kernel: [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Dec 3 22:06:01 edge2 kernel: [<ffffffff81071ebf>] ? warn_slowpath_fmt_taint+0x3f/0x50
Dec 3 22:06:01 edge2 kernel: [<ffffffff8103f6dd>] ? native_set_pte_at+0xd/0x40
Dec 3 22:06:01 edge2 kernel: [<ffffffff8103f019>] ? native_flush_tlb_single+0x9/0x10
Dec 3 22:06:01 edge2 kernel: [<ffffffff812bad6a>] ? warn_invalid_dmar+0x7a/0x90
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c56dfd>] ? check_zero_address+0xd6/0x118
Dec 3 22:06:01 edge2 kernel: [<ffffffff8130c193>] ? acpi_get_table_with_size+0x5a/0xb4
Dec 3 22:06:01 edge2 kernel: [<ffffffff81533565>] ? _etext+0x0/0x3
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c56e51>] ? detect_intel_iommu+0x12/0x91
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c2f758>] ? pci_iommu_alloc+0x5e/0x6c
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c42ac0>] ? mem_init+0x19/0xec
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c26d8c>] ? start_kernel+0x221/0x430
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129
Dec 3 22:06:01 edge2 kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124
Dec 3 22:06:01 edge2 kernel: ---[ end trace a7919e7f17c0a725 ]---
(0018543)
JohnnyHughes (administrator)
2013-12-03 23:45

FireMaster, it seems you installed an external driver .. is that also the case for vip8439 (external driver)?

Does the stock e1000e driver from the kernel work as is for either of you?
(0018544)
toracat (developer)
2013-12-03 23:52

FireMaster has bnx2. Alexey and vip8439 have e1000e. This type of error seems to be observed with several different network devices/drivers.
(0018545)
toracat (developer)
2013-12-03 23:54

... and I don't see any indication of "external" driver. ??
(0018546)
vip8439 (reporter)
2013-12-04 02:06

JohnnyHughes, I am only using the stock e1000e driver. These machines are scheduled to be postfix servers in a production hosting environment so I have only stuck with what comes with CentOS. I have not altered the yum configurations in any way and have not added any third-party repos. The only packages installed outside of a standard minimal install are screen and vim-enhanced. The output from dmesg reports the NIC as eth1 Intel(R) PRO/1000. All of my hardware is using the built-in NICs on SuperMicro motherboards; however I do have one machine using a different model SuperMicro board and it is not having any issues with CentOS 6.5. The machine with the different model board also reports Intel(R) PRO/1000. These machines will not be live for quite some time so I am willing to try some things. After all, re-installing from a minimal ISO is very quick. I do want to mention that after adding the kernel parameters that toracat suggested, the machines did not drop off the network again as of end of work day; however, they are under zero load at the moment.
(0018548)
Alexey (reporter)
2013-12-04 06:18

intremap=off didn't help
(0018549)
FireMaster (reporter)
2013-12-04 09:46

Hi All.
After install external driver from here: http://www.broadcom.com/support/ethernet_nic/downloaddrivers.php [^]

Before 6 month I was report for the same problem with e1000e driver
http://bugs.centos.org/view.php?id=6080 [^]

Please check the 6080 bug.
Best regards,
(0018550)
FireMaster (reporter)
2013-12-04 09:47

After install external driver works correcly ( > 24hrs uptime)
(0018556)
toracat (developer)
2013-12-04 17:14

A similar case has been reported in bug 0006814 with the e1000e driver. A workaround in there was to use the kernel option pcie_aspm=off.
(0018557)
toracat (developer)
2013-12-04 17:24

@FireMaster

In bug 0006080, the problem was resolved by installing a newer version of the e1000e driver from ELRepo. How about this time? The driver version in the CentOS 6.5 kernel is 2.3.2-k. kmod-e1000e currently has 2.5.4.
(0018558)
toracat (developer)
2013-12-04 17:27

> intremap=off didn't help

Alexey, how about the other one pcie_aspm=off ?

Also, you, too, can try the kmod-e1000e package from ELRepo to see if the latest version resolves the issue.
(0018562)
kstange (reporter)
2013-12-04 18:52

I've installed the e1000e driver from ELRepo on my server and the connection is thus far stable. I am going to leave the server for a while and see if this remains the case.
(0018563)
kstange (reporter)
2013-12-04 18:53

The above ELRepo driver test is with the pcie_aspm=off option removed from the kernel config, so ASPM will operate normally.
(0018564)
toracat (developer)
2013-12-04 21:32

@kstange

I was going to ask about the pcie_aspm option. :) Please do report back once you have run the system for some prolonged time.
(0018565)
vip8439 (reporter)
2013-12-04 21:51

I wanted to report that our machines still have not dropped off of the network since using the pcie_aspm=off and intremap=off parameters. I'm half way wondering if it will work without incident if I remove those parameters and simply disable ASPM in the BIOS. I think I will leave it as is and wait for a newer e1000e driver.
(0018566)
Alexey (reporter)
2013-12-05 05:31

> Alexey, how about the other one pcie_aspm=off ?
This workaround works! Uptime 23 hours. Thanks.
(0018567)
univ (reporter)
2013-12-05 06:25

What a crazy bug. Do you agree "yum update" should never break a system?

I just ran "yum update", rebooted the server afterwards because of kernel update and voila - server offline, NICs not working. Logged in via IPMI, rebooted again, same thing.

pcie_aspm=off and another reboot fixed it, fortunately.

I'm only lucky I didn't do the same with the other 10+ production servers.

PS: Supermicro board here too.
(0018568)
univ (reporter)
2013-12-05 06:45

PS:
DMI: Supermicro X8SIE/X8SIE, BIOS 1.2 08/19/11
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
CentOS release 6.5 (Final)
Linux server 2.6.32-431.el6.x86_64 0000001 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
(0018570)
kstange (reporter)
2013-12-05 07:35

Just wanted to note that I have experienced no issues since switching to the ELrepo driver *without* disabling ASPM, so this does seem to be a suitable correction.
(0018571)
toracat (developer)
2013-12-05 08:40

Thanks for reporting back, @kstange.

So far, updating the e1000e driver seems to be the best solution. I will note once again that the distro kernel has version 2.3.2-k and kmod-e1000e is currently at 2.5.4.

Note also that ELRepo's kmods are kABI-tracking, meaning they survive kernel updates transparently - no need to reinstall.

As for the bnx2 driver, getting the latest version from Broadcom has worked (per the OP, FireMaster).
(0018574)
vip8439 (reporter)
2013-12-05 16:22

univ, you have the same Supermicro board as we do. The bug did not seem to affect all of our other Supermicro boards that are not X8SIE so I am really interested in what is different in the X8SIE. Running lshw on all of our Supermicro boards reports an Intel R Pro/1000. Perhaps the messages I see in /var/log/messages really are correct and the BIOS on the X8SIE really does have a bug? I agree updating the driver seems to be the best solution, but I did want to point out that univ and I have the same motherboard since that knowledge may help someone in the future.
(0018575)
vip8439 (reporter)
2013-12-05 16:45

Ok, I lied. The affected machine is an X8SIL BIOS version 1.1.
(0018576)
toracat (developer)
2013-12-05 16:51

Yes, it will help if we could collect the info regarding what hardware / BIOS / firmware are affected.

Depending on the situation, the solution (or workaround) is to use a new version of the driver or to use (a) kernel parameter(s).
(0018577)
kstange (reporter)
2013-12-05 17:33

As I earlier noted my board is an X8SIE-F (the -F means this is a board with a Supermicro IPMI controller). I am running BIOS 1.0b. I can try flashing to 1.2a, but Supermicro helpfully publishes no changelog for BIOS updates, so I have no idea what to expect out of this revision.
(0018579)
enechai (reporter)
2013-12-05 17:38

Hi all.

Confirm the problem.

Dec 5 05:44:25 db5 kernel: ------------[ cut here ]------------
Dec 5 05:44:25 db5 kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted)
Dec 5 05:44:25 db5 kernel: Hardware name: X8SIE
Dec 5 05:44:25 db5 kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
Dec 5 05:44:25 db5 kernel: Modules linked in: xt_NOTRACK iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ext3 jbd sr_mod cdrom iTCO_wdt iTCO_vendor_support serio_raw i2c_i801 i2c_core lpc_ich mfd_core e1000e ptp pps_core sg ext4 jbd2 mbcache usb_storage sd_mod crc_t10dif pata_acpi ata_generic ata_piix aacraid dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Dec 5 05:44:25 db5 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 0000001
Dec 5 05:44:25 db5 kernel: Call Trace:
Dec 5 05:44:25 db5 kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Dec 5 05:44:25 db5 kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
Dec 5 05:44:25 db5 kernel: [<ffffffff8147b74b>] ? dev_watchdog+0x26b/0x280
Dec 5 05:44:25 db5 kernel: [<ffffffff8147b4e0>] ? dev_watchdog+0x0/0x280
Dec 5 05:44:25 db5 kernel: [<ffffffff81084b07>] ? run_timer_softirq+0x197/0x340
Dec 5 05:44:25 db5 kernel: [<ffffffff810ac8f5>] ? tick_dev_program_event+0x65/0xc0
Dec 5 05:44:25 db5 kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
Dec 5 05:44:25 db5 kernel: [<ffffffff810ac9ca>] ? tick_program_event+0x2a/0x30
Dec 5 05:44:25 db5 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Dec 5 05:44:25 db5 kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Dec 5 05:44:25 db5 kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90
Dec 5 05:44:25 db5 kernel: [<ffffffff815310aa>] ? smp_apic_timer_interrupt+0x4a/0x60
Dec 5 05:44:25 db5 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Dec 5 05:44:25 db5 kernel: <EOI> [<ffffffff812e09be>] ? intel_idle+0xde/0x170
Dec 5 05:44:25 db5 kernel: [<ffffffff812e09a1>] ? intel_idle+0xc1/0x170
Dec 5 05:44:25 db5 kernel: [<ffffffff814266f7>] ? cpuidle_idle_call+0xa7/0x140
Dec 5 05:44:25 db5 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Dec 5 05:44:25 db5 kernel: [<ffffffff81520e13>] ? start_secondary+0x2ac/0x2ef
Dec 5 05:44:25 db5 kernel: ---[ end trace f550dccd6de4c3d8 ]---
Dec 5 05:44:25 db5 kernel: e1000e 0000:03:00.0: eth0: Reset adapter unexpectedly
Dec 5 05:44:25 db5 kernel: e1000e 0000:03:00.0: eth0: Timesync Tx Control register not set as expected

Kernel - 2.6.32-431.el6.x86_64

Network card: Intel Corporation 82574L Gigabit Network

BIOS Information
        Vendor: American Megatrends Inc.
        Version: 1.2
        Release Date: 08/19/11
        BIOS Revision: 8.16

System Information
        Manufacturer: Supermicro
        Product Name: X8SIE
        Version: 0123456789
(0018581)
kstange (reporter)
2013-12-05 17:44

enechai's data seems to suggest that the 1.2 BIOS will not help, so I'm not going to waste the effort.
(0018582)
univ (reporter)
2013-12-05 17:45

kstange: Yeah, and even if the BIOS update would help, this is a driver bug which needs to get fixed.
(0018584)
toracat (developer)
2013-12-05 17:50

I am not so sure about declaring this to be a "driver bug". It could still be due to a BIOS / firmware issue, which can seemingly be "fixed" by use of certain kernel parameter(s) or an updated device driver.
(0018585)
univ (reporter)
2013-12-05 18:01

toracat: First, thanks for the effort you are putting into this! Driver bug or not: ok, maybe, but if I run a production server just fine with CentOS, then type "yum update" and reboot afterwards, and then the NICs stop working, it sounds to me, as an ordinary person, very much like a driver bug. :) On the other hand, if I would install a new system from scratch, and then find out that with the latest CentOS version the NICs don't work, I could argue whether it is a BIOS / have-to-update-something issue, but not if I run a production machine which suddenly stops working. Maybe we want to call it a quality assurance bug ;) I'm guessing this will have to get fixed at RedHat?
(0018586)
toracat (developer)
2013-12-05 18:05

> univ wrote:
> I'm guessing this will have to get fixed at RedHat?

Precisely. Would you be willing to file a report there? ( http://bugzilla.redhat.com [^] )
(0018590)
kstange (reporter)
2013-12-05 18:30

I have reported this bug to RH at https://bugzilla.redhat.com/show_bug.cgi?id=1038754. [^]
(0018593)
toracat (developer)
2013-12-05 18:48

Thanks, kstange. The BZ is 'private'. Please update this thread with new development. :)
(0018594)
univ (reporter)
2013-12-05 18:55

kstange: I'm getting "You are not authorized to access bug #1038754." - can you make it public?
(0018595)
kstange (reporter)
2013-12-05 19:20

I can't. RH policy makes kernel bugs private. I will provide any relevant news from the bug here.
(0018603)
Fluoros (reporter)
2013-12-08 09:10

Having the same problem with Supermicro X8SIL.
BIOS version is 1.2a .

Adding pcie_aspm=off to the kernel parameter helped.

Thanks!
(0018610)
kstange (reporter)
2013-12-09 19:06

I just thought I'd note that while my NICs did not have their EEPROM patched using Intel's fixeep-82574_83.sh, running this had no effect.

There are no real updates yet from bugs, but if anyone has a Supermicro contact that wants to get involved in this, RH suggested doing so.
(0018612)
univ (reporter)
2013-12-09 19:11

I just sent an eMail to Supermicro with a description and link to this thread. Did RH say what the best way for Supermicro would be to get involved?
(0018613)
kstange (reporter)
2013-12-09 19:17

They just suggested we get them involved. I also contacted their main support email, since I don't have anyone to contact directly.

If we could get someone from Intel involved that would likely help even more, but I don't have the info to do that.
(0018614)
univ (reporter)
2013-12-09 19:24

Ok, I contacted Intel through their general support mechanism. Maybe we're lucky.
(0018627)
RobHost (reporter)
2013-12-12 20:54

We can also confirm this bug. Is there any chance to set the param without reboot the machines?
The RHEL docs [1] means /sys/module/pcie_aspm/parameters/policy can be used. But it does not change from "default performance [powersave]" even with kernel param "pcie_aspm=off".

[1] https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Power_Management_Guide/ASPM.html [^]
(0019026)
stg Ideeel (reporter)
2014-01-11 20:46

Same bug here on a Supermicro X8SIE-F with CentOS-6.5-x86_64
Found a another workaround:
Enter BIOS >> Advanced >> Advanced - Chipset Control >>
Change "Active State Power Management" to Enabled.
Default Bios setting is disabled.

After charging this setting no problems with "dead network" anymore,
before network would die in just 5 a 10 minutes.
(0019055)
mdieterich (reporter)
2014-01-14 20:12

Same issue here on a Supermicro X8SIE, 1.2a firmware, running CentOS-6.5-x86_64.
 
I initially stumbled upon it when trying to kickstart with CentOS 6.5 and reported it as an issue there (0006899), before they kindly pointed me over here. I was able to complete the install by tossing "pcie_aspm=off". I'm now trying the "Active State Power Management" bios setting to see if it can remain up without requiring I turn off PCIe power management in the kernel.
(0019069)
mdieterich (reporter)
2014-01-16 13:03

Coming up on two days of uptime since I enabled the "Active State Power Management" BIOS setting. The last 12 hours have been under active load. Things seem OK. It looks like there is some bug related to the e1000e driver and power management, perhaps only when power management is not enabled?
(0019111)
mdieterich (reporter)
2014-01-23 15:21

I think we are still tickling a bug. The machine managed to stay up with load for about 28 hours and then dropped off the network again. We managed to grab a call trace this time, which may be of some help to someone:

Call Trace:
[<ffffffffa03a0b46>] ? _ZN12FileMetadata22setInodeDirtyAndVerifyEj+0x16/0x50 [mmfs26]
[<ffffffffa02abcd3>] cxiWaitEventWait+0x143/0x220 [mmfslinux]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffffa031ffc0>] _Z20DeclareResourceUsagejij+0x310/0x4c0 [mmfs26]
[<ffffffff8105571d>] ? check_preempt_curr+0x6d/0x90
[<ffffffffa03202d5>] ? _Z18HoldDaemonSegAndSGP12SegMapStatusPiP13gpfsVfsData_tPP11StripeGroupS1_Pji16VfsOperationType+0x165/0x200 [mmfs26]
[<ffffffffa0346801>] ? _Z9gpfsWriteP13gpfsVfsData_tP15KernelOperationP9cxiNode_tiP8cxiUio_tP9MMFSVInfoP10cxiVattr_tSA_P10ext_cred_tPvi+0x2d71/0x56b0 [mmfs26]
[<ffffffff81297e48>] ? swiotlb_dma_mapping_error+0x18/0x30
[<ffffffffa00f47f6>] ? e1000_xmit_frame+0x786/0xe70 [e1000e]
[<ffffffff812987c0>] ? swiotlb_map_page+0x0/0x100
[<ffffffff81460364>] ? dev_hard_start_xmit+0x224/0x480
[<ffffffff8147bc38>] ? sch_direct_xmit+0x78/0x1c0
[<ffffffff8146075f>] ? dev_queue_xmit+0x11f/0x320
[<ffffffff810a12df>] ? up+0x2f/0x50
[<ffffffff810a12df>] ? up+0x2f/0x50
...

There is more, but I suspect the e1000e driver is still at fault here and that it was at the root of the hang. I'm going to need to downgrade this machine soon, since I can't leave it out of commission. Is there anything else that would be useful to try before I do this?

Thanks,

Mark
(0019112)
univ (reporter)
2014-01-23 15:29

I think "pcie_aspm=off" is the workaround of choice. It seems you had it only enabled during the install and removed it afterwards? It fixed the issue for me and my machines are continuously up since then.
(0019113)
toracat (developer)
2014-01-23 16:49

@mdieterich

You can try the e1000e driver from ELRepo. Please see the comments by kstange in this thread. The newer version of the driver offered by ELRepo seems to have fixed the problem.
(0019115)
mdieterich (reporter)
2014-01-23 17:09

@toracat

Thanks. I somehow missed the references to ELRepo. I'll give this a shot.
(0019129)
mdieterich (reporter)
2014-01-25 22:28

I now have 7 of these machines upgraded to CentOS 6.5, so I've got some additional data points. It looks like the combination of the e1000e driver from ELRepo combined with the BIOS setting enabling the "Active State Power Management" option results in stability. Either one of these alone is not enough.

Enabling the BIOS option with the distributed e1000e driver, my test machines made it about 12 hours under load before they dropped off the network.

Running the e1000e driver from ELRepo without the BIOS option enabled and my machines were dropping off the network fairly quickly, some within 5 minutes of being put under load. Captured a stack trace during the most recent version of this crash:

Call Trace:
kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
kernel: [<ffffffff81014969>] ? sched_clock+0x9/0x10
kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
kernel: [<ffffffff8147b75b>] ? dev_watchdog+0x26b/0x280
kernel: [<ffffffff810a7159>] ? ktime_get+0x69/0xf0
kernel: [<ffffffff81083e75>] ? internal_add_timer+0xb5/0x110
kernel: [<ffffffff8147b4f0>] ? dev_watchdog+0x0/0x280
kernel: [<ffffffff81084b07>] ? run_timer_softirq+0x197/0x340
kernel: [<ffffffff8109fdcb>] ? hrtimer_interrupt+0x14b/0x260
kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
kernel: [<ffffffff810e6ed0>] ? handle_IRQ_event+0x60/0x170
kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90
kernel: [<ffffffff81530ff5>] ? do_IRQ+0x75/0xf0
kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
kernel: <EOI> [<ffffffff812e09ce>] ? intel_idle+0xde/0x170
kernel: [<ffffffff812e09b1>] ? intel_idle+0xc1/0x170
kernel: [<ffffffff81426707>] ? cpuidle_idle_call+0xa7/0x140
kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
kernel: [<ffffffff81520e2c>] ? start_secondary+0x2ac/0x2ef
kernel: ---[ end trace 7cb9cfd2aff3cf36 ]---
kernel: e1000e 0000:03:00.0: eth0: Reset adapter unexpectedly
kernel: e1000e 0000:03:00.0: eth0: Timesync Tx Control register not set as expected

The BIOS version does not seem to make a difference.
(0019130)
toracat (developer)
2014-01-26 13:45

@mdieterich

Thanks for reporting back with the result. So, the newer version of e1000e (from ELRepo) works for you. At least RH is aware of the issue. Let's see how they fix it.
(0019138)
ghammond-adfonic (reporter)
2014-01-28 13:30

Just hit this myself on Supermicro hardware with -431.3.1.

I reverted to kernel-2.6.32-358.14.1.el6.x86_64 (from 6.4) which was fine before and still seems to be OK on 6.5. An potential option for those of us who don't have BIOS/console access to machines.
(0019184)
kstange (reporter)
2014-01-31 19:02

I just wanted to copy some information from the RH bug that has been ongoing. RH had me check the same BIOS option indicated and I did find that turning on "Active State Power Management" in my BIOS (which defaults to disabled on my board; Supermicro X8SIE-F) has caused the connection to remain stable for me so far.

We verified that the hardware registers are different depending on the ASPM option in the BIOS, so Linux should be able to make an intelligent and correct decision, but isn't.

At this point, they are trying to pass it off as a configuration issue, but I'm trying to get them to acknowledge that since the ELRepo driver appears to improve the behavior, it is a problem with the driver or with Linux not honoring the ASPM setting correctly for the NIC. If it's a configuration issue, then RH 6.5 has a system requirement that your BIOS has ASPM enabled or no support for ASPM, and I think that makes no sense.

Personally for me, any of ELRepo driver, pcie_aspm=off, or turning on ASPM in the BIOS have resolved the stability problem, or at least made it extremely unlikely such that it's not been possible to reproduce.
(0019185)
kstange (reporter)
2014-01-31 19:04

I wouldn't recommend going back to the 358 kernel. There are security vulnerabilities that are fixed since then.
(0019186)
mdieterich (reporter)
2014-01-31 19:14

@kstange

I'd encourage you to keep after them. This is one of the various options I tried; e10000e driver shipping with 6.5 and "Active State Power Management" enabled in the BIOS. The system was more stable with this setup, but still crashed after about 12 hours once the machine was under full load. The combination of this BIOS option enabled and the ELRepo driver is the only thing I've found that has kept the systems stable. Even with the ELRepo driver installed, without this BIOS option on, the machines were still dropping off the network.

Mark
(0019200)
save (reporter)
2014-02-04 08:17

Hi,
this night we had the same issue. We are on a X9SCM-F with both Ethernet used in bonding. Tonight one of the two crashed, the system was updated to last kernel 2 days ago. Now we have installed ELRepo driver and will enable ASPM from BIOS, hoping this will solve the problem.

Saverio
(0019202)
save (reporter)
2014-02-04 09:16

Update: in my BIOS under ASPM there were three option:

Diabled (default)
Auto
Force L0

I selected Auto.

Could someone who installed ELRepo drivers please check if I have the correct version?

modinfo e1000e

filename: /lib/modules/2.6.32-431.3.1.el6.x86_64/weak-updates/e1000e/e1000e.ko
version: 2.5.4-NAPI
license: GPL
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: 14FC0D45EE1DAA1B5E0DBBA
(0019210)
AlanBartlett (reporter)
2014-02-04 16:20

Re: Note 19202

Being responsible for creating that kmod-e1000e package for the ELRepo Project, I can confirm that you have the latest version installed.

[Build64R6 tmp]$ modinfo ./e1000e.ko | grep -vE 'alias|depends|parm'
filename: ./e1000e.ko
version: 2.5.4-NAPI
license: GPL
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: 14FC0D45EE1DAA1B5E0DBBA
vermagic: 2.6.32-279.el6.x86_64 SMP mod_unload modversions
[Build64R6 tmp]$
(0019232)
MikeDVB (reporter)
2014-02-06 00:27

Performed a fresh CentOS 6.4 x86_64 Minimal installation on a server with a X8SIL-F motherboard with bios version 1.2a. The install went fine but once we got into the OS there was no network connectivity.

Upon restarting the network [just to make sure mostly that it was started on boot] we got this message for eth0:
"Timesync Tx Control register not set as expected"

This happened on both of these kernels:
2.6.32-431.3.1.el6.x86_64
2.6.32-431.el6.x86_64

This message also showed on the console:
irq 37: nobody cared (try booting with the "irqpoll" option)
handlers:
[<ffffffffa0125260>] (e1000_msix_other+0x0/0x1f0 [e1000e])
Disabling IRQ 0000037

Adding "pcie_aspm=off" to the kernel parameter did restore stability in the limited testing I was able to perform so far. This, however, doesn't seem to be a proper long-term solution. ASPM is disabled in the bios, however, we will not be enabling it even as an attempted work-around for this issue.

I have not tried the more updated drivers but this is for a backup server that will be relatively infrequently rebooted [private network server] so it's not much of a huge issue for us at this point but if this were any of our other production servers it would be a huge issue.

If my understanding is correct we're essentially waiting on RedHat to acknowledge and resolve this issue. I suppose I won't hold my breath.
(0019233)
MikeDVB (reporter)
2014-02-06 00:31

Forgot to add we installed 6.4 because it was the ISO we had handy - the issue occurred on 6.4 and also on 6.5 [after a yum update + reboot].
(0019264)
medgen (reporter)
2014-02-12 07:31

We run Scientific Linux and having the same problem here. As it is based on RHEL6 like CentOS, this isn't surprising, I know. But I only wanted to document that for others, that SL is affected, too (SL doesn't have a bugtracker any more).

We have the problem on SL 6.4 (appeared with kernel-2.6.32-431.1.2.el6.x86_64) and on SL 6.5 (2.6.32-431.3.1.el6.x86_64).

Setting "Active State Power Management" (ASPM) in the Bios of our SuperMicro X8SIE-LN4 boards to "yes" fixes/workaround the issue great and the "Timesync Tx Control register not set as expected" messages are gone. Thanks for that information.
(0019496)
univ (reporter)
2014-03-13 14:27

@kstange

Any news from RedHat? The issue is still there. :-( CentOS release 6.5 (Final) + 2.6.32-431.5.1.el6.x86_64
(0019667)
BonOnlines (reporter)
2014-04-24 07:59

@All

Is there any update on this issue. I had also some of my servers which fail to this issue.
(0019674)
twopoint71 (reporter)
2014-04-25 02:07

This problem occurred on my server as well after running a YUM update and rebooting. After reading the notes, I decided to try medgen's solution, and I can confirm stability returned after enabling "Active State Power Management" in the BIOS.

Here is my server setup
OS version.............: CentOS 6.5 (after update)
BIOS version...........: 1.0b
Board version..........: X8SIE-F
Kernel verison.........: 2.6.32-431.11.2.el6.x86_64
Ethernet driver........: e1000e
Ethernet driver version: 2.3.2-k
Ethernet controller....: Intel 82574L
Ethernet firmware ver..: 1.9-0

Thanks medgen!
(0019749)
ppyy (reporter)
2014-05-11 14:03

same problem on Dell 1950.

any progress of BZ of redhat?
(0020176)
tmueko (reporter)
2014-07-04 10:40

Same Problem on Asus P7F-E
e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
e1000e: Copyright(c) 1999 - 2013 Intel Corporation.
e1000e 0000:05:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
e1000e 0000:05:00.0: setting latency timer to 64
e1000e 0000:05:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
e1000e 0000:05:00.0: irq 29 for MSI/MSI-X
  alloc irq_desc for 30 on node -1
  alloc kstat_irqs on node -1
e1000e 0000:05:00.0: irq 30 for MSI/MSI-X
  alloc irq_desc for 31 on node -1
  alloc kstat_irqs on node -1
e1000e 0000:05:00.0: irq 31 for MSI/MSI-X
e1000e 0000:05:00.0: eth0: registered PHC clock
e1000e 0000:05:00.0: eth0: (PCI Express:2.5GT/s:Width x1) e0:cb:4e:12:79:15
e1000e 0000:05:00.0: eth0: Intel(R) PRO/1000 Network Connection
e1000e 0000:05:00.0: eth0: MAC: 3, PHY: 8, PBA No: FFFFFF-0FF
(0020763)
xander (reporter)
2014-08-28 08:50

same problem here
HP ML350 G5

Aug 28 08:42:11 ga2 kernel: ------------[ cut here ]------------
Aug 28 08:42:11 ga2 kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted)
Aug 28 08:42:11 ga2 kernel: Hardware name: ProLiant ML350 G5
Aug 28 08:42:11 ga2 kernel: NETDEV WATCHDOG: eth1 (bnx2): transmit queue 0 timed out
Aug 28 08:42:11 ga2 kernel: Modules linked in: ebtable_nat ebtables mptctl autofs4 ipmi_devintf bridge bonding 8021q garp stp llc ipt_REJECT iptable_filter iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun kvm_intel kvm iTCO_wdt iTCO_vendor_support osst st ch microcode hpilo hpwdt bnx2 sg lpc_ich mfd_core i5000_edac edac_core i5k_amb shpchp ext4 jbd2 mbcache sr_mod cdrom hpsa cciss mptsas mptscsih mptbase scsi_transport_sas pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Aug 28 08:42:11 ga2 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.23.3.el6.x86_64 0000001
Aug 28 08:42:11 ga2 kernel: Call Trace:
Aug 28 08:42:11 ga2 kernel: <IRQ> [<ffffffff81071b37>] ? warn_slowpath_common+0x87/0xc0
Aug 28 08:42:11 ga2 kernel: [<ffffffff81071c26>] ? warn_slowpath_fmt+0x46/0x50
Aug 28 08:42:11 ga2 kernel: [<ffffffff8147c5db>] ? dev_watchdog+0x26b/0x280
Aug 28 08:42:11 ga2 kernel: [<ffffffff813c6ed0>] ? rh_timer_func+0x0/0x10
Aug 28 08:42:11 ga2 kernel: [<ffffffff813c6600>] ? usb_hcd_poll_rh_status+0x140/0x180
Aug 28 08:42:11 ga2 kernel: [<ffffffff8105dc2c>] ? scheduler_tick+0xcc/0x260
Aug 28 08:42:11 ga2 kernel: [<ffffffff8147c370>] ? dev_watchdog+0x0/0x280
Aug 28 08:42:11 ga2 kernel: [<ffffffff810847f7>] ? run_timer_softirq+0x197/0x340
Aug 28 08:42:11 ga2 kernel: [<ffffffff810ac585>] ? tick_dev_program_event+0x65/0xc0
Aug 28 08:42:11 ga2 kernel: [<ffffffff8107a5f1>] ? __do_softirq+0xc1/0x1e0
Aug 28 08:42:11 ga2 kernel: [<ffffffff810ac65a>] ? tick_program_event+0x2a/0x30
Aug 28 08:42:11 ga2 kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 28 08:42:11 ga2 kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Aug 28 08:42:11 ga2 kernel: [<ffffffff8107a4a5>] ? irq_exit+0x85/0x90
Aug 28 08:42:11 ga2 kernel: [<ffffffff8153239a>] ? smp_apic_timer_interrupt+0x4a/0x60
Aug 28 08:42:11 ga2 kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
Aug 28 08:42:11 ga2 kernel: <EOI> [<ffffffff81315d5c>] ? acpi_idle_enter_bm+0x29c/0x2d0
Aug 28 08:42:11 ga2 kernel: [<ffffffff81315d55>] ? acpi_idle_enter_bm+0x295/0x2d0
Aug 28 08:42:11 ga2 kernel: [<ffffffff81427477>] ? cpuidle_idle_call+0xa7/0x140
Aug 28 08:42:11 ga2 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Aug 28 08:42:11 ga2 kernel: [<ffffffff8150de1a>] ? rest_init+0x7a/0x80
Aug 28 08:42:11 ga2 kernel: [<ffffffff81c26f8f>] ? start_kernel+0x424/0x430
Aug 28 08:42:11 ga2 kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129
Aug 28 08:42:11 ga2 kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124
Aug 28 08:42:11 ga2 kernel: ---[ end trace eeb0eab76937e445 ]---
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- start FTQ dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: RV2P_PFTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: RV2P_TFTQ_CTL 00020000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: RV2P_MFTQ_CTL 00020000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TBDR_FTQ_CTL 00004000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TDMA_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TXP_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TXP_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TPAT_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: RXP_CFTQ_CTL 00008000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: RXP_FTQ_CTL 00100000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: COM_COMXQ_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: COM_COMTQ_FTQ_CTL 00020000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: COM_COMQ_FTQ_CTL 00010000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: CP_CPQ_FTQ_CTL 00008000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: CPU states:
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 045000 mode b84c state 80001000 evt_mask 500 pc 8000bf8 pc 8000bf8 instr 1440ffed
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 085000 mode b84c state 80001000 evt_mask 500 pc 800067c pc 8000678 instr 8d2f0000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0c5000 mode b84c state 80001000 evt_mask 500 pc 80044c0 pc 80044cc instr 8f550000
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 105000 mode b84c state 80001000 evt_mask 500 pc 8000730 pc 80007b0 instr 4021
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 145000 mode b880 state 80004000 evt_mask 500 pc 8004dc0 pc 8003e60 instr 30420100
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 185000 mode b84c state 80000000 evt_mask 500 pc 8000730 pc 8000708 instr 3e00008
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- end FTQ dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- start TBDC dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: TBDC free cnt: 32
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: LINE CID BIDX CMD VALIDS
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 00 000800 07d0 00 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 01 000800 07d0 00 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 02 13bf80 fd68 fb [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 03 16ef80 fff8 ff [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 04 0efd80 f728 dd [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 05 01fd80 fcd8 a3 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 06 1dfd80 f790 ff [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 07 1ff980 def8 2d [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 08 1fff80 fe78 7f [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 09 14ad00 aff8 ff [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0a 121f00 5b78 1e [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0b 1d5e80 3228 7e [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0c 1fe780 bbf8 d5 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0d 1bef00 77c8 dd [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0e 1ffa80 fbe0 d7 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 0f 17cf80 f3f0 d3 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 10 1f7980 e758 a7 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 11 1e5b00 f768 de [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 12 0ffe80 57f8 cf [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 13 03d700 5ef8 fb [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 14 0ffe80 5fc8 f6 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 15 07e700 b4f8 fc [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 16 17bf80 59f8 7e [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 17 1f7e80 ee98 9d [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 18 1fef80 fe78 ef [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 19 1bfb80 f7e8 af [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1a 1fba80 bef8 f5 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1b 0fbf80 cff8 e9 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1c 1fff80 ffb8 6f [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1d 1f7f80 39f8 3a [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1e 1fbf00 e318 78 [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: 1f 1bbf80 7fb8 7a [0]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- end TBDC dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: intr_sem[0] PCI_CMD[02b80556]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: PCI_PM[64002000] PCI_MISC_CFG[81020088]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: RPM_MGMT_PKT_CTRL[00000000]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: HC_STATS_INTERRUPT_STATUS[00000000]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- start MCP states dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: MCP_STATE_P0[00000106] MCP_STATE_P1[f7fbffff]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: MCP mode[0000b880] state[80004000] evt_mask[00000500]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: pc[08000eb0] pc[08004f18] instr[00601821]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: shmem states:
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: drv_mb[01030008] fw_mb[00000008] link_status[0000006f] drv_pulse_mb[000048cf]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: dev_info_signature[44564903] reset_type[01005254] condition[00000106]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: 000001c0: 01005254 4253800a 00000106 efffff7c
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: 000003cc: 44444444 44444444 44444444 00000a3c
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: 000003dc: 0ffeffff 0000ffff ffffffff ffffffff
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: 000003ec: 00000000 00000000 00000000 00000002
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: DEBUG: 0x3fc[0000ffff]
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: <--- end MCP states dump --->
Aug 28 08:42:11 ga2 kernel: bnx2 0000:03:00.0: eth1: NIC Copper Link is Down
Aug 28 08:42:11 ga2 kernel: bonding: bond0: link status definitely down for interface eth1, disabling it

- Issue History
Date Modified Username Field Change
2013-12-03 07:47 FireMaster New Issue
2013-12-03 08:46 FireMaster Note Added: 0018530
2013-12-03 10:56 Alexey Note Added: 0018531
2013-12-03 16:47 Ave Note Added: 0018533
2013-12-03 16:50 toracat Relationship added has duplicate 0006813
2013-12-03 16:53 toracat Note Added: 0018536
2013-12-03 17:16 vip8439 Note Added: 0018538
2013-12-03 17:31 toracat Note Added: 0018539
2013-12-03 19:34 vip8439 Note Added: 0018540
2013-12-03 22:34 vip8439 Note Added: 0018542
2013-12-03 23:45 JohnnyHughes Note Added: 0018543
2013-12-03 23:52 toracat Note Added: 0018544
2013-12-03 23:54 toracat Note Added: 0018545
2013-12-04 02:06 vip8439 Note Added: 0018546
2013-12-04 06:18 Alexey Note Added: 0018548
2013-12-04 09:46 FireMaster Note Added: 0018549
2013-12-04 09:47 FireMaster Note Added: 0018550
2013-12-04 17:14 toracat Note Added: 0018556
2013-12-04 17:24 toracat Note Added: 0018557
2013-12-04 17:27 toracat Note Added: 0018558
2013-12-04 18:41 toracat Relationship added has duplicate 0006814
2013-12-04 18:52 kstange Note Added: 0018562
2013-12-04 18:53 kstange Note Added: 0018563
2013-12-04 21:32 toracat Note Added: 0018564
2013-12-04 21:51 vip8439 Note Added: 0018565
2013-12-05 05:31 Alexey Note Added: 0018566
2013-12-05 06:25 univ Note Added: 0018567
2013-12-05 06:45 univ Note Added: 0018568
2013-12-05 07:35 kstange Note Added: 0018570
2013-12-05 08:40 toracat Note Added: 0018571
2013-12-05 16:22 vip8439 Note Added: 0018574
2013-12-05 16:45 vip8439 Note Added: 0018575
2013-12-05 16:46 toracat Status new => assigned
2013-12-05 16:46 toracat Steps to Reproduce Updated View Revisions
2013-12-05 16:51 toracat Note Added: 0018576
2013-12-05 17:33 kstange Note Added: 0018577
2013-12-05 17:38 enechai Note Added: 0018579
2013-12-05 17:44 kstange Note Added: 0018581
2013-12-05 17:45 univ Note Added: 0018582
2013-12-05 17:50 toracat Note Added: 0018584
2013-12-05 18:01 univ Note Added: 0018585
2013-12-05 18:05 toracat Note Added: 0018586
2013-12-05 18:30 kstange Note Added: 0018590
2013-12-05 18:48 toracat Note Added: 0018593
2013-12-05 18:55 univ Note Added: 0018594
2013-12-05 19:20 kstange Note Added: 0018595
2013-12-08 09:10 Fluoros Note Added: 0018603
2013-12-09 19:06 kstange Note Added: 0018610
2013-12-09 19:11 univ Note Added: 0018612
2013-12-09 19:17 kstange Note Added: 0018613
2013-12-09 19:24 univ Note Added: 0018614
2013-12-12 20:54 RobHost Note Added: 0018627
2014-01-11 20:46 stg Ideeel Note Added: 0019026
2014-01-14 20:08 toracat Relationship added has duplicate 0006899
2014-01-14 20:12 mdieterich Note Added: 0019055
2014-01-16 13:03 mdieterich Note Added: 0019069
2014-01-23 15:21 mdieterich Note Added: 0019111
2014-01-23 15:29 univ Note Added: 0019112
2014-01-23 16:49 toracat Note Added: 0019113
2014-01-23 17:09 mdieterich Note Added: 0019115
2014-01-25 22:28 mdieterich Note Added: 0019129
2014-01-26 13:45 toracat Note Added: 0019130
2014-01-28 13:30 ghammond-adfonic Note Added: 0019138
2014-01-31 19:02 kstange Note Added: 0019184
2014-01-31 19:04 kstange Note Added: 0019185
2014-01-31 19:14 mdieterich Note Added: 0019186
2014-02-04 08:17 save Note Added: 0019200
2014-02-04 09:16 save Note Added: 0019202
2014-02-04 16:20 AlanBartlett Note Added: 0019210
2014-02-06 00:27 MikeDVB Note Added: 0019232
2014-02-06 00:31 MikeDVB Note Added: 0019233
2014-02-12 07:31 medgen Note Added: 0019264
2014-03-13 14:27 univ Note Added: 0019496
2014-04-24 07:59 BonOnlines Note Added: 0019667
2014-04-25 02:07 twopoint71 Note Added: 0019674
2014-05-11 14:03 ppyy Note Added: 0019749
2014-07-04 10:40 tmueko Note Added: 0020176
2014-08-28 08:50 xander Note Added: 0020763


Copyright © 2000 - 2014 MantisBT Team
Powered by Mantis Bugtracker