View Issue Details

IDProjectCategoryView StatusLast Update
0014175CentOS-6kernelpublic2019-02-27 22:45
Reporterliu13551700260 
PriorityurgentSeveritycrashReproducibilitysometimes
Status newResolutionopen 
Product Version6.9 
Target VersionFixed in Version 
Summary0014175: kernel: Pid: 0, comm: swapper Not tainted 2.6.32-642.15.1.el6.x86_64 #1
DescriptionNov 23 18:52:12 ip-192-168-1-22 kernel: __ratelimit: 171 callbacks suppressed
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 102.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 103.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 104.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 105.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 106.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 107.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 108.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 109.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 110.
Nov 23 18:52:12 ip-192-168-1-22 kernel: net eth0: Bad rx response id 111.
Nov 23 18:52:12 ip-192-168-1-22 kernel: BUG: scheduling while atomic: swapper/0/0x10000100
Nov 23 18:52:12 ip-192-168-1-22 kernel: Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_multiport iptable_filter ip_tables ipv6 xen_netfront i2c_piix4 i2c_core ext4 jbd2 mbcache xen_blkfront pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Nov 23 18:52:12 ip-192-168-1-22 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-642.15.1.el6.x86_64 #1
Nov 23 18:52:12 ip-192-168-1-22 kernel: Call Trace:
Nov 23 18:52:12 ip-192-168-1-22 kernel: <IRQ> [<ffffffff810682a4>] ? __schedule_bug+0x44/0x50
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8154925c>] ? schedule+0xa4c/0xb70
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81074cf4>] ? enqueue_task_fair+0x64/0x100
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010eba7>] ? xennet_alloc_rx_buffers+0x367/0x380 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8107405a>] ? __cond_resched+0x2a/0x40
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010eba7>] ? xennet_alloc_rx_buffers+0x367/0x380 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81549650>] ? _cond_resched+0x30/0x40
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100fd96>] ? is_valid_bugaddr+0x16/0x40
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8129943f>] ? report_bug+0x1f/0xc0
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8101104f>] ? die+0x7f/0x90
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8154cb94>] ? do_trap+0xc4/0x160
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100cd95>] ? do_invalid_op+0x95/0xb0
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010eba7>] ? xennet_alloc_rx_buffers+0x367/0x380 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff814b533d>] ? ip_local_deliver_finish+0xdd/0x2d0
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100c01b>] ? invalid_op+0x1b/0x20
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010eba7>] ? xennet_alloc_rx_buffers+0x367/0x380 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010f712>] ? xennet_poll+0x852/0xd50 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffffa010f49e>] ? xennet_poll+0x5de/0xd50 [xen_netfront]
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81007bc1>] ? xen_clocksource_read+0x21/0x30
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8147f473>] ? net_rx_action+0x103/0x300
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81085275>] ? __do_softirq+0xe5/0x230
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff810f6208>] ? handle_edge_irq+0x98/0x180
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100c38c>] ? call_softirq+0x1c/0x30
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100fca5>] ? do_softirq+0x65/0xa0
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81085105>] ? irq_exit+0x85/0x90
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8133691c>] ? xen_evtchn_do_upcall+0x3c/0x50
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8100c4b3>] ? xen_hvm_callback_vector+0x13/0x20
Nov 23 18:52:12 ip-192-168-1-22 kernel: <EOI> [<ffffffff8104601b>] ? native_safe_halt+0xb/0x10
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8101696d>] ? default_idle+0x4d/0xb0
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81009fe6>] ? cpu_idle+0xb6/0x110
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff8152f40a>] ? rest_init+0x7a/0x80
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81c3b127>] ? start_kernel+0x429/0x436
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81c3a33a>] ? x86_64_start_reservations+0x125/0x129
Nov 23 18:52:12 ip-192-168-1-22 kernel: [<ffffffff81c3a453>] ? x86_64_start_kernel+0x115/0x124
Nov 23 18:54:16 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:54:22 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:54:28 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:54:36 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:54:51 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:55:02 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 18:55:13 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
Nov 23 19:02:28 ip-192-168-1-22 kernel: INFO: task crond:26967 blocked for more than 120 seconds.
Nov 23 19:02:28 ip-192-168-1-22 kernel: Not tainted 2.6.32-642.15.1.el6.x86_64 #1
Nov 23 19:02:28 ip-192-168-1-22 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 19:02:28 ip-192-168-1-22 kernel: crond D 0000000000000002 0 26967 971 0x00000000
Nov 23 19:02:28 ip-192-168-1-22 kernel: ffff88032d433d18 0000000000000082 ffff88032d433ce0 ffff88032d433cdc
Nov 23 19:02:28 ip-192-168-1-22 kernel: 0000000000010246 ffff88040fc245e0 002e75dff4bc6ac8 ffff880028216ec0
Nov 23 19:02:28 ip-192-168-1-22 kernel: 0000000000000300 000000040b758788 ffff88040822e5f8 ffff88032d433fd8
Nov 23 19:02:28 ip-192-168-1-22 kernel: Call Trace:
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff81549b25>] schedule_timeout+0x215/0x2e0
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff81341515>] ? extract_entropy+0x125/0x1f0
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff81549783>] wait_for_common+0x123/0x180
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff8106c500>] ? default_wake_function+0x0/0x20
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff815498bd>] wait_for_completion+0x1d/0x20
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff810a2118>] synchronize_sched+0x58/0x60
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff810a20a0>] ? wakeme_after_rcu+0x0/0x20
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff812370dc>] install_session_keyring_to_cred+0x6c/0xd0
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff8123ae06>] ? security_prepare_creds+0x16/0x20
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff8123727e>] join_session_keyring+0x13e/0x170
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff81235da8>] keyctl_join_session_keyring+0x38/0x70
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff81236a50>] sys_keyctl+0x180/0x1a0
Nov 23 19:02:28 ip-192-168-1-22 kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
Nov 23 19:02:29 ip-192-168-1-22 dhclient[883]: DHCPREQUEST on eth0 to 192.168.1.1 port 67 (xid=0x3d6c3642)
TagsNo tags attached.

Activities

tru

tru

2017-11-24 07:56

administrator   ~0030644

Current CentOS-6 kernel is kernel-2.6.32-696.16.1.el6.x86_64... Please upgrade your system, maybe that is fixed in the latest version.
liu13551700260

liu13551700260

2017-11-24 08:03

reporter   ~0030645

what's the reason, please?thanks
griffenater

griffenater

2019-02-27 22:45

reporter   ~0033912

Noticed issue after users reported unable to login.

Anyone confirmed upgrade resolves issue?

Messages from kernel on node with issue. All messages occurred with same timestamp

nfsd: page allocation failure. order:3, mode:0x20
Pid: 3639, comm: nfsd Not tainted 2.6.32-696.18.7.el6.x86_64 #1
Call Trace:
[<ffffffff8113efdc>] ? __alloc_pages_nodemask+0x7dc/0x950
[<ffffffff8117fa82>] ? kmem_getpages+0x62/0x170
[<ffffffff8118069a>] ? fallback_alloc+0x1ba/0x270
[<ffffffff811800ef>] ? cache_grow+0x2cf/0x320
[<ffffffff81180419>] ? ____cache_alloc_node+0x99/0x160
[<ffffffff81470af2>] ? pskb_expand_head+0x62/0x280
[<ffffffff81181069>] ? __kmalloc+0x199/0x230
[<ffffffff81470af2>] ? pskb_expand_head+0x62/0x280
[<ffffffff814713da>] ? __pskb_pull_tail+0x2aa/0x360
[<ffffffffa0356ecd>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa01f1a1f>] ? bnx2x_start_xmit+0xa6f/0x17d0 [bnx2x]
[<ffffffff81480e09>] ? harmonize_features+0x29/0x70
[<ffffffff814814cc>] ? dev_hard_start_xmit+0x21c/0x490
[<ffffffff81478664>] ? skb_tx_hash+0x14/0x20
[<ffffffff8149eeba>] ? sch_direct_xmit+0x15a/0x1c0
[<ffffffff81481a28>] ? dev_queue_xmit+0x268/0x360
[<ffffffffa055a821>] ? bond_dev_queue_xmit+0x31/0xa0 [bonding]
[<ffffffffa055a8ef>] ? bond_xmit_slave_id+0x5f/0xd0 [bonding]
[<ffffffffa055c505>] ? bond_start_xmit+0x195/0x380 [bonding]
[<ffffffff814814cc>] ? dev_hard_start_xmit+0x21c/0x490
[<ffffffff814acb19>] ? nf_iterate+0x69/0xb0
[<ffffffff814819b0>] ? dev_queue_xmit+0x1f0/0x360
[<ffffffff814bde04>] ? ip_finish_output+0x184/0x360
[<ffffffff814be098>] ? ip_output+0xb8/0xc0
[<ffffffff814bd2df>] ? __ip_local_out+0x9f/0xb0
[<ffffffff814bd315>] ? ip_local_out+0x25/0x30
[<ffffffff814bd820>] ? ip_queue_xmit+0x190/0x420
[<ffffffff814d3262>] ? tcp_transmit_skb+0x4d2/0x910
[<ffffffff814d587c>] ? tcp_write_xmit+0x1dc/0xa90
[<ffffffff814d6460>] ? __tcp_push_pending_frames+0x30/0xe0
[<ffffffff814cd78e>] ? tcp_data_snd_check+0x2e/0x100
[<ffffffff814d161d>] ? tcp_rcv_established+0x3fd/0x800
[<ffffffff814d9c03>] ? tcp_v4_do_rcv+0x2e3/0x490
[<ffffffff814d3262>] ? tcp_transmit_skb+0x4d2/0x910
[<ffffffff8146a765>] ? release_sock+0x65/0x110
[<ffffffff814c86d1>] ? tcp_recvmsg+0x8e1/0x10f0
[<ffffffff814e9ffa>] ? inet_recvmsg+0x5a/0x90
[<ffffffff81151d99>] ? zone_statistics+0x99/0xc0
[<ffffffff81468fc3>] ? sock_recvmsg+0x133/0x160
[<ffffffff810a71a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8108f84c>] ? lock_timer_base+0x3c/0x70
[<ffffffff81469034>] ? kernel_recvmsg+0x44/0x60
[<ffffffffa05bbee5>] ? svc_recvfrom+0x65/0xa0 [sunrpc]
[<ffffffffa05bcf47>] ? svc_tcp_recvfrom+0x347/0x710 [sunrpc]
[<ffffffffa05c9d7b>] ? svc_recv+0x80b/0x840 [sunrpc]
[<ffffffff8106cb30>] ? default_wake_function+0x0/0x20
[<ffffffffa06dfc65>] ? nfsd+0xa5/0x160 [nfsd]
[<ffffffffa06dfbc0>] ? nfsd+0x0/0x160 [nfsd]
[<ffffffff810a6d0e>] ? kthread+0x9e/0xc0
[<ffffffff81557afa>] ? child_rip+0xa/0x20
[<ffffffff810a6c70>] ? kthread+0x0/0xc0
[<ffffffff81557af0>] ? child_rip+0x0/0x20

Issue History

Date Modified Username Field Change
2017-11-24 05:30 liu13551700260 New Issue
2017-11-24 07:56 tru Note Added: 0030644
2017-11-24 08:03 liu13551700260 Note Added: 0030645
2019-02-27 22:45 griffenater Note Added: 0033912