| View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||||||
| ID | Project | Category | View Status | Date Submitted | Last Update | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0005225 | CentOS-5 | nfs-utils | public | 2011-11-04 23:13 | 2012-01-20 22:38 | ||||||||
| Reporter | sykosoft | ||||||||||||
| Priority | normal | Severity | major | Reproducibility | random | ||||||||
| Status | assigned | Resolution | open | ||||||||||
| Platform | x86_64 | OS | CentOS 5 | OS Version | 5.5-5.7 | ||||||||
| Product Version | 5.5 | ||||||||||||
| Target Version | Fixed in Version | ||||||||||||
| Summary | 0005225: NFS deadlocking mountpoint (XFS+NFS) | ||||||||||||
| Description | We've been having a problem across 4 different machines, each with different hardware and configs, but some commonality between them. In all cases, we are exporting a large (20-40TB) XFS mountpoint via NFS. Additionally, these are all supermicro chassis. That's basically where the commonality ends. Details as follows: Systems: 1. 16bay with 3ware RAID card. 2. 24 bay with adaptec RAID card. 3. 24 bay with LSI RAID card. 4. 24 bay with Areca RAID card. CentOS Versions: 1. 5.5 2. 5.6 3. 5.7 Kernels: 1. Anything up to 2.6.18-274.3.1.el5.centos.plus starting with 2.6.18-164.15.1.el5.centos.plus Several of the systems utilize DRBD for HA, some do not (ruling out DRBD). All systems utilize LVM. In all cases, NFS seems to cause an I/O deadlock on the exported XFS filesystem, where no I/O can proceed (even a simple ls). Other filesystems on the same RAID volume (and thus, RAID card) work normally during these events. nfsd is in D state, unkillable, and any processes that attempt I/O on the XFS mountpoint also go into D state. This does not (or at least, has not in 2 years) occurred on other mountpoints in the same configuration using either ext4 or reiserfs, only the XFS mountpoints. Below is an example of the dmesg output: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enab led SGI XFS Quota Management subsystem XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800cee54>] zone_statistics+0x3e/0x6d [<ffffffff8000f470>] __alloc_pages+0x78/0x308 [<ffffffff8003c659>] do_unlinkat+0xe8/0x141 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d116>] system_call+0x7e/0x83 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed Adding 10223608k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1 across :10223608k IA-32 Microcode Update Driver: v1.14a <tigran@veritas.com> microcode: CPU1 updated from revision 0xa07 to 0xa0b, date = 09282010 microcode: CPU3 updated from revision 0xa07 to 0xa0b, date = 09282010 microcode: CPU2 updated from revision 0xa07 to 0xa0b, date = 09282010 microcode: CPU0 updated from revision 0xa07 to 0xa0b, date = 09282010 Loading iSCSI transport class v2.0-871. 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com> All bugs added by David S. Miller <davem@redhat.com> libcxgbi:libcxgbi_init_module: tag itt 0x1fff, 13 bits, age 0xf, 4 bits. libcxgbi:ddp_setup_host_page_size: system PAGE 4096, ddp idx 0. Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010) iscsi: registered transport (cxgb3i) NET: Registered protocol family 10 lo: Disabled Privacy Extensions IPv6 over IPv4 tunneling driver cnic: Broadcom NetXtreme II CNIC Driver cnic v2.2.13 (Jan 31, 2011) Broadcom NetXtreme II iSCSI Driver bnx2i v2.6.2.3 (Dec 31, 2010) iscsi: registered transport (bnx2i) iscsi: registered transport (tcp) iscsi: registered transport (iser) iscsi: registered transport (be2iscsi) 8021q: adding VLAN 0 to HW filter on device eth0 ADDRCONF(NETDEV_UP): eth0: link is not ready ADDRCONF(NETDEV_UP): eth0.4: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ADDRCONF(NETDEV_CHANGE): eth0.4: link becomes ready eth0: no IPv6 routers present eth0.4: no IPv6 routers present eth0.6: no IPv6 routers present eth0.89: no IPv6 routers present XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800cee54>] zone_statistics+0x3e/0x6d [<ffffffff8000f470>] __alloc_pages+0x78/0x308 [<ffffffff800e8fd7>] sys_readlinkat+0x98/0xa9 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed Bluetooth: Core ver 2.10 NET: Registered protocol family 31 Bluetooth: HCI device and connection manager initialized Bluetooth: HCI socket layer initialized Bluetooth: L2CAP ver 2.8 Bluetooth: L2CAP socket layer initialized Bluetooth: HIDP (Human Interface Emulation) ver 1.1 Installing knfsd (copyright (C) 1996 okir@monad.swb.de). NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory NFSD: starting 90-second grace period nfsd: last server has exited nfsd: unexporting all filesystems XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800cee54>] zone_statistics+0x3e/0x6d [<ffffffff8000f470>] __alloc_pages+0x78/0x308 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff80009165>] __handle_mm_fault+0x9f6/0x103b [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002239a>] __up_read+0x19/0x7f [<ffffffff80067225>] do_page_fault+0x4cc/0x842 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800f05e1>] copy_mount_options+0xce/0x127 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff80009165>] __handle_mm_fault+0x9f6/0x103b [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002239a>] __up_read+0x19/0x7f [<ffffffff80067225>] do_page_fault+0x4cc/0x842 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800f05e1>] copy_mount_options+0xce/0x127 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed XFS mounting filesystem dm-2 Starting XFS recovery on filesystem: dm-2 (logdev: internal) XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1545 of file fs/xfs/xfs_alloc .c. Caller 0xffffffff8835ca71 Call Trace: [<ffffffff8835af37>] :xfs:xfs_free_ag_extent+0x19e/0x67e [<ffffffff8835ca71>] :xfs:xfs_free_extent+0xa9/0xc9 [<ffffffff8838d874>] :xfs:xlog_recover_process_efi+0x112/0x16c [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff8838ea53>] :xfs:xlog_recover_process_efis+0x4f/0x8d [<ffffffff8838eaa5>] :xfs:xlog_recover_finish+0x14/0x9e [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff883936c6>] :xfs:xfs_mountfs+0x47a/0x5ac [<ffffffff883a76c8>] :xfs:xfs_fs_fill_super+0x0/0x3dc [<ffffffff88393daa>] :xfs:xfs_mru_cache_create+0x113/0x143 [<ffffffff883a78cb>] :xfs:xfs_fs_fill_super+0x203/0x3dc [<ffffffff800e7401>] get_sb_bdev+0x10a/0x16c [<ffffffff800e6d9e>] vfs_kern_mount+0x93/0x11a [<ffffffff800e6e67>] do_kern_mount+0x36/0x4d [<ffffffff800f1865>] do_mount+0x6a9/0x719 [<ffffffff80009165>] __handle_mm_fault+0x9f6/0x103b [<ffffffff8000c816>] _atomic_dec_and_lock+0x39/0x57 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000a83a>] __link_path_walk+0xf91/0xfd1 [<ffffffff8002239a>] __up_read+0x19/0x7f [<ffffffff80067225>] do_page_fault+0x4cc/0x842 [<ffffffff8002cc44>] mntput_no_expire+0x19/0x89 [<ffffffff8000ebef>] link_path_walk+0xac/0xb8 [<ffffffff800f05e1>] copy_mount_options+0xce/0x127 [<ffffffff8004c0df>] sys_mount+0x8a/0xcd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Failed to recover EFIs on filesystem: dm-2 XFS: log mount finish failed XFS mounting filesystem dm-2 Ending clean XFS mount for filesystem: dm-2 XFS mounting filesystem dm-2 Ending clean XFS mount for filesystem: dm-2 XFS mounting filesystem dm-2 Ending clean XFS mount for filesystem: dm-2 NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory NFSD: starting 90-second grace period 3w-9xxx: scsi0: AEN: INFO (0x04:0x0055): Battery charging started:. 3w-9xxx: scsi0: AEN: INFO (0x04:0x0056): Battery charging completed:. INFO: task nfsd:5222 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81000101d7a0 0 5222 1 5223 5220 (L-TLB) ffff810145a0ba80 0000000000000046 ffff81022d844500 ffff810018c0e6c0 ffffc2001008f870 000000000000000a ffff8101f59287e0 ffff81022fc18100 00019acb295e7a3f 00000000000012cf ffff8101f59289c8 0000000300000000 Call Trace: [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff884133fc>] :8021q:vlan_dev_hwaccel_hard_start_xmit+0x7c/0x81 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5223 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81013e2a8860 0 5223 1 5224 5222 (L-TLB) ffff81014c1e5a80 0000000000000046 ffff81022d844500 ffff8101c7c18280 ffffc20010090d88 000000000000000a ffff8101f5928080 ffff81013e2a8860 00019acae6c0eea7 000000000000176a ffff8101f5928268 0000000300000000 Call Trace: [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff884133fc>] :8021q:vlan_dev_hwaccel_hard_start_xmit+0x7c/0x81 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5224 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81000101d7a0 0 5224 1 5225 5223 (L-TLB) ffff81013e2aba80 0000000000000046 ffff81022d844500 ffff810028155ec0 ffffc20010091710 000000000000000a ffff81014c1e7820 ffff81022fc18100 00019ac96e5f9884 00000000000016c9 ffff81014c1e7a08 0000000300000000 Call Trace: [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff884133fc>] :8021q:vlan_dev_hwaccel_hard_start_xmit+0x7c/0x81 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5225 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff810132856080 0 5225 1 5226 5224 (L-TLB) ffff8101ca6afa80 0000000000000046 ffff8101ca6afaa4 ffffffff8022f40c 0000000000000000 000000000000000a ffff81014c1e70c0 ffff810132856080 00019ac97a8f429d 00000000000010ef ffff81014c1e72a8 0000000300000000 Call Trace: [<ffffffff8022f40c>] sock_alloc_send_pskb+0x7d/0x282 [<ffffffff882490a7>] :e1000e:e1000_maybe_stop_tx+0x1d/0x61 [<ffffffff8824cad6>] :e1000e:e1000_xmit_frame+0x9be/0xa16 [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5226 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81000101d7a0 0 5226 1 5227 5225 (L-TLB) ffff81021a545730 0000000000000046 ffff81021a5456a0 ffffffff8008e20d 00019acae714a337 000000000000000a ffff81013e2a8860 ffff81022fc18100 00019acae7149cc2 00000000000029f7 ffff81013e2a8a48 0000000300000003 Call Trace: [<ffffffff8008e20d>] __activate_task+0x56/0x6d [<ffffffff80064a0b>] __down+0xc3/0xd8 [<ffffffff8008e7f7>] default_wake_function+0x0/0xe [<ffffffff800646c9>] __down_failed+0x35/0x3a [<ffffffff883a0f0b>] :xfs:.text.lock.xfs_buf+0xf/0x34 [<ffffffff8839f8de>] :xfs:_xfs_buf_find+0x154/0x1de [<ffffffff883a0134>] :xfs:xfs_buf_get_flags+0x52/0x137 [<ffffffff883a08fb>] :xfs:xfs_buf_read_flags+0x12/0x80 [<ffffffff88396f6f>] :xfs:xfs_trans_read_buf+0x47/0x2af [<ffffffff8837e3ba>] :xfs:xfs_ialloc_read_agi+0x6c/0x110 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff8837e593>] :xfs:xfs_imap_lookup+0x46/0x1a5 [<ffffffff880fc9a4>] :dm_mod:dm_request+0x11d/0x124 [<ffffffff8837e864>] :xfs:xfs_dilocate+0x172/0x1da [<ffffffff883845ed>] :xfs:xfs_imap+0x69/0x152 [<ffffffff882490a7>] :e1000e:e1000_maybe_stop_tx+0x1d/0x61 [<ffffffff88384e9b>] :xfs:xfs_itobp+0x47/0xe7 [<ffffffff8839d4fa>] :xfs:kmem_zone_alloc+0x5a/0xa7 [<ffffffff8838739a>] :xfs:xfs_iread+0x73/0x1e9 [<ffffffff80236b0b>] dev_hard_start_xmit+0x1b7/0x28a [<ffffffff8838291b>] :xfs:xfs_iget_core+0x2fc/0x56f [<ffffffff800259a2>] alloc_inode+0xeb/0x192 [<ffffffff88382c60>] :xfs:xfs_iget+0xd2/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5227 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81000101d7a0 0 5227 1 5228 5226 (L-TLB) ffff8101f9b297c0 0000000000000046 ffff8101f9b29730 ffffffff8008e20d 00019ac97a30a2ac 000000000000000a ffff81013e2a8100 ffff81022fc18100 00019ac97a30a2fa 0000000000005183 ffff81013e2a82e8 0000000300000003 Call Trace: [<ffffffff8008e20d>] __activate_task+0x56/0x6d [<ffffffff80064a0b>] __down+0xc3/0xd8 [<ffffffff8008e7f7>] default_wake_function+0x0/0xe [<ffffffff800646c9>] __down_failed+0x35/0x3a [<ffffffff883a0f0b>] :xfs:.text.lock.xfs_buf+0xf/0x34 [<ffffffff8839f8de>] :xfs:_xfs_buf_find+0x154/0x1de [<ffffffff883a0134>] :xfs:xfs_buf_get_flags+0x52/0x137 [<ffffffff883a08fb>] :xfs:xfs_buf_read_flags+0x12/0x80 [<ffffffff883970c2>] :xfs:xfs_trans_read_buf+0x19a/0x2af [<ffffffff8837e3ba>] :xfs:xfs_ialloc_read_agi+0x6c/0x110 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff8837f4e7>] :xfs:xfs_ialloc_ag_select+0x1de/0x271 [<ffffffff8001dd41>] tcp_recvmsg+0x956/0xa69 [<ffffffff8837f5ad>] :xfs:xfs_dialloc+0x33/0x80a [<ffffffff80031ac8>] sock_common_recvmsg+0x2d/0x43 [<ffffffff800303b8>] sock_recvmsg+0xfd/0x155 [<ffffffff88385e65>] :xfs:xfs_ialloc+0x5e/0x57f [<ffffffff800a2dfd>] autoremove_wake_function+0x0/0x2e [<ffffffff88397dbb>] :xfs:xfs_dir_ialloc+0x86/0x2bf [<ffffffff8838c643>] :xfs:xlog_grant_log_space+0x204/0x25c [<ffffffff8839a8b1>] :xfs:xfs_create+0x237/0x45c [<ffffffff8835fe57>] :xfs:xfs_attr_get+0x8e/0x9f [<ffffffff883a4300>] :xfs:xfs_vn_mknod+0x144/0x215 [<ffffffff8003a4a2>] vfs_create+0xe8/0x15e [<ffffffff886cbc1c>] :nfsd:nfsd_create_v3+0x2c9/0x42e [<ffffffff886d1652>] :nfsd:nfsd3_proc_create+0x130/0x141 [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5228 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff81000101d7a0 0 5228 1 5229 5227 (L-TLB) ffff81020a3f1a80 0000000000000046 ffff81020a3f1aa4 ffffffff8022f40c 0000000038743bd0 000000000000000a ffff810149da77a0 ffff81022fc18100 00019ac9c36dabfa 000000000000173e ffff810149da7988 0000000300000000 Call Trace: [<ffffffff8022f40c>] sock_alloc_send_pskb+0x7d/0x282 [<ffffffff882490a7>] :e1000e:e1000_maybe_stop_tx+0x1d/0x61 [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5229 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff8101cbc91860 0 5229 1 5230 5228 (L-TLB) ffff8101fc31ba80 0000000000000046 ffff81022d844500 ffff8101822d73c0 ffffc20010091198 000000000000000a ffff810149da7040 ffff8101cbc91860 00019aca2484a8f1 00000000000016a4 ffff810149da7228 0000000300000000 Call Trace: [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff800ef593>] inode_wait+0x9/0xd [<ffffffff800639fa>] __wait_on_bit+0x40/0x6e [<ffffffff800ef58a>] inode_wait+0x0/0xd [<ffffffff80063a94>] out_of_line_wait_on_bit+0x6c/0x78 [<ffffffff800a2e2b>] wake_bit_function+0x0/0x23 [<ffffffff884133fc>] :8021q:vlan_dev_hwaccel_hard_start_xmit+0x7c/0x81 [<ffffffff8003d98d>] ifind_fast+0x6e/0x83 [<ffffffff8002355d>] iget_locked+0x59/0x149 [<ffffffff88382bdd>] :xfs:xfs_iget+0x4f/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5230 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff810001015120 0 5230 1 5231 5229 (L-TLB) ffff8101f5fe5730 0000000000000046 0000000000000010 ffff81022e4fbf40 0000000000000000 000000000000000a ffff8101328567e0 ffff810107ba3080 00019acae769530d 00000000000002bd ffff8101328569c8 000000027dd77a38 Call Trace: [<ffffffff80064a0b>] __down+0xc3/0xd8 [<ffffffff8008e7f7>] default_wake_function+0x0/0xe [<ffffffff800646c9>] __down_failed+0x35/0x3a [<ffffffff883a0f0b>] :xfs:.text.lock.xfs_buf+0xf/0x34 [<ffffffff8839f8de>] :xfs:_xfs_buf_find+0x154/0x1de [<ffffffff883a0134>] :xfs:xfs_buf_get_flags+0x52/0x137 [<ffffffff883a08fb>] :xfs:xfs_buf_read_flags+0x12/0x80 [<ffffffff88396f6f>] :xfs:xfs_trans_read_buf+0x47/0x2af [<ffffffff8837e3ba>] :xfs:xfs_ialloc_read_agi+0x6c/0x110 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff8837e593>] :xfs:xfs_imap_lookup+0x46/0x1a5 [<ffffffff8837e864>] :xfs:xfs_dilocate+0x172/0x1da [<ffffffff80030be2>] release_sock+0x13/0xc1 [<ffffffff883845ed>] :xfs:xfs_imap+0x69/0x152 [<ffffffff80236b0b>] dev_hard_start_xmit+0x1b7/0x28a [<ffffffff88384e9b>] :xfs:xfs_itobp+0x47/0xe7 [<ffffffff8839d4fa>] :xfs:kmem_zone_alloc+0x5a/0xa7 [<ffffffff8838739a>] :xfs:xfs_iread+0x73/0x1e9 [<ffffffff8838291b>] :xfs:xfs_iget_core+0x2fc/0x56f [<ffffffff800259a2>] alloc_inode+0xeb/0x192 [<ffffffff88382c60>] :xfs:xfs_iget+0xd2/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task nfsd:5231 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nfsd D ffff8101cbc91100 0 5231 1 5232 5230 (L-TLB) ffff810135043730 0000000000000046 ffff8101350436a0 ffffffff8008e20d 00019acae75f7d66 000000000000000a ffff810132856080 ffff8101cbc91100 00019acae75f834b 0000000000001c9c ffff810132856268 0000000300000003 Call Trace: [<ffffffff8008e20d>] __activate_task+0x56/0x6d [<ffffffff8014f524>] deadline_set_request+0x38/0x6e [<ffffffff80064a0b>] __down+0xc3/0xd8 [<ffffffff8008e7f7>] default_wake_function+0x0/0xe [<ffffffff800646c9>] __down_failed+0x35/0x3a [<ffffffff883a0f0b>] :xfs:.text.lock.xfs_buf+0xf/0x34 [<ffffffff8839f8de>] :xfs:_xfs_buf_find+0x154/0x1de [<ffffffff883a0134>] :xfs:xfs_buf_get_flags+0x52/0x137 [<ffffffff883a08fb>] :xfs:xfs_buf_read_flags+0x12/0x80 [<ffffffff88396f6f>] :xfs:xfs_trans_read_buf+0x47/0x2af [<ffffffff8837e3ba>] :xfs:xfs_ialloc_read_agi+0x6c/0x110 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff8837e593>] :xfs:xfs_imap_lookup+0x46/0x1a5 [<ffffffff8002239a>] __up_read+0x19/0x7f [<ffffffff80154aa7>] __next_cpu+0x19/0x28 [<ffffffff8837e864>] :xfs:xfs_dilocate+0x172/0x1da [<ffffffff883845ed>] :xfs:xfs_imap+0x69/0x152 [<ffffffff8005bd0c>] cache_alloc_refill+0x88/0x188 [<ffffffff88384e9b>] :xfs:xfs_itobp+0x47/0xe7 [<ffffffff8839d4fa>] :xfs:kmem_zone_alloc+0x5a/0xa7 [<ffffffff8838739a>] :xfs:xfs_iread+0x73/0x1e9 [<ffffffff80236b0b>] dev_hard_start_xmit+0x1b7/0x28a [<ffffffff8838291b>] :xfs:xfs_iget_core+0x2fc/0x56f [<ffffffff800259a2>] alloc_inode+0xeb/0x192 [<ffffffff88382c60>] :xfs:xfs_iget+0xd2/0x17a [<ffffffff883a11a0>] :xfs:xfs_fs_get_dentry+0x3e/0xae [<ffffffff886bb36d>] :exportfs:find_exported_dentry+0x43/0x486 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff886cc80b>] :nfsd:exp_get_by_name+0x5b/0x71 [<ffffffff886ccdfa>] :nfsd:exp_find_key+0x89/0x9c [<ffffffff80046c44>] try_to_wake_up+0x472/0x484 [<ffffffff886c8739>] :nfsd:nfsd_acceptable+0x0/0xdc [<ffffffff883a1046>] :xfs:xfs_fs_decode_fh+0xce/0xd8 [<ffffffff886c8ab1>] :nfsd:fh_verify+0x29c/0x4cf [<ffffffff886d0760>] :nfsd:nfsd3_proc_getattr+0x8a/0xbe [<ffffffff886c61db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 [<ffffffff885fa80d>] :sunrpc:svc_process+0x44c/0x713 [<ffffffff80064614>] __down_read+0x12/0x92 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6725>] :nfsd:nfsd+0x1a5/0x2c8 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff886c6580>] :nfsd:nfsd+0x0/0x2c8 [<ffffffff8005dfa7>] child_rip+0x0/0x11 When it happens on another system, I will paste the output in this bug. | ||||||||||||
| Steps To Reproduce | Random, unknown cause. | ||||||||||||
| Tags | No tags attached. | ||||||||||||
| Attached Files |
| ||||||||||||
 Relationships |
|
 Relationships |
| Notes |
|
|
sykosoft (reporter) 2011-11-04 23:14 |
Additional information: This never occurs on the same mountpoint when it is not exported via NFS. Michael |
|
tru (administrator) 2011-11-04 23:47 |
does that also happen without the centosplus kernels with the stock xfs kernel module? (not the deprecated kmod-xfs one)? [tru@diane ~]$ modinfo xfs filename: /lib/modules/2.6.18-274.7.1.el5/kernel/fs/xfs/xfs.ko license: GPL description: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled author: Silicon Graphics, Inc. srcversion: 4A41C05CBD42F5525F11CBD depends: vermagic: 2.6.18-274.7.1.el5 SMP mod_unload gcc-4.1 module_sig: 883f3504ea08a82e35359b9fcadd1511227309d1790b8833b57324ca8fc298a79b3bab28830e44d0a083d0fdeab95aab86814aeeae45b077a922b72 |
|
tru (administrator) 2011-11-04 23:52 |
your logs are showing mounting XFS issue, not related to the not yet started NFS exports. Are you sure that NFS + XFS is culprit? not just XFS (and hardware?) ? df -hTlP /pvs/vgs/lvs could also be usefull xfs_check on "damaged" partitions? |
|
sykosoft (reporter) 2011-11-04 23:53 |
[root@avl-filer05 ~]# uname -a Linux avl-filer05.OBFUSCATED 2.6.18-274.7.1.el5.centos.plus #1 SMP Thu Oct 20 19:28:06 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@avl-filer05 ~]# modinfo xfs filename: /lib/modules/2.6.18-274.7.1.el5.centos.plus/kernel/fs/xfs/xfs.ko license: GPL description: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled author: Silicon Graphics, Inc. srcversion: 4A41C05CBD42F5525F11CBD depends: vermagic: 2.6.18-274.7.1.el5.centos.plus SMP mod_unload gcc-4.1 module_sig: 883f3504ea0b77754c68be09185409a112df5b0a0ba9b83ac1fc2e9a482351f44113a1db41cf009f7c8b9880dbda8963a59652e19c7177a25d3b55c3 |
|
sykosoft (reporter) 2011-11-04 23:54 |
[root@backup ~]# uname -a Linux backup.OBFUSCATED 2.6.18-274.3.1.el5.centos.plus #1 SMP Wed Sep 7 05:38:58 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@backup ~]# modinfo xfs filename: /lib/modules/2.6.18-274.3.1.el5.centos.plus/kernel/fs/xfs/xfs.ko license: GPL description: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled author: Silicon Graphics, Inc. srcversion: 4A41C05CBD42F5525F11CBD depends: vermagic: 2.6.18-274.3.1.el5.centos.plus SMP mod_unload gcc-4.1 module_sig: 883f3504e67445326061d8b7f693c112b78e0a0ab143d9b4bfab1dbb7f66765c573aa9a678c4509d15b05ef444594fd055cd3436653488403a4e7ec3 |
|
tru (administrator) 2011-11-04 23:54 |
see also: http://oss.sgi.com/archives/xfs/2011-04/msg00125.html |
|
sykosoft (reporter) 2011-11-04 23:57 |
In answer to your question: 1. 4 different machines, with different cpus, raid cards, and even hard drive models all exhibiting the same behavior 2. Does not occur unless nfs is exporting the xfs mountpoint 3. In the instance of the traces above, those came from a machine that we ran an xfs_repair on less than 1 week ago (some issues, all fixed, still occurring). Michael |
|
tru (administrator) 2011-11-05 00:09 |
-> XFS: log mount finish failed that not good :( but that could be related to the hard reboot (BBU on your hardware raid?). |
|
sykosoft (reporter) 2011-11-05 00:11 |
Agreed that it wasn't good, however, that's why we did the xfs_repair. Those messages do not appear on the other machines at all, nor are there any indications of XFS filesystem problems on the other machines. Michael |
|
sykosoft (reporter) 2011-11-28 22:03 |
We believe this is caused by mounting via UDP vs TCP. We have modified the mount options of a few clients on a few of these systems, and have not had the issue any longer (vs at least once per week). The network is a fully gigabit network, and we had a mix of some UDP and TCP NFS clients. For each NFS server that we were testing this fix on, we have modified all connecting clients to TCP instead of UDP . No further problems have been noted, though it continues to occur on the UDP mounted servers. Perhaps the problem is related to mmap-ing over NFS mounted via UDP. I hope to change the rest of the clients, and show conclusively that it does not occur when TCP mounted but only when UDP mounted. Michael |
|
tru (administrator) 2011-11-28 22:18 |
thanks for the feedback! |
|
hjmangalam (reporter) 2012-01-20 19:26 |
this just happened to us and was resolved by increasing the number of nfsd's running. Cour cluster head/storage node (64b CentOS 5.7, Areca 16port controller, quad opteron, 16GBRAM, kernel 2.6.18-274.17.1.el5 #1 SMP) was intermittantly locking up with the same error messages and behavior reported above. Increasing the default 8 NFSDs to 256 has apparently solved the problem . We're starting to get lots of very large IO hits and when that happens I think the low # of NFSDs saturate and block. A larger number of them allows the Q to grow long enough to survive the IO storm. Just after I set this, we had another such IO storm where the 1m load went to 170 and the node kept working. this is set in /etc/sysconfig/nfs where the value to increase is: RPCNFSDCOUNT I set it to 256; another sysadmin has his set to 2048 (which seems excessive, but he has more RAM on his machine). |
|
sykosoft (reporter) 2012-01-20 21:41 |
As a note, depending on the load on your NFS server, and the underlying hardware, we keep rpcnfsdcount low at times, to prevent i/o contention on the underlying devices. |
|
sykosoft (reporter) 2012-01-20 21:42 |
Also, for completeness sake, were you mounted TCP or UDP? Michael |
|
hjmangalam (reporter) 2012-01-20 22:38 |
Re:rpcnfsdcount being kept low, how does this prevent i/o contention? I haven't noticed any problems (performance or otherwise) since increasing the number of nfsd's and as noted, another storage server is running about 10x this number. In our case, it's a matter of staying alive and allowing IO, versus marginally slower perf. We are mounted TCP only, except that a few nodes report mountprot=UDP; but the rest of the transport is TCP. |
| Notes |
| Issue History |
|||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2011-11-04 23:13 | sykosoft | New Issue | |
| 2011-11-04 23:14 | sykosoft | Note Added: 0013713 | |
| 2011-11-04 23:47 | tru | Note Added: 0013714 | |
| 2011-11-04 23:52 | tru | Note Added: 0013715 | |
| 2011-11-04 23:52 | tru | Status | new => feedback |
| 2011-11-04 23:53 | sykosoft | Note Added: 0013716 | |
| 2011-11-04 23:53 | sykosoft | Status | feedback => assigned |
| 2011-11-04 23:54 | sykosoft | Note Added: 0013717 | |
| 2011-11-04 23:54 | tru | Note Added: 0013718 | |
| 2011-11-04 23:57 | sykosoft | Note Added: 0013719 | |
| 2011-11-05 00:09 | tru | Note Added: 0013720 | |
| 2011-11-05 00:11 | sykosoft | Note Added: 0013721 | |
| 2011-11-28 22:03 | sykosoft | Note Added: 0013842 | |
| 2011-11-28 22:18 | tru | Note Added: 0013843 | |
| 2012-01-20 19:26 | hjmangalam | Note Added: 0014280 | |
| 2012-01-20 21:41 | sykosoft | Note Added: 0014281 | |
| 2012-01-20 21:42 | sykosoft | Note Added: 0014282 | |
| 2012-01-20 22:38 | hjmangalam | Note Added: 0014284 | |
| Issue History |
