2017-03-23 18:18 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0012487CentOS-7kernelpublic2017-03-15 17:35
Reporterrellik.hsu 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusnewResolutionopen 
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012487: fs/xfs/xfs_aops.c:1244 xfs_vm_releasepage+0xcb/0x100 [xfs]
DescriptionI running a server on Amazon EC2 and find this on dmesg


[275403.907670] ------------[ cut here ]------------
[275403.913056] WARNING: at fs/xfs/xfs_aops.c:1244 xfs_vm_releasepage+0xcb/0x100 [xfs]()
[275403.919514] Modules linked in: binfmt_misc xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop isofs raid10 intel_powerclamp crc32_pclmul ppdev ghash_clmulni_intel cirrus aesni_intel ttm lrw gf128mul glue_helper ablk_helper drm_kms_helper cryptd syscopyarea sysfillrect sysimgblt fb_sys_fops drm parport_pc parport i2c_piix4 i2c_core pcspkr nfsd auth_rpcgss nfs_acl lockd grace ip_tables xfs libcrc32c ata_generic pata_acpi xen_blkfront crct10dif_pclmul ata_piix crct10dif_common crc32c_intel libata ixgbevf(OE) serio_raw floppy fjes sunrpc dm_mirror dm_region_hash dm_log dm_mod
[275403.982974] CPU: 22 PID: 220 Comm: kswapd0 Tainted: G OE ------------ 3.10.0-514.2.2.el7.x86_64 #1
[275403.990509] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016
[275403.995272] 0000000000000000 00000000ae26c730 ffff881e15db7aa0 ffffffff816861cc
[275404.000963] ffff881e15db7ad8 ffffffff81085940 ffffea0001238960 ffffea0001238940
[275404.006700] ffff883c898923b8 ffff881e15db7da0 ffffea0001238940 ffff881e15db7ae8
[275404.012709] Call Trace:
[275404.015457] [<ffffffff816861cc>] dump_stack+0x19/0x1b
[275404.019309] [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
[275404.023410] [<ffffffff81085a8a>] warn_slowpath_null+0x1a/0x20
[275404.039151] [<ffffffffa026e56b>] xfs_vm_releasepage+0xcb/0x100 [xfs]
[275404.045714] [<ffffffff811805c2>] try_to_release_page+0x32/0x50
[275404.049798] [<ffffffff81196546>] shrink_active_list+0x3d6/0x3e0
[275404.054074] [<ffffffff81196941>] shrink_lruvec+0x3f1/0x770
[275404.057891] [<ffffffff81196d36>] shrink_zone+0x76/0x1a0
[275404.061645] [<ffffffff81197fdc>] balance_pgdat+0x48c/0x5e0
[275404.065519] [<ffffffff811982a3>] kswapd+0x173/0x450
[275404.069014] [<ffffffff810b1720>] ? wake_up_atomic_t+0x30/0x30
[275404.072940] [<ffffffff81198130>] ? balance_pgdat+0x5e0/0x5e0
[275404.076802] [<ffffffff810b064f>] kthread+0xcf/0xe0
[275404.080240] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[275404.084446] [<ffffffff81696718>] ret_from_fork+0x58/0x90
[275404.088210] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[275404.092427] ---[ end trace 7eee283d93ab9e18 ]---
Additional Informationmdadm-3.4-14.el7.x86_64

kernel option in /etc/default/grub : GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto console=ttyS0,115200 net.ifnames=0 processor.max_cstate=0 intel_idle.max_cstate=0 transparent_hugepage=never clocksource=tsc elevator=noop"

TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0028398

TuxHandwerker (reporter)

WARNING: at fs/xfs/xfs_aops.c:1246 xfs_vm_releasepage+0xe7/0x100 [xfs]()
Same here, running many rsync's parallel.

~0028405

dragle (reporter)

Ditto here, got this message last night. I also frequently run rsyncs on that machine, but none were running at the time of the message.

Also CentOS 7.3.

:WARNING: at fs/xfs/xfs_aops.c:1244 xfs_vm_releasepage+0xcb/0x100 [xfs]()
:Modules linked in: uas usb_storage ext4 mbcache jbd2 dm_crypt drbg ansi_cprng iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bonding xt_conntrack ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle ip6table_filter ip6_tables iptable_filter nf_conntrack_ftp intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mxm_wmi ipmi_devintf iTCO_wdt iTCO_vendor_support pcspkr mei_me mei dcdbas shpchp sb_edac sg edac_core lpc_ich acpi_power_meter ipmi_si ipmi_msghandler wmi binfmt_misc nf_conntrack_proto_gre nf_conntrack ip_gre ip_tunnel
: gre ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_pclmul crct10dif_common mgag200 crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm igb ahci tg3 dca libahci ptp megaraid_sas libata pps_core i2c_algo_bit i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
:CPU: 4 PID: 185 Comm: kswapd0 Not tainted 3.10.0-514.2.2.el7.x86_64 #1
:Hardware name: Dell Inc. PowerEdge R530/0HFG24, BIOS 2.2.5 09/08/2016
: 0000000000000000 000000006656b8c1 ffff88085cea7aa0 ffffffff816861cc
: ffff88085cea7ad8 ffffffff81085940 ffffea00019c68e0 ffffea00019c68c0
: ffff88105cf9ce38 ffff88085cea7da0 ffffea00019c68c0 ffff88085cea7ae8
:Call Trace:
: [<ffffffff816861cc>] dump_stack+0x19/0x1b
: [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
: [<ffffffff81085a8a>] warn_slowpath_null+0x1a/0x20
: [<ffffffffa020356b>] xfs_vm_releasepage+0xcb/0x100 [xfs]
: [<ffffffff811805c2>] try_to_release_page+0x32/0x50
: [<ffffffff81196546>] shrink_active_list+0x3d6/0x3e0
: [<ffffffff81196941>] shrink_lruvec+0x3f1/0x770
: [<ffffffff81196d36>] shrink_zone+0x76/0x1a0
: [<ffffffff81197fdc>] balance_pgdat+0x48c/0x5e0
: [<ffffffff811982a3>] kswapd+0x173/0x450
: [<ffffffff810b1720>] ? wake_up_atomic_t+0x30/0x30
: [<ffffffff81198130>] ? balance_pgdat+0x5e0/0x5e0
: [<ffffffff810b064f>] kthread+0xcf/0xe0
: [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
: [<ffffffff81696718>] ret_from_fork+0x58/0x90
: [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140

~0028417

cullmann (reporter)

Same here, with normal or here with "plus" kernel (on a NFS server with high load):

[62937.680213] ------------[ cut here ]------------
[62937.680283] WARNING: at fs/xfs/xfs_aops.c:1244 xfs_vm_releasepage+0xcb/0x100 [xfs]()
[62937.680286] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osst st bonding nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log vfat fat intel_powerclamp iTCO_wdt coretemp intel_rapl iosf_mbi iTCO_vendor_support ipmi_devintf kvm_intel kvm irqbypass crc32_pclmul dm_mod ghash_clmulni_intel aesni_intel lrw gf128mul
[62937.680333] glue_helper ablk_helper cryptd pcspkr sb_edac ses enclosure edac_core i2c_i801 sg lpc_ich mei_me mei ioatdma shpchp wmi ipmi_ssif ipmi_si ipmi_msghandler acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper mpt2sas syscopyarea sysfillrect sysimgblt fb_sys_fops raid_class ttm scsi_transport_sas igb ahci drm libahci ptp uas pps_core usb_storage dca libata i2c_algo_bit megaraid_sas i2c_core fjes
[62937.680401] CPU: 3 PID: 145 Comm: kswapd0 Tainted: G I ------------ 3.10.0-514.2.2.el7.centos.plus.x86_64 #1
[62937.680404] Hardware name: Supermicro X10DRC/X10DRi-LN4+, BIOS 1.0 07/14/2014
[62937.680407] 0000000000000000 00000000305f5189 ffff88103ddffaa0 ffffffff816953ec
[62937.680414] ffff88103ddffad8 ffffffff81085940 ffffea0001a11fe0 ffffea0001a11fc0
[62937.680419] ffff8800604a78b8 ffff88103ddffda0 ffffea0001a11fc0 ffff88103ddffae8
[62937.680425] Call Trace:
[62937.680436] [<ffffffff816953ec>] dump_stack+0x19/0x1b
[62937.680444] [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
[62937.680450] [<ffffffff81085a8a>] warn_slowpath_null+0x1a/0x20
[62937.680499] [<ffffffffa043756b>] xfs_vm_releasepage+0xcb/0x100 [xfs]
[62937.680506] [<ffffffff811805c2>] try_to_release_page+0x32/0x50
[62937.680509] [<ffffffff81196546>] shrink_active_list+0x3d6/0x3e0
[62937.680512] [<ffffffff81196941>] shrink_lruvec+0x3f1/0x770
[62937.680515] [<ffffffff81196d36>] shrink_zone+0x76/0x1a0
[62937.680518] [<ffffffff81197fdc>] balance_pgdat+0x48c/0x5e0
[62937.680521] [<ffffffff811982a3>] kswapd+0x173/0x450
[62937.680526] [<ffffffff810b1720>] ? wake_up_atomic_t+0x30/0x30
[62937.680528] [<ffffffff81198130>] ? balance_pgdat+0x5e0/0x5e0
[62937.680531] [<ffffffff810b064f>] kthread+0xcf/0xe0
[62937.680534] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[62937.680538] [<ffffffff816a5958>] ret_from_fork+0x58/0x90
[62937.680541] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[62937.680543] ---[ end trace 96d62c2561a24f7d ]---

~0028461

dragle (reporter)

Got another one last night.

If it's meaningful to anyone else, we have both a remote mounted iSCSI volume (ext4) on this machine, as well as a USB attached external drive bay (these are xfs). When this error hit last night we were writing to the USB drive bay, but not to the iSCSI drive (but the iSCSI was mounted/logged in; we do write to it throughout the day, just not at that particular time). With my first post on 1/18 we were not writing to either of the external drives.

~0028619

alienth (reporter)

Also started getting this recently on Cent7. Not in EC2, this is bare metal hardware. Also not doing any fancy RAID or mdadm setup, and the disk is direct attached.

$ uname -a
Linux ny-tsdb03.ds.stackexchange.com 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux


Feb 20 17:58:40 ny-tsdb03 kernel: Call Trace:
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff816862ac>] dump_stack+0x19/0x1b
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81085a8a>] warn_slowpath_null+0x1a/0x20
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffffa042756b>] xfs_vm_releasepage+0xcb/0x100 [xfs]
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff811805b2>] try_to_release_page+0x32/0x50
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81196536>] shrink_active_list+0x3d6/0x3e0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81196931>] shrink_lruvec+0x3f1/0x770
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81196d26>] shrink_zone+0x76/0x1a0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81197fcc>] balance_pgdat+0x48c/0x5e0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81198293>] kswapd+0x173/0x450
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff810b1720>] ? wake_up_atomic_t+0x30/0x30
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81198120>] ? balance_pgdat+0x5e0/0x5e0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff810b064f>] kthread+0xcf/0xe0
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff81696818>] ret_from_fork+0x58/0x90
Feb 20 17:58:40 ny-tsdb03 kernel: [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
Feb 20 17:58:40 ny-tsdb03 kernel: ---[ end trace 38bce61035257926 ]---

~0028754

ascherbakov-ya (reporter)

The same message began to receive on 15 new servers under load, all the same configuration with configured hardware RAID5 on Megaraid. A fragment of the log with one of the servers:

# uname -a
Linux es-node24.avp.ru 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# uptime
17:57:21 up 14 days, 3:40, 1 user, load average: 0,02, 0,12, 0,31

[1144590.651414] WARNING: at fs/xfs/xfs_aops.c:1244 xfs_vm_releasepage+0xcb/0x100 [xfs]()
[1144590.651418] Modules linked in: rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm iw_cm ib_ipoib ib_cm mlx4_ib ib_core mlx4_en mlx4_core devlink mpt3sas mpt2sas raid_class mptctl mptbase 8021q garp mrp ipt_REJECT nf_reject_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter bridge stp llc bonding binfmt_misc iTCO_wdt iTCO_vendor_support ipmi_devintf mxm_wmi intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure mei_me scsi_transport_sas sg mei cdc_ether usbnet lpc_ich mii sb_edac edac_core i2c_i801 ipmi_ssif pcspkr
[1144590.651481] ipmi_si ipmi_msghandler wmi shpchp acpi_power_meter acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ptp i2c_core pps_core megaraid_sas fjes
[1144590.651512] CPU: 1 PID: 226 Comm: kswapd0 Not tainted 3.10.0-514.2.2.el7.x86_64 #1
[1144590.651514] Hardware name: LENOVO System x3650 M5: -[8871AC1]-/01GR174, BIOS -[TCE126O-2.20]- 09/07/2016
[1144590.651517] 0000000000000000 000000006867b81f ffff881ffdf2baa0 ffffffff816861cc
[1144590.651521] ffff881ffdf2bad8 ffffffff81085940 ffffea0001345820 ffffea0001345800
[1144590.651525] ffff883febab3d38 ffff881ffdf2bda0 ffffea0001345800 ffff881ffdf2bae8
[1144590.651532] Call Trace:
[1144590.651542] [<ffffffff816861cc>] dump_stack+0x19/0x1b
[1144590.651549] [<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
[1144590.651552] [<ffffffff81085a8a>] warn_slowpath_null+0x1a/0x20
[1144590.651582] [<ffffffffa028f56b>] xfs_vm_releasepage+0xcb/0x100 [xfs]
[1144590.651589] [<ffffffff811805c2>] try_to_release_page+0x32/0x50
[1144590.651598] [<ffffffff81196546>] shrink_active_list+0x3d6/0x3e0
[1144590.651606] [<ffffffff81196941>] shrink_lruvec+0x3f1/0x770
[1144590.651612] [<ffffffff81196d36>] shrink_zone+0x76/0x1a0
[1144590.651615] [<ffffffff81197fdc>] balance_pgdat+0x48c/0x5e0
[1144590.651619] [<ffffffff811982a3>] kswapd+0x173/0x450
[1144590.651627] [<ffffffff810b1720>] ? wake_up_atomic_t+0x30/0x30
[1144590.651637] [<ffffffff81198130>] ? balance_pgdat+0x5e0/0x5e0
[1144590.651642] [<ffffffff810b064f>] kthread+0xcf/0xe0
[1144590.651647] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[1144590.651656] [<ffffffff81696718>] ret_from_fork+0x58/0x90
[1144590.651663] [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
[1144590.651667] ---[ end trace ca0023e0a8c7262b ]---

~0028867

vitaliyf (reporter)

We are seeing this crash on physical servers (LSI MegaRAID). On some of them journald/rsyslog are the only processes doing disk I/O (though quite a bit of it).

We have a few servers running CentOS 7.2-era kernels and we've never seen xfs crashes there.

There is upstream bug report here: https://access.redhat.com/solutions/2779111
+Notes

-Issue History
Date Modified Username Field Change
2016-12-23 05:43 rellik.hsu New Issue
2016-12-23 05:43 rellik.hsu File Added: dmesg.png
2017-01-18 11:48 TuxHandwerker Note Added: 0028398
2017-01-18 15:05 dragle Note Added: 0028405
2017-01-19 12:03 cullmann Note Added: 0028417
2017-01-23 20:09 dragle Note Added: 0028461
2017-02-21 03:57 alienth Note Added: 0028619
2017-03-03 15:08 ascherbakov-ya Note Added: 0028754
2017-03-15 15:38 vitaliyf Note Added: 0028867
+Issue History