View Issue Details

IDProjectCategoryView StatusLast Update
0016761CentOS-7filesystempublic2019-11-22 07:15
Reportersushilmax 
PriorityurgentSeveritytrivialReproducibilityalways
Status newResolutionopen 
PlatformredhatOSCentosOS Version7.6
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0016761: Pod crashes when attached to volumes having xfs file system.
DescriptionCreating volumes with xfs file system and attaching them to fio pods in kubernetes, leading to pod crash after 5-6 minutes.
Steps To Reproduce1) Create a volume with xfs file system.
2) Attach pod to that volume.
Additional Informationdmesg -T output:

[Thu Sep 26 05:54:00 2019] perf: interrupt took too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[Thu Sep 26 06:16:40 2019] nvme nvme1: pci function 0000:18:10.3
[Thu Sep 26 06:16:40 2019] nvme 0000:18:10.3: enabling device (0140 -> 0142)
[Thu Sep 26 06:16:40 2019] nvme 0000:18:10.3: irq 421 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 421 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 422 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 423 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 424 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 425 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 426 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 427 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 428 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 429 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 430 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 431 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 432 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 433 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 434 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 435 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 436 for MSI/MSI-X
[Thu Sep 26 06:16:41 2019] nvme 0000:18:10.3: irq 437 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme nvme2: pci function 0000:18:10.4
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: enabling device (0140 -> 0142)
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 438 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 438 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 439 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 440 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 441 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 442 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 443 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 444 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 445 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 446 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 447 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 448 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 449 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 450 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 451 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 452 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 453 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] nvme 0000:18:10.4: irq 454 for MSI/MSI-X
[Thu Sep 26 06:16:42 2019] XFS (nvme1n1): Mounting V5 Filesystem
[Thu Sep 26 06:16:42 2019] XFS (nvme1n1): Ending clean mount
[Thu Sep 26 06:16:43 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:43 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Thu Sep 26 06:16:43 2019] IPv6: ADDRCONF(NETDEV_UP): vethfcb10f82: link is not ready
[Thu Sep 26 06:16:43 2019] IPv6: ADDRCONF(NETDEV_CHANGE): vethfcb10f82: link becomes ready
[Thu Sep 26 06:16:43 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Thu Sep 26 06:16:43 2019] device vethfcb10f82 entered promiscuous mode
[Thu Sep 26 06:16:44 2019] nvme nvme3: pci function 0000:18:10.5
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: enabling device (0140 -> 0142)
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 455 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] XFS (nvme2n1): Mounting V5 Filesystem
[Thu Sep 26 06:16:44 2019] XFS (nvme2n1): Ending clean mount
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 455 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 456 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 457 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 458 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 459 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 460 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 461 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 462 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 463 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 464 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 465 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 466 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 467 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 468 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 469 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 470 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] nvme 0000:18:10.5: irq 471 for MSI/MSI-X
[Thu Sep 26 06:16:44 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:44 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Thu Sep 26 06:16:44 2019] IPv6: ADDRCONF(NETDEV_UP): veth467f1ce9: link is not ready
[Thu Sep 26 06:16:44 2019] IPv6: ADDRCONF(NETDEV_CHANGE): veth467f1ce9: link becomes ready
[Thu Sep 26 06:16:44 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Thu Sep 26 06:16:44 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:44 2019] device veth467f1ce9 entered promiscuous mode
[Thu Sep 26 06:16:45 2019] nvme nvme4: pci function 0000:18:10.6
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: enabling device (0140 -> 0142)
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 472 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 472 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 473 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 474 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 475 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 476 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 477 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 478 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 479 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 480 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 481 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 482 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 483 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 484 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 485 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 486 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 487 for MSI/MSI-X
[Thu Sep 26 06:16:45 2019] nvme 0000:18:10.6: irq 488 for MSI/MSI-X
[Thu Sep 26 06:16:46 2019] XFS (nvme3n1): Mounting V5 Filesystem
[Thu Sep 26 06:16:46 2019] XFS (nvme3n1): Ending clean mount
[Thu Sep 26 06:16:46 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:46 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Thu Sep 26 06:16:46 2019] IPv6: ADDRCONF(NETDEV_UP): veth5d61697e: link is not ready
[Thu Sep 26 06:16:46 2019] IPv6: ADDRCONF(NETDEV_CHANGE): veth5d61697e: link becomes ready
[Thu Sep 26 06:16:46 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Thu Sep 26 06:16:46 2019] device veth5d61697e entered promiscuous mode
[Thu Sep 26 06:16:46 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:47 2019] XFS (nvme4n1): Mounting V5 Filesystem
[Thu Sep 26 06:16:47 2019] XFS (nvme4n1): Ending clean mount
[Thu Sep 26 06:16:47 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:47 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Thu Sep 26 06:16:47 2019] IPv6: ADDRCONF(NETDEV_UP): vethc0adbe54: link is not ready
[Thu Sep 26 06:16:47 2019] IPv6: ADDRCONF(NETDEV_CHANGE): vethc0adbe54: link becomes ready
[Thu Sep 26 06:16:47 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Thu Sep 26 06:16:48 2019] device vethc0adbe54 entered promiscuous mode
[Thu Sep 26 06:16:48 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:16:49 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:36 2019] XFS (nvme2n1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3258 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:36 2019] CPU: 10 PID: 365109 Comm: kworker/10:73 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:36 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:26 2019] Filesystem "nvme2n1": reserve blocks depleted! Consider increasing reserve pool size.
[Thu Sep 26 06:17:36 2019] Workqueue: dio/nvme2n1 dio_aio_complete_work

[Thu Sep 26 06:17:36 2019] Call Trace:
[Thu Sep 26 06:17:36 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:36 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0353ce8>] ? xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc036246b>] xfs_btree_insert+0x1db/0x1f0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0353ce8>] xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0355246>] xfs_bmapi_convert_unwritten+0x116/0x1c0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc035a2e9>] xfs_bmapi_write+0x269/0xab0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0396327>] xfs_iomap_write_unwritten+0x117/0x300 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:36 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:36 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:36 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:36 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:36 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:36 2019] XFS (nvme2n1): Internal error xfs_trans_cancel at line 984 of file fs/xfs/xfs_trans.c. Caller xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:36 2019] CPU: 10 PID: 365109 Comm: kworker/10:73 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:36 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:36 2019] Workqueue: dio/nvme2n1 dio_aio_complete_work
[Thu Sep 26 06:17:36 2019] Call Trace:
[Thu Sep 26 06:17:36 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:36 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc03964c2>] ? xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc03a677d>] xfs_trans_cancel+0xbd/0xe0 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc03964c2>] xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:36 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:36 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:36 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:36 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:36 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:36 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:36 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:36 2019] XFS (nvme2n1): xfs_do_force_shutdown(0x8) called from line 985 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc03a6796
[Thu Sep 26 06:17:36 2019] XFS (nvme2n1): Corruption of in-memory data detected. Shutting down filesystem
[Thu Sep 26 06:17:36 2019] XFS (nvme2n1): Please umount the filesystem and rectify the problem(s)
[Thu Sep 26 06:17:37 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:37 2019] XFS (nvme1n1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3258 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:37 2019] CPU: 24 PID: 36051 Comm: kworker/24:1 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:37 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:37 2019] Workqueue: dio/nvme1n1 dio_aio_complete_work
[Thu Sep 26 06:17:37 2019] Call Trace:
[Thu Sep 26 06:17:37 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:37 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0353c84>] ? xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc036246b>] xfs_btree_insert+0x1db/0x1f0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0353c84>] xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0355246>] xfs_bmapi_convert_unwritten+0x116/0x1c0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc035a2e9>] xfs_bmapi_write+0x269/0xab0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0396327>] xfs_iomap_write_unwritten+0x117/0x300 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:37 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:37 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:37 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:37 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:37 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:37 2019] XFS (nvme1n1): Internal error xfs_trans_cancel at line 984 of file fs/xfs/xfs_trans.c. Caller xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:37 2019] CPU: 24 PID: 36051 Comm: kworker/24:1 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:37 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:37 2019] Workqueue: dio/nvme1n1 dio_aio_complete_work
[Thu Sep 26 06:17:37 2019] Call Trace:
[Thu Sep 26 06:17:37 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:37 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc03964c2>] ? xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc03a677d>] xfs_trans_cancel+0xbd/0xe0 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc03964c2>] xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:37 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:37 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:37 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:37 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:37 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:37 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:37 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:37 2019] XFS (nvme1n1): xfs_do_force_shutdown(0x8) called from line 985 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc03a6796
[Thu Sep 26 06:17:37 2019] XFS (nvme1n1): Corruption of in-memory data detected. Shutting down filesystem
[Thu Sep 26 06:17:37 2019] XFS (nvme1n1): Please umount the filesystem and rectify the problem(s)
[Thu Sep 26 06:17:39 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:39 2019] XFS (nvme4n1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3258 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:39 2019] CPU: 37 PID: 367877 Comm: kworker/37:11 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:39 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:39 2019] Workqueue: dio/nvme4n1 dio_aio_complete_work
[Thu Sep 26 06:17:39 2019] Call Trace:
[Thu Sep 26 06:17:39 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:39 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0353c84>] ? xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc036246b>] xfs_btree_insert+0x1db/0x1f0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0353c84>] xfs_bmap_add_extent_unwritten_real+0x444/0x18f0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0355246>] xfs_bmapi_convert_unwritten+0x116/0x1c0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc035a2e9>] xfs_bmapi_write+0x269/0xab0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0396327>] xfs_iomap_write_unwritten+0x117/0x300 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:39 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:39 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:39 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:39 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:39 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:39 2019] XFS (nvme4n1): Internal error xfs_trans_cancel at line 984 of file fs/xfs/xfs_trans.c. Caller xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:39 2019] CPU: 37 PID: 367877 Comm: kworker/37:11 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:39 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:39 2019] Workqueue: dio/nvme4n1 dio_aio_complete_work
[Thu Sep 26 06:17:39 2019] Call Trace:
[Thu Sep 26 06:17:39 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:39 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc03964c2>] ? xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc03a677d>] xfs_trans_cancel+0xbd/0xe0 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc03964c2>] xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:39 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:39 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:39 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:39 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:39 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:39 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:39 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:39 2019] XFS (nvme4n1): xfs_do_force_shutdown(0x8) called from line 985 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc03a6796
[Thu Sep 26 06:17:39 2019] XFS (nvme4n1): Corruption of in-memory data detected. Shutting down filesystem
[Thu Sep 26 06:17:39 2019] XFS (nvme4n1): Please umount the filesystem and rectify the problem(s)
[Thu Sep 26 06:17:40 2019] XFS (nvme3n1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3258 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:40 2019] CPU: 4 PID: 366871 Comm: kworker/4:107 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:40 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:40 2019] Workqueue: dio/nvme3n1 dio_aio_complete_work
[Thu Sep 26 06:17:40 2019] Call Trace:
[Thu Sep 26 06:17:40 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:40 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0353ce8>] ? xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc036246b>] xfs_btree_insert+0x1db/0x1f0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0353ce8>] xfs_bmap_add_extent_unwritten_real+0x4a8/0x18f0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0355246>] xfs_bmapi_convert_unwritten+0x116/0x1c0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc035a2e9>] xfs_bmapi_write+0x269/0xab0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0396327>] xfs_iomap_write_unwritten+0x117/0x300 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:40 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:40 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:40 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:40 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:40 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:40 2019] XFS (nvme3n1): Internal error xfs_trans_cancel at line 984 of file fs/xfs/xfs_trans.c. Caller xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:40 2019] CPU: 4 PID: 366871 Comm: kworker/4:107 Kdump: loaded Tainted: G E ------------ T 3.10.0-957.el7.x86_64 #1
[Thu Sep 26 06:17:40 2019] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.00.01.2016.031420190801 03/14/2019
[Thu Sep 26 06:17:40 2019] Workqueue: dio/nvme3n1 dio_aio_complete_work
[Thu Sep 26 06:17:40 2019] Call Trace:
[Thu Sep 26 06:17:40 2019] [<ffffffff86161dc1>] dump_stack+0x19/0x1b
[Thu Sep 26 06:17:40 2019] [<ffffffffc0389a1b>] xfs_error_report+0x3b/0x40 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc03964c2>] ? xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc03a677d>] xfs_trans_cancel+0xbd/0xe0 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc03964c2>] xfs_iomap_write_unwritten+0x2b2/0x300 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffffc0380f63>] xfs_end_io_direct_write+0x133/0x170 [xfs]
[Thu Sep 26 06:17:40 2019] [<ffffffff85c824f5>] dio_complete+0x125/0x2a0
[Thu Sep 26 06:17:40 2019] [<ffffffff85c827f1>] dio_aio_complete_work+0x21/0x30
[Thu Sep 26 06:17:40 2019] [<ffffffff85ab9d4f>] process_one_work+0x17f/0x440
[Thu Sep 26 06:17:40 2019] [<ffffffff85abade6>] worker_thread+0x126/0x3c0
[Thu Sep 26 06:17:40 2019] [<ffffffff85abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1c31>] kthread+0xd1/0xe0
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:40 2019] [<ffffffff86174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[Thu Sep 26 06:17:40 2019] [<ffffffff85ac1b60>] ? insert_kthread_work+0x40/0x40
[Thu Sep 26 06:17:40 2019] XFS (nvme3n1): xfs_do_force_shutdown(0x8) called from line 985 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc03a6796
[Thu Sep 26 06:17:40 2019] XFS (nvme3n1): Corruption of in-memory data detected. Shutting down filesystem
[Thu Sep 26 06:17:40 2019] XFS (nvme3n1): Please umount the filesystem and rectify the problem(s)
[Thu Sep 26 06:17:40 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:41 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:53 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:55 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:55 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:17:55 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:18:19 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:18:20 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:18:24 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
[Thu Sep 26 06:18:26 2019] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
Tags7.6.1810, centos 7, corruption, crash
abrt_hash
URL

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2019-11-22 07:15 sushilmax New Issue
2019-11-22 07:15 sushilmax Tag Attached: 7.6.1810
2019-11-22 07:15 sushilmax Tag Attached: centos 7
2019-11-22 07:15 sushilmax Tag Attached: corruption
2019-11-22 07:15 sushilmax Tag Attached: crash