2017-10-23 20:33 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013648CentOS-7kernelpublic2017-08-16 06:13
Reporterqwxz_wh@hotmail.com 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusnewResolutionopen 
PlatformX64OSCentosOS Version7.2
Product Version7.2.1511 
Target VersionFixed in Version 
Summary0013648: XFS SWAPEXT xlog_write: reservation ran out
DescriptionAug 4 23:40:01 localhost systemd: Starting Session 963 of user root.
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xlog_write: reservation summary:
#012 trans type = SWAPEXT (40)
#012 unit res = 8652816 bytes
#012 current res = -36 bytes
#012 total reg = 2048 bytes (o/flow = 537553056 bytes)
#012 ophdrs = 303294 (ophdr space = 3639528 bytes)
#012 ophdr + reg = 541194632 bytes
#012 num regions = 5
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): region[0]: LR header - 512 bytes
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): region[1]: LR header - 512 bytes
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): region[2]: LR header - 512 bytes
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): region[3]: LR header - 512 bytes
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): region[4]: commit - 0 bytes
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xlog_write: reservation ran out. Need to up reservation
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xfs_do_force_shutdown(0x2) called from line 2070 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06cdea8
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): Log I/O Error Detected. Shutting down filesystem
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): Please umount the filesystem and rectify the problem(s)
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xfs_do_force_shutdown(0x2) called from line 1526 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06ce684
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:46:30 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:46:54 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:47:24 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:47:54 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xlog_write: reservation summary:#012 trans type = SWAPEXT (40)#012 unit res = 8991844 bytes#012 current res = -392 bytes#012 total reg = 4096 bytes (o/flow = 558594992 bytes)#012 ophdrs = 316724 (ophdr space = 3800688 bytes)#012 ophdr + reg = 562399776 bytes#012 num regions = 9
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[0]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[1]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[2]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[3]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[4]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[5]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[6]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[7]: LR header - 512 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): region[8]: commit - 0 bytes
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xlog_write: reservation ran out. Need to up reservation
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xfs_do_force_shutdown(0x2) called from line 2070 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06cdea8
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): Log I/O Error Detected. Shutting down filesystem
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): Please umount the filesystem and rectify the problem(s)
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xfs_do_force_shutdown(0x2) called from line 1526 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06ce684
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:48:14 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:48:21 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:48:24 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:48:51 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:48:54 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:49:21 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:49:24 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 4 23:49:51 localhost kernel: XFS (nvme0n1): xfs_log_force: error -5 returned.
Aug 4 23:49:54 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Steps To Reproduce1. mkfs.xfs /dev/nvme0n1
2. mount /dev/nvme0n1 /nvme0n1
3. fio script
[global]
ioengine=libaio
direct=1
thread
randrepeat=0
norandommap
random_generator=tausworthe
directory=/nvme1n1
size=152649M
bs=4k

[randwrite4k]
stonewall
group_reporting
rw=randwrite
numjobs=16
iodepth=16
time_based
runtime=3600
Additional InformationI found alike issue in the https://patchwork.kernel.org/patch/9706313/, but did not work after applied the patch 9706313.
TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0029817

TrevorH (developer)

a) Don't use 7.2, it's out of date
b) https://bugzilla.redhat.com/show_bug.cgi?id=1402533

So update your Intel SSD firmware to the one listed there and try again.

~0029844

qwxz_wh@hotmail.com (reporter)

In the Centos 7.3, repeated the same test, and xfs log resevation ran out again.

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# grep XFS /var/log/messages
Aug 10 16:26:02 localhost kernel: SGI XFS with ACLs, security attributes, no debug enabled
Aug 10 16:26:02 localhost kernel: XFS (nvme1n1): Mounting V5 Filesystem
Aug 10 16:26:02 localhost kernel: XFS (nvme1n1): Ending clean mount
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): xlog_write: reservation summary:
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): trans type = CHECKPOINT (40)
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): unit res = 10882960 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): current res = -412 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): total reg = 5632 bytes (o/flow = 676140672 bytes)
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): ophdrs = 380210 (ophdr space = 4562520 bytes)
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): ophdr + reg = 680708824 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): num regions = 12
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[0]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[1]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[2]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[3]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[4]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[5]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[6]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[7]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[8]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[9]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[10]: LR header - 512 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): region[11]: commit - 0 bytes
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): xlog_write: reservation ran out. Need to up reservation
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): xfs_do_force_shutdown(0x2) called from line 2111 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06b1fa8
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): Log I/O Error Detected. Shutting down filesystem
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): Please umount the filesystem and rectify the problem(s)
Aug 10 18:08:22 localhost kernel: XFS (nvme1n1): xfs_do_force_shutdown(0x2) called from line 1550 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06b2794
Aug 10 18:08:23 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 10 18:08:23 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 10 18:08:26 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 10 18:08:56 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.
Aug 10 18:09:26 localhost kernel: XFS (nvme1n1): xfs_log_force: error -5 returned.

~0029845

TrevorH (developer)

From the reports before it appears the important bit is the SSD firmware. This is an Intel 600p?

~0029846

qwxz_wh@hotmail.com (reporter)

No. It's Memblaze Pblaze5 ,fm version is 6a0.

~0029877

qwxz_wh@hotmail.com (reporter)

Using intel p3600/p3700/DC p3500 concatenated a 4TB lvm, and did the belows test

1. fio 128k seq write whole drive twice
2. fio 4k randwrite queue depth 64 with 2 jobs
3. Mkfs.xfs and mount to folder
4. run fio 4k randwrite 16 * 190GB


# uname -a
Linux localhost.localdomain 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 CVMD422200011P6DGN INTEL SSDPEDME016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10131
/dev/nvme1n1 CVPD514300KC400NGN INTEL SSDPE2MX400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV10135
/dev/nvme2n1 CVPD51460038400NGN INTEL SSDPE2MX400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV10135
/dev/nvme3n1 CVPD51620011400NGN INTEL SSDPE2MX400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV10135
/dev/nvme4n1 CVFT421600491P6DGN INTEL SSDPEDMD016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10171
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/nvme0n1 myvg lvm2 a-- 1.46t 0
  /dev/nvme1n1 myvg lvm2 a-- 372.61g 2.36g
  /dev/nvme2n1 myvg lvm2 a-- 372.61g 0
  /dev/nvme3n1 myvg lvm2 a-- 372.61g 0
  /dev/nvme4n1 myvg lvm2 a-- 1.46t 0
# lvdisplay
  --- Logical volume ---
  LV Path /dev/myvg/lvol0
  LV Name lvol0
  VG Name myvg
  LV UUID ylBC12-4wyS-1c7M-2sDO-c6Tr-E0aG-qCrVVD
  LV Write Access read/write
  LV Creation host, time target, 2017-08-15 17:49:38 +0800
  LV Status available
  # open 1
  LV Size 4.00 TiB
  Current LE 1048576
  Segments 5
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:0

# mount |grep myvg
/dev/mapper/myvg-lvol0 on /lvol0 type xfs (rw,relatime,attr2,inode64,noquota)

[ 6491.583826] XFS (dm-0): xlog_write: reservation summary:
[ 6491.583830] XFS (dm-0): trans type = CHECKPOINT (40)
[ 6491.583831] XFS (dm-0): unit res = 7862100 bytes
[ 6491.583832] XFS (dm-0): current res = -156 bytes
[ 6491.583833] XFS (dm-0): total reg = 4608 bytes (o/flow = 488251128 bytes)
[ 6491.583834] XFS (dm-0): ophdrs = 289539 (ophdr space = 3474468 bytes)
[ 6491.583835] XFS (dm-0): ophdr + reg = 491730204 bytes
[ 6491.583835] XFS (dm-0): num regions = 10
[ 6491.583836] XFS (dm-0): region[0]: LR header - 512 bytes
[ 6491.583837] XFS (dm-0): region[1]: LR header - 512 bytes
[ 6491.583838] XFS (dm-0): region[2]: LR header - 512 bytes
[ 6491.583839] XFS (dm-0): region[3]: LR header - 512 bytes
[ 6491.583840] XFS (dm-0): region[4]: LR header - 512 bytes
[ 6491.583840] XFS (dm-0): region[5]: LR header - 512 bytes
[ 6491.583841] XFS (dm-0): region[6]: LR header - 512 bytes
[ 6491.583842] XFS (dm-0): region[7]: LR header - 512 bytes
[ 6491.583843] XFS (dm-0): region[8]: LR header - 512 bytes
[ 6491.583843] XFS (dm-0): region[9]: commit - 0 bytes
[ 6491.583844] XFS (dm-0): xlog_write: reservation ran out. Need to up reservation
[ 6491.583847] XFS (dm-0): xfs_do_force_shutdown(0x2) called from line 2111 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa06d6fa8
[ 6491.583936] XFS (dm-0): Log I/O Error Detected. Shutting down filesystem
[ 6491.583938] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
+Notes

-Issue History
Date Modified Username Field Change
2017-08-09 10:12 qwxz_wh@hotmail.com New Issue
2017-08-09 10:22 TrevorH Note Added: 0029817
2017-08-10 11:29 qwxz_wh@hotmail.com Note Added: 0029844
2017-08-10 11:37 TrevorH Note Added: 0029845
2017-08-10 11:54 qwxz_wh@hotmail.com Note Added: 0029846
2017-08-16 06:13 qwxz_wh@hotmail.com Note Added: 0029877
+Issue History