View Issue Details

IDProjectCategoryView StatusLast Update
0015681CentOS-7kernelpublic2019-01-11 14:40
Reporterr0mr0m 
PriorityhighSeveritycrashReproducibilitysometimes
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0015681: Kernel panic while using ceph rbd
DescriptionOnce moved to centos7.6 about a week ago, we suddenly started encountering kernel panics during writes that performed via rbd.

[ 8941.026611] libceph: mon2 2.104.194.77:6789 session established

[ 8941.033738] libceph: client50241 fsid ea0df043-7b25-4447-a43d-e9b2af8fe069

[ 8941.053063] rbd: rbd0: capacity 1073741824 features 0x1

[ 8941.252456] XFS (rbd0): Mounting V5 Filesystem

[ 8941.362658] XFS (rbd0): Ending clean mount

[ 8947.674186] rbd: rbd1: capacity 1073741824 features 0x1

[ 8952.568957] rbd: rbd2: capacity 1073741824 features 0x1

[ 8953.147799] rbd2: p1

[ 8974.184516] XFS (rbd0): Unmounting Filesystem

[ 8974.192754] usercopy: kernel memory exposure attempt detected from ffff8d83b56f7000 (kmalloc-512) (1024 bytes)

[ 8974.203999] ------------[ cut here ]------------

[ 8974.209149] kernel BUG at mm/usercopy.c:72!

[ 8974.213817] invalid opcode: 0000 [#1] SMP

[ 8974.218410] Modules linked in: ip6table_raw xt_physdev xt_CHECKSUM veth iptable_mangle iptable_raw ebtable_filter rbd libceph ebtables dns_resolver ip6table_filter ip6_tables fuse btrfs raid6_pq xor msdos fat xt_comment mlx4_en(OE) mlx4_core(OE) xt_multiport ipt_REJECT nf_reject_ipv4 nf_conntrack_netlink nfnetlink iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc xfs openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat mlx5_core(OE) nf_conntrack mlxfw(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass pcspkr joydev sg lpc_ich i2c_i801 mei_me mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter dm_multipath ip_tables ext4 mbcache jbd2 dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel igb mgag200 ghash_clmulni_intel dca drm_kms_helper ahci aesni_intel syscopyarea lrw gf128mul sysfillrect glue_helper sysimgblt fb_sys_fops ablk_helper ttm cryptd libahci ptp drm mlx_compat(OE) pps_core i2c_algo_bit libata drm_panel_orientation_quirks devlink wmi scsi_transport_iscsi sunrpc dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mlx4_core]

[ 8974.348153] CPU: 4 PID: 97983 Comm: msgr-worker-1 Kdump: loaded Tainted: G IOE ------------ 3.10.0-957.1.3.el7.x86_64 #1

[ 8974.361450] Hardware name: Intel Corporation S2600KP/S2600KP, BIOS SE5C610.86B.01.01.0005.101720141054 10/17/2014

[ 8974.372908] task: ffff8d8d676f1040 ti: ffff8d8cb8b70000 task.ti: ffff8d8cb8b70000

[ 8974.381263] RIP: 0010:[<ffffffffbd83e167>] [<ffffffffbd83e167>] __check_object_size+0x87/0x250

[ 8974.390991] RSP: 0018:ffff8d8cb8b73b98 EFLAGS: 00010246

[ 8974.396921] RAX: 0000000000000062 RBX: ffff8d83b56f7000 RCX: 0000000000000000

[ 8974.404886] RDX: 0000000000000000 RSI: ffff8d9378913898 RDI: ffff8d9378913898

[ 8974.412851] RBP: ffff8d8cb8b73bb8 R08: 0000000000000000 R09: 0000000000000000

[ 8974.420814] R10: 0000000000000e17 R11: ffff8d8cb8b73896 R12: 0000000000000400

[ 8974.428777] R13: 0000000000000001 R14: ffff8d83b56f7400 R15: 0000000000000400

[ 8974.436739] FS: 00007fe4c0dc3700(0000) GS:ffff8d9378900000(0000) knlGS:0000000000000000

[ 8974.445768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 8974.452180] CR2: 00007f82a46f0018 CR3: 00000007867f8000 CR4: 00000000001607e0

[ 8974.460142] Call Trace:

[ 8974.462875] [<ffffffffbd98c0dd>] memcpy_toiovec+0x4d/0xb0

[ 8974.469015] [<ffffffffbdc2a858>] skb_copy_datagram_iovec+0x128/0x280

[ 8974.476193] [<ffffffffbdc9172a>] tcp_recvmsg+0x22a/0xb30

[ 8974.482218] [<ffffffffbdcc00e0>] inet_recvmsg+0x80/0xb0

[ 8974.488146] [<ffffffffbdc186ec>] sock_aio_read.part.9+0x14c/0x170

[ 8974.495043] [<ffffffffbdc18731>] sock_aio_read+0x21/0x30

[ 8974.501069] [<ffffffffbd840743>] do_sync_read+0x93/0xe0

[ 8974.507002] [<ffffffffbd841225>] vfs_read+0x145/0x170

[ 8974.512737] [<ffffffffbd84203f>] SyS_read+0x7f/0xf0

[ 8974.518277] [<ffffffffbdd74ddb>] system_call_fastpath+0x22/0x27

[ 8974.524978] Code: 45 d1 48 c7 c6 d4 b6 07 be 48 c7 c1 e0 4b 08 be 48 0f 45 f1 49 89 c0 4d 89 e1 48 89 d9 48 c7 c7 d0 1a 08 be 31 c0 e8 20 d5 51 00 <0f> 0b 0f 1f 80 00 00 00 00 48 c7 c0 00 00 60 bd 4c 39 f0 73 0d

[ 8974.546748] RIP [<ffffffffbd83e167>] __check_object_size+0x87/0x250

[ 8974.553851] RSP <ffff8d8cb8b73b98>

Additional InformationHappens pretty often

versions:
CentOS Linux release 7.6.1810 (Core)
Linux stratonode1.node.strato 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

librbd1-12.2.8-0.el7.x86_64


[root@stratonode1 ~]# modinfo libceph
filename: /lib/modules/3.10.0-957.1.3.el7.x86_64/kernel/net/ceph/libceph.ko.xz
license: GPL
description: Ceph core library
author: Patience Warnick <patience@newdream.net>
author: Yehuda Sadeh <yehuda@hq.newdream.net>
author: Sage Weil <sage@newdream.net>
retpoline: Y
rhelversion: 7.6
srcversion: 4F8CE6AEFA99B11C267981D
depends: libcrc32c,dns_resolver
intree: Y
vermagic: 3.10.0-957.1.3.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: E7:CE:F3:61:3A:9B:8B:D0:12:FA:E7:49:82:72:15:9B:B1:87:9C:65
sig_hashalgo: sha256
[root@stratonode1 ~]# modinfo rbd
filename: /lib/modules/3.10.0-957.1.3.el7.x86_64/kernel/drivers/block/rbd.ko.xz
license: GPL
description: RADOS Block Device (RBD) driver
author: Jeff Garzik <jeff@garzik.org>
author: Yehuda Sadeh <yehuda@hq.newdream.net>
author: Sage Weil <sage@newdream.net>
author: Alex Elder <elder@inktank.com>
retpoline: Y
rhelversion: 7.6
srcversion: 5386BBBD00C262C66CB81F5
depends: libceph
intree: Y
vermagic: 3.10.0-957.1.3.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: E7:CE:F3:61:3A:9B:8B:D0:12:FA:E7:49:82:72:15:9B:B1:87:9C:65
sig_hashalgo: sha256
parm: single_major:Use a single major number for all rbd devices (default: true) (bool)

TagsNo tags attached.
abrt_hash
URL

Activities

r0mr0m

r0mr0m

2019-01-11 14:40

reporter   ~0033566

After contacting redhat, apparently it's known issue.
This link can add more info:
https://bugzilla.redhat.com/show_bug.cgi?id=1665248

Issue History

Date Modified Username Field Change
2019-01-09 11:31 r0mr0m New Issue
2019-01-11 14:40 r0mr0m Note Added: 0033566