View Issue Details

IDProjectCategoryView StatusLast Update
0008039CentOS-7kernelpublic2015-02-24 23:30
Reportergrossws 
PrioritynormalSeveritycrashReproducibilityhave not tried
Status resolvedResolutionfixed 
Platformx86_64OScentosOS Version7.0
Product Version7.0-1406 
Target VersionFixed in Version 
Summary0008039: kernel crash in netns cleanup_net (nf_nat_cleanup_conntrack)
DescriptionKernel crashes after several days (about 3-4 days in my case). I have some short-lived docker containers (created and destroyed every 15 mins), so it's netns is destroyed every 15 mins.

Maybe upstream bug: https://bugzilla.kernel.org/show_bug.cgi?id=65191.

Currently I applied upstream patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=945b2b2d259d1a4364a2799e80e8ff32f8c6ee6f to standard centos kernel (3.10.0-123.13.2.el7) and use it.

Can this patch be included to standard centos kernel?
Additional InformationCrash dump:
[289778.842664] CPU: 20 PID: 69556 Comm: kworker/u164:2 Not tainted 3.10.0-123.el7.x86_64 #1
[289778.842729] Hardware name: Supermicro X8QB6/X8QB6, BIOS 2.0c 06/11/2
[289778.842795] Workqueue: netns cleanup_net
[289778.842834] task: ffff8819ecdf38e0 ti: ffff881342172000 task.ti: ffff881342172000
[289778.842894] RIP: 0010:[<ffffffffa047c200>] [<ffffffffa047c200>] nf_nat_cleanup_conntrack+0x40/0x70 [nf_nat]
[289778.842974] RSP: 0000:ffff881342173ca0 EFLAGS: 00010246
[289778.843019] RAX: 0000000000000000 RBX: ffff881a17d94c88 RCX: ffff88128bed3b90
[289778.843072] RDX: ffffc900238a3338 RSI: 0000000000000006 RDI: ffffffffa047f498
[289778.843126] RBP: ffff881342173ca8 R08: 0000000000000286 R09: 0000000180400030
[289778.843180] R10: ffffea0026eb7e00 R11: ffffffff8117b372 R12: ffff881a17d94c00
[289778.843232] R13: ffff8801102f8000 R14: ffff8801102f8008 R15: ffff88128bed3180
[289778.843282] FS: 0000000000000000(0000) GS:ffff881a5f080000(0000) knlGS:0000000000000000
[289778.843344] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[289778.843387] CR2: ffffc900238a3338 CR3: 0000000e0f0f1000 CR4: 00000000000007e0
[289778.843440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[289778.843494] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[289778.843553] Stack:
[289778.843573] 0000000000000001 ffff881342173cd0 ffffffffa03bfdb4 ffff8801102f8000
[289778.843663] ffff88128bed3180 ffffffffa03d82c0 ffff881342173cf0 ffffffffa03b72d5
[289778.843740] ffff8801102f8000 ffff88128bed3180 ffff881342173d10 ffffffffa03b883d
[289778.843818] Call Trace:
[289778.843865] [<ffffffffa03bfdb4>] __nf_ct_ext_destroy+0x44/0x60 [nf_conntrack]
[289778.843923] [<ffffffffa03b72d5>] nf_conntrack_free+0x25/0x60 [nf_conntrack]
[289778.843976] [<ffffffffa03b883d>] destroy_conntrack+0xbd/0x110 [nf_conntrack]
[289778.844031] [<ffffffffa03bce20>] ? nf_conntrack_helper_fini+0x30/0x30 [nf_conntrack]
[289778.844102] [<ffffffff815002a7>] nf_conntrack_destroy+0x17/0x20
[289778.844168] [<ffffffffa03b824b>] nf_ct_iterate_cleanup+0xcb/0x160 [nf_conntrack]
[289778.844237] [<ffffffffa03bd0fd>] nf_ct_l3proto_pernet_unregister+0x1d/0x20 [nf_conntrack]
[289778.844308] [<ffffffffa03d6329>] ipv4_net_exit+0x19/0x50 [nf_conntrack_ipv4]
[289778.844368] [<ffffffff814c82b9>] ops_exit_list.isra.1+0x39/0x60
[289778.844423] [<ffffffff814c8b60>] cleanup_net+0x110/0x260
[289778.844480] [<ffffffff8107e02b>] process_one_work+0x17b/0x460
[289778.844524] [<ffffffff8107edfb>] worker_thread+0x11b/0x400
[289778.844574] [<ffffffff8107ece0>] ? rescuer_thread+0x400/0x400
[289778.844624] [<ffffffff81085aef>] kthread+0xcf/0xe0
[289778.844669] [<ffffffff81085a20>] ? kthread_create_on_node+0x140/0x140
[289778.844730] [<ffffffff815f206c>] ret_from_fork+0x7c/0xb0
[289778.844773] [<ffffffff81085a20>] ? kthread_create_on_node+0x140/0x140
[289778.844825] Code: 53 0f b6 58 11 84 db 74 45 48 01 c3 74 40 48 83 7b 10 00 74 39 48 c7 c7 98 f4 47 a0 e8 0a cf 16 e1 48 8b 03 48 8b 53 08 48 85 c0 <48> 89 02 74 04 48 89 50 08 48 b8 00 02 20 00 00 00 ad de 48 c7
[289778.845392] RIP [<ffffffffa047c200>] nf_nat_cleanup_conntrack+0x40/0x70 [nf_nat]
[289778.845460] RSP <ffff881342173ca0>
[289778.845493] CR2: ffffc900238a3338
TagsNo tags attached.
abrt_hash
URL

Activities

toracat

toracat

2014-12-29 07:42

manager   ~0022058

The distro kernel cannot be patched because it is a bug-for-bug rebuild of upstream (RHEL) kernel. You need to file a bug report upstream at http://bugzilla.redhat.com . In the mean time, the patch can be added to the centosplus kernel.
toracat

toracat

2015-01-29 01:32

manager   ~0022249

kernel-3.10.0-123.20.1.el7 is out. Applying the patch resulted in a compilation error. Therefore it was not included in this update. Pending investigation.
grossws

grossws

2015-01-29 09:35

reporter   ~0022257

I'll check and fix it soon
grossws

grossws

2015-02-24 17:22

reporter   ~0022396

Sorry for delay. This patch is present in kernel-3.10.0-123.20.1.el7.
toracat

toracat

2015-02-24 17:44

manager   ~0022397

Hrm, I don't really see the referenced patch in kernel-3.10.0-123.20.1.el7. Do you mean the reported bug has been fixed in this kernel? If so, can we close this bug report?
grossws

grossws

2015-02-24 18:02

reporter   ~0022398

It was fixed merged into mainline kernel long ago (June, 2014), see https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=945b2b2d259d1a4364a2799e80e8ff32f8c6ee6f

It was fixed in Fedora 20 core kernel-3.15.3-200.fc20, it also was in 3.16 mainline kernel.

To check it presence in kernel-3.10.0-123.20.1.el7 you can prepare its build from srpm, run rpmbuild -bp kernel.spec and look at BUILD/*/*/net/netfilter/nf_nat_core.c at nf_nat_proto_clean (string num about 520).
grossws

grossws

2015-02-24 18:03

reporter   ~0022399

And this bug can be closed, since patch is present in current build
toracat

toracat

2015-02-24 23:30

manager   ~0022403

Closing as 'resolved' as per the submitter's note.

Issue History

Date Modified Username Field Change
2014-12-29 03:02 grossws New Issue
2014-12-29 07:42 toracat Note Added: 0022058
2014-12-29 07:43 toracat Status new => assigned
2015-01-29 01:32 toracat Note Added: 0022249
2015-01-29 09:35 grossws Note Added: 0022257
2015-02-24 17:22 grossws Note Added: 0022396
2015-02-24 17:44 toracat Note Added: 0022397
2015-02-24 17:45 toracat Status assigned => feedback
2015-02-24 18:02 grossws Note Added: 0022398
2015-02-24 18:02 grossws Status feedback => assigned
2015-02-24 18:03 grossws Note Added: 0022399
2015-02-24 23:30 toracat Note Added: 0022403
2015-02-24 23:30 toracat Status assigned => resolved
2015-02-24 23:30 toracat Resolution open => fixed