View Issue Details

IDProjectCategoryView StatusLast Update
0015528CentOS-7bindpublic2018-12-11 20:48
Reportermhagen94 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
PlatformCentOSOS7OS Version7.6.1810
Product Version 
Target VersionFixed in Version 
Summary0015528: BIND 9.9.4 keeps crashing with `buffer.c:420: REQUIRE(l <= ((b)->length - (b)->used))`
DescriptionSo i'm trying to understand why but bind just randomly keeps crashing on me.
There is no good pattern to it. It crashed with some websites. (https://bugs.launchpad.net/)

Version: BIND 9.9.4-RedHat-9.9.4-72.el7 (Extended Support Version)

:: gdb coredump
(gdb) bt
#0 0x00007fd50f896207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fd50f8978f8 in __GI_abort () at abort.c:90
#2 0x0000557cdb7b925e in assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>,
    cond=<optimized out>) at ./main.c:222
#3 0x00007fd510d1731a in isc_assertion_failed (file=file@entry=0x7fd510d54a48 "buffer.c", line=line@entry=420,
    type=type@entry=isc_assertiontype_require, cond=cond@entry=0x7fd510d54cb0 "l <= ((b)->length - (b)->used)")
    at assertions.c:57
#4 0x00007fd510d19b10 in isc__buffer_putstr (b=b@entry=0x7fd50cf68070, source=source@entry=0x7fd5124ecafe ";") at buffer.c:420
#5 0x00007fd5123d25e7 in rdataset_totext (rdataset=rdataset@entry=0x7fd50860d320, owner_name=owner_name@entry=0x7fd50860c150,
    ctx=ctx@entry=0x7fd50cf67c10, omit_final_dot=omit_final_dot@entry=isc_boolean_false, target=target@entry=0x7fd50cf68070)
    at masterdump.c:454
#6 0x00007fd5123d4133 in dns_master_rdatasettotext (owner_name=0x7fd50860c150, rdataset=rdataset@entry=0x7fd50860d320,
    style=style@entry=0x7fd5124ed120 <dns_master_style_comment>, target=target@entry=0x7fd50cf68070) at masterdump.c:708
#7 0x00007fd5123dced8 in dns_message_sectiontotext (msg=msg@entry=0x7fd5086081d8, section=section@entry=2,
    style=style@entry=0x7fd5124ed120 <dns_master_style_comment>, flags=flags@entry=0, target=target@entry=0x7fd50cf68070)
    at message.c:3296
#8 0x00007fd5123ddbea in dns_message_totext (msg=msg@entry=0x7fd5086081d8,
    style=style@entry=0x7fd5124ed120 <dns_master_style_comment>, flags=flags@entry=0, target=target@entry=0x7fd50cf68070)
    at message.c:3520
#9 0x00007fd5123ddf6f in dns_message_logfmtpacket (message=message@entry=0x7fd5086081d8,
    description=description@entry=0x7fd5124fa453 "received packet:\n", category=0x7fd512735bf0 <dns_categories+80>,
    module=0x7fd512735b60 <dns_modules+480>, style=0x7fd5124ed120 <dns_master_style_comment>, level=level@entry=10,
    mctx=0x557cdc9031d0) at message.c:3615
#10 0x00007fd5124689c6 in resquery_response (task=0x7fd508ff2190, event=<optimized out>) at resolver.c:6977
#11 0x00007fd510d3a276 in dispatch (manager=0x7fd512b19010) at task.c:1116
#12 run (uap=0x7fd512b19010) at task.c:1286
#13 0x00007fd5108eadd5 in start_thread (arg=0x7fd50cf69700) at pthread_create.c:307
#14 0x00007fd50f95dead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


:: general.log
04-Dec-2018 20:06:34.873 buffer.c:420: REQUIRE(l <= ((b)->length - (b)->used)) failed, back trace
04-Dec-2018 20:06:34.873 #0 0x558401376090 in ??
04-Dec-2018 20:06:34.873 #1 0x7fd86798f31a in ??
04-Dec-2018 20:06:34.873 #2 0x7fd867991b10 in ??
04-Dec-2018 20:06:34.873 #3 0x7fd86904a5e7 in ??
04-Dec-2018 20:06:34.873 #4 0x7fd86904c133 in ??
04-Dec-2018 20:06:34.873 #5 0x7fd869054ed8 in ??
04-Dec-2018 20:06:34.873 #6 0x7fd869055bea in ??
04-Dec-2018 20:06:34.873 #7 0x7fd869055f6f in ??
04-Dec-2018 20:06:34.873 #8 0x7fd8690e09c6 in ??
04-Dec-2018 20:06:34.873 #9 0x7fd8679b2276 in ??
04-Dec-2018 20:06:34.873 #10 0x7fd867562dd5 in ??
04-Dec-2018 20:06:34.873 #11 0x7fd8665d5ead in ??
04-Dec-2018 20:06:34.873 exiting (due to assertion failure)
Steps To ReproduceI have no clue, updated from BIND 9.9.4-61.el7_5.1.x86_64 and kept crashing.

Tags7.6, bind, crash, security
abrt_hash
URL

Activities

mhagen94

mhagen94

2018-12-04 20:15

reporter   ~0033212

JFTR RedHat folks know and they are working on the fix already.
johnny2k5

johnny2k5

2018-12-06 11:14

reporter   ~0033247

I'm also experiencing this crash after upgrade from 7.5 to 7.6, it's really a shame that this kind of problems make it to a stable release.

SHAME ON REDHAT!

@mhagen94: do you have a link to the upstream bug issue?
kevbo

kevbo

2018-12-07 03:52

reporter   ~0033257

Same problem here. This is....really bad. I had to figure out how to get systemd to restart the darn thing when it died, and set a cron job to delete the core files, because all of them made my box run out of space.

The link to the upstream bug appears to be here:

https://access.redhat.com/solutions/3683641

I can't see much on that page. Troubling, however, is the date: "Updated November 9 2018 at 8:20 PM"

How is this not biting everyone who uses Bind? Maybe there just aren't that many of us.
jrd

jrd

2018-12-10 06:51

manager   ~0033290

Would you be willing to share your named.conf, redacting keys as necessary?
andrew918

andrew918

2018-12-10 18:02

reporter   ~0033297

We've seen this exact same issue on 3 servers that have been updated to CentOS 7.6. Two of them are authoritative name servers with 14K zones and after several systemd restarts they manage to reach a stable state and will remain functioning without issue until restarted again.

The 3rd server is the most problematic and is a private DNS resolver only. It does receive a lot of queries, mainly RBL lookups for spam filtering. The configuration is basically the default with a few additional logging options. As it turns out, disabling those logs appears to resolve the crashing. Before this change, systemd was having to restart the service many times per minute. Now it has remained stable for two 15 minutes periods with a manual restart in between. I've attached the named.conf where you can see the commented logging.

The other two impact servers have similar additional logging configured as well. Since they've remained stable for the past 10 hours I won't be touching them to evaluate this possible fix.

named.conf (1,771 bytes)
TrevorH

TrevorH

2018-12-10 19:14

manager   ~0033298

Do any of you run rndc dumpdb on a regular basis?
andrew918

andrew918

2018-12-10 19:19

reporter   ~0033299

No @TrevorH.
kevbo

kevbo

2018-12-10 19:53

reporter   ~0033300

Here is my config. Also very simple. _Also_ has additional logging. I need the querylog.

This is a simple, small setup. There are a couple of extra zones for local names, and then some zones that I redirect to 127.0.0.1.

I could attempt to turn off the logging, but honestly, the logging is required.

I've also run with this config for years.

With this exercise, I diffed my config to the rpm's current default. The only option I had in my config that seems to no longer be needed was "dnssec-lookaside auto;". I've removed that, but I'm still seeing the crashing.

Thank you.

named-2.conf (2,766 bytes)
pgreco

pgreco

2018-12-10 19:58

developer   ~0033301

We're working on a temporary fix until we get an official one from RH. I hope to have a test build available tomorrow
mhagen94

mhagen94

2018-12-10 22:44

reporter   ~0033305

"Do any of you run rndc dumpdb on a regular basis?" No

So i "fixed" my issue just to force systemctl to restart named when it crashes. It happens when local computers/servers do a query's for some websites.
This is my experience.

Attached logging.conf and named.conf
For long time i have been using this log format, just copied somewhere from the internet but has everything i need, and maybe bit more (don't judge)

My setup contains a primary local/public DNS server and a offsite slave DNS.
Using views on the primary for internal clients/server

logging.conf (2,924 bytes)
named-3.conf (2,333 bytes)
pgreco

pgreco

2018-12-11 16:39

developer   ~0033311

Well, we have a test build here https://buildlogs.centos.org/c7.1810.u.x86_64/bind/20181211154620/9.9.4-72.el7.0.1.x86_64/
Please use it if you can, with your default config.

Thanks, Pablo.
koons5159

koons5159

2018-12-11 18:34

reporter   ~0033312

I experienced the same issue today after performing a system update. I installed the test build and so far so good. Haven't crashed yet.
mhagen94

mhagen94

2018-12-11 20:17

reporter   ~0033314

Same result here, installed it on my local dev dns (which has the same issue and config) and no crashes or errors so far.
kevbo

kevbo

2018-12-11 20:48

reporter   ~0033315

I have installed it here: no problems so far. I will keep monitoring it.

Issue History

Date Modified Username Field Change
2018-12-04 19:21 mhagen94 New Issue
2018-12-04 19:21 mhagen94 Tag Attached: 7.6
2018-12-04 19:21 mhagen94 Tag Attached: bind
2018-12-04 19:21 mhagen94 Tag Attached: crash
2018-12-04 19:41 mhagen94 Tag Attached: security
2018-12-04 20:15 mhagen94 Note Added: 0033212
2018-12-06 11:14 johnny2k5 Note Added: 0033247
2018-12-07 03:52 kevbo Note Added: 0033257
2018-12-10 06:51 jrd Note Added: 0033290
2018-12-10 18:02 andrew918 File Added: named.conf
2018-12-10 18:02 andrew918 Note Added: 0033297
2018-12-10 19:14 TrevorH Note Added: 0033298
2018-12-10 19:19 andrew918 Note Added: 0033299
2018-12-10 19:53 kevbo File Added: named-2.conf
2018-12-10 19:53 kevbo Note Added: 0033300
2018-12-10 19:58 pgreco Note Added: 0033301
2018-12-10 22:44 mhagen94 File Added: logging.conf
2018-12-10 22:44 mhagen94 File Added: named-3.conf
2018-12-10 22:44 mhagen94 Note Added: 0033305
2018-12-11 16:39 pgreco Note Added: 0033311
2018-12-11 18:34 koons5159 Note Added: 0033312
2018-12-11 20:17 mhagen94 Note Added: 0033314
2018-12-11 20:48 kevbo Note Added: 0033315