View Issue Details

IDProjectCategoryView StatusLast Update
0009680CentOS-6ntppublic2019-03-03 18:47
ReporterJiriHorky 
PrioritynormalSeveritymajorReproducibilitysometimes
Status closedResolutionno change required 
PlatformOSCentOS 6OS Version6.7
Product Version 
Target VersionFixed in Version 
Summary0009680: 1.centos.pool.ntp.org resolves to just one offline server in Mexico from time to time
DescriptionWe run our servers in ~30 datacenters around the world and we recently came across a strange issue in Mexico. From time to time, the 1.centos.ntp.org domain returns just one IP instead of usual 4. Moreover, that IP does not provide NTP service or is completely down.
The issue does not appear at any of our others DCs, so BGP and/or DNS GeoBalancing may be involved.

The dig +trace output is attached below. The issue ussually lasts around 20minutes. If ntpd is restarted in this time with just 1.centos.pool.ntp.org configuration, it won't work (as version of ntpd does resolving only during startup).

I am happy to provide more information, if needed.

--------

Sat 31 Oct 2015 06:52:21 AM CET

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.4 <<>> +trace 1.centos.pool.ntp.org
;; global options: +cmd
. 518400 IN NS k.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS a.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS f.root-servers.net.
. 518400 IN NS h.root-servers.net.
;; Received 496 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

org. 172800 IN NS d0.org.afilias-nst.org.
org. 172800 IN NS b2.org.afilias-nst.org.
org. 172800 IN NS b0.org.afilias-nst.org.
org. 172800 IN NS c0.org.afilias-nst.info.
org. 172800 IN NS a2.org.afilias-nst.info.
org. 172800 IN NS a0.org.afilias-nst.info.
;; Received 441 bytes from 192.58.128.30#53(192.58.128.30) in 25 ms

ntp.org. 86400 IN NS ns2.ntp.org.
ntp.org. 86400 IN NS ns1.ntp.org.
ntp.org. 86400 IN NS dns2.udel.edu.
ntp.org. 86400 IN NS dns1.udel.edu.
;; Received 153 bytes from 199.19.56.1#53(199.19.56.1) in 194 ms

pool.ntp.org. 3600 IN NS f.ntpns.org.
pool.ntp.org. 3600 IN NS h.ntpns.org.
pool.ntp.org. 3600 IN NS e.ntpns.org.
pool.ntp.org. 3600 IN NS i.ntpns.org.
pool.ntp.org. 3600 IN NS b.ntpns.org.
pool.ntp.org. 3600 IN NS c.ntpns.org.
pool.ntp.org. 3600 IN NS g.ntpns.org.
pool.ntp.org. 3600 IN NS d.ntpns.org.
pool.ntp.org. 3600 IN NS a.ntpns.org.
;; Received 189 bytes from 149.20.68.4#53(149.20.68.4) in 138 ms

1.centos.pool.ntp.org. 150 IN A 132.248.30.3
;; Received 76 bytes from 174.127.124.192#53(174.127.124.192) in 50 ms

Steps To Reproducedig +trace 1.centos.pool.ntp.org
TagsNo tags attached.

Activities

range

range

2015-11-01 15:46

administrator   ~0024758

Last edited: 2015-11-01 15:47

View 2 revisions

There's actually not that much we can do about this - these names are assigned to projects by http://www.pool.ntp.org/ and they also are the ones running the mechanism which assigns ip addresses to those names.

I don't see any support venue on their site (except for the mailing lists), so I am not sure how to notfiy them about this.

On the other hand you probably should keep all 4 centos.pool.ntp.org hostnames in your ntp or chrony configuration - probably exactly because of issues like these.

JiriHorky

JiriHorky

2015-11-03 14:03

reporter   ~0024771

I have modified our script and did a recursive query for all the centos ntp pool servers while the issue is occurring, see the attached log file. The result is that for all the {0..3}.centos.pool.ntp.org we get:

 150 IN A 132.248.30.3

So the issue is related to all the pool hostnames. Since the hostnames says "centos" in them, and the pool servers are configured like that by default, would you try to reach the ntp.org administrator please?
Don't hesitate to CC me, as I am willing to add more information if needed.
JiriHorky

JiriHorky

2015-11-03 14:03

reporter  

centos.pool.txt (10,373 bytes)

Tue 03 Nov 2015 02:31:32 PM CET 0.centos.pool.ntp.org
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.4 <<>> +trace 0.centos.pool.ntp.org
;; global options: +cmd
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
;; Received 500 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
;; Received 441 bytes from 199.7.83.42#53(199.7.83.42) in 58 ms

ntp.org.                86400   IN      NS      dns2.udel.edu.
ntp.org.                86400   IN      NS      dns1.udel.edu.
ntp.org.                86400   IN      NS      ns1.ntp.org.
ntp.org.                86400   IN      NS      ns2.ntp.org.
;; Received 153 bytes from 199.19.57.1#53(199.19.57.1) in 59 ms

pool.ntp.org.           3600    IN      NS      g.ntpns.org.
pool.ntp.org.           3600    IN      NS      c.ntpns.org.
pool.ntp.org.           3600    IN      NS      a.ntpns.org.
pool.ntp.org.           3600    IN      NS      i.ntpns.org.
pool.ntp.org.           3600    IN      NS      b.ntpns.org.
pool.ntp.org.           3600    IN      NS      e.ntpns.org.
pool.ntp.org.           3600    IN      NS      f.ntpns.org.
pool.ntp.org.           3600    IN      NS      h.ntpns.org.
pool.ntp.org.           3600    IN      NS      d.ntpns.org.
;; Received 189 bytes from 128.175.13.16#53(128.175.13.16) in 82 ms

0.centos.pool.ntp.org.  150     IN      A       132.248.30.3
;; Received 76 bytes from 46.29.176.73#53(46.29.176.73) in 335 ms


; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.4 <<>> +trace 1.centos.pool.ntp.org
;; global options: +cmd
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
;; Received 500 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
;; Received 441 bytes from 199.7.83.42#53(199.7.83.42) in 58 ms

ntp.org.                86400   IN      NS      ns1.ntp.org.
ntp.org.                86400   IN      NS      ns2.ntp.org.
ntp.org.                86400   IN      NS      dns1.udel.edu.
ntp.org.                86400   IN      NS      dns2.udel.edu.
;; Received 153 bytes from 199.249.112.1#53(199.249.112.1) in 45 ms

pool.ntp.org.           3600    IN      NS      i.ntpns.org.
pool.ntp.org.           3600    IN      NS      g.ntpns.org.
pool.ntp.org.           3600    IN      NS      c.ntpns.org.
pool.ntp.org.           3600    IN      NS      e.ntpns.org.
pool.ntp.org.           3600    IN      NS      a.ntpns.org.
pool.ntp.org.           3600    IN      NS      h.ntpns.org.
pool.ntp.org.           3600    IN      NS      f.ntpns.org.
pool.ntp.org.           3600    IN      NS      b.ntpns.org.
pool.ntp.org.           3600    IN      NS      d.ntpns.org.
;; Received 189 bytes from 149.20.68.3#53(149.20.68.3) in 57 ms

1.centos.pool.ntp.org.  150     IN      A       132.248.30.3

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.4 <<>> +trace 2.centos.pool.ntp.org
;; global options: +cmd
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
;; Received 500 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
;; Received 441 bytes from 202.12.27.33#53(202.12.27.33) in 160 ms

ntp.org.                86400   IN      NS      ns1.ntp.org.
ntp.org.                86400   IN      NS      ns2.ntp.org.
ntp.org.                86400   IN      NS      dns1.udel.edu.
ntp.org.                86400   IN      NS      dns2.udel.edu.
;; Received 153 bytes from 199.249.120.1#53(199.249.120.1) in 46 ms

pool.ntp.org.           3600    IN      NS      b.ntpns.org.
pool.ntp.org.           3600    IN      NS      h.ntpns.org.
pool.ntp.org.           3600    IN      NS      i.ntpns.org.
pool.ntp.org.           3600    IN      NS      g.ntpns.org.
pool.ntp.org.           3600    IN      NS      f.ntpns.org.
pool.ntp.org.           3600    IN      NS      c.ntpns.org.
pool.ntp.org.           3600    IN      NS      a.ntpns.org.
pool.ntp.org.           3600    IN      NS      e.ntpns.org.
pool.ntp.org.           3600    IN      NS      d.ntpns.org.
;; Received 189 bytes from 149.20.68.3#53(149.20.68.3) in 57 ms

2.centos.pool.ntp.org.  150     IN      A       132.248.30.3

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.4 <<>> +trace 3.centos.pool.ntp.org
;; global options: +cmd
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
;; Received 500 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
;; Received 441 bytes from 199.7.91.13#53(199.7.91.13) in 45 ms

ntp.org.                86400   IN      NS      ns1.ntp.org.
ntp.org.                86400   IN      NS      ns2.ntp.org.
ntp.org.                86400   IN      NS      dns1.udel.edu.
ntp.org.                86400   IN      NS      dns2.udel.edu.
;; Received 153 bytes from 199.249.112.1#53(199.249.112.1) in 46 ms

pool.ntp.org.           3600    IN      NS      e.ntpns.org.
pool.ntp.org.           3600    IN      NS      b.ntpns.org.
pool.ntp.org.           3600    IN      NS      d.ntpns.org.
pool.ntp.org.           3600    IN      NS      i.ntpns.org.
pool.ntp.org.           3600    IN      NS      a.ntpns.org.
pool.ntp.org.           3600    IN      NS      h.ntpns.org.
pool.ntp.org.           3600    IN      NS      f.ntpns.org.
pool.ntp.org.           3600    IN      NS      g.ntpns.org.
pool.ntp.org.           3600    IN      NS      c.ntpns.org.
;; Received 189 bytes from 149.20.68.4#53(149.20.68.4) in 57 ms

3.centos.pool.ntp.org.  150     IN      A       132.248.30.3
;; Received 76 bytes from 50.31.240.214#53(50.31.240.214) in 172 ms
centos.pool.txt (10,373 bytes)
JiriHorky

JiriHorky

2015-11-14 22:29

reporter   ~0024845

Ok, so I reached the ntp list myself. The problem was there was just one or two servers in the pool for mexico, which became suddenly overloaded when announced. Once it went down, servers from generic pool was returned. This helped the Mexico NTP servers so they were returned back to the pool and the situation repeated.
There is now ongoing discussion on the list how to fix the situation in other countries as well.

Issue History

Date Modified Username Field Change
2015-10-31 07:38 JiriHorky New Issue
2015-11-01 15:46 range Note Added: 0024758
2015-11-01 15:47 range Note Edited: 0024758 View Revisions
2015-11-03 14:03 JiriHorky Note Added: 0024771
2015-11-03 14:03 JiriHorky File Added: centos.pool.txt
2015-11-14 22:29 JiriHorky Note Added: 0024845
2019-03-03 18:47 avij Status new => closed
2019-03-03 18:47 avij Resolution open => no change required