View Issue Details

IDProjectCategoryView StatusLast Update
0018066CentOS-7sudopublic2021-02-13 12:09
Reporterhoudinifxtd Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.7-1908 
Summary0018066: Sudo has 25 second delay when used in user data. Default network config is incorrect on boot for ami-0b2045146eb00b617
DescriptionWhen I try to use the Centos 7 AMI in ap-southeast-2 for AWS it has problems without any modification.

I can see 25 second delays every time a script may run that uses sudo in user data which slows down the ability to scale up a spot fleet.

2021-02-13 11:03:06 test sudo delay
2021-02-13 11:03:06 no sudo
sudo
2021-02-13 11:03:31 sudo
2021-02-13 11:03:31 no sudo
sudo
2021-02-13 11:03:56 sudo
2021-02-13 11:03:56 no sudo

This may be related, I also see network interfaces which do not exist also have remnant configuration files from the AMI build. deleting this file and rebuilding the ami has no effect. ifcfg-eth0 should not exist, there is no interface matching this name.
[root@ip-10-4-101-166 network-scripts]# ls
ifcfg-ens5 ifdown ifdown-ippp ifdown-post ifdown-sit ifdown-tunnel ifup-bnep ifup-ipv6 ifup-plusb ifup-routes ifup-TeamPort init.ipv6-global
ifcfg-eth0 ifdown-bnep ifdown-ipv6 ifdown-ppp ifdown-Team ifup ifup-eth ifup-isdn ifup-post ifup-sit ifup-tunnel network-functions
ifcfg-lo ifdown-eth ifdown-isdn ifdown-routes ifdown-TeamPort ifup-aliases ifup-ippp ifup-plip ifup-ppp ifup-Team ifup-wireless network-functions-ipv6
[root@ip-10-4-101-166 network-scripts]# ifconfig
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
        inet 10.4.101.166 netmask 255.255.255.0 broadcast 10.4.101.255
        inet6 fe80::60:d6ff:fee7:5d2 prefixlen 64 scopeid 0x20<link>
        ether 02:60:d6:e7:05:d2 txqueuelen 1000 (Ethernet)
        RX packets 9264 bytes 13208806 (12.5 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 3070 bytes 222673 (217.4 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 108 bytes 9740 (9.5 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 108 bytes 9740 (9.5 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I have read elsewhere that /etc/hosts must be configured properly and I believe this is valid from my user data:
[root@ip-10-4-101-166 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 ip-10-4-101-166.ap-southeast-2.compute.internal ip-10-4-101-166
Steps To ReproduceStart the ami with this user data:
#!/bin/bash

set -e

# Send the log output from this script to user-data.log, syslog, and the console
# From: https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# Log the given message. All logs are written to stderr with a timestamp.
function log {
 local -r message="$1"
 local -r timestamp=$(date +"%Y-%m-%d %H:%M:%S")
 >&2 echo -e "$timestamp $message"
}

function has_yum {
  [[ -n "$(command -v yum)" ]]
}

if $(has_yum); then
    hostname=$(hostname -s) # in centos, failed dns lookup can cause commands to slowdown
    echo "127.0.0.1 $hostname.${aws_domain} $hostname" | tee -a /etc/hosts
fi

log "hostname: $(hostname)"
log "hostname: $(hostname -f) $(hostname -s)"

log "test sudo delay"
log "no sudo"
sudo echo "sudo"
log "sudo"
log "no sudo"
sudo echo "sudo"
log "sudo"
log "no sudo"
Additional InformationI am using terraform to attach an EIP, which I can provide a snippet for:

resource "aws_eip" "bastionip" {
  count = var.create_vpc ? 1 : 0
  vpc = true
  instance = aws_instance.bastion[count.index].id
}

resource "aws_instance" "bastion" {
  count = var.create_vpc ? 1 : 0
  ami = var.bastion_ami_id
  instance_type = var.instance_type
  key_name = var.aws_key_name
  subnet_id = tolist(var.public_subnet_ids)[0]

  vpc_security_group_ids = local.vpc_security_group_ids

  root_block_device {
    delete_on_termination = true
  }
  user_data = data.template_file.user_data_auth_client.rendered
}

data "template_file" "user_data_auth_client" {
  template = file("${path.module}/user-data-broken.sh")
}
TagsNo tags attached.
abrt_hash
URL

Activities

houdinifxtd

houdinifxtd

2021-02-13 12:09

reporter   ~0038244

I forgot to add that the dig command can resolve the hostname correctly so if DNS is the cause of slowing down sudo (as I see when I google the issue), it doesn't seem to help here.

[root@ip-10-4-101-166 network-scripts]# dig $(hostname)

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.3 <<>> ip-10-4-101-166.ap-southeast-2.compute.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8755
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ip-10-4-101-166.ap-southeast-2.compute.internal. IN A

;; ANSWER SECTION:
ip-10-4-101-166.ap-southeast-2.compute.internal. 60 IN A 10.4.101.166

;; Query time: 0 msec
;; SERVER: 10.4.0.2#53(10.4.0.2)
;; WHEN: Sat Feb 13 11:29:45 UTC 2021
;; MSG SIZE rcvd: 81

Issue History

Date Modified Username Field Change
2021-02-13 11:25 houdinifxtd New Issue
2021-02-13 12:09 houdinifxtd Note Added: 0038244