View Issue Details

IDProjectCategoryView StatusLast Update
0015977CentOS-7crontabspublic2019-04-02 18:45
Reporterbuchsk 
PrioritynormalSeveritymajorReproducibilityrandom
Status newResolutionopen 
Platformgnu/linux x64OSCentOSOS Version6.* 7.*
Product Version 
Target VersionFixed in Version 
Summary0015977: /usr/bin/run-parts has a fatal programming error which can cause cron to hang perpetually
DescriptionI have seen the cron hang due to this error on CentOS, RHEL and Amazon Linux. The programming error appears to be common.

This run-parts script is part of the crontabs package. It is used to execute the cron.hourly, cron.daily, etc. actions.

*_Problem:_*
run-parts is a shell script which contains this pipeline at its heart (where $i is the command from, e.g. "/etc/cron.daily/logrotate", to be executed):
{code}
$i 2>&1 | awk -v "progname=$i" \
                              'progname {
                                   print progname ":\n"
                                   progname="";
                               }
                               { print; }'
{code}

That awk script is horrendous example of programming. It defines a variable *progname* on the command line, e.g. logrotate. The script also defines a function of the same name. In that AWK function, the AWK function is deleted. That is a race condition.

The purpose of this code is to echo the progname initially once and then from then on just echo its input. To keep it as an awk script, it should be the following:
{code}
$i 2>&1 | awk -v "progname=$i" \
                              'BEGIN { print progname ":\n" }
                               { print; }'
{code}

However, there is no need to add the additional burden of awk having to echo each line. This would work just fine:
{code}
echo -e "$i:\n"
$i
{code}

I posted this in regard to Amazon Linux in the appropriate AWS forum, but got no response. I know they should have escalated it upstream.
Steps To ReproduceHave cron running the daily, hourly, etc. parts using run-parts. Some day, you may see the race condition come out. I have three cases over the last 12 month, so it is rare. But, that code is terrible, so why not change it just because of that!
Additional InformationHere is what the processes look like when the race condition is hit:

ps axwu|grep cron
root 1793 0.0 0.0 116912 1188 ? Ss 2018 3:21 crond
root 12003 0.0 0.0 103328 860 pts/2 S+ 13:33 0:00 grep cron
root 14361 0.0 0.0 19052 948 ? Ss 2018 0:00 /usr/sbin/anacron -s
root 16875 0.0 0.0 106112 1268 ? SN 2018 0:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 16887 0.0 0.0 105972 948 ? SN 2018 0:00 awk -v progname=/etc/cron.daily/logrotate progname {????? print progname ":\n"????? progname="";???? }???? { print; }

The awk process never finishes (it is trying to find the function it was trying to run, I guess it gets lost when the function is deleted. Today is April 2, 2019 and this has been hung since Dec 21, 2018.

The process of running awk seems to have gotten nowhere:

# ps -p $pid H -www -o pid,cputime,state,lstart,time,etime,cmd
  PID TIME S STARTED TIME ELAPSED CMD
16887 00:00:00 S Fri Dec 21 14:13:01 2018 00:00:00 101-22:45:16 awk -v progname=/etc/cron.daily/logrotate progname {????? print progname ":\n"????? progname="";???? }???? { print; }

TagsNo tags attached.
abrt_hash
URL

Activities

buchsk

buchsk

2019-04-02 18:01

reporter  

Aws_Forum_Post (2,108 bytes)
Amazon Linux - run-parts command has a race condition in awk
Tags: AMI Linux EC2 Bug

*_Environment:_*
# cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.09"
PRETTY_NAME="Amazon Linux AMI 2017.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
# yum provides run-parts
Loaded plugins: priorities, update-motd, upgrade-helper
1039 packages excluded due to repository priority protections
crontabs-1.10-33.9.amzn1.noarch : Root crontab files used to schedule the execution of programs
Repo        : amzn-main
Matched from:
Filename    : /usr/bin/run-parts
crontabs-1.10-33.9.amzn1.noarch : Root crontab files used to schedule the execution of programs
Repo        : installed
Matched from:
Filename    : /usr/bin/run-parts

*_Problem:_*
run-parts is a shell script which contains this pipeline at its heart (where $i is the command from, e.g. /etc/cron.daily/logrotate, to be executed):
{code}
$i 2>&1 | awk -v "progname=$i" \
                              'progname {
                                   print progname ":\n"
                                   progname="";
                               }
                               { print; }'
{code}

That awk script is horrendous example of programming. It defines a variable *progname* on the command line, e.g. logrotate. The script defines a function of the same name. In that AWS function the AWK function is deleted. That is a race condition. The purpose is to echo the progname initially once and then from then on just echo its input. To keep it as an awk script, it should be the following:
{code}
$i 2>&1 | awk -v "progname=$i" \
                              'BEGIN { print progname ":\n" }
                               { print; }'
{code}

However, there is no need to add the additional burden of awk having to echo each line. This would work just fine:
{code}
echo -e "$i:\n"
$i
{code}

Please consider this a bug. I realize it may be necessary to push it upstream, but that is why you get paid the big bucks :-)
Aws_Forum_Post (2,108 bytes)
TrevorH

TrevorH

2019-04-02 18:45

manager   ~0034154

CentOS is a rebuild of the sources used to create RHEL. We do not modify anything except to remove branding and logos. You will need to submit your request to Redhat via bugzilla.redhat.com and if/when RH accepts it and incorporate it into RHEL and releases the patched version, then CentOS will pick it up and rebuild it.

Issue History

Date Modified Username Field Change
2019-04-02 18:01 buchsk New Issue
2019-04-02 18:01 buchsk File Added: Aws_Forum_Post
2019-04-02 18:45 TrevorH Note Added: 0034154