View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0018287||CentOS-8||-OTHER||public||2021-08-24 07:47||2021-08-26 15:05|
|Summary||0018287: WAAgentLinux package outputs Python errors to log, filling disk and making machine unusable|
Apologies if this has been filed before - I couldn't see anything listed.
There is seemingly an issue with the WAAgentLinux package that provides Azure utilities for CentOS.
On multiple Azure Linux VMs, the boot drives fill up to the point where the machine is unusable, because of the /var/log/waagent.log file becoming absolutely massive (~15GB on one machine). It seems that it cannot find a Python module, and re-attempts to run a process continuously.
The output printed to the log file is similar to the below:
2021-08-23T08:00:21.856964Z INFO Daemon Daemon Installed Agent WALinuxAgent-188.8.131.52 is the most current agent
2021-08-23T08:00:21.857200Z WARNING Daemon Daemon Agent WALinuxAgent-184.108.40.206 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined
2021-08-23T08:00:21.857406Z ERROR Daemon Daemon Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-220.127.116.11 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/azurelinuxagent/ga/update.py", line 156, in run_latest
cmds = get_python_cmd()
NameError: name 'get_python_cmd' is not defined
Distro and WALinuxAgent details (printed from 'waagent --version'):
WALinuxAgent-18.104.22.168 running on centos 8.4.2105
Goal state agent: 22.214.171.124
I logged this as a bug with the WAAgentLinux developers:
and they think it's a packaging error - in creating the 126.96.36.199 version, the call to 'get_python_cmd()' should have been removed (see comment https://github.com/Azure/WALinuxAgent/issues/2336#issuecomment-904027245 ).
Other users have successfully worked around the issue by downgrading the package to version 188.8.131.52.
I'd be grateful if you could look into this. Thanks for your work on CentOS!
|Steps To Reproduce||1. Install WAAgentLinux-184.108.40.206 from the OpenLogic repositories: http://olcentgbl.trafficmanager.net/openlogic/8/openlogic/x86_64/|
2. Leave the service running and wait. The /var/log/waagent.log file will repeatedly fill with multiple Python errors as described in the Description field here, to the point where the disk will fill up faster than Logrotate can archive the log files.
|Tags||No tags attached.|
Note on installed package versions that exhibit the issue:
$ sudo rpm -qa | grep -i WALinuxAgent
FYI, WALinuxAgent 220.127.116.11-1_ol001 is not a package published by CentOS or Red Hat. I was requested, by Microsoft, to publish this update in our OpenLogic repos.
The el8 version of the package contained an error that was not caught during testing, but I will be releasing an ol002 version of the package shortly to remedy the issue.
The el7 version of the package is not affected and does not require an update.
|WALinuxAgent 18.104.22.168-1_ol002.el8 was released yesterday and I have received confirmation that it resolved the issue.|
|This is resolved as of non-centos package ending in 002.|
|2021-08-24 07:47||capncodewash||New Issue|
|2021-08-24 07:51||capncodewash||Note Added: 0038590|
|2021-08-24 20:11||N3WWN||Note Added: 0038591|
|2021-08-26 14:13||N3WWN||Note Added: 0038592|
|2021-08-26 15:05||Evolution||Status||new => resolved|
|2021-08-26 15:05||Evolution||Resolution||open => fixed|
|2021-08-26 15:05||Evolution||Note Added: 0038593|