View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0018287 | CentOS-8 | -OTHER | public | 2021-08-24 07:47 | 2021-08-26 15:05 |
Reporter | capncodewash | Assigned To | |||
Priority | normal | Severity | minor | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Product Version | 8.4.2105 | ||||
Summary | 0018287: WAAgentLinux package outputs Python errors to log, filling disk and making machine unusable | ||||
Description | Hi there, Apologies if this has been filed before - I couldn't see anything listed. There is seemingly an issue with the WAAgentLinux package that provides Azure utilities for CentOS. On multiple Azure Linux VMs, the boot drives fill up to the point where the machine is unusable, because of the /var/log/waagent.log file becoming absolutely massive (~15GB on one machine). It seems that it cannot find a Python module, and re-attempts to run a process continuously. The output printed to the log file is similar to the below: 2021-08-23T08:00:21.856964Z INFO Daemon Daemon Installed Agent WALinuxAgent-2.3.1.1 is the most current agent 2021-08-23T08:00:21.857200Z WARNING Daemon Daemon Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined 2021-08-23T08:00:21.857406Z ERROR Daemon Daemon Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/azurelinuxagent/ga/update.py", line 156, in run_latest cmds[0] = get_python_cmd() NameError: name 'get_python_cmd' is not defined Distro and WALinuxAgent details (printed from 'waagent --version'): WALinuxAgent-2.3.1.1 running on centos 8.4.2105 Python: 3.6.8 Goal state agent: 2.3.1.1 I logged this as a bug with the WAAgentLinux developers: https://github.com/Azure/WALinuxAgent/issues/2336 and they think it's a packaging error - in creating the 2.3.1.1 version, the call to 'get_python_cmd()' should have been removed (see comment https://github.com/Azure/WALinuxAgent/issues/2336#issuecomment-904027245 ). Other users have successfully worked around the issue by downgrading the package to version 2.2.49.2. I'd be grateful if you could look into this. Thanks for your work on CentOS! | ||||
Steps To Reproduce | 1. Install WAAgentLinux-2.3.1.1 from the OpenLogic repositories: http://olcentgbl.trafficmanager.net/openlogic/8/openlogic/x86_64/ 2. Leave the service running and wait. The /var/log/waagent.log file will repeatedly fill with multiple Python errors as described in the Description field here, to the point where the disk will fill up faster than Logrotate can archive the log files. | ||||
Tags | No tags attached. | ||||
Note on installed package versions that exhibit the issue: $ sudo rpm -qa | grep -i WALinuxAgent WALinuxAgent-udev-2.3.1.1-1_ol001.el8.noarch WALinuxAgent-2.3.1.1-1_ol001.el8.noarch |
|
FYI, WALinuxAgent 2.3.1.1-1_ol001 is not a package published by CentOS or Red Hat. I was requested, by Microsoft, to publish this update in our OpenLogic repos. The el8 version of the package contained an error that was not caught during testing, but I will be releasing an ol002 version of the package shortly to remedy the issue. The el7 version of the package is not affected and does not require an update. |
|
WALinuxAgent 2.3.1.1-1_ol002.el8 was released yesterday and I have received confirmation that it resolved the issue. | |
This is resolved as of non-centos package ending in 002. | |
Date Modified | Username | Field | Change |
---|---|---|---|
2021-08-24 07:47 | capncodewash | New Issue | |
2021-08-24 07:51 | capncodewash | Note Added: 0038590 | |
2021-08-24 20:11 | N3WWN | Note Added: 0038591 | |
2021-08-26 14:13 | N3WWN | Note Added: 0038592 | |
2021-08-26 15:05 | Evolution | Status | new => resolved |
2021-08-26 15:05 | Evolution | Resolution | open => fixed |
2021-08-26 15:05 | Evolution | Note Added: 0038593 |