View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0018287CentOS-8-OTHERpublic2021-08-24 07:472021-08-26 15:05
Reportercapncodewash Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version8.4.2105 
Summary0018287: WAAgentLinux package outputs Python errors to log, filling disk and making machine unusable
DescriptionHi there,
Apologies if this has been filed before - I couldn't see anything listed.

There is seemingly an issue with the WAAgentLinux package that provides Azure utilities for CentOS.

On multiple Azure Linux VMs, the boot drives fill up to the point where the machine is unusable, because of the /var/log/waagent.log file becoming absolutely massive (~15GB on one machine). It seems that it cannot find a Python module, and re-attempts to run a process continuously.

The output printed to the log file is similar to the below:

2021-08-23T08:00:21.856964Z INFO Daemon Daemon Installed Agent WALinuxAgent-2.3.1.1 is the most current agent
2021-08-23T08:00:21.857200Z WARNING Daemon Daemon Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined
2021-08-23T08:00:21.857406Z ERROR Daemon Daemon Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.3.1.1 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' failed with exception: name 'get_python_cmd' is not defined Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/azurelinuxagent/ga/update.py", line 156, in run_latest
    cmds[0] = get_python_cmd()
NameError: name 'get_python_cmd' is not defined

Distro and WALinuxAgent details (printed from 'waagent --version'):
WALinuxAgent-2.3.1.1 running on centos 8.4.2105
Python: 3.6.8
Goal state agent: 2.3.1.1

I logged this as a bug with the WAAgentLinux developers:
https://github.com/Azure/WALinuxAgent/issues/2336

and they think it's a packaging error - in creating the 2.3.1.1 version, the call to 'get_python_cmd()' should have been removed (see comment https://github.com/Azure/WALinuxAgent/issues/2336#issuecomment-904027245 ).

Other users have successfully worked around the issue by downgrading the package to version 2.2.49.2.

I'd be grateful if you could look into this. Thanks for your work on CentOS!
Steps To Reproduce1. Install WAAgentLinux-2.3.1.1 from the OpenLogic repositories: http://olcentgbl.trafficmanager.net/openlogic/8/openlogic/x86_64/
2. Leave the service running and wait. The /var/log/waagent.log file will repeatedly fill with multiple Python errors as described in the Description field here, to the point where the disk will fill up faster than Logrotate can archive the log files.
TagsNo tags attached.

Activities

capncodewash

capncodewash

2021-08-24 07:51

reporter   ~0038590

Note on installed package versions that exhibit the issue:

$ sudo rpm -qa | grep -i WALinuxAgent
WALinuxAgent-udev-2.3.1.1-1_ol001.el8.noarch
WALinuxAgent-2.3.1.1-1_ol001.el8.noarch
N3WWN

N3WWN

2021-08-24 20:11

reporter   ~0038591

FYI, WALinuxAgent 2.3.1.1-1_ol001 is not a package published by CentOS or Red Hat. I was requested, by Microsoft, to publish this update in our OpenLogic repos.

The el8 version of the package contained an error that was not caught during testing, but I will be releasing an ol002 version of the package shortly to remedy the issue.

The el7 version of the package is not affected and does not require an update.
N3WWN

N3WWN

2021-08-26 14:13

reporter   ~0038592

WALinuxAgent 2.3.1.1-1_ol002.el8 was released yesterday and I have received confirmation that it resolved the issue.
Evolution

Evolution

2021-08-26 15:05

administrator   ~0038593

This is resolved as of non-centos package ending in 002.

Issue History

Date Modified Username Field Change
2021-08-24 07:47 capncodewash New Issue
2021-08-24 07:51 capncodewash Note Added: 0038590
2021-08-24 20:11 N3WWN Note Added: 0038591
2021-08-26 14:13 N3WWN Note Added: 0038592
2021-08-26 15:05 Evolution Status new => resolved
2021-08-26 15:05 Evolution Resolution open => fixed
2021-08-26 15:05 Evolution Note Added: 0038593