View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0015288||Buildsys||Ci.centos.org Ecosystem Testing||public||2018-09-18 07:35||2018-09-24 18:50|
|Summary||0015288: Running out of space in slave04 (the dedicated slave node of the Devtools project)|
|Description||We have run out of space in slave04.|
The devtools project has a very large amount of workspaces:
[devtools@slave04 workspace]$ ls -l | wc -l
Without counting the concurrent ones:
[devtools@slave04 workspace]$ ls |sed 's/@.*$// '|sort -u|wc -l
Is it possible to request a larger disk for this node?
|Tags||No tags attached.|
Some workspaces were using a lot of data, and the reason is because we had an rsync in those jobs that synced back from the duffy nodes into the slave node. The underlying problem is that whenever https://bugs.centos.org/view.php?id=14920 affected these jobs, the files were rsynced to the slave node, and the workspace kept on growing.
We have removed the rsync that syncs data from the duffy node to the slave, and cleaned the workspaces, and now we have enough disk space.
However, I would like to know what is the procedure to prevent this from happening in the future. Afaik, you are monitoring the slaves with zabbix, can we be notified whenever we reach a large percentage of usage?
The real fix is to not keep anything around at all in the workspace, and use artifacts.ci.centos.org