View Issue Details

IDProjectCategoryView StatusLast Update
0015288BuildsysCi.centos.org Ecosystem Testingpublic2018-09-24 18:50
Reporterjmelis 
PrioritynormalSeverityminorReproducibilityN/A
Status resolvedResolutionfixed 
Summary0015288: Running out of space in slave04 (the dedicated slave node of the Devtools project)
DescriptionWe have run out of space in slave04.

The devtools project has a very large amount of workspaces:

[devtools@slave04 workspace]$ ls -l | wc -l
1157

Without counting the concurrent ones:

[devtools@slave04 workspace]$ ls |sed 's/@.*$// '|sort -u|wc -l
398

Is it possible to request a larger disk for this node?
TagsNo tags attached.

Activities

jmelis

jmelis

2018-09-18 10:39

reporter   ~0032750

Some workspaces were using a lot of data, and the reason is because we had an rsync in those jobs that synced back from the duffy nodes into the slave node. The underlying problem is that whenever https://bugs.centos.org/view.php?id=14920 affected these jobs, the files were rsynced to the slave node, and the workspace kept on growing.

We have removed the rsync that syncs data from the duffy node to the slave, and cleaned the workspaces, and now we have enough disk space.

However, I would like to know what is the procedure to prevent this from happening in the future. Afaik, you are monitoring the slaves with zabbix, can we be notified whenever we reach a large percentage of usage?
bstinson

bstinson

2018-09-24 18:50

administrator   ~0032781

The real fix is to not keep anything around at all in the workspace, and use artifacts.ci.centos.org

Issue History

Date Modified Username Field Change
2018-09-18 07:35 jmelis New Issue
2018-09-18 10:39 jmelis Note Added: 0032750
2018-09-24 18:50 bstinson Status new => resolved
2018-09-24 18:50 bstinson Resolution open => fixed
2018-09-24 18:50 bstinson Note Added: 0032781