View Issue Details

IDProjectCategoryView StatusLast Update
0015288CentOS CI[All Projects] generalpublic2020-01-29 16:34
Status resolvedResolutionfixed 
Summary0015288: Running out of space in slave04 (the dedicated slave node of the Devtools project)
DescriptionWe have run out of space in slave04.

The devtools project has a very large amount of workspaces:

[devtools@slave04 workspace]$ ls -l | wc -l

Without counting the concurrent ones:

[devtools@slave04 workspace]$ ls |sed 's/@.*$// '|sort -u|wc -l

Is it possible to request a larger disk for this node?
TagsNo tags attached.




2018-09-18 10:39

reporter   ~0032750

Some workspaces were using a lot of data, and the reason is because we had an rsync in those jobs that synced back from the duffy nodes into the slave node. The underlying problem is that whenever affected these jobs, the files were rsynced to the slave node, and the workspace kept on growing.

We have removed the rsync that syncs data from the duffy node to the slave, and cleaned the workspaces, and now we have enough disk space.

However, I would like to know what is the procedure to prevent this from happening in the future. Afaik, you are monitoring the slaves with zabbix, can we be notified whenever we reach a large percentage of usage?


2018-09-24 18:50

administrator   ~0032781

The real fix is to not keep anything around at all in the workspace, and use

Issue History

Date Modified Username Field Change
2018-09-18 07:35 jmelis New Issue
2018-09-18 10:39 jmelis Note Added: 0032750
2018-09-24 18:50 bstinson Status new => resolved
2018-09-24 18:50 bstinson Resolution open => fixed
2018-09-24 18:50 bstinson Note Added: 0032781
2020-01-29 16:34 siddharthvipul1 Project Buildsys => CentOS CI
2020-01-29 16:34 siddharthvipul1 Category Ecosystem Testing => general