View Issue Details

IDProjectCategoryView StatusLast Update
0016208CentOS CIgeneralpublic2020-02-13 18:45
ReporterMartin.Pitt Assigned To 
Status resolvedResolutionfixed 
Summary0016208: cockpit-images volume (os-pv-100gi-00000002) inexplicably full

In the "cockpit" OpenShift project we use a persistent volume ( as VM image cache. This is now shown as full: 187G 187G 1.0M 100% /cache/images

and regularly leads to ENOSPC. However, all actual files on this device are only 30 GB:

$ du -hsc /cache/images/
30G /cache/images/
30G total

Yesterday I already tried to shut down all containers that mount this volume, just in case there are processes holding open fds to deleted files, but that didn't help at all. There are no hidden files on this device either. At this point I don't know what I could do on my end.

Is there something obviously wrong on the NFS server side?

In case repairing the volume isn't easy: It wouldn't be a problem to entirely delete, or remove/re-add the PV, and re-populate it. I just copied all files to our other (Red Hat internal) store.
TagsNo tags attached.




2019-06-26 08:54

reporter   ~0034724

Odd, today this looks different even nothing substantial changed on our end since I filed this 4 days ago (as I shut down all pods that work with the image): 187G 143G 44G 77% /cache/images

I re-enabled the pods for now, as it seems it's not dangerously close to ENOSPC again. However, we still only allocate ~ 30 GB, not 143.


2019-08-29 05:47

reporter   ~0035029

This is still an issue today:

187G 171G 16G 92% /cache/images

$ du -hs /cache/images/
38G /cache/images/


2019-09-04 14:51

reporter   ~0035066

I tried to completely empty the volume file system now:

$ ls -la /cache/images/
total 0
drwxrwxrwx. 2 nobody nobody 6 Sep 4 14:45 .
drwxrwsrwx. 4 root 1000190000 34 Sep 4 09:13 ..

 du -hs /cache/images/
0 /cache/images/

$ df /cache/images/
Filesystem 1K-blocks Used Available Use% Mounted on 195265536 142974976 52290560 74% /cache/images

I then shut down all pods that use that volume, just in case they hold open deleted fds there. But that didn't help either.

So there really is some inexplicable 137G usage that I just can't get rid of. The fun thing is that this is supposed to be a 100 GiB PV, not 187 -- 100 would be enough, but 50 is just too small. We need some 70 GiB for our stuff.


2020-02-13 18:45

administrator   ~0036290

Migrated this PV to different storage.

Issue History

Date Modified Username Field Change
2019-06-22 14:16 Martin.Pitt New Issue
2019-06-26 08:54 Martin.Pitt Note Added: 0034724
2019-08-29 05:47 Martin.Pitt Note Added: 0035029
2019-09-04 14:51 Martin.Pitt Note Added: 0035066
2020-01-29 15:42 siddharthvipul1 Project Buildsys => CentOS CI
2020-01-29 15:42 siddharthvipul1 Category Ecosystem Testing => general
2020-02-13 18:45 bstinson Status new => resolved
2020-02-13 18:45 bstinson Resolution open => fixed
2020-02-13 18:45 bstinson Note Added: 0036290