View Issue Details

IDProjectCategoryView StatusLast Update
0016208BuildsysCi.centos.org Ecosystem Testingpublic2019-06-26 08:54
ReporterMartin.Pitt 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Summary0016208: cockpit-images volume (os-pv-100gi-00000002) inexplicably full
DescriptionHello!

In the "cockpit" OpenShift project we use a persistent volume (https://console.apps.ci.centos.org:8443/console/project/cockpit/browse/persistentvolumeclaims/cockpit-images) as VM image cache. This is now shown as full:

    172.22.6.19:/exports/os-pv-100gi-00000002 187G 187G 1.0M 100% /cache/images

and regularly leads to ENOSPC. However, all actual files on this device are only 30 GB:

$ du -hsc /cache/images/
30G /cache/images/
30G total

Yesterday I already tried to shut down all containers that mount this volume, just in case there are processes holding open fds to deleted files, but that didn't help at all. There are no hidden files on this device either. At this point I don't know what I could do on my end.

Is there something obviously wrong on the NFS server side?

In case repairing the volume isn't easy: It wouldn't be a problem to entirely delete, or remove/re-add the PV, and re-populate it. I just copied all files to our other (Red Hat internal) store.
TagsNo tags attached.

Activities

Martin.Pitt

Martin.Pitt

2019-06-26 08:54

reporter   ~0034724

Odd, today this looks different even nothing substantial changed on our end since I filed this 4 days ago (as I shut down all pods that work with the image):

   172.22.6.19:/exports/os-pv-100gi-00000002 187G 143G 44G 77% /cache/images

I re-enabled the pods for now, as it seems it's not dangerously close to ENOSPC again. However, we still only allocate ~ 30 GB, not 143.

Issue History

Date Modified Username Field Change
2019-06-22 14:16 Martin.Pitt New Issue
2019-06-26 08:54 Martin.Pitt Note Added: 0034724