View Issue Details

IDProjectCategoryView StatusLast Update
0015684BuildsysCi.centos.org Ecosystem Testingpublic2019-01-17 18:41
Reporterlmilbaum 
PrioritynormalSeveritymajorReproducibilitysometimes
Status resolvedResolutionfixed 
Summary0015684: Error syncing pod
Descriptionnamespace: ember-csi

Today, I run into the following error: "Error syncing pod" which broke the pipeline.
TagsNo tags attached.

Activities

bstinson

bstinson

2019-01-09 18:58

administrator   ~0033549

I'm going to need more information here. You don't even mention which pod this pertains to.

I don't see anything interesting in our event logs
lmilbaum

lmilbaum

2019-01-10 08:28

reporter   ~0033551

The PODs in question are created dynamically during a pipeline and scrabed before the following.
I will try to reproduce the error today and extract the logs which I am have permissions to get.
lmilbaum

lmilbaum

2019-01-16 20:11

reporter   ~0033607

https://console.apps.ci.centos.org:8443/console/project/ember-csi/browse/pods/ember-csi-1-vf7rh?tab=events
lmilbaum

lmilbaum

2019-01-17 08:44

reporter   ~0033614

output of describe pod. it failed to pull the image since the docker storage in the node is full.
Events:
  FirstSeen LastSeen Count From SubObjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  14h 1h 83 kubelet, n16.humpty.ci.centos.org spec.containers{ember-csi} Warning Failed (combined from similar events): Failed to pull image "172.30.254.79:5000/ember-csi/ember-csi@sha256:ef2047c7777dc050a8fc69c45b9cda39c7687ad4a9b7666cf692de9287f7d271": rpc error: code = 2 desc = failed to register layer: devmapper: Thin Pool has 150455 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
  16h 1h 158 kubelet, n16.humpty.ci.centos.org spec.containers{ember-csi} Normal Pulling pulling image "172.30.254.79:5000/ember-csi/ember-csi@sha256:ef2047c7777dc050a8fc69c45b9cda39c7687ad4a9b7666cf692de9287f7d271"
  29m 12m 2 kubelet, n16.humpty.ci.centos.org spec.containers{ember-csi} Warning Failed Failed to pull image "172.30.254.79:5000/ember-csi/ember-csi@sha256:ef2047c7777dc050a8fc69c45b9cda39c7687ad4a9b7666cf692de9287f7d271": rpc error: code = 2 desc = failed to register layer: devmapper: Thin Pool has 151148 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
  16h 7m 3697 kubelet, n16.humpty.ci.centos.org spec.containers{ember-csi} Normal BackOff Back-off pulling image "172.30.254.79:5000/ember-csi/ember-csi@sha256:ef2047c7777dc050a8fc69c45b9cda39c7687ad4a9b7666cf692de9287f7d271"
  16h 2m 3888 kubelet, n16.humpty.ci.centos.org Warning FailedSync Error syncing pod
lmilbaum

lmilbaum

2019-01-17 12:08

reporter   ~0033615

BTW, I had to remove the failing pod to release the blocking object
bstinson

bstinson

2019-01-17 18:41

administrator   ~0033623

I checked on that particular node and one of our cleanup jobs (that removes old images from the thin pool on a regular basis) was stuck on that node.

It should be un-stuck now.

Issue History

Date Modified Username Field Change
2019-01-09 15:28 lmilbaum New Issue
2019-01-09 18:58 bstinson Status new => feedback
2019-01-09 18:58 bstinson Note Added: 0033549
2019-01-10 08:28 lmilbaum Note Added: 0033551
2019-01-10 08:28 lmilbaum Status feedback => assigned
2019-01-16 20:11 lmilbaum Note Added: 0033607
2019-01-17 08:44 lmilbaum Note Added: 0033614
2019-01-17 12:08 lmilbaum Note Added: 0033615
2019-01-17 18:41 bstinson Status assigned => resolved
2019-01-17 18:41 bstinson Resolution open => fixed
2019-01-17 18:41 bstinson Note Added: 0033623