View Issue Details

IDProjectCategoryView StatusLast Update
0016104CentOS-7kernelpublic2019-05-21 09:31
Reporteravanpourm 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0016104: Insufficient memory leads to a lot of IO reading
DescriptionWhen the container memory usage rises to the upper limit of "memory.memsw.limit_in_bytes", the kernel will discard the filecache, which contains the program code. This can result in a lot of IO reading. Most of the reads are program files . Those files is required during program execution. Seriously affects the health of other programs on the same server.
How to solve it?
Steps To Reproduce**Steps to reproduce the issue:**
1. Run my test program
   ```
docker run --memory=200M --memory-swap=200M --memory-swappiness=0 -p 8980:8989 harbor.meitu-int.com/hpatest/exampletest:v0.0.2-test /bin/sh -c "while(true) do sleep 1; done"
   ```
2. Send a request, the server will apply for a lot of memory
```
curl -sSL "http://127.0.0.1:8989/memory/incr?block=130&blocksize=1048576"
```
   Observing container memory
```
docker stats
```
3. Open another terminal to observe the IO status.
```
iostat -x 1
```
4. Open another terminal to observe the pmap status.
```
while(true) do date; pmap -Xp 47431 ;sleep 1; done
```
5. Sending a query request will result in a large number of IOs.
```
curl -sSL "http://127.0.0.1:8989/metrics
```
**Describe the results you received:**
```
# iostat -x /dev/dm-62 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-62 0.00 0.00 56.00 0.00 13144.00 0.00 312.29 2.3 5.61 5.61 0.00 2.70 100
```

```
# while(true) do date; pmap -Xp 47431 ;sleep 1; done
47431: /example-test -v 10 -logtostderr
         Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
        00400000 r-xp 00000000 fd:3e 262 6152 520 520 512 0 0 0 0 0 0 /example-test
        00a02000 r--p 00602000 fd:3e 262 7204 328 328 48 0 0 0 0 0 0 /example-test
        0110b000 rw-p 00d0b000 fd:3e 262 248 44 44 44 44 0 0 0 0 0 /example-test
        01149000 rw-p 00000000 00:00 0 156 72 72 72 72 0 0 0 0 0
      c000000000 rw-p 00000000 00:00 0 4096 4048 4048 4048 4048 0 0 0 0 0
      c000400000 rw-p 00000000 00:00 0 778240 0 0 0 0 0 0 0 0 0
      c02fc00000 rw-p 00000000 00:00 0 2048 200 200 200 200 0 0 0 0 0
      c02fe00000 rw-p 00000000 00:00 0 780288 0 0 0 0 0 0 0 0 0
      c05f800000 rw-p 00000000 00:00 0 2048 908 908 900 908 0 0 0 0 0
      c05fa00000 rw-p 00000000 00:00 0 59392 59392 59392 59392 59392 0 0 0 0 0
      c063400000 rw-p 00000000 00:00 0 2048 2048 2048 2048 2048 0 0 0 0 0
      c063600000 rw-p 00000000 00:00 0 10240 10240 10240 10240 10240 0 0 0 0 0
      c064000000 rw-p 00000000 00:00 0 65536 51200 51200 51200 51200 0 0 0 0 0
    7fb29329f000 rw-p 00000000 00:00 0 90880 56848 56848 56732 56848 0 0 0 0 0
    7ffc06952000 rw-p 00000000 00:00 0 132 16 16 16 16 0 0 0 0 0 [stack]
    7ffc069df000 r--p 00000000 00:00 0 12 0 0 0 0 0 0 0 0 0 [vvar]
    7ffc069e2000 r-xp 00000000 00:00 0 8 4 0 4 0 0 0 0 0 0 [vdso]
ffffffffff600000 r-xp 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 [vsyscall]
                                            ======= ====== ====== ========== ========= ============== =============== ==== ======= ======
                                            1808732 185868 185864 185456 185016 0 0 0 0 0 KB


```

**Describe the results you expected:**

**I hope that the executable page will not be swapped out.**
**And the page map is like this:**

```
# while(true) do date; pmap -Xp 47431 ;sleep 1; done
47431: /example-test -v 10 -logtostderr
         Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
        00400000 r-xp 00000000 fd:3e 262 6152 4108 4108 3964 0 0 0 0 0 0 /example-test
        00a02000 r--p 00602000 fd:3e 262 7204 4308 4308 4204 0 0 0 0 0 0 /example-test
        0110b000 rw-p 00d0b000 fd:3e 262 248 196 196 196 40 0 0 0 0 0 /example-test
        01149000 rw-p 00000000 00:00 0 156 72 72 68 72 0 0 0 0 0
      c000000000 rw-p 00000000 00:00 0 4096 4008 4008 4008 4008 0 0 0 0 0
      c000400000 rw-p 00000000 00:00 0 778240 0 0 0 0 0 0 0 0 0
      c02fc00000 rw-p 00000000 00:00 0 2048 188 188 188 188 0 0 0 0 0
      c02fe00000 rw-p 00000000 00:00 0 780288 0 0 0 0 0 0 0 0 0
      c05f800000 rw-p 00000000 00:00 0 2048 444 444 444 444 0 0 0 0 0
      c05fa00000 rw-p 00000000 00:00 0 71680 0 0 0 0 0 0 0 0 0
    7fb2934ef000 rw-p 00000000 00:00 0 88512 54392 54392 54368 54392 0 0 0 0 0
    7ffc06952000 rw-p 00000000 00:00 0 132 16 16 16 16 0 0 0 0 0 [stack]
    7ffc069df000 r--p 00000000 00:00 0 12 0 0 0 0 0 0 0 0 0 [vvar]
    7ffc069e2000 r-xp 00000000 00:00 0 8 4 0 4 0 0 0 0 0 0 [vdso]
ffffffffff600000 r-xp 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 [vsyscall]
                                            ======= ===== ===== ========== ========= ============== =============== ==== ======= ======
                                            1740828 67736 67732 67460 59160 0 0 0 0 0 KB

```


Tagscgroup
abrt_hash
URL

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2019-05-21 09:31 avanpourm New Issue
2019-05-21 09:31 avanpourm Tag Attached: cgroup