View Issue Details

IDProjectCategoryView StatusLast Update
0014864CentOS-7kernelpublic2018-06-16 12:07
Reporterricky rocker 
PrioritynormalSeverityminorReproducibilityalways
Status assignedResolutionopen 
PlatformVirtualBoxOSCentOSOS Version7.5
Product Version7.5.1804 
Target VersionFixed in Version 
Summary0014864: iotop fails with python error post upgrade to 7.5
Descriptionroot@server # iotop
Traceback (most recent call last):
  File "/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 620, in main
    main_loop()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 610, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 508, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib64/python2.7/curses/wrapper.py", line 43, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 501, in run_iotop_window
    ui.run()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 155, in run
    self.process_list.duration)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 434, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 415, in get_data
    return list(map(format, processes))
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 388, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python2.7/site-packages/iotop/data.py", line 292, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python2.7/site-packages/iotop/data.py", line 196, in parse_proc_pid_status
    key, value = line.split(':\t', 1)
ValueError: need more than 1 value to unpack
root@server #
Steps To Reproduceroot@server # iotop
Additional Informationhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844173
TagsNo tags attached.
abrt_hash
URL

Relationships

has duplicate 0014876 new [abrt] iotop: data.py:196:parse_proc_pid_status:ValueError: need more than 1 value to unpack 

Activities

jb044

jb044

2018-05-26 19:41

reporter   ~0031921

Same problem on x86_64, if I look at a process status file in /proc/<pid>/status I see both tab separated values and space separated values. Perhaps related to the Intel patches on the kernel?
TrevorH

TrevorH

2018-05-27 01:41

manager   ~0031922

What is the output from `uname -a`
jb044

jb044

2018-05-27 15:10

reporter   ~0031925

Linux trinity.lokaal.net 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
TrevorH

TrevorH

2018-05-28 14:16

manager   ~0031931

Confirmed, this is a breakage from the previous kernel kernel-3.10.0-862.2.3.el7.x86_64 which works and kernel-3.10.0-862.3.2.el7.x86_64 which does not.

Since both iotop and the kernel are packages from RHEL, you'll need to report this on bugzilla.redhat.com to get this fixed. Once RH fix it for RHEL then CentOS will inherit the fix.
TrevorH

TrevorH

2018-05-28 14:38

manager   ~0031932

I've checked this and the only difference that I can see in /proc/self/status between the two kernels is that the newer one includes a completely blank line in it just before a new line "SpeculationStoreBypass: vulnerable". The "fix" referenced in the Debian bug report does not work for this and iotop still crashes with the same stacktrace even with it included. Personally I think this is a kernel bug in the 862.3.2 kernel and the error is the presence of a blank line in /proc/$pid/status when there shouldn't be one.

I did hack a working copy of iotop together by applying the following though my knowledge of python is mostly readonly so this may be completely incorrect and invalid...

--- /usr/lib/python2.7/site-packages/iotop/data.py 2013-05-26 23:44:18.000000000 +0100
+++ /usr/lib/python2.7/site-packages/iotop/data.py.hack 2018-05-28 15:36:02.400093463 +0100
@@ -193,8 +193,12 @@
     result_dict = {}
     try:
         for line in open('/proc/%d/status' % pid):
- key, value = line.split(':\t', 1)
- result_dict[key] = value.strip()
+ try:
+ key, value = line.split(':\t', 1)
+ except ValueError:
+ error = True
+ else:
+ result_dict[key] = value.strip()
     except IOError:
         pass # No such process
     return result_dict
TrevorH

TrevorH

2018-05-28 14:48

manager   ~0031933

https://bugzilla.redhat.com/show_bug.cgi?id=1583248
toracat

toracat

2018-05-29 01:04

manager   ~0031934

I have uploaded the kernel patch that fixes the issue (centos-linux-3.10-array-bug14864.patch).

The kernel-plus package with this patch is now available for testing:

https://people.centos.org/toracat/kernel/7/plus/bug14864/

centos-linux-3.10-array-bug14864.patch (462 bytes)
--- a/fs/proc/array.c	2018-05-15 14:03:51.000000000 -0700
+++ b/fs/proc/array.c	2018-05-28 17:18:00.968882505 -0700
@@ -367,7 +367,7 @@ static inline void task_seccomp(struct s
 #ifdef CONFIG_SECCOMP
 	seq_printf(m, "Seccomp:\t%d\n", p->seccomp.mode);
 #endif
-	seq_printf(m, "\nSpeculationStoreBypass:\t");
+	seq_printf(m, "SpeculationStoreBypass:\t");
 	switch (arch_prctl_spec_ctrl_get(p, PR_SPEC_STORE_BYPASS)) {
 	case -EINVAL:
 		seq_printf(m, "unknown");
toracat

toracat

2018-06-12 15:00

manager   ~0032068

There is now a knowledgebase article, "iotop fails with "ValueError: need more than 1 value to unpack" on kernel 3.10.0-862.3.2.el7.x86_64":

https://access.redhat.com/solutions/3467211

RH is planning to fix the kernel, possibly the iotop as well.
toracat

toracat

2018-06-16 05:48

manager   ~0032097

kernel-plus-3.10.0-862.3.3.el7.centos.plus has been released. It has the patch from this bug report.
ricky rocker

ricky rocker

2018-06-16 10:51

reporter   ~0032101

sadly am still seeing the issue :(

# uname -a
Linux aa.xx.yy.zz 3.10.0-862.3.3.el7.x86_64 #1 SMP Fri Jun 15 04:15:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@aa:~ # iotop
Traceback (most recent call last):
  File "/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 620, in main
    main_loop()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 610, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 508, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib64/python2.7/curses/wrapper.py", line 43, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 501, in run_iotop_window
    ui.run()
...
TrevorH

TrevorH

2018-06-16 12:07

manager   ~0032102

You're not using the centosplus kernel which is the only one that has the fix for this. The main distro kernel is locked to the upstream one and contains only fixes made by Redhat. The plus kernel is built from the same source as that but contains extra options and patches and it's that one you need to run. If you run uname -r with the plus kernel you should see

3.10.0-862.3.3.el7.centos.plus.x86_64

Issue History

Date Modified Username Field Change
2018-05-25 10:13 ricky rocker New Issue
2018-05-26 19:41 jb044 Note Added: 0031921
2018-05-27 01:41 TrevorH Note Added: 0031922
2018-05-27 15:10 jb044 Note Added: 0031925
2018-05-28 14:16 TrevorH Note Added: 0031931
2018-05-28 14:38 TrevorH Note Added: 0031932
2018-05-28 14:48 TrevorH Note Added: 0031933
2018-05-28 14:51 TrevorH Reproducibility sometimes => always
2018-05-28 14:51 TrevorH Category iotop => kernel
2018-05-29 01:04 toracat File Added: centos-linux-3.10-array-bug14864.patch
2018-05-29 01:04 toracat Note Added: 0031934
2018-05-29 01:06 toracat Status new => assigned
2018-05-29 06:45 toracat Relationship added has duplicate 0014876
2018-06-12 15:00 toracat Note Added: 0032068
2018-06-16 05:48 toracat Note Added: 0032097
2018-06-16 10:51 ricky rocker Note Added: 0032101
2018-06-16 12:07 TrevorH Note Added: 0032102