View Issue Details

IDProjectCategoryView StatusLast Update
0014864CentOS-7kernelpublic2018-07-05 18:21
Reporterricky rocker 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformVirtualBoxOSCentOSOS Version7.5
Product Version7.5.1804 
Target VersionFixed in Version7.5.1804 
Summary0014864: iotop fails with python error post upgrade to 7.5
Descriptionroot@server # iotop
Traceback (most recent call last):
  File "/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 620, in main
    main_loop()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 610, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 508, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib64/python2.7/curses/wrapper.py", line 43, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 501, in run_iotop_window
    ui.run()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 155, in run
    self.process_list.duration)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 434, in refresh_display
    lines = self.get_data()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 415, in get_data
    return list(map(format, processes))
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 388, in format
    cmdline = p.get_cmdline()
  File "/usr/lib/python2.7/site-packages/iotop/data.py", line 292, in get_cmdline
    proc_status = parse_proc_pid_status(self.pid)
  File "/usr/lib/python2.7/site-packages/iotop/data.py", line 196, in parse_proc_pid_status
    key, value = line.split(':\t', 1)
ValueError: need more than 1 value to unpack
root@server #
Steps To Reproduceroot@server # iotop
Additional Informationhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844173
TagsNo tags attached.
abrt_hash
URL

Relationships

has duplicate 0014876 closedTrevorH [abrt] iotop: data.py:196:parse_proc_pid_status:ValueError: need more than 1 value to unpack 

Activities

jb044

jb044

2018-05-26 19:41

reporter   ~0031921

Same problem on x86_64, if I look at a process status file in /proc/<pid>/status I see both tab separated values and space separated values. Perhaps related to the Intel patches on the kernel?
TrevorH

TrevorH

2018-05-27 01:41

manager   ~0031922

What is the output from `uname -a`
jb044

jb044

2018-05-27 15:10

reporter   ~0031925

Linux trinity.lokaal.net 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
TrevorH

TrevorH

2018-05-28 14:16

manager   ~0031931

Confirmed, this is a breakage from the previous kernel kernel-3.10.0-862.2.3.el7.x86_64 which works and kernel-3.10.0-862.3.2.el7.x86_64 which does not.

Since both iotop and the kernel are packages from RHEL, you'll need to report this on bugzilla.redhat.com to get this fixed. Once RH fix it for RHEL then CentOS will inherit the fix.
TrevorH

TrevorH

2018-05-28 14:38

manager   ~0031932

I've checked this and the only difference that I can see in /proc/self/status between the two kernels is that the newer one includes a completely blank line in it just before a new line "SpeculationStoreBypass: vulnerable". The "fix" referenced in the Debian bug report does not work for this and iotop still crashes with the same stacktrace even with it included. Personally I think this is a kernel bug in the 862.3.2 kernel and the error is the presence of a blank line in /proc/$pid/status when there shouldn't be one.

I did hack a working copy of iotop together by applying the following though my knowledge of python is mostly readonly so this may be completely incorrect and invalid...

--- /usr/lib/python2.7/site-packages/iotop/data.py 2013-05-26 23:44:18.000000000 +0100
+++ /usr/lib/python2.7/site-packages/iotop/data.py.hack 2018-05-28 15:36:02.400093463 +0100
@@ -193,8 +193,12 @@
     result_dict = {}
     try:
         for line in open('/proc/%d/status' % pid):
- key, value = line.split(':\t', 1)
- result_dict[key] = value.strip()
+ try:
+ key, value = line.split(':\t', 1)
+ except ValueError:
+ error = True
+ else:
+ result_dict[key] = value.strip()
     except IOError:
         pass # No such process
     return result_dict
TrevorH

TrevorH

2018-05-28 14:48

manager   ~0031933

https://bugzilla.redhat.com/show_bug.cgi?id=1583248
toracat

toracat

2018-05-29 01:04

manager   ~0031934

I have uploaded the kernel patch that fixes the issue (centos-linux-3.10-array-bug14864.patch).

The kernel-plus package with this patch is now available for testing:

https://people.centos.org/toracat/kernel/7/plus/bug14864/

centos-linux-3.10-array-bug14864.patch (462 bytes)
--- a/fs/proc/array.c	2018-05-15 14:03:51.000000000 -0700
+++ b/fs/proc/array.c	2018-05-28 17:18:00.968882505 -0700
@@ -367,7 +367,7 @@ static inline void task_seccomp(struct s
 #ifdef CONFIG_SECCOMP
 	seq_printf(m, "Seccomp:\t%d\n", p->seccomp.mode);
 #endif
-	seq_printf(m, "\nSpeculationStoreBypass:\t");
+	seq_printf(m, "SpeculationStoreBypass:\t");
 	switch (arch_prctl_spec_ctrl_get(p, PR_SPEC_STORE_BYPASS)) {
 	case -EINVAL:
 		seq_printf(m, "unknown");
toracat

toracat

2018-06-12 15:00

manager   ~0032068

There is now a knowledgebase article, "iotop fails with "ValueError: need more than 1 value to unpack" on kernel 3.10.0-862.3.2.el7.x86_64":

https://access.redhat.com/solutions/3467211

RH is planning to fix the kernel, possibly the iotop as well.
toracat

toracat

2018-06-16 05:48

manager   ~0032097

kernel-plus-3.10.0-862.3.3.el7.centos.plus has been released. It has the patch from this bug report.
ricky rocker

ricky rocker

2018-06-16 10:51

reporter   ~0032101

sadly am still seeing the issue :(

# uname -a
Linux aa.xx.yy.zz 3.10.0-862.3.3.el7.x86_64 #1 SMP Fri Jun 15 04:15:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@aa:~ # iotop
Traceback (most recent call last):
  File "/sbin/iotop", line 17, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 620, in main
    main_loop()
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 610, in <lambda>
    main_loop = lambda: run_iotop(options)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 508, in run_iotop
    return curses.wrapper(run_iotop_window, options)
  File "/usr/lib64/python2.7/curses/wrapper.py", line 43, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/iotop/ui.py", line 501, in run_iotop_window
    ui.run()
...
TrevorH

TrevorH

2018-06-16 12:07

manager   ~0032102

You're not using the centosplus kernel which is the only one that has the fix for this. The main distro kernel is locked to the upstream one and contains only fixes made by Redhat. The plus kernel is built from the same source as that but contains extra options and patches and it's that one you need to run. If you run uname -r with the plus kernel you should see

3.10.0-862.3.3.el7.centos.plus.x86_64
toracat

toracat

2018-06-29 00:03

manager   ~0032152

The distro kernel has been fixed as of 3.10.0-862.6.3.el7.
Eugene.Dolgoff

Eugene.Dolgoff

2018-06-30 12:51

reporter   ~0032163

Another user experienced a similar problem:

Just launched iotop.

reporter: libreport-2.1.11.1
cmdline: /usr/bin/python /sbin/iotop
dso_list: python-libs-2.7.5-68.el7.x86_64
executable: /sbin/iotop
kernel: 3.10.0-862.3.3.el7.x86_64
package: iotop-0.6-2.el7
pkg_fingerprint: 24C6 A8A7 F4A8 0EB5
pkg_vendor: CentOS
reason: data.py:196:parse_proc_pid_status:ValueError: need more than 1 value to unpack
reproducible: Not sure how to reproduce the problem
runlevel: N 5
type: Python
uid: 0
toracat

toracat

2018-07-02 05:06

manager   ~0032164

@Eugene.Dolgoff

Please update the kernel to 3.10.0-862.6.3.el7 as soon as it is released. That version has the fix.
xlu

xlu

2018-07-03 06:20

reporter   ~0032167

Another user experienced a similar problem:

Started the app with sudo.

reporter: libreport-2.1.11.1
cmdline: /usr/bin/python /sbin/iotop
dso_list: python-libs-2.7.5-68.el7.x86_64
executable: /sbin/iotop
kernel: 3.10.0-862.3.3.el7.x86_64
package: iotop-0.6-2.el7
pkg_fingerprint: 24C6 A8A7 F4A8 0EB5
pkg_vendor: CentOS
reason: data.py:196:parse_proc_pid_status:ValueError: need more than 1 value to unpack
reproducible: Not sure how to reproduce the problem
runlevel: N 5
type: Python
uid: 0
jb044

jb044

2018-07-05 18:15

reporter   ~0032180

# uname -a
Linux trinity.lokaal.net 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

iotop works again
tigalch

tigalch

2018-07-05 18:20

manager   ~0032181

@xlu:
Please update the kernel to 3.10.0-862.6.3.el7.
tigalch

tigalch

2018-07-05 18:21

manager   ~0032182

kernel 3.10.0-862.6.3.el7 fiex the issue.

Issue History

Date Modified Username Field Change
2018-05-25 10:13 ricky rocker New Issue
2018-05-26 19:41 jb044 Note Added: 0031921
2018-05-27 01:41 TrevorH Note Added: 0031922
2018-05-27 15:10 jb044 Note Added: 0031925
2018-05-28 14:16 TrevorH Note Added: 0031931
2018-05-28 14:38 TrevorH Note Added: 0031932
2018-05-28 14:48 TrevorH Note Added: 0031933
2018-05-28 14:51 TrevorH Reproducibility sometimes => always
2018-05-28 14:51 TrevorH Category iotop => kernel
2018-05-29 01:04 toracat File Added: centos-linux-3.10-array-bug14864.patch
2018-05-29 01:04 toracat Note Added: 0031934
2018-05-29 01:06 toracat Status new => assigned
2018-05-29 06:45 toracat Relationship added has duplicate 0014876
2018-06-12 15:00 toracat Note Added: 0032068
2018-06-16 05:48 toracat Note Added: 0032097
2018-06-16 10:51 ricky rocker Note Added: 0032101
2018-06-16 12:07 TrevorH Note Added: 0032102
2018-06-29 00:03 toracat Note Added: 0032152
2018-06-30 12:51 Eugene.Dolgoff Note Added: 0032163
2018-07-02 05:06 toracat Note Added: 0032164
2018-07-03 06:20 xlu Note Added: 0032167
2018-07-05 18:15 jb044 Note Added: 0032180
2018-07-05 18:20 tigalch Note Added: 0032181
2018-07-05 18:21 tigalch Status assigned => resolved
2018-07-05 18:21 tigalch Resolution open => fixed
2018-07-05 18:21 tigalch Fixed in Version => 7.5.1804
2018-07-05 18:21 tigalch Note Added: 0032182