2017-12-11 13:04 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013474CentOS-7phppublic2017-11-08 18:59
Reporterdeadlydud 
PriorityhighSeveritymajorReproducibilityrandom
StatusnewResolutionopen 
PlatformintelOSLinuxOS VersionCentOS
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0013474: PHP opcache memory corruption still occurs in PHP-7.0.20
DescriptionInstated new Centos 7.3.1611 server last night. It ran for about 17 hours
under decent load.

Suddenly PHP errors occur, I traced it to corrupt strings within post-compiled
php opcode, googled a bit and came across:

https://bugs.php.net/bug.php?id=71135 Random memory corruption with strings

Doing an opcache reset without restarting Apache cured the problem,
but now I worry the problem will re-occur within the next 24 hours,
Random letter changes within PHP code can cause broken or blank web pages,
data corruption, and a non-functioning server causes loss of income
and an angry boss.
Steps To ReproduceLet Apache run for several hours, serving same PHP pages referencing a
large number of PHP data array files in own /cache directory,

Am using default Centos settings for /etc/php.d/opcache.ini
Additional Information
I will try tomorrow excluding a directory containing 10000+ php data array files
using the opcache blacklist,
and will apply opcache ini setting re opcache fast-shutdown = 0,
though it would be great if any existing PHP fixes addressing the opcache corruption is to be applied in a Centos PHP 7.0.21 update
so that the problem does not occur in the first place.
Tagscentos 7
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0029571

deadlydud (reporter)

Any temporary work-arounds to this problem will be highly appreciated.

I'm considering creating a cron-job that calls a php script every 10 hours to force an opcache reset, though will try editing /etc/php.d/opcache.ini
and setting opcache.fast_shutdown=0 first (googling around shown that this does not always cure the problem though) .. will try it out.. advise?

~0029593

deadlydud (reporter)

Update:

Noticed a pattern: Apache with mod_php running 5 hours without incident, then in Apache error log one entry about an Apache PHP thread segfaulting.

Immediately afterwards: many errors appearing about PHP errors encountered, which turns out to be a single character in the PHP opcocde cache that got flipped, randomly on several files. Checking each source file reveals no changes on the file on disk, only change is in shared memory.

A theory: though that having a script that uses (say) 63 Mb RAM with PHP process set to max 64Mb, PHP script does a shell call to an external binary that goes over the memory limit, upon return PHP process segfaults, though the theory got shot out of the water since there is not a single segfault once (spoiler alert) opcache got disabled.

Reset opcache using a small PHP opcache manager script immediately cures the problem, until it happens again.

Temporary solution was to edit Centos /etc/php.d/opcache.ini and setting opcache enabled to 0, and restarting Apache. Monitored ram usage and no further errors happened afterwards, even under load.

Hope the opcache memory corruption due to rare race condition can be solved.
Will watch out for Centos PHP updates, and will try again to activate it afterwards.

~0029594

TrevorH (developer)

Does your machine have ECC memory?

~0029595

deadlydud (reporter)

Guys at IT says "The Centos boxes runs on Hyper-v and the host server will have ECC memory installed.", though i do not see any errors in dmesg output regarding memory.

~0029597

deadlydud (reporter)

just to clarify: its Exactly the same as in https://bugs.php.net/bug.php?id=71135

Not sure if Centos guru's talk to dmitry@php.net .. he said:
[2016-02-24 21:03 UTC] dmitry@php.net
The problem was caused by incorrect synchronization of processes during opcache reload. Some processes might crash or might not crash because of race-conditions. The commit mentioned above and a couple of following fixes, should solve the problem.

though I do see other users reporting the same thing (on 2017-07-03)
 [2017-07-03 10:38 UTC] kalyanceg at gmail dot com
We are facing this issue on debian stretch with php7.0-opcache 7.0.19-1
We see opcache getting corrupted after some days of apache running continuously and gets fixed only on restart of apache.

we run php7.0.20 (latest) and the problem is still there.

~0030542

deadlydud (reporter)

Since there is no activity on this thread .. I think everyone should monitor

https://bugs.php.net/bug.php?id=71135

instead, it seems that this is still an ongoing problem.
(They mention in Sept 2017 that this bit flipping memory corrpution still
occurs even with PHP version 7.1.x)
+Notes

-Issue History
Date Modified Username Field Change
2017-06-27 22:38 deadlydud New Issue
2017-06-27 22:38 deadlydud Tag Attached: centos 7
2017-06-28 10:06 deadlydud Note Added: 0029571
2017-07-04 20:04 deadlydud Note Added: 0029593
2017-07-04 20:58 TrevorH Note Added: 0029594
2017-07-05 07:44 deadlydud Note Added: 0029595
2017-07-05 08:54 deadlydud Note Added: 0029597
2017-11-08 18:59 deadlydud Note Added: 0030542
+Issue History