View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0013474||CentOS-7||php||public||2017-06-27 22:38||2017-11-08 18:59|
|Target Version||Fixed in Version|
|Summary||0013474: PHP opcache memory corruption still occurs in PHP-7.0.20|
|Description||Instated new Centos 7.3.1611 server last night. It ran for about 17 hours|
under decent load.
Suddenly PHP errors occur, I traced it to corrupt strings within post-compiled
php opcode, googled a bit and came across:
https://bugs.php.net/bug.php?id=71135 Random memory corruption with strings
Doing an opcache reset without restarting Apache cured the problem,
but now I worry the problem will re-occur within the next 24 hours,
Random letter changes within PHP code can cause broken or blank web pages,
data corruption, and a non-functioning server causes loss of income
and an angry boss.
|Steps To Reproduce||Let Apache run for several hours, serving same PHP pages referencing a |
large number of PHP data array files in own /cache directory,
Am using default Centos settings for /etc/php.d/opcache.ini
I will try tomorrow excluding a directory containing 10000+ php data array files
using the opcache blacklist,
and will apply opcache ini setting re opcache fast-shutdown = 0,
though it would be great if any existing PHP fixes addressing the opcache corruption is to be applied in a Centos PHP 7.0.21 update
so that the problem does not occur in the first place.
Any temporary work-arounds to this problem will be highly appreciated.
I'm considering creating a cron-job that calls a php script every 10 hours to force an opcache reset, though will try editing /etc/php.d/opcache.ini
and setting opcache.fast_shutdown=0 first (googling around shown that this does not always cure the problem though) .. will try it out.. advise?
Noticed a pattern: Apache with mod_php running 5 hours without incident, then in Apache error log one entry about an Apache PHP thread segfaulting.
Immediately afterwards: many errors appearing about PHP errors encountered, which turns out to be a single character in the PHP opcocde cache that got flipped, randomly on several files. Checking each source file reveals no changes on the file on disk, only change is in shared memory.
A theory: though that having a script that uses (say) 63 Mb RAM with PHP process set to max 64Mb, PHP script does a shell call to an external binary that goes over the memory limit, upon return PHP process segfaults, though the theory got shot out of the water since there is not a single segfault once (spoiler alert) opcache got disabled.
Reset opcache using a small PHP opcache manager script immediately cures the problem, until it happens again.
Temporary solution was to edit Centos /etc/php.d/opcache.ini and setting opcache enabled to 0, and restarting Apache. Monitored ram usage and no further errors happened afterwards, even under load.
Hope the opcache memory corruption due to rare race condition can be solved.
Will watch out for Centos PHP updates, and will try again to activate it afterwards.
|Does your machine have ECC memory?|
|Guys at IT says "The Centos boxes runs on Hyper-v and the host server will have ECC memory installed.", though i do not see any errors in dmesg output regarding memory.|
just to clarify: its Exactly the same as in https://bugs.php.net/bug.php?id=71135
Not sure if Centos guru's talk to email@example.com .. he said:
[2016-02-24 21:03 UTC] firstname.lastname@example.org
The problem was caused by incorrect synchronization of processes during opcache reload. Some processes might crash or might not crash because of race-conditions. The commit mentioned above and a couple of following fixes, should solve the problem.
though I do see other users reporting the same thing (on 2017-07-03)
[2017-07-03 10:38 UTC] kalyanceg at gmail dot com
We are facing this issue on debian stretch with php7.0-opcache 7.0.19-1
We see opcache getting corrupted after some days of apache running continuously and gets fixed only on restart of apache.
we run php7.0.20 (latest) and the problem is still there.
Since there is no activity on this thread .. I think everyone should monitor
instead, it seems that this is still an ongoing problem.
(They mention in Sept 2017 that this bit flipping memory corrpution still
occurs even with PHP version 7.1.x)
|2017-06-27 22:38||deadlydud||New Issue|
|2017-06-27 22:38||deadlydud||Tag Attached: centos 7|
|2017-06-28 10:06||deadlydud||Note Added: 0029571|
|2017-07-04 20:04||deadlydud||Note Added: 0029593|
|2017-07-04 20:58||TrevorH||Note Added: 0029594|
|2017-07-05 07:44||deadlydud||Note Added: 0029595|
|2017-07-05 08:54||deadlydud||Note Added: 0029597|
|2017-11-08 18:59||deadlydud||Note Added: 0030542|