CURRENT, CLANG 6: apache24, uid 80: exited on signal 11

[O. Hartmann]

Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11) when
accessing the server, especially with setup of nextcloud, refdb, phpldapadmin and any
other access with LDAP backend (all https).

The message on console is:

pid 23108 (httpd), uid 80: exited on signal 11
pid 32283 (httpd), uid 80: exited on signal 11
pid 21286 (httpd), uid 80: exited on signal 11
pid 10292 (httpd), uid 80: exited on signal 11

I do not see anything else! Neither in the log (no matter what log level I switch on) nor
elsewhere.

Configuration hasn't changed.

What the ... is up with the system? How can I dig into the problem?

Thanks for advices in advance,

Oliver

[Dimitry Andric]

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11) when
accessing the server, especially with setup of nextcloud, refdb, phpldapadmin and any
other access with LDAP backend (all https).
pid 23108 (httpd), uid 80: exited on signal 11
pid 32283 (httpd), uid 80: exited on signal 11
pid 21286 (httpd), uid 80: exited on signal 11
pid 10292 (httpd), uid 80: exited on signal 11
I do not see anything else! Neither in the log (no matter what log level I switch on) nor
elsewhere.
Configuration hasn't changed.
What the ... is up with the system? How can I dig into the problem?

Try debugging httpd, e.g.:

sudo /usr/local/sbin/httpd -X

then "run", and see if you can get a backtrace.

-Dimitry

[Dimitry Andric]

Post by Dimitry Andric

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11) when
accessing the server, especially with setup of nextcloud, refdb, phpldapadmin and any
other access with LDAP backend (all https).
pid 23108 (httpd), uid 80: exited on signal 11
pid 32283 (httpd), uid 80: exited on signal 11
pid 21286 (httpd), uid 80: exited on signal 11
pid 10292 (httpd), uid 80: exited on signal 11
I do not see anything else! Neither in the log (no matter what log level I switch on) nor
elsewhere.
Configuration hasn't changed.
What the ... is up with the system? How can I dig into the problem?

sudo /usr/local/sbin/httpd -X

I meant "sudo gdb /usr/local/sbin/httpd -X", sorry.

-Dimitry

[O. Hartmann]

Am Tue, 30 Jan 2018 00:09:07 +0100

Post by Dimitry Andric

Post by Dimitry Andric

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11) when
accessing the server, especially with setup of nextcloud, refdb, phpldapadmin and any
other access with LDAP backend (all https).
pid 23108 (httpd), uid 80: exited on signal 11
pid 32283 (httpd), uid 80: exited on signal 11
pid 21286 (httpd), uid 80: exited on signal 11
pid 10292 (httpd), uid 80: exited on signal 11
I do not see anything else! Neither in the log (no matter what log level I switch
on) nor elsewhere.
Configuration hasn't changed.
What the ... is up with the system? How can I dig into the problem?

sudo /usr/local/sbin/httpd -X

I meant "sudo gdb /usr/local/sbin/httpd -X", sorry.
-Dimitry

I see this:


Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file
or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb 03
11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096 of
process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process 59320]
[New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]

Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

[O. Hartmann]

Am Sat, 3 Feb 2018 11:43:25 +0100

Post by O. Hartmann
Am Tue, 30 Jan 2018 00:09:07 +0100

Post by Dimitry Andric

Post by Dimitry Andric

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11)
when accessing the server, especially with setup of nextcloud, refdb, phpldapadmin
and any other access with LDAP backend (all https).
pid 23108 (httpd), uid 80: exited on signal 11
pid 32283 (httpd), uid 80: exited on signal 11
pid 21286 (httpd), uid 80: exited on signal 11
pid 10292 (httpd), uid 80: exited on signal 11
I do not see anything else! Neither in the log (no matter what log level I switch
on) nor elsewhere.
Configuration hasn't changed.
What the ... is up with the system? How can I dig into the problem?

sudo /usr/local/sbin/httpd -X

I meant "sudo gdb /usr/local/sbin/httpd -X", sorry.
-Dimitry

Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such
file or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb 03
11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096 of
process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process 59320]
[New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]
Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

I also filed a bug report:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225499

[Dimitry Andric]

Post by O. Hartmann
Am Tue, 30 Jan 2018 00:09:07 +0100

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11) when
accessing the server, especially with setup of nextcloud, refdb, phpldapadmin and any
other access with LDAP backend (all https).

...

Post by O. Hartmann
Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file
or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb 03
11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096 of
process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process 59320]
[New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]
Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

Run the "backtrace" command there, please. If it doesn't give any
function names, try setting WITH_DEBUG=1 in the environment, then
rebuilding the mod_php56 port (and optionally also the php56 and
apache24 ports themselves), to add debug info.

-Dimitry

[O. Hartmann]

Am Sun, 4 Feb 2018 12:28:54 +0100

Am Sat, 3 Feb 2018 14:04:17 +0100

Post by Dimitry Andric

Post by O. Hartmann
Am Tue, 30 Jan 2018 00:09:07 +0100

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and
performed updates of the ports tree. Since Sunday, I receive segmenatation
faults (SIG 11) when accessing the server, especially with setup of nextcloud,
refdb, phpldapadmin and any other access with LDAP backend (all https).

...

Post by O. Hartmann
Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such
file or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb
03 11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096
of process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process
59320] [New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]
Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

Run the "backtrace" command there, please. If it doesn't give any
function names, try setting WITH_DEBUG=1 in the environment, then
rebuilding the mod_php56 port (and optionally also the php56 and
apache24 ports themselves), to add debug info.
-Dimitry

www/mod_php56: env WITH_DEBUG=1 make reinstall (prior to that, I had to "pkg delete" the
port www/mod_php56 first). When having mod_php56 rebuilt with DEBUG on as recommended
and performing command sequence to enter gdb, run -X, the Apache 2.4 now crashes
immediately and not when the URL is called by the browser triggering the fault. This is
bt
#0 0x000000080208b382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#1 0x0000000802088e1a in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#2 0x0000000802085f12 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#3 0x0000000802085cf3 in zend_accel_script_optimize ()
from /usr/local/lib/php/20131226-zts/opcache.so #4 0x000000080207a393 in ?? ()
from /usr/local/lib/php/20131226-zts/opcache.so #5 0x00000008020799d6 in
persistent_compile_file () from /usr/local/lib/php/20131226-zts/opcache.so #6
0x000000080195d33c in zend_execute_scripts ()
from /usr/local/libexec/apache24/libphp5.so #7 0x00000008018f47a3 in
php_execute_script () from /usr/local/libexec/apache24/libphp5.so #8
0x00000008019f1015 in ?? () from /usr/local/libexec/apache24/libphp5.so #9
0x00000000002563d7 in ap_invoke_handler () #10 0x000000000029005d in
ap_process_async_request () #11 0x000000000028ca2d in ap_process_http_connection () #12
0x000000000026e3b7 in ap_run_process_connection () #13 0x000000080152827f in ?? ()
from /usr/local/libexec/apache24/mod_mpm_event.so #14 0x0000000801527e11 in ?? ()
from /usr/local/libexec/apache24/mod_mpm_event.so #15 0x00000008005d2476 in ?? ()
from /lib/libthr.so.3 #16 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffde1ef000

I forgot to mention, that I already have rebuilt www/apache24 via portmaster -df and I
also have rebuild all php ports/modules installed including php56 itself (but without
debugging, it takes a while on that box).

[O. Hartmann]

Am Sat, 3 Feb 2018 14:04:17 +0100

Post by Dimitry Andric

Post by O. Hartmann
Am Tue, 30 Jan 2018 00:09:07 +0100

Post by O. Hartmann
Last weekend I updated a CURRENT server to recent CURRENT, > r328400 and performed
updates of the ports tree. Since Sunday, I receive segmenatation faults (SIG 11)
when accessing the server, especially with setup of nextcloud, refdb, phpldapadmin
and any other access with LDAP backend (all https).

...

Post by O. Hartmann
Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such
file or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb 03
11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096 of
process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process 59320]
[New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]
Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

Run the "backtrace" command there, please. If it doesn't give any
function names, try setting WITH_DEBUG=1 in the environment, then
rebuilding the mod_php56 port (and optionally also the php56 and
apache24 ports themselves), to add debug info.
-Dimitry

www/mod_php56: env WITH_DEBUG=1 make reinstall (prior to that, I had to "pkg delete" the
port www/mod_php56 first). When having mod_php56 rebuilt with DEBUG on as recommended and
performing command sequence to enter gdb, run -X, the Apache 2.4 now crashes immediately
and not when the URL is called by the browser triggering the fault. This is what I get
then:

bt
#0 0x000000080208b382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#1 0x0000000802088e1a in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#2 0x0000000802085f12 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so
#3 0x0000000802085cf3 in zend_accel_script_optimize ()
from /usr/local/lib/php/20131226-zts/opcache.so #4 0x000000080207a393 in ?? ()
from /usr/local/lib/php/20131226-zts/opcache.so #5 0x00000008020799d6 in
persistent_compile_file () from /usr/local/lib/php/20131226-zts/opcache.so #6
0x000000080195d33c in zend_execute_scripts () from /usr/local/libexec/apache24/libphp5.so
#7 0x00000008018f47a3 in php_execute_script ()
from /usr/local/libexec/apache24/libphp5.so #8 0x00000008019f1015 in ?? ()
from /usr/local/libexec/apache24/libphp5.so #9 0x00000000002563d7 in ap_invoke_handler
() #10 0x000000000029005d in ap_process_async_request () #11 0x000000000028ca2d in
ap_process_http_connection () #12 0x000000000026e3b7 in ap_run_process_connection ()
#13 0x000000080152827f in ?? () from /usr/local/libexec/apache24/mod_mpm_event.so
#14 0x0000000801527e11 in ?? () from /usr/local/libexec/apache24/mod_mpm_event.so
#15 0x00000008005d2476 in ?? () from /lib/libthr.so.3
#16 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffde1ef000