it crashed at the function log_raw. here is the backtrace
== Backtrace #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49 #1 0x0000007f82ed54b4 in __GI_abort () at abort.c:79 #2 0x0000007f82f1f984 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f82fdc660 "%s\n") at ../sysdeps/posix/libc_fatal.c:155 #3 0x0000007f82f273dc in malloc_printerr (str=str@entry=0x7f82fd7bc8 "corrupted double-linked list") at malloc.c:5389 #4 0x0000007f82f27cc8 in unlink_chunk (p=p@entry=0x7f64012260, av=0x7f64000020) at malloc.c:1472 #5 0x0000007f82f27e48 in malloc_consolidate (av=av@entry=0x7f64000020) at malloc.c:4539 #6 0x0000007f82f29df0 in _int_malloc (av=av@entry=0x7f64000020, bytes=bytes@entry=8192) at malloc.c:3727 #7 0x0000007f82f2c5b0 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3448 #8 0x0000007f82f1e41c in __GI___open_memstream (bufloc=bufloc@entry=0x7f72ffc3f8, sizeloc=sizeloc@entry=0x7f72ffc400) at memstream.c:83 #9 0x0000007f82f7d274 in __vsyslog_internal (pri=31, pri@entry=7, fmt=fmt@entry=0x5584b5f930 "%s: %s", ap=..., mode_flags=mode_flags@entry=2) at ../misc/syslog.c:181 #10 0x0000007f82f7d810 in __syslog_chk (pri=pri@entry=7, flag=flag@entry=1, fmt=fmt@entry=0x5584b5f930 "%s: %s") at ../misc/syslog.c:136 #11 0x0000005584b45884 in syslog (__fmt=0x5584b5f930 "%s: %s", __pri=7) at /usr/include/bits/syslog.h:31 #12 log_raw (level=level@entry=7, stamp=stamp@entry=0x7f64003420 "2023.06.16 01:36:58", id=id@entry=0x7f64001120 "LOG7[19]", text=text@entry=0x7f6401b060 "Service [proxy-r] started", opt=<optimized out>, opt=<optimized out>) at ../../stunnel-5.57/src/log.c:263 #13 0x0000005584b45b88 in s_log (level=level@entry=7, format=format@entry=0x5584b5a780 "Service [%s] started") at ../../stunnel-5.57/src/log.c:192 #14 0x0000005584b454e8 in client_main (c=c@entry=0x7f82e30060) at ../../stunnel-5.57/src/client.c:178 #15 0x0000005584b45658 in client_thread (arg=0x7f82e30060) at ../../stunnel-5.57/src/client.c:130 #16 0x0000007f83026e64 in start_thread (arg=0x7fc21e457f) at pthread_create.c:463 #17 0x0000007f82f815dc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 == Thread apply all backtrace full
Hi Phan Anh,
The "corrupted double-linked list" error in malloc_consolidate() means that the heap data structures were already corrupted before executing this operation. Running stunnel with valgrind should identify the root cause. See https://valgrind.org/ for details.
Please also include your stunnel.conf and the output of "stunnel -version".
Best regards, Mike
On 7/4/23 20:52, [email protected] wrote:
it crashed at the function log_raw. here is the backtrace
== Backtrace #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49 #1 0x0000007f82ed54b4 in __GI_abort () at abort.c:79 #2 0x0000007f82f1f984 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f82fdc660 "%s\n") at ../sysdeps/posix/libc_fatal.c:155 #3 0x0000007f82f273dc in malloc_printerr (str=str@entry=0x7f82fd7bc8 "corrupted double-linked list") at malloc.c:5389 #4 0x0000007f82f27cc8 in unlink_chunk (p=p@entry=0x7f64012260, av=0x7f64000020) at malloc.c:1472 #5 0x0000007f82f27e48 in malloc_consolidate (av=av@entry=0x7f64000020) at malloc.c:4539 #6 0x0000007f82f29df0 in _int_malloc (av=av@entry=0x7f64000020, bytes=bytes@entry=8192) at malloc.c:3727 #7 0x0000007f82f2c5b0 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3448 #8 0x0000007f82f1e41c in __GI___open_memstream (bufloc=bufloc@entry=0x7f72ffc3f8, sizeloc=sizeloc@entry=0x7f72ffc400) at memstream.c:83 #9 0x0000007f82f7d274 in __vsyslog_internal (pri=31, pri@entry=7, fmt=fmt@entry=0x5584b5f930 "%s: %s", ap=..., mode_flags=mode_flags@entry=2) at ../misc/syslog.c:181 #10 0x0000007f82f7d810 in __syslog_chk (pri=pri@entry=7, flag=flag@entry=1, fmt=fmt@entry=0x5584b5f930 "%s: %s") at ../misc/syslog.c:136 #11 0x0000005584b45884 in syslog (__fmt=0x5584b5f930 "%s: %s", __pri=7) at /usr/include/bits/syslog.h:31 #12 log_raw (level=level@entry=7, stamp=stamp@entry=0x7f64003420 "2023.06.16 01:36:58", id=id@entry=0x7f64001120 "LOG7[19]", text=text@entry=0x7f6401b060 "Service [proxy-r] started", opt=<optimized out>, opt=<optimized out>) at ../../stunnel-5.57/src/log.c:263 #13 0x0000005584b45b88 in s_log (level=level@entry=7, format=format@entry=0x5584b5a780 "Service [%s] started") at ../../stunnel-5.57/src/log.c:192 #14 0x0000005584b454e8 in client_main (c=c@entry=0x7f82e30060) at ../../stunnel-5.57/src/client.c:178 #15 0x0000005584b45658 in client_thread (arg=0x7f82e30060) at ../../stunnel-5.57/src/client.c:130 #16 0x0000007f83026e64 in start_thread (arg=0x7fc21e457f) at pthread_create.c:463 #17 0x0000007f82f815dc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 == Thread apply all backtrace full _______________________________________________ stunnel-users mailing list [email protected] To unsubscribe send an email [email protected]
Hi Mike, thanks for the quick reply. It's not easy to reproduce the crash on the production system and also it's not possible to run valgrind on the production system as well. However I have tried to collect some more information as following:
stunnel-version: LOG5[ui]: stunnel 5.57 on aarch64-mbient-linux-gnu platform
stunnel.conf [proxy-r] ; local endpoint accept = 16666 ; remote endpoint connect = some-server-name:443 verifyChain = yes checkHost = some-server-name sslVersion = TLSv1.3
system log around the crash point:
593805 2023/06/16 01:36:59.284173 133.4751 105 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:58.878789 133.074348 stunnel[4346]: Debug: LOG7[ui]: Service [proxy-r] accepted (FD=16) from 127.0.0.1:38392 593807 2023/06/16 01:36:59.284181 133.4751 107 LNX SYS JOUR 927 log warn verbose 5 2023/06/16 01:36:58.881544 133.075156 kernel: Warning: CPU: 6 PID: 17279 Comm: stunnel Tainted: P W O 5.4.134-qgki #1 593830 2023/06/16 01:36:59.284839 133.4753 130 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:59.005061 133.200639 systemd[1]: Debug: Received SIGCHLD from PID 4346 (stunnel). 593831 2023/06/16 01:36:59.284948 133.4753 131 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:59.005130 133.200828 systemd[1]: Debug: Child 4346 (stunnel) died (code=killed, status=6/ABRT)
the logs for the working case should look like this: 2023.04.25 11:03:48 LOG7[ui]: Service [proxy-r] accepted (FD=16) from 127.0.0.1:56650 2023.04.25 11:03:48 LOG7[4]: Service [proxy-r] started
I have seen some refactoring regarding stunnel logging for the versions after 5.57, do you think it makes sense to upgrade the stunnel to the later version in the hope to resolve the crash? Many thanks. BR, Phan Anh
Hi Phan Anh,
Can you please execute "stunnel -version" on that system (the command "stunnel" with the "-version" parameter")?
Yes, updating both stunnel *and* OpenSSL to their latest stable versions (5.69 and 3.1.1 respectively) is a good idea.
What exactly is this "mbient-linux"? Which version of libc and OpenSSL does it use? Are there any public documentation for that project? I've seen similar errors caused by 3rd party modifications of OpenSSL or recently by a bug in musl that is used instead of glibc on Alpine Linux.
Best regards, Mike
On 7/6/23 16:54, [email protected] wrote:
Hi Mike, thanks for the quick reply. It's not easy to reproduce the crash on the production system and also it's not possible to run valgrind on the production system as well. However I have tried to collect some more information as following:
stunnel-version: LOG5[ui]: stunnel 5.57 on aarch64-mbient-linux-gnu platform
stunnel.conf [proxy-r] ; local endpoint accept = 16666 ; remote endpoint connect = some-server-name:443 verifyChain = yes checkHost = some-server-name sslVersion = TLSv1.3
system log around the crash point:
593805 2023/06/16 01:36:59.284173 133.4751 105 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:58.878789 133.074348 stunnel[4346]: Debug: LOG7[ui]: Service [proxy-r] accepted (FD=16) from 127.0.0.1:38392 593807 2023/06/16 01:36:59.284181 133.4751 107 LNX SYS JOUR 927 log warn verbose 5 2023/06/16 01:36:58.881544 133.075156 kernel: Warning: CPU: 6 PID: 17279 Comm: stunnel Tainted: P W O 5.4.134-qgki #1 593830 2023/06/16 01:36:59.284839 133.4753 130 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:59.005061 133.200639 systemd[1]: Debug: Received SIGCHLD from PID 4346 (stunnel). 593831 2023/06/16 01:36:59.284948 133.4753 131 LNX SYS JOUR 927 log debug verbose 5 2023/06/16 01:36:59.005130 133.200828 systemd[1]: Debug: Child 4346 (stunnel) died (code=killed, status=6/ABRT)
the logs for the working case should look like this: 2023.04.25 11:03:48 LOG7[ui]: Service [proxy-r] accepted (FD=16) from 127.0.0.1:56650 2023.04.25 11:03:48 LOG7[4]: Service [proxy-r] started
I have seen some refactoring regarding stunnel logging for the versions after 5.57, do you think it makes sense to upgrade the stunnel to the later version in the hope to resolve the crash? Many thanks. BR, Phan Anh _______________________________________________ stunnel-users mailing list [email protected] To unsubscribe send an email [email protected]
hi Mike, root@binz:~# stunnel -version Initializing inetd mode configuration stunnel 5.57 on aarch64-mbient-linux-gnu platform Compiled/running with OpenSSL 1.1.1q 5 Jul 2022 Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,OCSP,PSK,SNI Auth:LIBWRAP
Global options: RNDbytes = 1024 RNDoverwrite = yes
Service-level options: ciphers = HIGH:!aNULL:!SSLv2:!DH:!kDHEPSK ciphersuites = TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256 (with TLSv1.3) curves = X25519:P-256:X448:P-521:P-384 debug = daemon.notice logId = sequential options = NO_SSLv2 options = NO_SSLv3 securityLevel = 2 sessionCacheSize = 1000 sessionCacheTimeout = 300 seconds stack = 65536 bytes TIMEOUTbusy = 300 seconds TIMEOUTclose = 60 seconds TIMEOUTconnect = 10 seconds TIMEOUTidle = 43200 seconds verify = none
GNU C Library (GNU libc) stable release version 2.32. Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 9.3.0. libc ABIs: UNIQUE ABSOLUTE For bug reporting instructions, please see: https://www.gnu.org/software/libc/bugs.html.
"mbient-linux" is a customized linux which is not public. It uses also another non-public client certificate engine (qautheng). Could it be that the qautheng somehow interferes with stunnel and causes the crash? Following are the sample logs from stunnel regarding qautheng:
2023.04.25 11:03:05 LOG6[ui]: Client certificate engine (qautheng) not supported 2023.04.25 11:03:05 LOG6[ui]: Loading certificate from engine ID: /mnt/cathi/certs/client_cert.pem 2023.04.25 11:03:05 LOG3[ui]: ENGINE_ctrl_cmd: Peer suddenly disconnected
BR, Phan Anh
-
Michał Trojnara -
phananh.nguyen＠dxc.com