Hi,
We have a problem in our production system. Need help to isolate the cause. Want to know whether issue is with stunnel. We are not in situation to upgrade, since we are in production. Your help is appreciated.
Environment:
1) Client program(java in win desktops) connect to server (solaris) Stunnel port which is forwarded locally to an inetd program which connects to oracle database. ( 2 Tier Architecture )
2) Setup worked for almost 2 years without any issue, suddenly since last September, we have connection reset issue on app clients intermittently.
3) almost lots of clients ( in 100's ) get reset and we find 100's of below messages in /var/adm/messages file of solaris.
stunnel: [ID 821868 daemon.info] LOG6[21204:5272]: s_poll_wait timeout: connection reset
stunnel: [ID 821868 daemon.notice] LOG5[21204:4804]: Connecti
on reset: 1867341 bytes sent to SSL, 55211 bytes sent to socket
4) After reset, Client attempt to relogin works without any issue. there is no pattern of load, time, no of users.
5) all the database, inetd program, stunnel runs with oracle userid
6) No other OS related, resource related errors in /var/adm/messages.
7) At this point, i dont have stunnel debug log during the problem occurrence , since we have cron job that recycles logs. if required will send later.
8) Interms of change happend before the first occurance of issue, we did CPU/RAM upgrade, Oracle Database CPU Patch. But database team says no errors reported on oracle side during reset.
9) Max users on database is around 1024, which our max users is 300.
********************************************************************
Config:
#/usr/local/bin/stunnel -version
stunnel 4.25 on
sparc-sun-solaris2.10 with OpenSSL 0.9.7d 17 Mar 2004 (+ security fixes for: CVE-2005-2969 CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-5135 CVE-2007-3108 CVE-2008-5077 CVE-2009-0590 CVE-2009-3555)
Threading:PTHREAD SSL:ENGINE Sockets:POLL,IPv6 Auth:LIBWRAP
Global options
debug = 5
pid
= /usr/local/var/run/stunnel/stunnel.pid
RNDbytes = 64
RNDfile = /dev/urandom
RNDoverwrite = yes
Service-level options
cert
= /usr/local/etc/stunnel/stunnel.pem
ciphers = ALL:!DHE-RSA-AES256-SHA:!DHE-DSS-AES256-SHA:!AES256-SHA:!ADH:+RC4:@STRENGTH
key = /usr/local/etc/stunnel/stunnel.pem
session = 300 seconds
stack = 65536 bytes
sslVersion
= SSLv3 for client, all for server
TIMEOUTbusy = 300 seconds
TIMEOUTclose = 60 seconds
TIMEOUTconnect = 10 seconds
TIMEOUTidle = 43200 seconds
verify = none
-------------------------------------------------
#uname -a
SunOS xxxxxxxxxx 5.10 Generic_144488-02 sun4u sparc SUNW,SPARC-Enterprise
---------------------------------------------------------------------
# gcc -v
Reading specs from /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/specs
Configured with: /sfw10/builds/build/sfw10-patch/usr/src/cmd/gcc/gcc-3.4.3/configure --prefix=/usr/sfw --with-as=/usr/ccs/bin/as --without-gnu-as --with-ld=/usr/ccs/bin/ld --without-gnu-ld --enable-languages=c,c++ --enable-shared
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
------------------------------------------------------------------------
# openssl version
OpenSSL 0.9.7d 17 Mar 2004 (+ security fixes for: CVE-2005-2969 CVE-2006-2937
CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-5135 CVE-2007-3108 CVE-2008-5077 CVE-2009-0590 CVE-2009-3555)
-----------------------------------------------------------------------------
# more avaloq.conf ( Ssl config file we use /usr/local/bin/stunnel /opt/app/ssl/stunnel.conf
; Sample stunnel configuration file by Michal Trojnara 2002-2006
; Some options used here may not be adequate for your particular configuration
; Please make sure you understand them (especially the effect of chroot jail)
; Certificate/key is needed in server mode and optional in client mode
cert = /opt/app/ssl/cert.pem
key = /opt/app/ssl/cert.pem
; Protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = all
; Some security enhancements for UNIX systems - comment them out on
Win32
;chroot = /usr/local/var/lib/stunnel/
;setuid = nobody
;setgid = nogroup
; PID is created inside chroot jail
pid = /opt/app/ssl/avaloq_ssl/avaloq.pid
; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
;compression = rle
; Workaround for Eudora bug
;options = DONT_INSERT_EMPTY_FRAGMENTS
; Authentication stuff
;verify = 2
; Don't forget to c_rehash CApath
; CApath is located inside chroot jail
;CApath = /certs
; It's often easier to use CAfile
;CAfile = /usr/local/etc/stunnel/certs.pem
; Don't forget to c_rehash CRLpath
; CRLpath is located inside chroot jail
;CRLpath = /crls
; Alternatively you can use CRLfile
;CRLfile =
/usr/local/etc/stunnel/crls.pem
; Some debugging stuff useful for troubleshooting
debug = 7
output = /db/avaloq/export/stunnel.log
; Use it for client mode
;client = yes
; Service-level configuration
[application]
accept =7766
connect =localhost:7767
TIMEOUTconnect = 60
;[pop3s]
;accept = 995
;connect = 110
;[imaps]
;accept = 993
;connect = 143
;[ssmtp]
;accept = 465
;connect = 25
;[https]
;accept = 443
;connect = 80
;TIMEOUTclose = 0
;
vim:ft=dosini
--------------------------------------------------------------------------------
#netstat -an |grep 7766 |wc -l
249
Regards,
kumar