Hi All, We have been doing some throughput testing and are wondering of you can help explain some of the results we have been getting.
We are trying to establish the maximum number of handshakes per second our appliances can do when using Stunnel. Our tests have been done on a 4 core Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz machine with 8Gb ram. Using stunnel 5.33 openssl 1.0.2j (We use stunnel 5.33 here, but this was also tested with stunnel 5.40 and we got the same issue)
Stunnel was compiled with - ./configure --enable-fips --enable-ipv6 --with-threads=pthread --sysconfdir=/etc --with-ssl=/usr/src/built/OpenSSL_1_0_2j-no_march/usr/local/ --disable-libwrap --disable-systemd
I was using a python script of my own design for load testing when I noticed the problem we then switched to an open source package called vegeta https://github.com/tsenart/vegeta the command I was using to run it was -
echo "GET https://10.10.10.10:443" | ./vegeta attack -workers=30 -rate=3000 -duration=300s -insecure > test.out
We are essentially seeing a degradation in throughput at a fixed point in time. I am seeing the problem when using 3000 transactions per second. We are doing https gets through stunnel to a http backend, 1 transaction is an ssl handshake the http get and waiting for the response. Interestingly if I try 5.06 it works fine I get a smooth constant loading.
Please take a look at 30worker3000rate5.33-300s.png where you can see after about 55 seconds there is a jump in latency. I get the same issue if I run the test with 2000 transactions. Please see 30worker20005.33-700s.png but with the lower transaction rate the issue takes longer to rear its head. The issue doesnt happen if I run the test with 3000 transaction using 5.06. Please see 30worker3000rate5.06-300s.png
From looking at the graphs our thoughts were that we were hitting some sort
of cache limit and used that as a starting point. We have taken a look at the code changes between versions we know were working and the problematic version and the main thing that stands out is the changes to the leak detection. Previously this was partially based on the number of clients but has been changed to use the sections. This led to the decision to try a different limit value and found the number of connections we saw before seeing an issue corresponds to the change of the limit value.
We think we have identified the code that is causing the problem/doing something funny it was added in 5.33
NOEXPORT long leak_threshold() { long limit;
limit=10000*((int)number_of_sections+1); #ifndef USE_FORK limit+=100*num_clients; #endif return limit; }
its src/str.c
when increasing the limit number so limit=10000*((int)number_of_sections+1) to limit=20000*((int)number_of_sections+1)
This still results in an increase in response time but at a later time in the test, please see 30worker3000rate5.33-ben-highlimit.png
Interestingly once I end up in this state I am stuck with high latencies until I restart stunnel. (The box I am testing on was left over night to make sure its not any hanging tcp connections or anything like that)
I must admit we are at the limit of our C understanding and your imput would be welcomed.
