<div dir="ltr"><div class="gmail_quote">Hi All, We have been doing some throughput testing and are wondering<br>
of you can help explain some of the results we have been getting.<br>
<br>
We are trying to establish the maximum number of handshakes per second<br>
our appliances can do when using Stunnel.<br>
Our tests have been done on a 4 core Intel(R) Xeon(R) CPU E3-1230 v5 @<br>
3.40GHz machine with 8Gb ram. Using stunnel 5.33 openssl 1.0.2j (We<br>
use stunnel 5.33 here, but this was also tested with stunnel 5.40 and<br>
we got the same issue)<br>
<br>
Stunnel was compiled with - ./configure --enable-fips --enable-ipv6<br>
--with-threads=pthread --sysconfdir=/etc<br>
--with-ssl=/usr/src/built/Open<wbr>SSL_1_0_2j-no_march/usr/local/<br>
--disable-libwrap --disable-systemd<br>
<br>
I was using a python script of my own design for load testing when I<br>
noticed the problem we then switched to an open source package called<br>
vegeta <a href="https://github.com/tsenart/vegeta" rel="noreferrer" target="_blank">https://github.com/tsenart/veg<wbr>eta</a> the command I was using to<br>
run it was -<br>
<br>
echo "GET <a href="https://10.10.10.10:443" rel="noreferrer" target="_blank">https://10.10.10.10:443</a>" | ./vegeta attack -workers=30<br>
-rate=3000 -duration=300s -insecure > test.out<br>
<br>
We are essentially seeing a degradation in throughput at a fixed point in time.<br>
I am seeing the problem when using 3000 transactions per second. We<br>
are doing https gets through stunnel to a http backend, 1 transaction<br>
is an ssl handshake the http get and waiting for the response.<br>
Interestingly if I try 5.06 it works fine I get a smooth constant<br>
loading.<br>
<br>
Please take a look at 30worker3000rate5.33-300s.png where you can see<br>
after about 55 seconds there is a jump in latency.<br>
I get the same issue if I run the test with 2000 transactions. Please<br>
see 30worker20005.33-700s.png but with the lower transaction rate the<br>
issue takes longer to rear its head.<br>
The issue doesnt happen if I run the test with 3000 transaction using<br>
5.06. Please see 30worker3000rate5.06-300s.png<br>
<br>
>From looking at the graphs our thoughts were that we were hitting some sort<br>
of cache limit and used that as a starting point. We have taken a look at the<br>
code changes between versions we know were working and the problematic<br>
version and the main thing that stands out is the changes to the leak detection.<br>
Previously this was partially based on the number of clients but has been<br>
changed to use the sections. This led to the decision to try a<br>
different limit value<br>
and found the number of connections we saw before seeing an issue corresponds<br>
to the change of the limit value.<br>
<br>
We think we have identified the code that is causing the problem/doing<br>
something funny it was added in 5.33<br>
<br>
NOEXPORT long leak_threshold() {<br>
long limit;<br>
<br>
limit=10000*((int)number_of_s<wbr>ections+1);<br>
#ifndef USE_FORK<br>
limit+=100*num_clients;<br>
#endif<br>
return limit;<br>
}<br>
<br>
its src/str.c<br>
<br>
when increasing the limit number so<br>
limit=10000*((int)number_of_s<wbr>ections+1)<br>
to<br>
limit=20000*((int)number_of_s<wbr>ections+1)<br>
<br>
This still results in an increase in response time but at a later time<br>
in the test, please see 30worker3000rate5.33-ben-highl<wbr>imit.png<br>
<br>
Interestingly once I end up in this state I am stuck with high<br>
latencies until I restart stunnel. (The box I am testing on was left<br>
over night to make sure its not any hanging tcp connections or<br>
anything like that)<br>
<br>
I must admit we are at the limit of our C understanding and your imput<br>
would be welcomed.<br>
</div><br><br clear="all"><br>-- <br><div class="m_-6937871115618681570gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">With Kind Regards.<br><br>Scott McKeown<br>Loadbalancer.org<br><a href="http://www.loadbalancer.org" target="_blank">http://www.loadbalancer.org</a><br><div>Tel (UK) - +44 (<span style="font-family:arial;font-size:small"><a href="tel:0330%20380%201064" value="+443303801064" target="_blank">0) 3303801064</a> (24x7)</span></div><div><span style="font-family:arial;font-size:small">Tel (US) - <a href="tel:+1%20888-867-9504" value="+18888679504" target="_blank">+1 888.867.9504</a> (Toll Free)(24x7)</span></div></div></div>
</div>