Hello all on the list, This mail is a follow up to this old discussion : http://stunnel.mirt.net/pipermail/stunnel-users/2007-November/001728.html
It talks about Webdav over SSL renaming problem, explanations and solutions; ie when one setups stunnel between an http client and an httpS server to access DAV location, some features work (load documents up and down) but renaming files fails.
Well, after having encountered a similar problem and googled for almost nothing, I have written a memo on the particular problem of (AND SOLUTION TO) "MOVE/COPY http methods returning error 502 when one wants to rename files on a DAV location accessed from a client such as "Windows WebDAV Miniredir" or other, via stunnel".
To ease googling of this page, all this stuff relates to WebDAV over SSL, Webdav over Stunnel, CANNOT rename file on a webdav location accessed via SSL/stunnel on a HTTPS DAV server, and also relates to "move big file is very long when accessing a DAV location over SSL/stunnel".Other relevant keywords: "net use over stunnel to dav location fails in renaming files", "Windows WebDav MiniRedir over stunnel fails in renaming files".
Anyway I can say that STUNNEL has no bug in this matter, but as being involved in the particular failing configuration commonly presented in various online tutorials, I thought that this forum was a good place for this memo.
I also put this memo in apache bug tracker here : https://issues.apache.org/bugzilla/show_bug.cgi?id=45449
Let me now include my memo, and apologize for all in the forum if it is a little bit "boring"...
====================================== I/ The problem: Briefly: one cannot rename, move or copy (from remote to remote) files on a https webdav location accessed via stunnel, from dav client such as Windows Webdav Mini-Redir (ie "net use" mapped drive).
However rename or copy work with "Add Network Place" wizard, but is very long for big files. Also, via stunnel, accessing a classic www location works fine (provided that all url in html files are relative!)
II/ CAUSE:
Quick comments : there is no bug in Apache, no bug in stunnel, and no bug (related to this pb) in Windows webdav miniredir client
so what?...
The method MOVE and COPY of the dav protocol are most often implemented in dav clients by using an absolute destination url such as "http://myserver.mydomain/folder1" or "https://sslserver.otherdomain/folder2", WHILE the "source" document url is just expressed as a relative path to the "Host" designated in the http header of the request. All other http and dav methods, which are essentially some kind of "get something methods" are most often implemented by using relative path such "/folderX/adoc" to designate a document of interest, "relative" means relative to the "host" field defined in the header of the http request. Note: The PUT method is built the same way as the GET methods: ie with a relative path to the implicit "document root" of the server designated in the "Host:" http header field of the request (see RFC 2616 for http, RFC4918 for dav). Well, from a vocabulary point of view, rfc2616 call my "relative path" as "abs-path" because they start with a "/", and not a dot (see chapter 3.2.1 of rfc2616).
That said, to secure a dav communication between a XP client and a DAV server, some people need to set up an ssl (s)tunnel between a WXP/Dav MiniRedir client and a HTTPS/dav server: this because WXP client are not able to manage "https connection" directly through "net use". So they usually use a setup like this on their client machine stunnel.conf file: accept: localhost:80 connect: remotehost:443 + some other stuff for cert management. The XP webdav miniredir client is only able to connect to 80 location. On XP "net use" does not support https locations nor forcing port number to any desired value.
So, with such setup, clients are about to build url like this "http://localhost/remotefolder/remotedoc" in their dav client, then put this request in an ssl tunnel (stunnel) which will transport this request to the 443 port of a remote server. Let's say that the remote server is an Apache one (for example).
A classic configuration of Apache is to setup two ip-based vhosts to access the same locations (it is very simple and even recommended because apache does not support ssl access on name-based vhost): - one ip-based on 80 port (0.0.0.0:80 is the good way to write this, _default_ and * are not appropriate in that case) - the other ip-based on 443 port Some may restrict the 80 vhost to "apache server local network", and forbid access from the whole internet, and keep 443 access with client cert validation for external access.
So what happens when MOVE/RENAME is rejected : the http client will build a MOVE request specifying : - a source document as a relative "file path", relative to "localhost", which is an "happily" AMBIGUOUS name (or address if used as 127.0.0.1) that will allow BOTH the client to send its resquest in its local tunnel-end AND the server to interpret url as relative to ITSELF (ie the server, not the client) at the other end of the (s)tunnel, so that requested document can be reaaly reached. I will explain this further but you have to understand that it is really a chance that the name "localhost" can serve two different purposes on two different machines, when it is used an an url exchanged between the two machines, because an url should express a unic "place" -not two! I call that the ubiquity of localhost based url, such as "/localhost/src_doc", relative to the "localhost machine" but in fact relevant on the SERVER machine - a destination location as an ABSOLUTE url, such as "http://localhost/newpath/newdoc_name", also relevant in fact only on the server machine
Then there are at least two potential sources of pb: 1/ nothing assures the client that the server will be able/allowed to reach/write to such a location "http://someserver" (even if someserver is the "itself" remote server contacted by the client, because http urls are not "the same" as https urls) 2/ even if that location is reachable, the server receives this request on its 443 port and may expect that url be specified as https, and may consider receiving an http url as a bug (because Apache does not want to proxy by default between two of its vhosts: you have to ask it ...and I think this in NORMAL). This is where the bug mainly is !
III/ Various comments and questions
1/ the "localhost" ubiquity good bug ----------------------------------------- There should be a 3rd problem, but this will fortunately not be the case because of the "ubiquity/ambivalence" of the "locahost" hostname both in the client network space, and on the server network space: normally, when the client builds an url beginning with "http://localhost", it really wants to address and truly adresses a http request to ITS own host machine: by the stunnel magic that request comes to the REMOTE server which, except on particular requests, usually accepts this and serves it : that means that the server, when seeing such request, interprets "localhost" as "ITSELF", and as "itself" is in general allowed to access all its web locations, then the server accepts to serve the request and send back a document in the tunnel back to the client. You have to understand that this is NOT (so) normal because the "locahost" requested by the client is TRULY not the same host machine as that "understood" by the server. This is what I call the "localhost" ubiquity good-bug : without this bug no ssl access through stunnel would work (no www, no dav, no http, no https, nothing!). This property is not at all related to a kind of "proxying" role of stunnel: this is something totally different. Remember also that stunnel is NOT an http proxy so it transports http requests "as is".
2/ HTTP + SSL is NOT HTTPS ------------------------------------ Because absolute http url requested by the client are directed to an https server: as the client does not know that it is talking to an httpS server, it does not build correct urls. So it is even a chance that most request works (due to localhost ambivalence and usage of relative path)...
3/ role of stunnel in HTTP over SSL -----------------------------------------
In that matter stunnel is not acting as a transparent proxy (this is not related to the "transparent proxy option" of stunnel) : if it were perfectly transparent it would not redirect a request to port 80 to a 443 port, and will not "route" requests for THE localhost to a machine on the internet. It is also not used as a proxy because the client does not deal with the remote adress but needs to know the adress of the stunnel client machine (localhost in the present case).
In fact stunnel, when used only on one side of a end-to-end communication, is acting as a GATEWAY between two networks : one talking ssl, the other not. As such it can translate protocols by removing the payload from an ordinary tcp/ip packet on one side and putting it in a new ssl tcp/ip packet on the other side. We can also say that stunnel acts as a "reverse proxy" hiding a remote machine behind itself...
Well do not think that tricking your stunnel conf file with "accept 443" and "connect 443" will solve your problem: because absolute urls in your requests still begin with http ! and Apache will still not appreciate it (no implicit proxying). Moreover MS XP clients will not allow you to mix http / 443 in a net use command (it is possible on Vista but useless for other reasons: on vista you can directly connect to https locations with net use)
Stunnel is not either acting as a ...tunnel ! No it does not, because a pure tunnel will encapsulate requests at one end and remove encapsulation at the other end.
4/ Some workarounds (but the true solution is later) ----------------------------------------------------------- WORKAROUND 1 : if you are impatient to solve your problem, a good solution is to put stunnel at both ends AND remove SSL support from Apache: so put stunnel both on the client and on the server to REALLY build a TRANSPARENT tunnel: I mean transparent to the fact that both ends believes to talk directly "http on 80 port", as they are in fact tunneled through an ssl session. Also note that this true ssl tunnel will work thanks to the "localhost ambivalence": without that ambivalence, not even a simple get request would work. NOTE: please check your apache conf with various directives such as "usecanonicalname" : I put if off in my conf,
WORKAROUND 2: of course something else which works is "true https-able client connecting directly to true https server": this is possible by using the Windows Vista webdav mini-redir directly talking to apache server on 443. Well in that case you will still have ANOTHER MS BUG : "net use" DOES not support BOTH "client cert auth" AND "basic auth by username/password" to access directly a folder on the server (whereas "add network place" does!). If you want the Vista client to work you will be obliged to remove "basic auth restriction" at least at the (dav-)root / location on the server: this is not a good new. Anyway this will work perfectly if you only use SSL to authenticate the server, not the client, and keep your basic auth settings unchanged. Well this solution does not involve stunnel so let us continue...
5/ BUT why other requests such as put/get are working? -------------------------------------------------------------------- Thanks to the localhost ambivalence AND to relative path in http requests other than MOVE and COPY (which are dav requests, and which are the SOLE requests having a destination field, and where this field is implemented as an absolute URL).
6/ NOTE about absolute url in MOVE/COPY destination field ---------------------------------------------------------------------- Some people think that the RFC4918 does not allow "relative url" in the destination field : this is WRONG. But it is a fact that most clients implement "destination" just as "absolute urls": this is like it is.
Some people may think "absolute url, well ok, so it implies that moving a doc between two servers or two "services" on the same server should be possible!" : yes BUT: 1/ the RFC does not say that it is mandatory ! and 2/ Moreover "miracles" are just not possible if various restrictions disallow this. 3/ Moreover requesting a "write access" on a location that is not in the "name space" of the server currently servicing the request is just not logical, here is why : if it were the case, it would mean that any server should act as a proxy to any other : although proxy server exist and are described in the rfc, a http server is not automatically a proxy server; proxy servers are a special family of servers that require EXPLICIT configuration, NOT implicit configuration. So "implicit proxying", although not explicitely forbidden by the RFC, is a NON SENSE vs the RFC. Further it must be noted that the proxying mechanisms described in the RFC2616 are NOT involved when performing HTTP-DAV over SSL ! because this is a particular case of "proxying at tcp/ip level", not http level, no the RFC is not relevant to solve this case: ie a server is not supposed to serve a request in a situation not described in the RFC.
7/ BUT Why "rename is working with "add network place wizard", or some other dav client: -------------------------------------------------------------------------------------------------------- Because your dav client implements some tricks : Some clients (such as Windows "Add Network Place" wizard) first try a true MOVE or COPY method, then they receive a 502 error ("Bad gateway") that I really think is a logical reason for rejecting the request: the reached host (127.0.0.1:443) is NOT implicitely a PROXY to 127.0.0.1:80 (as implied by the destination url in the MOVE/COPY request, forged as such by a client believing to act in a http/80 world), so it just CANNOT serve your request. So when copy/move fail, those clients (as "Add Network Place") try a GET (to local hard drive) and PUT (to remote machine) : well ok, it seems a good idea but it is NOT because of network speed (often low in upload) and delays in case of big files.
IV/ SO what is the solution ?
Remember I already stated two workarounds : 1/ true SSL tunneling by using stunnel BOTH on the client AND the server, to join a http client/80 and a http/80 server, and removing of SSL support in Apache: to secure your server just use apache or firewall restrictions such as "port 80 only reachable from lan or loopback or localhost!"). If 80 is restricted to localhost, the stunnel server must be installed on the http server, of course...
2/ On Vista, by directly using "net use https://", forgetting stunnel, but provided that you disable "basic auth" at least at the root of the mounted remote location.
But these are just workarounds.
**********
A solution based on stunnel client and apache classic conf directives exists:
Well you have understood that the problem comes from the fact that Apache DOES NOT want to act AS A PROXY from "127.0.0.1:443 service" to "127.0.0.1:80" service. And I really think it is normal.
So you may have 2 solutions : 1/ by doing some "rewrite" magic in your 443 vhost: 1.1/ either rewrite the destination field as A RELATIVE url 1.2/ either rewrite the destination field as beginning with https://localhost BUT ALL THIS WILL NOT be possible using standard apache directives because apache rewrite engine does not give the ability to rewrite this field. Should it be able to do that : not sure...and I do not request the apache team to do that.
2/ the TRUE solution is here : So you cannot rewrite the request, hmm, BUT you COULD REDIRECT IT to a service that COULD understand it as it is (because in fact the request can be considered as well-formed when received by an http server, although not by an httpS server), by using one of the apache redirection and/or proxying features. Well, in that particular case, redirect is NOT the good solution. The solution is in "PROXYING" !
The PROXYING solution : When you have finished this paper you will say "all this blah blah" for a few lines of config...yes but it is necessary to really master apache conf, in general and because this conf is also able to open big security holes.
"Forward proxy" is not the solution (you are lucky, I will NOT explain why). The solution is in "reverse proxying". See the apache doc about ProxyPass directive, as this is the basis for setting up reverse proxy...
Anyway, If you roughly configure "ProxyPass" directives, it will work for MOVE/COPY with stunnel and "net use" but will make FAIL MANY OTHER things, and in particular for "true https clients talking directly to the 443 server": because if you have a WELL formed destination url beginning with HTTPS, a raw proxying will send this to the "80 service" which will REJECT IT !
SO the solution is to only (reverse) proxy requests that really need to be rev-proxied ! For that the only solution is to use the " rewritecond...rewriterule ...[P] flag" magics, to ONLY rev-proxy <<MOVE and COPY methods arriving on 443 vhost AND having a destination field beginning with "http://%22%3E%3E, which defines perfectly http requests sent via a client SSL stunnel gateway.
Then here is the code to add this slective reverse proxying in the apache configuration :
# ================================================================ # IP BASED VIRTUAL HOSTS # ================================================================
<VirtualHost 0.0.0.0:80> # ServerName (not used, this is a reminder related to usecanonicalname-on; we prefer to use usecanonicalname Off!) ServerName http://LAN-DNS-NAME:80 DocumentRoot "/var/www/html" </VirtualHost>
<VirtualHost 0.0.0.0:443>
# ServerName useless but it is a reminder (see recommendation in ServerName apache doc however fort https vhost) ServerName https://WAN-DNS-NAME:443 DocumentRoot "/var/www/html"
# SSL configuration: PUT WHAT YOU WANT HERE provided that you know what you are doing ! # -----------------
SSLEngine on SSLOptions +StrictRequire
# SSL Cipher Suite: SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
# SSL Protocol support: SSLProtocol all -SSLv2
# I only accept users from my CA SSLCACertificateFile /etc/pki/tls/cacert.pem
# CRL check SSLCARevocationFile /etc/pki/tls/cacrl.pem
# Server Certificate Chain File SSLCertificateChainFile /etc/pki/tls/certs/localhost-chain.crt
# Server Certificate: SSLCertificateFile /etc/pki/tls/certs/localhost.crt
# Server Private Key: SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
# VERY IMPORTANT
SSLVerifyClient require
SSLVerifyDepth 1
<Location /> SSLRequireSSL </Location>
# REVERSE PROXYING : solution to "WEBDAV over STUNNEL and RENAME issue", the solution starts here # ----------------------------------------------------------------------------------------------------------------------------
RewriteEngine On
# Some env vars useful later
RewriteRule "(.*)" - [env=V443_URL:$1]
RewriteCond %{SSL:SSL_CLIENT_S_DN} "^(.*)$" [NC] RewriteRule "(.*)" - [env=MY_SSL_CLIENT:%1]
# REVERSE PROXYING : solution to "WEBDAV over STUNNEL and RENAME issue" # ----------------------------------------------------------------------------------------- # This is also the solution to problem in renaming files on a subversion server accessed through dav.
RewriteCond %{REQUEST_METHOD} (MOVE|COPY) [NC] RewriteCond %{HTTP:Destination} "^(http://.*)$" [NC] RewriteRule ^/(.*)$ http://127.0.0.1:80/$1 [proxy] # Note : [proxy] can be abreviated as [P] # RAF : b flag not supported ? (cf unescape)
# SOME security on the proxy: DO NOT HESITATE TO HARDEN THIS more than I did <Proxy http://127.0.0.1:80/%3E Order allow,deny Allow from env=MY_SSL_CLIENT </Proxy>
</VirtualHost>
# Some useful logs CustomLog logs/mylogs.log "SSL_DN is %{MY_SSL_CLIENT}e" CustomLog logs/mylogs.log "V443URL is %{V443_URL}e"
V/ Miscellaneous comments --------------------------------------
1/ Some people have the same kind of problem: - when using dav access to subversion location, - with various dav clients (cadaver, goliath) - when dealing with error 502 and MOVE/COPY methods All this problems have the same cause and solution described above : reverse proxying config on your https server.
2/ a little note about www access (not dav): Be careful to only have relative url in your various "href" properties in your links, and "usecanonicalname off" on apache, in order to have both http and https access working on the same directories.
3/ General note Think also to have SSLRequireSSL directive in apache conf: do not think to solve the "move/rename/copy" problem with dav over stunnel by removing this important directive: you will only open security holes.
4/ Some people have proposed some similar solutions either by rewriting or by proxying, but these solutions usually propose to write some kind of Apache module or filters, and in fact circumvent apache security by activating a kind of "implicit proxying" which is really a bad idea.
5/ use apache logs (and particularly logs of envvars) and wireshark (ex-ethereal) to diagnose your dav-client behavior: but remember that you can only sniff clear traffic SO it in only possible on the client BUT unfortunately it is NOT possible to sniff the loopback interface on a XP/Vista machine : the trick is to sniff dav traffic on a clear local lan without SSL support, to understand how your client manage various http methods, then to extrapolate when this traffic is transferred through ssl. In pure ssl traffic, apache logs can be enriched to track the "Destination field", with an envvar extracted by the headers through some rewriterule.
VI/ Some useful links ------------------------. Good discussion about dav+https (follow the 2 links at the bottom of the page: one to simple-groupware CMS, the other to "smallvoid" nett use guide) http://devwfb.blogspot.com/2008/07/how-to-map-https-only-web-folder-to.html http://www.simple-groupware.de/cms/WebDAV http://smallvoid.com/article/winnt-webdav-network-drive.html
DAV and SUBVERSION: http://www.phwinfo.com/forum/alt-apache-configuration/351885-webdav-over-ssl... http://sethd.org/Site/SubversionProxyFix.html http://svn.haxx.se/users/archive-2006-03/0544.shtml
Good info about webdav implementation by MS: http://greenbytes.de/tech/webdav/webdav-redirector-list.html
an howto to setup dav+stunnel http://www.autohotkey.com/forum/topic44471.html
some other howto: http://librenix.com/?page=SSL http://www.howtoforge.com/webdav_with_ssl_and_two_factor_authentication
Apache "bug" that I recommend NOT to fix : https://issues.apache.org/bugzilla/show_bug.cgi?id=45449
Some stunnel discussion about this: http://stunnel.mirt.net/pipermail/stunnel-users/2007-November/001728.html
========================
Yours sincerely, Pierre Delaage