Downloading files over TLS. This seems that it's something that
should be quite simple to do. My high level goals are
cross-platform and easy distribution. I don't need anything fancy
just a simple API like this:
download("https://url.com";, "/local/file");
Because of these goals, I have a few requirements, which I don't
think are unreasonable:
* Cross-platform (macOS, Linux, Windows, possible FreeBSD)
* Minimal runtime requirements (I don't mind a few compile time
requirements). I also don't mind runtime requirements that are
guaranteed to exist
* Ideally using the platform provided TLS implementation, or at
least platform provided certificates (I've been bitten several
times due to old certificates making request fail)
* On Linux I would really like to have a fully statically linked
binary, this will make distribution easier
* Due to the previous requirement, I think using musl will be
required. For most parts glibc works fine for static linking, but
I think in the case of networking it does not work. The DNS
lookup (or something like that) doesn't work with static linking
I think the main problem of all this is picking the right TLS
implementation.
The obvious candidate is std.net.curl, but I'm going to explain
why that is not ideal, at least not how it's currently
implemented. std.net.curl relies on OpenSSL, which is deprecated
on several platforms. On several platforms it's not the main,
platform provided TLS implementation. As far as I know, it also
provides its own certificates and doesn't rely on the ones
provided by the platform.
I going to break down some alternatives per platform.
macOS:
* std.net.curl. Depends on libcurl and OpenSSL. OpenSSL has been
deprecated on macOS for many years. In one wants to use OpenSSL
anyway, one need to ship it with the binary (statically link it).
For other problems, see above
* requests [1]. This is a potential better alternative than
std.net.curl since it doesn't rely on libcurl. But, it still
relies on OpenSSL, so it has the same problems as std.net.curl
* vibe.d [2]. Does not depend on libcurl, but does depend on
OpenSSL. But, it also supports Botan, which is promising. I'll
discuss Botan separately
* Hunt [3]. Same problem as the other ones, relies on OpenSSL
* Arsd [4]. Relies on OpenSSL
* SecureTransport [5]. This is an Apple specific library provided
by the platform. It seems to more or less follow the same idea as
OpenSSL, it's independent on the transport layer and works with
BSD sockets. Initially this looks perfect, it's provided by the
platform and uses certificates provided by the platform. The main
problem is that it has been deprecated. It also only supports TLS
up to version 1.2 and since it's deprecated, it's likely that it
won't support any newer versions
* Network [6]. This is an Apple specific library provided by the
platform. This is the recommend alternative to SecureTransport.
The problem is that this is not just an alternative TLS
implementation, it's a completely different alternative to BSD
sockets. The API is completely different and will require some
extra layers to to provide a cross-platform API. This means that
I cannot use any of the existing library to just add TLS, it will
be a completely different implementation, which might be ok.
Another big problem is that it only available on macOS 10.14 and
later. I have not decided yet if this is acceptable or not
* NSURLSession [7]. This is an Apple specific Objective-C class,
provided by the platform. It's higher level than the Network
framework and is available on all versions of macOS. This will
again require a some extra layers to hide the platform specific
API. There's also a problem with LDC which has very limited
support for Objective-C interoperability.
Linux:
As far as I know, OpenSSL (or some version of it, like LibreSSL)
is what's provided by the platform. So I guess that has to be
acceptable on Linux.
* std.net.curl. The main problem with std.net.curl on Linux is
that it uses `dlopen` to load the TLS library. `dlopen` doesn't
work for a statically linked executable. Side note, `dlopen`
doesn't work on iOS (for other reasons) so it would be nice if
this could be fixed
* requests. Seems to do that same as std.net.curl, uses `dlopen`
* For the other alternatives (mentioned in the macOS section), I
haven't investigated if they use `dlopen` or not
Windows:
I don't know that much of this platform.
* std.net.curl and basically all other options already mentioned
relies on OpenSSL, which is not provided by the platform
* SChannel. As far as I know, this the the platform provided
implementation of TLS on Windows.
* Are there any high level APIs, like NSURLSession, on Windows
that can be used to download files?
Botan:
This seems like a good candidate. It even exists a full D port of
it [8].
* This is not used by any specific platform but it can use the
platform provided certificates, which is a plus.
* The downside is that the code to access the platform provided
certificates has not been ported to D (or it was added to a newer
version of the C++ implementation).
* I think the the D port of Botan does not verify certificates,
which is not great. It might be due to the previous point
* I also don't know if the D port is updated to match the latest
version of the C++ implementation. By experience I know this can
be a huge task, so I don't expect it
* Botan only supports up to TLS 1.2 (even the C++
implementation). Due to the previous point it's unclear if the D
port of Botan will get support for TLS 1.3
I know there are other alternative TLS implementation, but I have
not looked into detail on these.
A question regarding libcurl. I know that curl (the CLI tool) can
take advantage of SecureTransport and SChannel as alternatives to
OpenSSL. Due to std.net.curl containing some OpenSSL related
code, is TLS handled completely outside of libcurl? So
std.net.curl can't easily take advantage of SecureTransport and
SChannel the same way as curl does?
Is there anything obvious I'm missing or why does this seem so
difficult? Do I have too high expectations and requirements?
[1] https://code.dlang.org/packages/requests
[2] https://code.dlang.org/packages/vibe-d
[3] https://code.dlang.org/packages/hunt
[4] https://code.dlang.org/packages/arsd-official%3Ahttp
[5]
https://developer.apple.com/documentation/security/secure_transport
[6] https://developer.apple.com/documentation/network
[7]
https://developer.apple.com/documentation/foundation/nsurlsession
[8] https://code.dlang.org/packages/botan
--
/Jacob Carlborg
