Hi, I merged your first patch on top of mine and it all works well now. It also passes all the tests. I also removed old code that appeared in my previous patch (in connect.c) that should _not_ go to production.
You added the support for proxy for FTPS, which I ended up forgotting. Well done! Tell me if you still see something broken. Regards, - AJ On Fri, 2015-09-11 at 10:54 +0200, Tim Ruehsen wrote: > > Hi Ander, > > two things that I found. > > 1. [PATCH 0002] Some FTPS pieces seem to be missing. > I went through the code by searching for SCHEME_FTP and "ftp" and added FTPS > stuff where it was missing. I did not think throughly about what I did - so > please just don't apply those changes blindly. > > With those changes, Wget tried to work recursively but hangs on the first > PASSIVE data transfer. > > 2. [PATCH 0003 ]'using_data_security' in getftp() is a local variable, reset > for each file, and only set if (prot != PROT_CLEAR). I turned the logic and > voila, recursion works. Again, think if this change might break something > else > that I didn't test. > > Regards, Tim
From e8e3033f0ed4a5ffd7ade8325c86ccd3da495fb3 Mon Sep 17 00:00:00 2001 From: Ander Juaristi <[email protected]> Date: Thu, 27 Aug 2015 16:32:36 +0200 Subject: [PATCH] Added support for FTPS * doc/wget.texi: updated documentation to reflect the new FTPS functionality. * src/ftp-basic.c (ftp_greeting): new function to read the server's greeting. (ftp_login): greeting code was previously here. Moved to ftp_greeting to support FTPS implicit mode. (ftp_auth): wrapper around the AUTH TLS command. (ftp_ccc): wrapper around the CCC command. (ftp_pbsz): wrapper around the PBSZ command. (ftp_prot): wraooer around the PROT command. * src/ftp.c (get_ftp_greeting): new static function. (init_control_ssl_connection): new static function to start SSL/TLS on the control channel. (getftp): added hooks to support FTPS commands (RFCs 2228 and 4217). (ftp_loop_internal): test for new FTPS error codes. * src/ftp.h: new enum 'prot_level' with available FTPS protection levels + prototypes of previous functions. New flag for enum 'wget_ftp_fstatus' to track whether the data channel has some security mechanism enabled or not. * src/gnutls.c (struct wgnutls_transport_context): new field 'session_data'. (wgnutls_close): free GnuTLS session data before exiting. (ssl_connect_wget): save/resume SSL/TLS session. * src/http.c (establish_connection): refactor ssl_connect_wget call. (metalink_from_http): take into account SCHEME_FTPS as well. * src/init.c, src/main.c, src/options.h: new command line/wgetrc options. (main): in recursive downloads, check for SCHEME_FTPS as well. * src/openssl.c (struct openssl_transport_context): new field 'sess'. (ssl_connect_wget): save/resume SSL/TLS session. * src/retr.c (retrieve_url): check new scheme SCHEME_FTPS. * src/ssl.h (ssl_connect_wget): refactor. New parameter of type 'int *'. * src/url.c. src/url.h: new scheme SCHEME_FTPS. * src/wget.h: new FTPS error codes. * src/metalink.h: support FTPS scheme. --- doc/wget.texi | 37 +++++++++ src/ftp-basic.c | 140 ++++++++++++++++++++++++++++++--- src/ftp.c | 237 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- src/ftp.h | 21 ++++- src/gnutls.c | 35 ++++++++- src/http.c | 5 +- src/init.c | 10 +++ src/main.c | 25 +++++- src/metalink.h | 2 +- src/openssl.c | 16 +++- src/options.h | 4 + src/recur.c | 15 ++-- src/retr.c | 15 +++- src/ssl.h | 2 +- src/url.c | 11 ++- src/url.h | 4 + src/wget.h | 3 +- 17 files changed, 545 insertions(+), 37 deletions(-) diff --git a/doc/wget.texi b/doc/wget.texi index 6666b27..f1244aa 100644 --- a/doc/wget.texi +++ b/doc/wget.texi @@ -2008,6 +2008,43 @@ this option has no effect. Symbolic links are always traversed in this case. @end table +@section FTPS Options + +@table @samp +@item --ftps-implicit +This option tells Wget to use FTPS implicitly. Implicit FTPS consists of initializing +SSL/TLS from the very beginning of the control connection. This option does not send +an @code{AUTH TLS} command: it assumes the server speaks FTPS and directly starts an +SSL/TLS connection. If the attempt is successful, the session continues just like +regular FTPS (@code{PBSZ} and @code{PROT} are sent, etc.). +Implicit FTPS is no longer a requirement for FTPS implementations, and thus +many servers may not support it. If @samp{--ftps-implicit} is passed and no explicit +port number specified, the default port for implicit FTPS, 990, will be used, instead +of the default port for the "normal" (explicit) FTPS which is the same as that of FTP, +21. + +@item --no-ftps-resume-ssl +Do not resume the SSL/TLS session in the data channel. When starting a data connection, +Wget tries to resume the SSL/TLS session previously started in the control connection. +SSL/TLS session resumption avoids performing an entirely new handshake by reusing +the SSL/TLS parameters of a previous session. Typically, the FTPS servers want it that way, +so Wget does this by default. Under rare circumstances however, one might want to +start an entirely new SSL/TLS session in every data connection. +This is what @samp{--no-ftps-resume-ssl} is for. + +@item --ftps-clear-data-connection +All the data connections will be in plain text. Only the control connection will be +under SSL/TLS. Wget will send a @code{PROT C} command to achieve this, which must be +approved by the server. + +@item --ftps-fallback-to-ftp +Fall back to FTP if FTPS is not supported by the target server. For security reasons, +this option is not asserted by default. The default behaviour is to exit with an error. +If a server does not successfully reply to the initial @code{AUTH TLS} command, or in the +case of implicit FTPS, if the initial SSL/TLS connection attempt is rejected, it is +considered that such server does not support FTPS. +@end table + @node Recursive Retrieval Options, Recursive Accept/Reject Options, FTP Options, Invoking @section Recursive Retrieval Options diff --git a/src/ftp-basic.c b/src/ftp-basic.c index b058af9..bcb7847 100644 --- a/src/ftp-basic.c +++ b/src/ftp-basic.c @@ -135,6 +135,23 @@ ftp_request (const char *command, const char *value) return res; } +uerr_t +ftp_greeting (int csock) +{ + uerr_t err = FTPOK; + char *response = NULL; + + err = ftp_response (csock, &response); + if (err != FTPOK) + goto bail; + if (*response != '2') + err = FTPSRVERR; + +bail: + if (response) + xfree (response); + return err; +} /* Sends the USER and PASS commands to the server, to control connection socket csock. */ uerr_t @@ -144,16 +161,6 @@ ftp_login (int csock, const char *acc, const char *pass) char *request, *respline; int nwritten; - /* Get greeting. */ - err = ftp_response (csock, &respline); - if (err != FTPOK) - return err; - if (*respline != '2') - { - xfree (respline); - return FTPSRVERR; - } - xfree (respline); /* Send USER username. */ request = ftp_request ("USER", acc); nwritten = fd_write (csock, request, strlen (request), -1); @@ -422,6 +429,119 @@ ip_address_to_eprt_repr (const ip_address *addr, int port, char *buf, buf[buflen - 1] = '\0'; } +#ifdef HAVE_SSL +/* + * The following three functions defined into this #ifdef block + * wrap the extended FTP commands defined in RFC 2228 (FTP Security Extensions). + * Currently, only FTPS is supported, so these functions are only compiled when SSL + * support is available, because there's no point in using FTPS when there's no SSL. + * Shall someone add new secure FTP protocols in the future, feel free to remove this + * #ifdef, or add new constants to it. + */ + +/* + * Sends an AUTH command as defined by RFC 2228, + * deriving its argument from the scheme. For example, if the provided scheme + * is SCHEME_FTPS, the command sent will be "AUTH TLS". Currently, this is the only + * scheme supported, so this function will return FTPNOAUTH when supplied a different + * one. It will also return FTPNOAUTH if the target server does not support FTPS. + */ +uerr_t +ftp_auth (int csock, enum url_scheme scheme) +{ + uerr_t err = 0; + int written = 0; + char *request = NULL, *response = NULL; + + if (scheme == SCHEME_FTPS) + { + request = ftp_request ("AUTH", "TLS"); + written = fd_write (csock, request, strlen (request), -1); + if (written < 0) + { + err = WRITEFAILED; + goto bail; + } + err = ftp_response (csock, &response); + if (err != FTPOK) + goto bail; + if (*response != '2') + err = FTPNOAUTH; + } + else + err = FTPNOAUTH; + +bail: + xfree (request); + xfree (response); + + return err; +} + +uerr_t +ftp_pbsz (int csock, int pbsz) +{ + uerr_t err = 0; + int written = 0; + char spbsz[5]; + char *request = NULL, *response = NULL; + + snprintf (spbsz, 5, "%d", pbsz); + request = ftp_request ("PBSZ", spbsz); + written = fd_write (csock, request, strlen (request), -1); + if (written < 0) + { + err = WRITEFAILED; + goto bail; + } + + err = ftp_response (csock, &response); + if (err != FTPOK) + goto bail; + if (*response != '2') + err = FTPNOPBSZ; + +bail: + xfree (request); + xfree (response); + + return err; +} + +uerr_t +ftp_prot (int csock, enum prot_level prot) +{ + uerr_t err = 0; + int written = 0; + char *request = NULL, *response = NULL; + /* value must be a single character value */ + char value[2]; + + value[0] = prot; + value[1] = '\0'; + + request = ftp_request ("PROT", value); + written = fd_write (csock, request, strlen (request), -1); + if (written < 0) + { + err = WRITEFAILED; + goto bail; + } + + err = ftp_response (csock, &response); + if (err != FTPOK) + goto bail; + if (*response != '2') + err = FTPNOPROT; + +bail: + xfree (request); + xfree (response); + + return err; +} +#endif /* HAVE_SSL */ + /* Bind a port and send the appropriate PORT command to the FTP server. Use acceptport after RETR, to get the socket of data connection. */ diff --git a/src/ftp.c b/src/ftp.c index 9dab99c..80420ef 100644 --- a/src/ftp.c +++ b/src/ftp.c @@ -44,6 +44,7 @@ as that of the covered work. */ #include "url.h" #include "retr.h" #include "ftp.h" +#include "ssl.h" #include "connect.h" #include "host.h" #include "netrc.h" @@ -237,6 +238,78 @@ print_length (wgint size, wgint start, bool authoritative) static uerr_t ftp_get_listing (struct url *, ccon *, struct fileinfo **); +static uerr_t +get_ftp_greeting(int csock, ccon *con) +{ + uerr_t err = 0; + + /* Get the server's greeting */ + err = ftp_greeting (csock); + if (err != FTPOK) + { + logputs (LOG_NOTQUIET, "Error in server response. Closing.\n"); + fd_close (csock); + con->csock = -1; + } + + return err; +} + +#ifdef HAVE_SSL +static uerr_t +init_control_ssl_connection (int csock, struct url *u, bool *using_control_security) +{ + bool using_security = false; + + /* If '--ftps-implicit' was passed, perform the SSL handshake directly, + * and do not send an AUTH command. + * Otherwise send an AUTH sequence before login, + * and perform the SSL handshake if accepted by server. + */ + if (!opt.ftps_implicit && !opt.server_response) + logputs (LOG_VERBOSE, "==> AUTH TLS ... "); + if (opt.ftps_implicit || ftp_auth (csock, SCHEME_FTPS) == FTPOK) + { + if (!ssl_connect_wget (csock, u->host, NULL)) + { + fd_close (csock); + return CONSSLERR; + } + else if (!ssl_check_certificate (csock, u->host)) + { + fd_close (csock); + return VERIFCERTERR; + } + + if (!opt.ftps_implicit && !opt.server_response) + logputs (LOG_VERBOSE, " done.\n"); + + /* If implicit FTPS was requested, we act as "normal" FTP, but over SSL. + * We're not using RFC 2228 commands. + */ + using_security = true; + } + else + { + /* The server does not support 'AUTH TLS'. + * Check if --ftps-fallback-to-ftp was passed. */ + if (opt.ftps_fallback_to_ftp) + { + logputs (LOG_NOTQUIET, "Server does not support AUTH TLS. Falling back to FTP.\n"); + using_security = false; + } + else + { + fd_close (csock); + return FTPNOAUTH; + } + } + + *using_control_security = using_security; + return NOCONERROR; +} +#endif + /* Retrieves a file with denoted parameters through opening an FTP connection to the server. It always closes the data connection, and closes the control connection in case of error. If warc_tmp @@ -260,6 +333,15 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread, char type_char; bool try_again; bool list_a_used = false; +#ifdef HAVE_SSL + enum prot_level prot = (opt.ftps_clear_data_connection ? PROT_CLEAR : PROT_PRIVATE); + /* these variables tell whether the target server + * accepts the security extensions (RFC 2228) or not, + * and whether we're actually using any of them + * (encryption at the control connection only, + * or both at control and data connections) */ + bool using_control_security = false, using_data_security = false; +#endif assert (con != NULL); assert (con->target != NULL); @@ -285,8 +367,34 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread, local_sock = -1; con->dltime = 0; +#ifdef HAVE_SSL + if (u->scheme == SCHEME_FTPS) + { + /* Initialize SSL layer first */ + if (!ssl_init ()) + { + scheme_disable (SCHEME_FTPS); + logprintf (LOG_NOTQUIET, _("Could not initialize SSL. It will be disabled.")); + err = SSLINITFAILED; + return err; + } + + /* If we're using the default FTP port and implicit FTPS was requested, + * rewrite the port to the default *implicit* FTPS port. + */ + if (opt.ftps_implicit && u->port == DEFAULT_FTP_PORT) + { + DEBUGP (("Implicit FTPS was specified. Rewriting default port to %d.\n", DEFAULT_FTPS_IMPLICIT_PORT)); + u->port = DEFAULT_FTPS_IMPLICIT_PORT; + } + } +#endif + if (!(cmd & DO_LOGIN)) - csock = con->csock; + { + csock = con->csock; + using_data_security = con->st & DATA_CHANNEL_SECURITY; + } else /* cmd & DO_LOGIN */ { char *host = con->proxy ? con->proxy->host : u->host; @@ -308,6 +416,43 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread, else con->csock = -1; +#ifdef HAVE_SSL + if (u->scheme == SCHEME_FTPS) + { + /* If we're in implicit FTPS mode, we have to set up SSL/TLS before everything else. + * Otherwise we first read the server's greeting, and then send an "AUTH TLS". + */ + if (opt.ftps_implicit) + { + err = init_control_ssl_connection (csock, u, &using_control_security); + if (err != NOCONERROR) + return err; + err = get_ftp_greeting (csock, con); + if (err != FTPOK) + return err; + } + else + { + err = get_ftp_greeting (csock, con); + if (err != FTPOK) + return err; + err = init_control_ssl_connection (csock, u, &using_control_security); + if (err != NOCONERROR) + return err; + } + } + else + { + err = get_ftp_greeting (csock, con); + if (err != FTPOK) + return err; + } +#else + err = get_ftp_greeting (csock, con); + if (err != FTPOK) + return err; +#endif + /* Second: Login with proper USER/PASS sequence. */ logprintf (LOG_VERBOSE, _("Logging in as %s ... "), quotearg_style (escape_quoting_style, user)); @@ -365,6 +510,46 @@ Error in server response, closing control connection.\n")); default: abort (); } + +#ifdef HAVE_SSL + if (using_control_security) + { + /* Send the PBSZ and PROT commands, in that order. + * If we are here it means that the server has already accepted + * some form of FTPS. Thus, these commands must work. + * If they don't work, that's an error. There's no sense in honoring + * --ftps-fallback-to-ftp or similar options. */ + if (u->scheme == SCHEME_FTPS) + { + if (!opt.server_response) + logputs (LOG_VERBOSE, "==> PBSZ 0 ... "); + if ((err = ftp_pbsz (csock, 0)) == FTPNOPBSZ) + { + logputs (LOG_NOTQUIET, _("Server did not accept the 'PBSZ 0' command.\n")); + return err; + } + if (!opt.server_response) + logputs (LOG_VERBOSE, "done."); + + if (!opt.server_response) + logprintf (LOG_VERBOSE, " ==> PROT %c ... ", prot); + if ((err = ftp_prot (csock, prot)) == FTPNOPROT) + { + logprintf (LOG_NOTQUIET, _("Server did not accept the 'PROT %c' command.\n"), prot); + return err; + } + if (!opt.server_response) + logputs (LOG_VERBOSE, "done.\n"); + + if (prot != PROT_CLEAR) + { + using_data_security = true; + con->st |= DATA_CHANNEL_SECURITY; + } + } + } +#endif + /* Third: Get the system type */ if (!opt.server_response) logprintf (LOG_VERBOSE, "==> SYST ... "); @@ -1313,6 +1498,36 @@ Error in server response, closing control connection.\n")); else if (expected_bytes) print_length (expected_bytes, restval, false); +#ifdef HAVE_SSL + if (u->scheme == SCHEME_FTPS && using_data_security) + { + /* We should try to restore the existing SSL session in the data connection + * and fall back to establishing a new session if the server doesn't want to restore it. + */ + if (!opt.ftps_resume_ssl || !ssl_connect_wget (dtsock, u->host, &csock)) + { + if (opt.ftps_resume_ssl) + logputs (LOG_NOTQUIET, "Server does not want to resume the SSL session. Trying with a new one.\n"); + if (!ssl_connect_wget (dtsock, u->host, NULL)) + { + fd_close (csock); + fd_close (dtsock); + logputs (LOG_NOTQUIET, "Could not perform SSL handshake.\n"); + return CONERROR; + } + } + else + logputs (LOG_NOTQUIET, "Resuming SSL session in data connection.\n"); + + if (!ssl_check_certificate (dtsock, u->host)) + { + fd_close (csock); + fd_close (dtsock); + return CONERROR; + } + } +#endif + /* Get the contents of the document. */ flags = 0; if (restval && rest_failed) @@ -1377,10 +1592,18 @@ Error in server response, closing control connection.\n")); become apparent later. */ if (*respline != '2') { - xfree (respline); if (res != -1) logprintf (LOG_NOTQUIET, "%s (%s) - ", tms, tmrate); logputs (LOG_NOTQUIET, _("Data transfer aborted.\n")); +#ifdef HAVE_SSL + if (!c_strncasecmp (respline, "425", 3) && u->scheme == SCHEME_FTPS) + { + logputs (LOG_NOTQUIET, "FTPS server rejects new SSL sessions in the data connection.\n"); + xfree (respline); + return FTPRESTFAIL; + } +#endif + xfree (respline); return FTPRETRINT; } xfree (respline); @@ -1700,8 +1923,14 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi switch (err) { case HOSTERR: case CONIMPOSSIBLE: case FWRITEERR: case FOPENERR: - case FTPNSFOD: case FTPLOGINC: case FTPNOPASV: case CONTNOTSUPPORTED: - case UNLINKERR: case WARC_TMP_FWRITEERR: + case FTPNSFOD: case FTPLOGINC: case FTPNOPASV: case FTPNOAUTH: case FTPNOPBSZ: case FTPNOPROT: + case UNLINKERR: case WARC_TMP_FWRITEERR: case CONSSLERR: case CONTNOTSUPPORTED: +#ifdef HAVE_SSL + if (err == FTPNOAUTH) + logputs (LOG_NOTQUIET, "Server does not support AUTH TLS.\n"); + if (opt.ftps_implicit) + logputs (LOG_NOTQUIET, "Server does not like implicit FTPS connections.\n"); +#endif /* Fatal errors, give up. */ if (warc_tmp != NULL) fclose (warc_tmp); diff --git a/src/ftp.h b/src/ftp.h index 737ce2e..84ded26 100644 --- a/src/ftp.h +++ b/src/ftp.h @@ -33,6 +33,7 @@ as that of the covered work. */ #define FTP_H #include "host.h" +#include "url.h" /* System types. */ enum stype @@ -53,10 +54,27 @@ enum ustype UST_OTHER }; +#ifdef HAVE_SSL +/* Data channel protection levels (to be used with PBSZ) */ +enum prot_level +{ + PROT_CLEAR = 'C', + PROT_SAFE = 'S', + PROT_CONFIDENTIAL = 'E', + PROT_PRIVATE = 'P' +}; +#endif + uerr_t ftp_response (int, char **); +uerr_t ftp_greeting (int); uerr_t ftp_login (int, const char *, const char *); uerr_t ftp_port (int, int *); uerr_t ftp_pasv (int, ip_address *, int *); +#ifdef HAVE_SSL +uerr_t ftp_auth (int, enum url_scheme); +uerr_t ftp_pbsz (int, int); +uerr_t ftp_prot (int, enum prot_level); +#endif #ifdef ENABLE_IPV6 uerr_t ftp_lprt (int, int *); uerr_t ftp_lpsv (int, ip_address *, int *); @@ -142,11 +160,12 @@ enum wget_ftp_fstatus AVOID_LIST = 0x0008, /* It tells us if during this session we have to avoid to use "LIST". */ - LIST_AFTER_LIST_A_CHECK_DONE = 0x0010 + LIST_AFTER_LIST_A_CHECK_DONE = 0x0010, /* It tells us if we have already checked "LIST" after the first "LIST -a" to handle the case of file/folders named "-a". */ + DATA_CHANNEL_SECURITY = 0x0020 /* Establish a secure data channel */ }; struct fileinfo *ftp_parse_ls (const char *, const enum stype); diff --git a/src/gnutls.c b/src/gnutls.c index be04342..a38301a 100644 --- a/src/gnutls.c +++ b/src/gnutls.c @@ -219,6 +219,7 @@ cert to be of the same type.\n")); struct wgnutls_transport_context { gnutls_session_t session; /* GnuTLS session handle */ + gnutls_datum_t *session_data; int last_error; /* last error returned by read/write/... */ /* Since GnuTLS doesn't support the equivalent to recv(..., @@ -405,6 +406,11 @@ wgnutls_close (int fd, void *arg) { struct wgnutls_transport_context *ctx = arg; /*gnutls_bye (ctx->session, GNUTLS_SHUT_RDWR);*/ + if (ctx->session_data) + { + gnutls_free (ctx->session_data->data); + gnutls_free (ctx->session_data); + } gnutls_deinit (ctx->session); xfree (ctx); close (fd); @@ -420,7 +426,7 @@ static struct transport_implementation wgnutls_transport = }; bool -ssl_connect_wget (int fd, const char *hostname) +ssl_connect_wget (int fd, const char *hostname, int *continue_session) { #ifdef F_GETFL int flags = 0; @@ -531,6 +537,27 @@ ssl_connect_wget (int fd, const char *hostname) return false; } + if (continue_session) + { + ctx = (struct wgnutls_transport_context *) fd_transport_context (*continue_session); + if (!gnutls_session_is_resumed (session)) + { + if (!ctx || !ctx->session_data || gnutls_session_set_data (session, ctx->session_data->data, ctx->session_data->size)) + { + /* server does not want to continue the session */ + gnutls_free (ctx->session_data->data); + gnutls_free (ctx->session_data); + gnutls_deinit (session); + return false; + } + } + else + { + logputs (LOG_ALWAYS, "SSL session has already been resumed. Continuing.\n"); + continue_session = NULL; + } + } + if (opt.connect_timeout) { #ifdef F_GETFL @@ -612,7 +639,13 @@ ssl_connect_wget (int fd, const char *hostname) } ctx = xnew0 (struct wgnutls_transport_context); + ctx->session_data = xnew0 (gnutls_datum_t); ctx->session = session; + if (gnutls_session_get_data2 (session, ctx->session_data)) + { + xfree (ctx->session_data); + logprintf (LOG_NOTQUIET, "WARNING: Could not save SSL session data for socket %d\n", fd); + } fd_register_transport (fd, &wgnutls_transport, ctx); return true; } diff --git a/src/http.c b/src/http.c index 8a36f41..4c1052d 100644 --- a/src/http.c +++ b/src/http.c @@ -2141,7 +2141,7 @@ establish_connection (struct url *u, struct url **conn_ref, if (conn->scheme == SCHEME_HTTPS) { - if (!ssl_connect_wget (sock, u->host)) + if (!ssl_connect_wget (sock, u->host, NULL)) { CLOSE_INVALIDATE (sock); return CONSSLERR; @@ -2742,6 +2742,9 @@ metalink_from_http (const struct response *resp, const struct http_stat *hs, case SCHEME_HTTPS: mres.type = xstrdup ("https"); break; + case SCHEME_FTPS: + mres.type = xstrdup ("ftps"); + break; #endif case SCHEME_FTP: mres.type = xstrdup ("ftp"); diff --git a/src/init.c b/src/init.c index ea074cc..dd1506c 100644 --- a/src/init.c +++ b/src/init.c @@ -188,6 +188,12 @@ static const struct { { "ftppasswd", &opt.ftp_passwd, cmd_string }, /* deprecated */ { "ftppassword", &opt.ftp_passwd, cmd_string }, { "ftpproxy", &opt.ftp_proxy, cmd_string }, +#ifdef HAVE_SSL + { "ftpscleardataconnection", &opt.ftps_clear_data_connection, cmd_boolean }, + { "ftpsfallbacktoftp", &opt.ftps_fallback_to_ftp, cmd_boolean }, + { "ftpsimplicit", &opt.ftps_implicit, cmd_boolean }, + { "ftpsresumessl", &opt.ftps_resume_ssl, cmd_boolean }, +#endif #ifdef __VMS { "ftpstmlf", &opt.ftp_stmlf, cmd_boolean }, #endif /* def __VMS */ @@ -408,6 +414,10 @@ defaults (void) #ifdef HAVE_SSL opt.check_cert = true; + opt.ftps_resume_ssl = true; + opt.ftps_fallback_to_ftp = false; + opt.ftps_implicit = false; + opt.ftps_clear_data_connection = false; #endif /* The default for file name restriction defaults to the OS type. */ diff --git a/src/main.c b/src/main.c index 3e74cd7..92d1bee 100644 --- a/src/main.c +++ b/src/main.c @@ -293,6 +293,12 @@ static struct cmdline_option option_data[] = { "ftp-stmlf", 0, OPT_BOOLEAN, "ftpstmlf", -1 }, #endif /* def __VMS */ { "ftp-user", 0, OPT_VALUE, "ftpuser", -1 }, +#ifdef HAVE_SSL + { "ftps-clear-data-connection", 0, OPT_BOOLEAN, "ftpscleardataconnection", -1 }, + { "ftps-fallback-to-ftp", 0, OPT_BOOLEAN, "ftpsfallbacktoftp", -1 }, + { "ftps-implicit", 0, OPT_BOOLEAN, "ftpsimplicit", -1 }, + { "ftps-resume-ssl", 0, OPT_BOOLEAN, "ftpsresumessl", -1 }, +#endif { "glob", 0, OPT_BOOLEAN, "glob", -1 }, { "header", 0, OPT_VALUE, "header", -1 }, { "help", 'h', OPT_FUNCALL, (void *)print_help, no_argument }, @@ -820,6 +826,20 @@ FTP options:\n"), --retr-symlinks when recursing, get linked-to files (not dir)\n"), "\n", +#ifdef HAVE_SSL + N_("\ +FTPS options:\n"), + N_("\ + --ftps-implicit use implicit FTPS (default port is 990)\n"), + N_("\ + --ftps-resume-ssl resume the SSL/TLS session started in the control connection when\n" + " opening a data connection\n"), + N_("\ + --ftps-clear-data-connection cipher the control channel only; all the data will be in plaintext\n"), + N_("\ + --ftps-fallback-to-ftp fall back to FTP if FTPS is not supported in the target server\n"), +#endif + N_("\ WARC options:\n"), N_("\ @@ -1810,12 +1830,13 @@ outputting to a regular file.\n")); else { if ((opt.recursive || opt.page_requisites) - && (url_scheme (*t) != SCHEME_FTP || url_uses_proxy (url_parsed))) + && ((url_scheme (*t) != SCHEME_FTP && url_scheme (*t) != SCHEME_FTPS) + || url_uses_proxy (url_parsed))) { int old_follow_ftp = opt.follow_ftp; /* Turn opt.follow_ftp on in case of recursive FTP retrieval */ - if (url_scheme (*t) == SCHEME_FTP) + if (url_scheme (*t) == SCHEME_FTP || url_scheme (*t) == SCHEME_FTPS) opt.follow_ftp = 1; retrieve_tree (url_parsed, NULL); diff --git a/src/metalink.h b/src/metalink.h index ec91b07..e98c210 100644 --- a/src/metalink.h +++ b/src/metalink.h @@ -34,7 +34,7 @@ as that of the covered work. */ #ifdef HAVE_SSL # define RES_TYPE_SUPPORTED(x)\ - ((!x) || !strcmp (x, "ftp") || !strcmp (x, "http") || !strcmp (x, "https")) + ((!x) || !strcmp (x, "http") || !strcmp (x, "https") || !strcmp (x, "ftp") || !strcmp (x, "ftps")) #else # define RES_TYPE_SUPPORTED(x)\ ((!x) || !strcmp (x, "ftp") || !strcmp (x, "http")) diff --git a/src/openssl.c b/src/openssl.c index 3ac0f44..4876048 100644 --- a/src/openssl.c +++ b/src/openssl.c @@ -334,6 +334,7 @@ ssl_init (void) struct openssl_transport_context { SSL *conn; /* SSL connection handle */ + SSL_SESSION *sess; /* SSL session info */ char *last_error; /* last error printed with openssl_errstr */ }; @@ -514,7 +515,7 @@ ssl_connect_with_timeout_callback(void *arg) Returns true on success, false on failure. */ bool -ssl_connect_wget (int fd, const char *hostname) +ssl_connect_wget (int fd, const char *hostname, int *continue_session) { SSL *conn; struct scwt_context scwt_ctx; @@ -527,7 +528,7 @@ ssl_connect_wget (int fd, const char *hostname) if (!conn) goto error; #if OPENSSL_VERSION_NUMBER >= 0x0090806fL && !defined(OPENSSL_NO_TLSEXT) - /* If the SSL library was build with support for ServerNameIndication + /* If the SSL library was built with support for ServerNameIndication then use it whenever we have a hostname. If not, don't, ever. */ if (! is_valid_ip_address (hostname)) { @@ -539,6 +540,14 @@ ssl_connect_wget (int fd, const char *hostname) } #endif + if (continue_session) + { + /* attempt to resume a previous SSL session */ + ctx = (struct openssl_transport_context *) fd_transport_context (*continue_session); + if (!ctx || !ctx->sess || !SSL_set_session (conn, ctx->sess)) + goto error; + } + #ifndef FD_TO_SOCKET # define FD_TO_SOCKET(X) (X) #endif @@ -557,6 +566,9 @@ ssl_connect_wget (int fd, const char *hostname) ctx = xnew0 (struct openssl_transport_context); ctx->conn = conn; + ctx->sess = SSL_get0_session (conn); + if (!ctx->sess) + logprintf (LOG_NOTQUIET, "WARNING: Could not save SSL session data for socket %d\n", fd); /* Register FD with Wget's transport layer, i.e. arrange that our functions are used for reading, writing, and polling. */ diff --git a/src/options.h b/src/options.h index 24ddbb5..0bb7d71 100644 --- a/src/options.h +++ b/src/options.h @@ -229,6 +229,10 @@ struct options char *random_file; /* file with random data to seed the PRNG */ char *egd_file; /* file name of the egd daemon socket */ bool https_only; /* whether to follow HTTPS only */ + bool ftps_resume_ssl; + bool ftps_fallback_to_ftp; + bool ftps_implicit; + bool ftps_clear_data_connection; #endif /* HAVE_SSL */ bool cookies; /* whether cookies are used. */ diff --git a/src/recur.c b/src/recur.c index ce55362..25cdbb7 100644 --- a/src/recur.c +++ b/src/recur.c @@ -610,7 +610,7 @@ download_child (const struct urlpos *upos, struct url *parent, int depth, u_scheme_like_http = schemes_are_similar_p (u->scheme, SCHEME_HTTP); /* 1. Schemes other than HTTP are normally not recursed into. */ - if (!u_scheme_like_http && !(u->scheme == SCHEME_FTP && opt.follow_ftp)) + if (!u_scheme_like_http && !((u->scheme == SCHEME_FTP || u->scheme == SCHEME_FTPS) && opt.follow_ftp)) { DEBUGP (("Not following non-HTTP schemes.\n")); reason = WG_RR_NONHTTP; @@ -832,12 +832,13 @@ write_reject_log_url (FILE *fp, const struct url *url) switch (url->scheme) { - case SCHEME_HTTP: scheme_str = "SCHEME_HTTP"; break; - #ifdef HAVE_SSL - case SCHEME_HTTPS: scheme_str = "SCHEME_HTTPS"; break; - #endif - case SCHEME_FTP: scheme_str = "SCHEME_FTP"; break; - default: scheme_str = "SCHEME_INVALID"; break; + case SCHEME_HTTP: scheme_str = "SCHEME_HTTP"; break; +#ifdef HAVE_SSL + case SCHEME_HTTPS: scheme_str = "SCHEME_HTTPS"; break; + case SCHEME_FTPS: scheme_str = "SCHEME_FTPS"; break; +#endif + case SCHEME_FTP: scheme_str = "SCHEME_FTP"; break; + default: scheme_str = "SCHEME_INVALID"; break; } fprintf (fp, "%s\t%s\t%s\t%i\t%s\t%s\t%s\t%s", diff --git a/src/retr.c b/src/retr.c index e2b737e..318b09c 100644 --- a/src/retr.c +++ b/src/retr.c @@ -817,7 +817,11 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file, result = http_loop (u, orig_parsed, &mynewloc, &local_file, refurl, dt, proxy_url, iri); } - else if (u->scheme == SCHEME_FTP) + else if (u->scheme == SCHEME_FTP +#ifdef HAVE_SSL + || u->scheme == SCHEME_FTPS +#endif + ) { /* If this is a redirection, temporarily turn off opt.ftp_glob and opt.recursive, both being undesirable when following @@ -833,7 +837,7 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file, FTP. In these cases we must decide whether the text is HTML according to the suffix. The HTML suffixes are `.html', `.htm' and a few others, case-insensitive. */ - if (redirection_count && local_file && u->scheme == SCHEME_FTP) + if (redirection_count && local_file && (u->scheme == SCHEME_FTP || u->scheme == SCHEME_FTPS)) { if (has_html_suffix_p (local_file)) *dt |= TEXTHTML; @@ -1095,12 +1099,12 @@ retrieve_from_file (const char *file, bool html, int *count) proxy = getproxy (cur_url->url); if ((opt.recursive || opt.page_requisites) - && (cur_url->url->scheme != SCHEME_FTP || proxy)) + && ((cur_url->url->scheme != SCHEME_FTP && cur_url->url->scheme != SCHEME_FTPS) || proxy)) { int old_follow_ftp = opt.follow_ftp; /* Turn opt.follow_ftp on in case of recursive FTP retrieval */ - if (cur_url->url->scheme == SCHEME_FTP) + if (cur_url->url->scheme == SCHEME_FTP || cur_url->url->scheme == SCHEME_FTPS) opt.follow_ftp = 1; status = retrieve_tree (parsed_url ? parsed_url : cur_url->url, @@ -1281,6 +1285,9 @@ getproxy (struct url *u) case SCHEME_HTTPS: proxy = opt.https_proxy ? opt.https_proxy : getenv ("https_proxy"); break; + case SCHEME_FTPS: + proxy = opt.ftp_proxy ? opt.ftp_proxy : getenv ("ftps_proxy"); + break; #endif case SCHEME_FTP: proxy = opt.ftp_proxy ? opt.ftp_proxy : getenv ("ftp_proxy"); diff --git a/src/ssl.h b/src/ssl.h index 23a16ec..eeb6b34 100644 --- a/src/ssl.h +++ b/src/ssl.h @@ -33,7 +33,7 @@ as that of the covered work. */ #define GEN_SSLFUNC_H bool ssl_init (void); -bool ssl_connect_wget (int, const char *); +bool ssl_connect_wget (int, const char *, int *); bool ssl_check_certificate (int, const char *); #endif /* GEN_SSLFUNC_H */ diff --git a/src/url.c b/src/url.c index d8d6d95..56079cd 100644 --- a/src/url.c +++ b/src/url.c @@ -78,6 +78,13 @@ static struct scheme_data supported_schemes[] = { "https", "https://";, DEFAULT_HTTPS_PORT, scm_has_query|scm_has_fragment }, #endif { "ftp", "ftp://";, DEFAULT_FTP_PORT, scm_has_params|scm_has_fragment }, +#ifdef HAVE_SSL + /* + * Explicit FTPS uses the same port as FTP. + * Implicit FTPS has its own port (990), but it is disabled by default. + */ + { "ftps", "ftps://", DEFAULT_FTP_PORT, scm_has_params|scm_has_fragment }, +#endif /* SCHEME_INVALID */ { NULL, NULL, -1, 0 } @@ -1772,7 +1779,7 @@ path_simplify (enum url_scheme scheme, char *path) else if (h[0] == '.' && h[1] == '.' && (h[2] == '/' || h[2] == '\0')) { /* Handle "../" by retreating the tortoise by one path - element -- but not past beggining. */ + element -- but not past beginning. */ if (t > beg) { /* Move backwards until T hits the beginning of the @@ -1780,7 +1787,7 @@ path_simplify (enum url_scheme scheme, char *path) for (--t; t > beg && t[-1] != '/'; t--) ; } - else if (scheme == SCHEME_FTP) + else if (scheme == SCHEME_FTP || scheme == SCHEME_FTPS) { /* If we're at the beginning, copy the "../" literally and move the beginning so a later ".." doesn't remove diff --git a/src/url.h b/src/url.h index a543f3d..7c77737 100644 --- a/src/url.h +++ b/src/url.h @@ -36,6 +36,7 @@ as that of the covered work. */ #define DEFAULT_HTTP_PORT 80 #define DEFAULT_FTP_PORT 21 #define DEFAULT_HTTPS_PORT 443 +#define DEFAULT_FTPS_IMPLICIT_PORT 990 /* This represents how many characters less than the OS max name length a file * should be. More precisely, a file name should be at most @@ -70,6 +71,9 @@ enum url_scheme { SCHEME_HTTPS, #endif SCHEME_FTP, +#ifdef HAVE_SSL + SCHEME_FTPS, +#endif SCHEME_INVALID }; diff --git a/src/wget.h b/src/wget.h index c04761d..ab6270f 100644 --- a/src/wget.h +++ b/src/wget.h @@ -349,7 +349,8 @@ typedef enum FTPSRVERR, FTPRETRINT, FTPRESTFAIL, URLERROR, FOPENERR, FOPEN_EXCL_ERR, FWRITEERR, HEOF, GATEWAYTIMEOUT, HERR, RETROK, RECLEVELEXC, WRONGCODE, - FTPINVPASV, FTPNOPASV, CONTNOTSUPPORTED, RETRUNNEEDED, RETRFINISHED, + FTPINVPASV, FTPNOPASV, FTPNOPBSZ, FTPNOPROT, FTPNOAUTH, + CONTNOTSUPPORTED, RETRUNNEEDED, RETRFINISHED, READERR, TRYLIMEXC, FILEBADFILE, RANGEERR, RETRBADPATTERN, PROXERR, AUTHFAILED, QUOTEXC, WRITEFAILED, SSLINITFAILED, VERIFCERTERR, -- 2.1.4
