I just noticed that in refactoring the ftp uploader, the function _make_remote_dir() never gets called! Try this version of weeutil/ftpupload.py.
-tk On Thu, May 28, 2020 at 12:50 PM Tom Keffer <tkef...@gmail.com> wrote: > Hard to say, but I think you're getting hit by issue #284 > <https://github.com/weewx/weewx/issues/284>. Try adding an option > secure_data and setting it to false: > > [[FTP]] > ... > secure_ftp = True > secure_data = False > > -tk > > On Thu, May 28, 2020 at 9:36 AM Richard Cropper < > richardcropper...@gmail.com> wrote: > >> Thanks Tom >> >> I had in fact tried those steps but did so again, without success. >> >> Suspecting that the directory might be the cause even though its >> permissions looked good, I deleted the weewx directory and then created it >> again. >> >> This had made a difference but not solved the problem. >> >> Here's an extract from the log: >> >> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt >> #1. Failed uploading /public_html/weewx/daytempfeel.png to >> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >> >> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt >> #2. Failed uploading /public_html/weewx/daytempfeel.png to >> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >> >> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt >> #3. Failed uploading /public_html/weewx/daytempfeel.png to >> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >> >> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Failed to >> upload file /public_html/weewx/daytempfeel.png >> >> I have tried all possible file paths again, as you suggest but I always >> get this broken pipe message. If I extend the file path any higher, I get >> the 'cannot open this file' message. >> >> I have been able to upload the index file from /var/www/html/weewx to the >> weewx directory on my server using curl. >> >> Richard >> >> On Wednesday, May 27, 2020 at 7:56:28 PM UTC+1, Tom Keffer wrote: >>> >>> Two things to try: >>> >>> 1. Make sure the FTP username has write permissions on the server. >>> >>> 2. Try different variations on the FTP path. For example, >>> >>> path = public_html/weewx/ # What you have >>> path = public_html/weewx >>> path = /public_html/weewx/ >>> path = /public_html/weewx >>> path = weewx >>> path = /weewx >>> >>> >>> >>> >>> >>> >>> On Wed, May 27, 2020 at 10:02 AM Richard Cropper <richardc...@gmail.com> >>> wrote: >>> >>>> Thanks Tom >>>> >>>> Yes all the files are in /var/www/html/weewx >>>> >>>> If I understand you correctly, this has more to do with the file path >>>> of the remote host server. >>>> >>>> I am at a loss to understand what is going on at that end. >>>> >>>> I can at least connect but I either get a message as you have seen or a >>>> 'broken pipe' message if I play around with the file path. >>>> >>>> I guess this is more to do with the remote server than weewx but thanks >>>> for your advice. >>>> >>>> Regards >>>> >>>> Richard >>>> On Wednesday, May 27, 2020 at 3:17:35 PM UTC+1, Tom Keffer wrote: >>>>> >>>>> Hello, Richard >>>>> >>>>> Unfortunately, the error message is misleading (and should be >>>>> changed). When it says >>>>> >>>>> Failed uploading public_html/weewx/monthtempfeel.png to >>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>> directory >>>>> >>>>> what it means is it could not upload to the *target* >>>>> public_html/weewx/monthtempfeel.png*.* The unfortunate part is that >>>>> the error message doesn't give the source. >>>>> >>>>> Take a look in /var/www/html/weewx and make sure the files you expect >>>>> to be in there are actually in there. >>>>> >>>>> -tk >>>>> >>>>> >>>>> On Wed, May 27, 2020 at 5:50 AM Richard Cropper <richardc...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi >>>>>> Having secured a connection, I still don't seem to be able to upload >>>>>> files >>>>>> >>>>>> I get the following error message on the log >>>>>> >>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #1. Failed uploading public_html/weewx/monthtempfeel.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #2. Failed uploading public_html/weewx/monthtempfeel.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #3. Failed uploading public_html/weewx/monthtempfeel.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Failed to upload file public_html/weewx/monthtempfeel.png >>>>>> >>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #1. Failed uploading public_html/weewx/yearwind.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #2. Failed uploading public_html/weewx/yearwind.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #3. Failed uploading public_html/weewx/yearwind.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Failed to upload file public_html/weewx/yearwind.png >>>>>> >>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #1. Failed uploading public_html/weewx/statistics.html to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #2. Failed uploading public_html/weewx/statistics.html to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #3. Failed uploading public_html/weewx/statistics.html to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Failed to upload file public_html/weewx/statistics.html >>>>>> >>>>>> May 27 13:30:41 raspberrypi weewx[899] ERROR weeutil.ftpupload: >>>>>> Attempt #1. Failed uploading public_html/weewx/weekbarometer.png to >>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or >>>>>> directory >>>>>> >>>>>> >>>>>> Here's the relevant bit of my weewx.conf: >>>>>> >>>>>> >>>>>> [[FTP]] >>>>>> >>>>>> # FTP'ing the results to a webserver is treated as just >>>>>> another report, >>>>>> >>>>>> # albeit one with an unusual report generator! >>>>>> >>>>>> skin = Ftp >>>>>> >>>>>> >>>>>> # If you wish to use FTP, set "enable" to "true", then >>>>>> >>>>>> # fill out the next four lines. >>>>>> >>>>>> # Use quotes around passwords to guard against parsing errors. >>>>>> >>>>>> enable = true >>>>>> >>>>>> user =[user name] >>>>>> >>>>>> password = [password] >>>>>> >>>>>> server = [server name] # The ftp server name, e.g, >>>>>> www.myserver.org >>>>>> >>>>>> path = public_html/weewx/ # The destination directory, >>>>>> e.g., /weather >>>>>> >>>>>> >>>>>> # Set to True for an FTP over TLS (FTPS) connection. Not all >>>>>> servers >>>>>> >>>>>> # support this. >>>>>> >>>>>> secure_ftp = True >>>>>> >>>>>> secure_data = True >>>>>> >>>>>> >>>>>> # To upload files from something other than what HTML_ROOT is >>>>>> set >>>>>> >>>>>> # to above, specify a different HTML_ROOT here. >>>>>> >>>>>> HTML_ROOT = /var/www/html/weewx/ >>>>>> >>>>>> >>>>>> # Most FTP servers use port 21 >>>>>> >>>>>> port = 21 >>>>>> >>>>>> >>>>>> # Set to 1 to use passive mode, zero for active mode >>>>>> >>>>>> passive = 1 >>>>>> >>>>>> >>>>>> The connection is FTP over TLS. >>>>>> >>>>>> >>>>>> My weather data is from an Aercus Weather Sleuth using interceptor on >>>>>> a Raspberry pi. >>>>>> >>>>>> >>>>>> It's as if FTP is looking for the wrong path on my pi but maybe I >>>>>> misunderstand the error message. That's why I tried removing the hash >>>>>> before HTML_ROOT but I got the same result with the hash. >>>>>> >>>>>> >>>>>> Otherwise very pleased with weewx which I can view on my pi using >>>>>> nginx >>>>>> >>>>>> >>>>>> Any help gratefully received. >>>>>> >>>>>> >>>>>> Richard >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to weewx...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to weewx...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to weewx-user+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com >> <https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAFv3edXX4RN0HWC7E6_WDQpoeRm8sDzDewtOHeQ4wKAw%40mail.gmail.com.
# # Copyright (c) 2009-2020 Tom Keffer <tkef...@gmail.com> # # See the file LICENSE.txt for your full rights. # """For uploading files to a remove server via FTP""" from __future__ import absolute_import from __future__ import print_function from __future__ import with_statement import ftplib import logging import os import sys import time from six.moves import cPickle log = logging.getLogger(__name__) class FtpUpload(object): """Uploads a directory and all its descendants to a remote server. Keeps track of when a file was last uploaded, so it is uploaded only if its modification time is newer.""" def __init__(self, server, user, password, local_root, remote_root, port=21, name="FTP", passive=True, secure=False, debug=0, secure_data=True, reuse_ssl=False): """Initialize an instance of FtpUpload. After initializing, call method run() to perform the upload. server: The remote server to which the files are to be uploaded. user, password : The user name and password that are to be used. name: A unique name to be given for this FTP session. This allows more than one session to be uploading from the same local directory. [Optional. Default is 'FTP'.] passive: True to use passive mode; False to use active mode. [Optional. Default is True (passive mode)] secure: Set to True to attempt an FTP over TLS (FTPS) session. debug: Set to 1 for extra debug information, 0 otherwise. secure_data: If a secure session is requested (option secure=True), should we attempt a secure data connection as well? This option is useful due to a bug in the Python FTP client library. See Issue #284. [Optional. Default is True] reuse_ssl: Work around a bug in the Python library that closes ssl sockets that should be reused. See https://bit.ly/3dKq4JY [Optional. Default is False] """ self.server = server self.user = user self.password = password self.local_root = os.path.normpath(local_root) self.remote_root = os.path.normpath(remote_root) self.port = port self.name = name self.passive = passive self.secure = secure self.debug = debug self.secure_data = secure_data self.reuse_ssl = reuse_ssl if self.reuse_ssl and sys.version < '3.6': raise ValueError("Reusing an SSL connection requires Python version 3.6 or greater") def run(self): """Perform the actual upload. returns: the number of files uploaded.""" # Get the timestamp and members of the last upload: timestamp, fileset = self.get_last_upload() n_uploaded = 0 try: if self.secure: log.debug("Attempting secure connection to %s" % self.server) if self.reuse_ssl: # Activate the workaround for the Python ftplib library. from ssl import SSLSocket class ReusedSslSocket(SSLSocket): def unwrap(self): pass class WeeFTPTLS(ftplib.FTP_TLS): """Explicit FTPS, with shared TLS session""" def ntransfercmd(self, cmd, rest=None): conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest) if self._prot_p: conn = self.context.wrap_socket(conn, server_hostname=self.host, session=self.sock.session) conn.__class__ = ReusedSslSocket return conn, size log.debug("Reusing SSL connections.") ftp_server = WeeFTPTLS() else: ftp_server = ftplib.FTP_TLS() else: log.debug("Attempting connection to %s" % self.server) ftp_server = ftplib.FTP() if self.debug: ftp_server.set_debuglevel(self.debug) ftp_server.set_pasv(self.passive) ftp_server.connect(self.server, self.port) ftp_server.login(self.user, self.password) if self.secure and self.secure_data: ftp_server.prot_p() log.debug("Secure data connection to %s" % self.server) else: log.debug("Connected to %s" % self.server) # Walk the local directory structure for (dirpath, unused_dirnames, filenames) in os.walk(self.local_root): # Strip out the common local root directory. What is left # will be the relative directory both locally and remotely. local_rel_dir_path = dirpath.replace(self.local_root, '.') if self._skip_this_dir(local_rel_dir_path): continue # This is the absolute path to the remote directory: remote_dir_path = os.path.normpath(os.path.join(self.remote_root, local_rel_dir_path)) # Make the remote directory if necessary: _make_remote_dir(ftp_server, remote_dir_path) # Now iterate over all members of the local directory: for filename in filenames: full_local_path = os.path.join(dirpath, filename) # See if this file can be skipped: if self._skip_this_file(timestamp, fileset, full_local_path): continue full_remote_path = os.path.join(remote_dir_path, filename) stor_cmd = "STOR %s" % full_remote_path with open(full_local_path, 'rb') as fd: try: ftp_server.storbinary(stor_cmd, fd) except ftplib.all_errors as e: # Unsuccessful. Log it, then reraise the exception log.error("Failed uploading %s to server %s. Reason: '%s'" % (full_local_path, self.server, e)) raise # Success. n_uploaded += 1 fileset.add(full_local_path) log.debug("Uploaded file %s to %s" % (full_local_path, full_remote_path)) finally: try: ftp_server.quit() except Exception as e: log.debug("quit() exception %s: '%s'", type(e), e) timestamp = time.time() self.save_last_upload(timestamp, fileset) return n_uploaded def get_last_upload(self): """Reads the time and members of the last upload from the local root""" timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name) # If the file does not exist, an IOError exception will be raised. # If the file exists, but is truncated, an EOFError will be raised. # Either way, be prepared to catch it. try: with open(timestamp_file_path, "rb") as f: timestamp = cPickle.load(f) fileset = cPickle.load(f) except (IOError, EOFError, cPickle.PickleError, AttributeError): timestamp = 0 fileset = set() # Either the file does not exist, or it is garbled. # Either way, it's safe to remove it. try: os.remove(timestamp_file_path) except OSError: pass return timestamp, fileset def save_last_upload(self, timestamp, fileset): """Saves the time and members of the last upload in the local root.""" timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name) with open(timestamp_file_path, "wb") as f: cPickle.dump(timestamp, f) cPickle.dump(fileset, f) def _skip_this_dir(self, local_dir): """Determine whether to skip a directory.""" return os.path.basename(local_dir) in ('.svn', 'CVS') def _skip_this_file(self, timestamp, fileset, full_local_path): """Determine whether to skip a specific file.""" filename = os.path.basename(full_local_path) if filename[-1] == '~' or filename[0] == '#': return True if full_local_path not in fileset: return False if os.stat(full_local_path).st_mtime > timestamp: return False # Filename is in the set, and is up to date. return True def _make_remote_dir(ftp_server, remote_dir_path): """Make a remote directory if necessary.""" try: ftp_server.mkd(remote_dir_path) except ftplib.all_errors as e: # Got an exception. It might be because the remote directory already exists: if sys.exc_info()[0] is ftplib.error_perm: msg = str(e).strip() # If a directory already exists, some servers respond with a '550' ("Requested # action not taken") code, others with a '521' ("Access denied" or "Pathname # already exists") code. if msg.startswith('550') or msg.startswith('521'): # Directory already exists return # It's a real error. Re-raise the exception. raise log.debug("Made directory %s" % remote_dir_path) return