I just noticed that in refactoring the ftp uploader, the function
_make_remote_dir() never gets called! Try this version of
weeutil/ftpupload.py.

-tk



On Thu, May 28, 2020 at 12:50 PM Tom Keffer <tkef...@gmail.com> wrote:

> Hard to say, but I think you're getting hit by issue #284
> <https://github.com/weewx/weewx/issues/284>. Try adding an option
> secure_data and setting it to false:
>
> [[FTP]]
>    ...
>    secure_ftp = True
>    secure_data = False
>
> -tk
>
> On Thu, May 28, 2020 at 9:36 AM Richard Cropper <
> richardcropper...@gmail.com> wrote:
>
>> Thanks Tom
>>
>> I had in fact tried those steps but did so again, without success.
>>
>> Suspecting that the directory might be the cause even though its
>> permissions looked good, I deleted the weewx directory and then created it
>> again.
>>
>> This had made a difference but not solved the problem.
>>
>> Here's an extract from the log:
>>
>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt
>> #1. Failed uploading /public_html/weewx/daytempfeel.png to
>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>
>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt
>> #2. Failed uploading /public_html/weewx/daytempfeel.png to
>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>
>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Attempt
>> #3. Failed uploading /public_html/weewx/daytempfeel.png to
>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe
>>
>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: Failed to
>> upload file /public_html/weewx/daytempfeel.png
>>
>> I have tried all possible file paths again, as you suggest but I always
>> get this broken pipe message. If I extend the file path any higher, I get
>> the 'cannot open this file' message.
>>
>> I have been able to upload the index file from /var/www/html/weewx to the
>> weewx directory on my server using curl.
>>
>> Richard
>>
>> On Wednesday, May 27, 2020 at 7:56:28 PM UTC+1, Tom Keffer wrote:
>>>
>>> Two things to try:
>>>
>>> 1. Make sure the FTP username has write permissions on the server.
>>>
>>> 2. Try different variations on the FTP path. For example,
>>>
>>> path = public_html/weewx/    # What you have
>>> path = public_html/weewx
>>> path = /public_html/weewx/
>>> path = /public_html/weewx
>>> path = weewx
>>> path = /weewx
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 27, 2020 at 10:02 AM Richard Cropper <richardc...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Tom
>>>>
>>>> Yes all the files are in /var/www/html/weewx
>>>>
>>>> If I understand you correctly, this has more to do with the file path
>>>> of the remote host server.
>>>>
>>>> I am at a loss to understand what is going on at that end.
>>>>
>>>> I can at least connect but I either get a message as you have seen or a
>>>> 'broken pipe' message if I play around with the file path.
>>>>
>>>> I guess this is more to do with the remote server than weewx but thanks
>>>> for your advice.
>>>>
>>>> Regards
>>>>
>>>> Richard
>>>> On Wednesday, May 27, 2020 at 3:17:35 PM UTC+1, Tom Keffer wrote:
>>>>>
>>>>> Hello, Richard
>>>>>
>>>>> Unfortunately, the error message is misleading (and should be
>>>>> changed). When it says
>>>>>
>>>>> Failed uploading public_html/weewx/monthtempfeel.png to
>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>> directory
>>>>>
>>>>> what it means is it could not upload to the *target*
>>>>> public_html/weewx/monthtempfeel.png*.* The unfortunate part is that
>>>>> the error message doesn't give the source.
>>>>>
>>>>> Take a look in /var/www/html/weewx and make sure the files you expect
>>>>> to be in there are actually in there.
>>>>>
>>>>> -tk
>>>>>
>>>>>
>>>>> On Wed, May 27, 2020 at 5:50 AM Richard Cropper <richardc...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>> Having secured a connection, I still don't  seem to be able to upload
>>>>>> files
>>>>>>
>>>>>> I get the following error message on the log
>>>>>>
>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #1. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #2. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #3. Failed uploading public_html/weewx/monthtempfeel.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Failed to upload file public_html/weewx/monthtempfeel.png
>>>>>>
>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #1. Failed uploading public_html/weewx/yearwind.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #2. Failed uploading public_html/weewx/yearwind.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #3. Failed uploading public_html/weewx/yearwind.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Failed to upload file public_html/weewx/yearwind.png
>>>>>>
>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #1. Failed uploading public_html/weewx/statistics.html to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #2. Failed uploading public_html/weewx/statistics.html to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #3. Failed uploading public_html/weewx/statistics.html to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Failed to upload file public_html/weewx/statistics.html
>>>>>>
>>>>>> May 27 13:30:41 raspberrypi weewx[899] ERROR weeutil.ftpupload:
>>>>>> Attempt #1. Failed uploading public_html/weewx/weekbarometer.png to
>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such file or
>>>>>> directory
>>>>>>
>>>>>>
>>>>>> Here's the relevant bit of my weewx.conf:
>>>>>>
>>>>>>
>>>>>>  [[FTP]]
>>>>>>
>>>>>>         # FTP'ing the results to a webserver is treated as just
>>>>>> another report,
>>>>>>
>>>>>>         # albeit one with an unusual report generator!
>>>>>>
>>>>>>         skin = Ftp
>>>>>>
>>>>>>
>>>>>>         # If you wish to use FTP, set "enable" to "true", then
>>>>>>
>>>>>>         # fill out the next four lines.
>>>>>>
>>>>>>         # Use quotes around passwords to guard against parsing errors.
>>>>>>
>>>>>>         enable = true
>>>>>>
>>>>>>         user =[user name]
>>>>>>
>>>>>>         password = [password]
>>>>>>
>>>>>>         server =  [server name]    # The ftp server name, e.g,
>>>>>> www.myserver.org
>>>>>>
>>>>>>         path = public_html/weewx/    # The destination directory,
>>>>>> e.g., /weather
>>>>>>
>>>>>>
>>>>>>         # Set to True for an FTP over TLS (FTPS) connection. Not all
>>>>>> servers
>>>>>>
>>>>>>         # support this.
>>>>>>
>>>>>>         secure_ftp = True
>>>>>>
>>>>>>         secure_data = True
>>>>>>
>>>>>>
>>>>>>        # To upload files from something other than what HTML_ROOT is
>>>>>> set
>>>>>>
>>>>>>         # to above, specify a different HTML_ROOT here.
>>>>>>
>>>>>>          HTML_ROOT = /var/www/html/weewx/
>>>>>>
>>>>>>
>>>>>>         # Most FTP servers use port 21
>>>>>>
>>>>>>         port = 21
>>>>>>
>>>>>>
>>>>>>         # Set to 1 to use passive mode, zero for active mode
>>>>>>
>>>>>>         passive = 1
>>>>>>
>>>>>>
>>>>>> The connection is FTP over TLS.
>>>>>>
>>>>>>
>>>>>> My weather data is from an Aercus Weather Sleuth using interceptor on
>>>>>> a Raspberry pi.
>>>>>>
>>>>>>
>>>>>> It's as if FTP is looking for the wrong path on my  pi but maybe I
>>>>>> misunderstand the error message. That's why I tried removing the hash
>>>>>> before HTML_ROOT but I got the same result with the hash.
>>>>>>
>>>>>>
>>>>>> Otherwise very pleased with weewx  which I can view on my pi using
>>>>>> nginx
>>>>>>
>>>>>>
>>>>>> Any help gratefully received.
>>>>>>
>>>>>>
>>>>>> Richard
>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAFv3edXX4RN0HWC7E6_WDQpoeRm8sDzDewtOHeQ4wKAw%40mail.gmail.com.
#
#    Copyright (c) 2009-2020 Tom Keffer <tkef...@gmail.com>
#
#    See the file LICENSE.txt for your full rights.
#
"""For uploading files to a remove server via FTP"""

from __future__ import absolute_import
from __future__ import print_function
from __future__ import with_statement

import ftplib
import logging
import os
import sys
import time

from six.moves import cPickle

log = logging.getLogger(__name__)


class FtpUpload(object):
    """Uploads a directory and all its descendants to a remote server.
    
    Keeps track of when a file was last uploaded, so it is uploaded only
    if its modification time is newer."""

    def __init__(self, server,
                 user, password,
                 local_root, remote_root,
                 port=21,
                 name="FTP",
                 passive=True,
                 secure=False,
                 debug=0,
                 secure_data=True,
                 reuse_ssl=False):
        """Initialize an instance of FtpUpload.
        
        After initializing, call method run() to perform the upload.
        
        server: The remote server to which the files are to be uploaded.
        
        user,
        password : The user name and password that are to be used.
        
        name: A unique name to be given for this FTP session. This allows more
        than one session to be uploading from the same local directory. [Optional.
        Default is 'FTP'.]
        
        passive: True to use passive mode; False to use active mode. [Optional.
        Default is True (passive mode)]
        
        secure: Set to True to attempt an FTP over TLS (FTPS) session.
        
        debug: Set to 1 for extra debug information, 0 otherwise.
        
        secure_data: If a secure session is requested (option secure=True),
        should we attempt a secure data connection as well? This option is useful
        due to a bug in the Python FTP client library. See Issue #284. 
        [Optional. Default is True]

        reuse_ssl: Work around a bug in the Python library that closes ssl sockets that should
        be reused. See https://bit.ly/3dKq4JY [Optional. Default is False]
        """
        self.server = server
        self.user = user
        self.password = password
        self.local_root = os.path.normpath(local_root)
        self.remote_root = os.path.normpath(remote_root)
        self.port = port
        self.name = name
        self.passive = passive
        self.secure = secure
        self.debug = debug
        self.secure_data = secure_data
        self.reuse_ssl = reuse_ssl

        if self.reuse_ssl and sys.version < '3.6':
            raise ValueError("Reusing an SSL connection requires Python version 3.6 or greater")

    def run(self):
        """Perform the actual upload.
        
        returns: the number of files uploaded."""

        # Get the timestamp and members of the last upload:
        timestamp, fileset = self.get_last_upload()

        n_uploaded = 0

        try:
            if self.secure:
                log.debug("Attempting secure connection to %s" % self.server)
                if self.reuse_ssl:
                    # Activate the workaround for the Python ftplib library.
                    from ssl import SSLSocket

                    class ReusedSslSocket(SSLSocket):
                        def unwrap(self):
                            pass

                    class WeeFTPTLS(ftplib.FTP_TLS):
                        """Explicit FTPS, with shared TLS session"""

                        def ntransfercmd(self, cmd, rest=None):
                            conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
                            if self._prot_p:
                                conn = self.context.wrap_socket(conn,
                                                                server_hostname=self.host,
                                                                session=self.sock.session)
                                conn.__class__ = ReusedSslSocket
                            return conn, size
                    log.debug("Reusing SSL connections.")
                    ftp_server = WeeFTPTLS()
                else:
                    ftp_server = ftplib.FTP_TLS()
            else:
                log.debug("Attempting connection to %s" % self.server)
                ftp_server = ftplib.FTP()

            if self.debug:
                ftp_server.set_debuglevel(self.debug)

            ftp_server.set_pasv(self.passive)
            ftp_server.connect(self.server, self.port)
            ftp_server.login(self.user, self.password)
            if self.secure and self.secure_data:
                ftp_server.prot_p()
                log.debug("Secure data connection to %s" % self.server)
            else:
                log.debug("Connected to %s" % self.server)

            # Walk the local directory structure
            for (dirpath, unused_dirnames, filenames) in os.walk(self.local_root):

                # Strip out the common local root directory. What is left
                # will be the relative directory both locally and remotely.
                local_rel_dir_path = dirpath.replace(self.local_root, '.')
                if self._skip_this_dir(local_rel_dir_path):
                    continue
                # This is the absolute path to the remote directory:
                remote_dir_path = os.path.normpath(os.path.join(self.remote_root,
                                                                local_rel_dir_path))

                # Make the remote directory if necessary:
                _make_remote_dir(ftp_server, remote_dir_path)

                # Now iterate over all members of the local directory:
                for filename in filenames:

                    full_local_path = os.path.join(dirpath, filename)
                    # See if this file can be skipped:
                    if self._skip_this_file(timestamp, fileset, full_local_path):
                        continue

                    full_remote_path = os.path.join(remote_dir_path, filename)
                    stor_cmd = "STOR %s" % full_remote_path

                    with open(full_local_path, 'rb') as fd:
                        try:
                            ftp_server.storbinary(stor_cmd, fd)
                        except ftplib.all_errors as e:
                            # Unsuccessful. Log it, then reraise the exception
                            log.error("Failed uploading %s to server %s. Reason: '%s'"
                                      % (full_local_path, self.server, e))
                            raise
                    # Success.
                    n_uploaded += 1
                    fileset.add(full_local_path)
                    log.debug("Uploaded file %s to %s" % (full_local_path, full_remote_path))
        finally:
            try:
                ftp_server.quit()
            except Exception as e:
                log.debug("quit() exception %s: '%s'", type(e), e)

        timestamp = time.time()
        self.save_last_upload(timestamp, fileset)
        return n_uploaded

    def get_last_upload(self):
        """Reads the time and members of the last upload from the local root"""

        timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name)

        # If the file does not exist, an IOError exception will be raised. 
        # If the file exists, but is truncated, an EOFError will be raised.
        # Either way, be prepared to catch it.
        try:
            with open(timestamp_file_path, "rb") as f:
                timestamp = cPickle.load(f)
                fileset = cPickle.load(f)
        except (IOError, EOFError, cPickle.PickleError, AttributeError):
            timestamp = 0
            fileset = set()
            # Either the file does not exist, or it is garbled.
            # Either way, it's safe to remove it.
            try:
                os.remove(timestamp_file_path)
            except OSError:
                pass

        return timestamp, fileset

    def save_last_upload(self, timestamp, fileset):
        """Saves the time and members of the last upload in the local root."""
        timestamp_file_path = os.path.join(self.local_root, "#%s.last" % self.name)
        with open(timestamp_file_path, "wb") as f:
            cPickle.dump(timestamp, f)
            cPickle.dump(fileset, f)

    def _skip_this_dir(self, local_dir):
        """Determine whether to skip a directory."""

        return os.path.basename(local_dir) in ('.svn', 'CVS')

    def _skip_this_file(self, timestamp, fileset, full_local_path):
        """Determine whether to skip a specific file."""

        filename = os.path.basename(full_local_path)
        if filename[-1] == '~' or filename[0] == '#':
            return True

        if full_local_path not in fileset:
            return False

        if os.stat(full_local_path).st_mtime > timestamp:
            return False

        # Filename is in the set, and is up to date. 
        return True


def _make_remote_dir(ftp_server, remote_dir_path):
    """Make a remote directory if necessary."""

    try:
        ftp_server.mkd(remote_dir_path)
    except ftplib.all_errors as e:
        # Got an exception. It might be because the remote directory already exists:
        if sys.exc_info()[0] is ftplib.error_perm:
            msg = str(e).strip()
            # If a directory already exists, some servers respond with a '550' ("Requested
            # action not taken") code, others with a '521' ("Access denied" or "Pathname
            # already exists") code.
            if msg.startswith('550') or msg.startswith('521'):
                # Directory already exists
                return
        # It's a real error. Re-raise the exception.
        raise

    log.debug("Made directory %s" % remote_dir_path)
    return

Reply via email to