Re: [Twisted-Python] Retry: Conch Issues
Thank you so much Aldi! Yes there are so many little decoupled pieces it's
difficult to figre out how to use them in concert. .Let's leave aside the
reconnecting option for a minute and just talk about how to percolate
errors up. Because currently I want to be able to always return from a
FileTransferClient call. By upleveling things do you mean like this?
Wait, ha, I actually don't know because the deferreds are one shot? Why
exactly is makeDirectory not firing any callbacks or errbacks and what can
I add at the Factory level to make it do so?
class MySSHClientFactory(SSHClientFactory):
def clientConnectionLost(self, connector, reason):
self.op.addErrback(reason)
def makeDirectory(self, path, attrs):
self.op = Deferred()
def _cbSuccess(result):
self.op.callback(result)
return self.sftpClient.makeDirectory(path,
attrs).addCallback(_cbSuccess)d
That won't work. If you can give me a breadcrumb or idiom I can apply it
across everything. I'm just too new to twisted to know the correct idiom
here. Totally good with moving the methods up to the factory, just not sure
how.
On Thu, Sep 24, 2020 at 5:36 PM Adi Roiban wrote:
> Hi
>
> On Fri, 25 Sep 2020 at 00:15, Robert DiFalco
> wrote:
>
>> I am having troubles with Conch. I've been able to reproduce them in a
>> unit test and all the code is here along with requirements.txt:
>>https://github.com/radifalco/conch_issues
>>
>> I'm trying to reuse my connection, but if it is idle too long the server
>> shuts it down. What I don't understand is why the next call to
>> `FileTransferClient` does not call back or errback? It will just hang
>> forever even though the Factory and Transport know the connection was
>> closed (as evidenced by the log messages).
>>
>> I don't ever want to hang, if I use FileTransferClient I would like it
>> always call a callback or errback. I feel like Conch is missing a deferred
>> somewhere but damned if I can find it.
>>
>> Thanks!
>>
>> Have you checked
> https://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
> ?
>
> I see that `connector.connect()` is commented in your code.
>
> The theory is that once a client connection is lost, you will need to call
> again the whole chain starting with reactor.connectTCP.
>
> -
>
> I know the pain here with SFTP as there is a lot of wrapping: TCP client,
> SSH high level protocol, SSH session, SSH channel, SSH subsystem.
>
> One option is to use MySSHClientFactory as a proxy for any SFTP operation.
> When the SSH channel is open and you have received the response from the
> subsystem request, pass the SFTP client all the way down to the factory.
> The factory will know when the connection is lost and then trigger a new
> connection... which will update the SFTP client instance on the factory.
>
> That is, instead of getting the SFTP client, the SFTP client will be
> pushed to the factory.
>
> I think that the main issue with your code is that it tries to work
> directly with the SFTP client, instead of using the factory as a proxy for
> SFTP operations.
>
> A protocol represents a single connection that is already made and it
> can't reconnect itself.
> The factory is responsible for handling new connections, or triggering new
> connections.
>
> So, your high-level API is using a very low-level object.
> The SFTP client represents an SSH connection that was already made,
> authenticated and the SFTP subsystem requested.
>
> For my production code I am doing something like this:
> MySSHClientFactory is instantiated with the reactor and the
> SFTPClientOptions.
> Then, when I want to list a directory I call
> sftp_factory.listDir('some/path')
> The factory can then check if it needs to get an SFTP client or reuse an
> existing one.
> A SFTP client is obtained by triggering a new connectTCP to the reactor
> with `self` as the factory.
>
> Does it make sense?
>
> --
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Retry: Conch Issues
Hi
On Fri, 25 Sep 2020 at 00:15, Robert DiFalco
wrote:
> I am having troubles with Conch. I've been able to reproduce them in a
> unit test and all the code is here along with requirements.txt:
>https://github.com/radifalco/conch_issues
>
> I'm trying to reuse my connection, but if it is idle too long the server
> shuts it down. What I don't understand is why the next call to
> `FileTransferClient` does not call back or errback? It will just hang
> forever even though the Factory and Transport know the connection was
> closed (as evidenced by the log messages).
>
> I don't ever want to hang, if I use FileTransferClient I would like it
> always call a callback or errback. I feel like Conch is missing a deferred
> somewhere but damned if I can find it.
>
> Thanks!
>
> Have you checked
https://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
?
I see that `connector.connect()` is commented in your code.
The theory is that once a client connection is lost, you will need to call
again the whole chain starting with reactor.connectTCP.
-
I know the pain here with SFTP as there is a lot of wrapping: TCP client,
SSH high level protocol, SSH session, SSH channel, SSH subsystem.
One option is to use MySSHClientFactory as a proxy for any SFTP operation.
When the SSH channel is open and you have received the response from the
subsystem request, pass the SFTP client all the way down to the factory.
The factory will know when the connection is lost and then trigger a new
connection... which will update the SFTP client instance on the factory.
That is, instead of getting the SFTP client, the SFTP client will be pushed
to the factory.
I think that the main issue with your code is that it tries to work
directly with the SFTP client, instead of using the factory as a proxy for
SFTP operations.
A protocol represents a single connection that is already made and it can't
reconnect itself.
The factory is responsible for handling new connections, or triggering new
connections.
So, your high-level API is using a very low-level object.
The SFTP client represents an SSH connection that was already made,
authenticated and the SFTP subsystem requested.
For my production code I am doing something like this:
MySSHClientFactory is instantiated with the reactor and the
SFTPClientOptions.
Then, when I want to list a directory I call
sftp_factory.listDir('some/path')
The factory can then check if it needs to get an SFTP client or reuse an
existing one.
A SFTP client is obtained by triggering a new connectTCP to the reactor
with `self` as the factory.
Does it make sense?
--
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Request for new Twisted release?
On Wed, 23 Sep 2020 at 18:11, Glyph wrote: > > > On Sep 22, 2020, at 3:42 PM, Craig Rodrigues > wrote: > > Amber, > > Can we have a new Twisted release within the next two months, say in > Nov./Dec. timeframe, or > sooner if you'd like? > > > Given that Amber hasn't been super responsive lately, and that I firmly > believe it should be possible to take breaks from open source (particularly > amidst the, you know, global collapse of civilization and stuff), I will > take the opportunity point out yet again that > http://twisted.readthedocs.org/en/latest/core/development/policy/release-process.html > documents > the release process and any sufficiently-motivated volunteer (with at least > a *bit* of history & trust within the community) can step forward and > receive all the permissions necessary to do all of these steps. > > I will personally volunteer to fill in the permissions gaps for any > interested parties, although I don't have much bandwidth for the release > itself. > > Amber, if you want to do it, go ahead and get started, I'm just saying we > don't need to put this on your shoulders :). > > Hi, To me, the current release process seems complicated and requires manual work and I think that the release document needs some update. The release documentation starts with "Tarballs for Twisted as a whole, and for each of its sub-projects". Is that still the case? Maybe the release can be simplified and automated: * Build the source and binary wheel in GitHub actions as this can be done for Linux (Ubuntu), Windows and macOS. * Host the download source and wheel files only on PyPi and publish them automatically from GitHub action on a new tag is created. * Host the documentation only on the Read The Docs. * Hos the API documentation on Read The Docs - might need some hacking, but at release we can create an API docs package as an artifact which is then pulled when Read The Docs documentation is created and copied as extra HTML files. * Move Twisted blog to GitHub pages... or even read the docs with a separate theme like Crate [1] ... if the blog is still required. -- With the above implemented the release should look like: * Each time the tests for a PR are executed, pydoctor will run and automatically create the API files as an artifact available for download. * *Manually* create a new branch in which the version is updated and the changelog/news/release note is created and all the news fragment files are removed/ . Have the branch reviewed and approved with all the tests passed. When the pydoctor tests are executed, the API docs are created. * The release branches can have a naming convention line 'release-20.0.0`. At first the release branch can have a release candidate version and a GitHub can automatically push the release to PyPi. * *Manually* send an email to Twisted mailing list to announce the pre-release. * Once the branch is merged, *manually* push a new tag * The new tag should trigger the GitHub action for publishing the release on PyPi * The new merge in master should trigger the Read The Docs build... which now will also include the static API docs pages. * Once the release is done, *manually* send a new email to the mailing list. There are still many manual steps, but the only permissions required is commit to the repo. This will no longer use any of the Twisted own infrastructure. [1] https://sphinx-themes.org/html/crate-docs-theme/crate/basic.html -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Retry: Conch Issues
I am having troubles with Conch. I've been able to reproduce them in a unit test and all the code is here along with requirements.txt: https://github.com/radifalco/conch_issues I'm trying to reuse my connection, but if it is idle too long the server shuts it down. What I don't understand is why the next call to `FileTransferClient` does not call back or errback? It will just hang forever even though the Factory and Transport know the connection was closed (as evidenced by the log messages). I don't ever want to hang, if I use FileTransferClient I would like it always call a callback or errback. I feel like Conch is missing a deferred somewhere but damned if I can find it. Thanks! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Conch SFTP Questions
Here's a gist and here are my high level questions, I hope they make sense and this is enough context. https://gist.github.com/radifalco/5a5cca4bf8d49d5c61113e36f9be7553 1. I would expect a `makeDirectory` to raise an exception when the SSH server closes a connection that is idle for too long (or any connection error/issue). However, it appears like `makeConnection` never fires any callbacks OR errbacks? In fact it doesn't seem to yield at all. It detects the connection is closed, calls closed, but `makeDirectory` never has any callbacks triggered. What am I missing? Is there a timeout I'm not setting? I wouldn't think so since the FileTransferClient and underlying transport already know the connection is no good. https://gist.github.com/radifalco/5a5cca4bf8d49d5c61113e36f9be7553#file-sftp-py-L280-L297 2. I can't quite figure out the reconnect/retry logic when the server closes the connection. Because when it recreates the connection the SFTPFileTransferClient has already been sent to the deferred. Maybe I need to retain that instance and call makeConnection on it again and not fire the deferred? As you can see I tried that but it doesn't work. On Wed, Sep 23, 2020 at 9:43 AM Glyph wrote: > > On Sep 22, 2020, at 3:47 PM, Robert DiFalco > wrote: > > Thanks! That is the full code. `connect` is from the conch library. > > > To clarify Adi's comment somewhat, the "full" code would mean following > the rules of http://sscce.org here; i.e. minimize the example to be > runnable on its own, then attach the code as you actually run it (as a .py > file, or in a link to a gist or a repo), rather than pasted into the body > of the email which can easily lead to transcription errors. > > For example, when I do paste your code into a file and run it, what I get > is: > > Traceback (most recent call last): > File "stuff.py", line 1, in > @attr.s(frozen=True) > NameError: name 'attr' is not defined > > > and yeah, one could guess as to where to get `import attr`, from — me > especially ;-) - but then we have to repeat that process for dozens of > names, one at a time, gradually reassembling your code into something that > approximates what it looked like to you; then we have to hook up your > connection example to be runnable on its own, and after we're done guessing > at all the imports we have to guess at the versions of things in your > environment. So a `requires.txt` generated by `pip freeze` or similar > would also be a helpful addition here, just so someone attempting to tinker > with your code can quickly get to the point rather than spending lots of > time setting up things you've already done. > > Thanks for using Twisted, and good luck with Conch! > > -glyph > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
