[Twisted-Python] twisted.web HTTPS client certificate

[Ian Haywood]

I am trying to work out how to retrieve on the server a X.509 
certificate presented by the HTTPS client.  This code tries to tell me 
the transport has no peer certificate.


same error when I use wget as a client so I think my problem is in the 
server code. I'm using self-signed certificates


Any clues as to what I'm doing wrong?

Ian

from twisted.internet import reactor, endpoints
from twisted.internet.ssl import Certificate
from twisted.web.server import Site
from twisted.web.resource import Resource


class CertPage(Resource):
isLeaf = True

def render_GET(self, request):
HTML = """

getPeerCertificate %r %r
"""
cert = Certificate.peerFromTransport(request.transport)
return bytes(HTML % (type(cert), cert), "ascii")


resource = CertPage()
site = Site(resource)
e = endpoints.serverFromString(
reactor, "ssl:8443:certKey=server.crt:privateKey=server.key"
)
e.listen(site)
reactor.run()
from zope.interface import implementer

from OpenSSL.crypto import FILETYPE_PEM
from twisted.internet.ssl import (
optionsForClientTLS,
Certificate,
PrivateCertificate,
KeyPair,
)
from twisted.trial import unittest
from twisted.internet import reactor
from twisted.web.iweb import IPolicyForHTTPS
from twisted.web.client import Agent, ResponseFailed, readBody


@implementer(IPolicyForHTTPS)
class LoadClientCert:
def __init__(self, hostmap, server_cert, client_cert, client_key):
self.hostmap = hostmap
with open(server_cert) as fd:
self.server_cert = Certificate.loadPEM(fd.read())
with open(client_cert) as fd:
client_cert = Certificate.loadPEM(fd.read())
with open(client_key) as fd:
client_key = KeyPair.load(fd.read(), FILETYPE_PEM)
self.client_cert = PrivateCertificate.fromCertificateAndKeyPair(
client_cert, client_key
)

def creatorForNetloc(self, hostname, port):
hostname = hostname.decode("ascii")
if hostname in self.hostmap:
hostname = self.hostmap[hostname]

import pdb

pdb.set_trace()
return optionsForClientTLS(hostname, self.server_cert, self.client_cert)


def getPage(url, server_cert, client_cert, client_key, hostmap={}):
a = Agent(reactor, LoadClientCert(hostmap, server_cert, client_cert, client_key))
d = a.request(b"GET", url.encode("ascii"))

def cb_getBody(response):
return readBody(response)

return d.addCallback(cb_getBody)


class HubTest(unittest.TestCase):
def test_getpage(self):
def cb_print(s):

print(s)

DIR = "/home/ian/athen/hub/"
d = getPage(
"https://localhost:8443/";,
DIR + "server.crt",
DIR + "client.crt",
DIR + "client.key",
{"localhost": "mintbox"},
)
d.addCallback(cb_print)
return d
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

On 12/03/2021 8:29 pm, Tobias Oberstein wrote:
since with io_uring the queue/ring to append to never blocks for 
writes, that buffering in twisted would lead to double buffering (as 
the ring already buffers)


then, io_uring follows a completion IO model (as IOCP). quite 
different from the "ready to write/read" model (as in epoll etc).


I am wondering, do you already have a design for these aspects of 
io_uring / twisted?
I haven't thought about it in detail, but python presents mmap as 
bytearray, so I would have thought it fairly easy to write directly to 
the ring from Protocol.write


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

On 12/03/2021 7:21 pm, Tobias Oberstein wrote:
sorry, I missed the beginning of the thread / discussion: are you 
working on a new, full Twisted reactor on top of io_uring?


and async filesystem is "just" one part / additional effort in that? 
the advantage of io_uring is being able to combine file and network I/O, 
so this PR is a precondition to working on the reactor.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

On 11/01/2021 8:26 pm, Glyph wrote:
seems somewhat irrelevant to the "asynchronous filesystem" part of 
this PR - do you think you could do a smaller version of this which 
decouples it from smb and ctypes?
but yes both can be spun out easily, so it's just the interface and a 
plain portable threads-based implementation. Where in the twisted 
tree should the interface and implementation go?
(It would also be nice to have an interface that acts as an 
IProducer to integrate more natively with Twisted's support for 
backpressure rather than only having a custom readChunk method.  I 
think readChunk is still necessary for completeness since you need 
to be able to seek and offset, though.)


certainly will look at this. It fits well with FTP's filesystem 
interaction.


OK this is done. I have changed the name to "asyncfs" in the hope this 
is more informative.


https://twistedmatrix.com/trac/ticket/10079

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

On 8/01/2021 7:23 am, Glyph wrote:


The mess of ctypes stuff


unclear what you mean: either the aio implementation or statx.py

seems somewhat irrelevant to the "asynchronous filesystem" part of 
this PR - do you think you could do a smaller version of this which 
decouples it from smb and ctypes?
but yes both can be spun out easily, so it's just the interface and a 
plain portable threads-based implementation. Where in the twisted tree 
should the interface and implementation go?
(It would also be nice to have an interface that acts as an IProducer 
to integrate more natively with Twisted's support for backpressure 
rather than only having a custom readChunk method.  I think readChunk 
is still necessary for completeness since you need to be able to seek 
and offset, though.)


certainly will look at this. It fits well with FTP's filesystem 
interaction.


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

On 6/01/2021 1:04 am, Adi Roiban wrote:
On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone 
mailto:exar...@twistedmatrix.com>> wrote:


On Tue, Jan 5, 2021 at 6:49 AM Barry Scott
mailto:barry.sc...@forcepoint.com>>
wrote:


What threads? Why do you call out file FDs different from
socket FDs?

If you give epoll()/select() a file FD it lies and says the FD is always 
"ready" even when, as you point out, file operations can block.


My suggestion is to find a real world / production use case for the 
new reactor so that we can run more than unit / functional tests.


--
Adi Roiban


As a "standard" reactor any twisted app would stress-test it for 
network/pipe operations, no?


For async file I/O my plan would be to export a new IFilesystem (which 
is closely based on conch.interfaces.ISFTPServer) apps would be have to 
be written to use it, conch could with minimal tweaking,  and 
unsurprisingly the SMB server will.


Apps can fallback to thread-based or aio(7) based implementations that 
already  exists as PoC: https://github.com/twisted/twisted/pull/1420


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

over engineered but don't exactly break new ground in a CS sense.

On 6/01/2021 11:35 am, Ian Haywood wrote:

Proprietary protocols like SMB tend to be over-eng

On 4/01/2021 4:41 pm, Glyph wrote:
I suspect that this may require somewhat less... cognitive surface 
area than your other contributions :).


And hey, we have a vaccine now, which means that maybe things will go 
back to normal or close enough to it that I'll have enough capacity 
to get back to it myself :)


-g


On Jan 3, 2021, at 8:01 PM, Ian Haywood  wrote:

In investigating async file I/O I came across this. In a nutshell 
it's the new epoll()


It's marginally more efficient although this is only apparent at 
very high loads. What's more interesting is that io_uring accepts 
files as well as network/pipe handles: avoiding the need for threads.


Here's a good intro: https://unixism.net/loti/index.html

If people think an IoUringReactor is worthwhile I'll open a ticket 
and make a start.


However it will need a reviewer... :-)

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

Proprietary protocols like SMB tend to be over-eng

On 4/01/2021 4:41 pm, Glyph wrote:

I suspect that this may require somewhat less... cognitive surface area than 
your other contributions :).

And hey, we have a vaccine now, which means that maybe things will go back to 
normal or close enough to it that I'll have enough capacity to get back to it 
myself :)

-g


On Jan 3, 2021, at 8:01 PM, Ian Haywood  wrote:

In investigating async file I/O I came across this. In a nutshell it's the new 
epoll()

It's marginally more efficient although this is only apparent at very high 
loads. What's more interesting is that io_uring accepts files as well as 
network/pipe handles: avoiding the need for threads.

Here's a good intro: https://unixism.net/loti/index.html

If people think an IoUringReactor is worthwhile I'll open a ticket and make a 
start.

However it will need a reviewer... :-)

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] reactor for Linux io_uring

[Ian Haywood]

In investigating async file I/O I came across this. In a nutshell it's 
the new epoll()


It's marginally more efficient although this is only apparent at very 
high loads. What's more interesting is that io_uring accepts files as 
well as network/pipe handles: avoiding the need for threads.


Here's a good intro: https://unixism.net/loti/index.html

If people think an IoUringReactor is worthwhile I'll open a ticket and 
make a start.


However it will need a reviewer... :-)

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Ian Haywood]

I think having a project coordinator as   a *first* priority isn't a 
good idea (not saying it isn't down the track)


Currently there's little to coordinate as the core team aren't active in 
planning or review. can the team be expanded ? (I'm not angling for a 
role myself)


Ian

On 4/01/2021 11:59 am, Adi Roiban wrote:



On Mon, 4 Jan 2021 at 00:54, Glyph > wrote:


It’s complicated and I’m not a lawyer, so maybe it is indeed not a
problem. But in brief it’s like trademark protection, kind of, in
that it becomes SFC’s problem to be aware that you’ve said these
things and tell you not to say them. The twisted project (which is
a bit of an amorphous concept to begin with) has authorized SFC to
be its fiscal sponsor, the SFC has gone through  the rigamarole
with the IRS to ensure this is an exempt-able public benefit
activity, and now someone is making claims about the project
hiring, which they’re on the hook for. How does the IRS know your
status of affiliation with the project or the conservancy for
sure? Somebody has to investigate it, investigating means asking a
bunch of questions and sucking up the SFC’s time and energy, even
if no enforcement action is ever formally taken.

In short: talk to the SFC first about the project’s status, get an
actual official recommendation and not my random opinion about
what may or may not be a problem, before doing anything related to
fundraising. I can’t say anything authoritative about what is
allowed, because as far as I understand it, *nothing* is allowed
without untangling the PLC/approval process first. :-)

(Except to raise money for the already-authorized expenses related
to the continued hosting of twistedmatrix.com
, of course.)


Thanks for the info and sorry for the trouble :)

I will try to send a (private) message to SFC these days to untangle 
the PLC team and I will add you to CC.


Until we solve the PLC issue, I would consider this discussion blocked.

Cheers
--
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Participating to GitHub Sponsors

[Ian Haywood]

No objection, but any plans for the money? I'd ask for more reviewers, 
I'd like to contribute more to twisted but there's little point without 
access to code review in a reasonable timeframe.


On 16/12/2020 11:56 am, Adi Roiban wrote:

Hi,

Does anyone have anything against applying so that Twisted can be on 
the waitlist of the GitHub Sponsors ?


https://github.com/sponsors/twisted/waitlist 



As far as I know, the money for the Twisted project is managed by the 
Software Freedom Conservancy organization and they are already 
approved hosts for GitHub Sponsors.


At this point, we only need to press the green button.
The bank account and other administrative issues are already sorted out.



If there are no complaints , I plan to press the green button in 1 week.

We will then have to see how to access those funds via Software 
Freedom Conservancy... but that is part 2.


Cheers
--
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Mailgun Email Service Replacement Proposal

[Ian Haywood]

On 3/10/2020 4:35 pm, Glyph wrote:

Basically, a signing / authenticating MX relay.

Anyone interested in attempting to write such a thing with Twisted? :)



I'm happy to it for  "fee": review of my SMB patches

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] smb component progress

[Ian Haywood]

I have continued to work slowly on this project. Named pipes are 
supported, the samba client can connect and list available shares.


i am currently working on a vfs layer using deferToThread to provide 
asynchronous file access


 I have tried to keep the code divided into logically separate chunks . 
my first chunk received comments from glyph 2 months ago but has 
remained in the review queue since.


https://twistedmatrix.com/trac/ticket/9818

is there anyone available to review this code to progress it ?

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] composition VS. inheritance

[Ian Haywood]

On 1/07/2020 1:41 am, Barry Scott wrote:

On Saturday, 27 June 2020 07:47:14 BST Ian Haywood wrote:


I've been told in code review to use composition instead of inheritance,
which is fine in a general sense but I have difficulty applying to
twisted-specific task




thanks for these pointers everyone.

I have changed the code in my PR to remove the subclass-of-a-subclass.

I'm now looking for a committer who is able to review it again.

https://github.com/twisted/twisted/pull/1274

Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] composition VS. inheritance

[Ian Haywood]

In smb I have a SMBPacketReceiver that inherits from t.i.p.Protocol, it 
breaks the incoming TCP stream into logical packets (the analogue of 
LineReceiver in line-based protocols).


I then subclass SMBPacketReceiver to SMBProtocol which does a lot of the 
"heavy lifting" analyzing incoming packets.


I've been told in code review to use composition instead of inheritance, 
which is fine in a general sense but I have difficulty applying to 
twisted-specific tasks.


1. how to do Factory.buildProtocol? It has to return a t.i.p.Protocol, 
but with composition the Protocol object is a private variable of 
SMBPacketReceiver, in turn a private variable of SMBProtocol.


2. what to do instead of overriding Protocol.dataReceived and access 
incoming data if not allowed to subclass it?


Now  its not that I cant think of workarounds to these two problems, but 
they're ugly


Is there some recent twisted code using composition that I can look at?


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] new code: 3.5 or 3.6?

[Ian Haywood]

I've followed the discussion re 3.6 and type annotations, but evidently 
not closely enough


when writing new code for twisted today, can we use 3.6 features or is 
3.5 still required? (it's actually a library feature enum.IntFlag I'm 
curious about, not type annotations)


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Ian Haywood]

On 16/05/2020 10:55 am, Ian Haywood wrote:


On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <mailto:wsanc...@wsanchez.net>


  I think it's great to get an SMB implementation in the Twisted 
org, but why would we even consider adding something like this to 
the main Twisted project?


The advantage of twisted itself is cross-protocol abstractions such as 
cred. Of course you don't have to be in the repo to use them, but in 
practice developers need the discipline of a single project to 
maintain consistency, otherwise the human tendency to reinvent wheels 
is too strong



Apropos we have two APIs for exporting filesystems which are broadly 
similar: SFTP in conch and FTP itself, SMB is about to be a third, 
ideally we should have one or have them descend from one another. It's 
slightly harder than it sounds as SMB is a file-access, as opposed to 
file-transfer, protocol and supports some extra features such as locking.


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SMB server component for twisted

[Ian Haywood]

On 15/05/2020 10:52 am, Glyph wrote:
y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega 


  I think it's great to get an SMB implementation in the Twisted org, 
but why would we even consider adding something like this to the main 
Twisted project?


The advantage of twisted itself is cross-protocol abstractions such as 
cred. Of course you don't have to be in the repo to use them, but in 
practice developers need the discipline of a single project to maintain 
consistency, otherwise the human tendency to reinvent wheels is too strong.


Ian

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SMB server component for twisted

[Ian Haywood]

On 7/05/2020 5:48 pm, Glyph wrote:

If you want to include it in Twisted itself, your best bet is to 
actually develop it /within/ twisted, as a series of small 
contributions, rather than as one gigantic one-shot one. 
 Contributions over, say, 400 lines, take exponentially longer to review.


Sounds great, I'll prepare a GitHub PR. Unfortunately the first one will 
be ~2000 lines just to login and connect to a share, but after that each 
new packet-type will be small.


Regarding unit-tests, I've found the best way to test the server is to 
use reactor.spawnProcess to launch the Samba command-line client, but 
that requires smbclient be available to run tests.


Ian



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] SMB server component for twisted

[Ian Haywood]

I have begun work on a SMB (Server Message Block; Windows filesharing) 
server protocol for twisted.


Work so far is here: https://github.com/ihaywood3/twsmb

I'm looking for any advice particularly around what I should be doing so 
this code is suitable for inclusion in twisted.


Ian Haywood

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python