Re: [Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[exarkun]

On 8 Oct, 11:38 pm, gl...@twistedmatrix.com wrote:
On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring 
 wrote:


Eliot 0.9 is out, with a new utility for pretty-printing log messages 
and native journald support 
. You can now 
route Eliot logs to journald and when an error occurs easily find all 
logged actions that led up to that particular error, as opposed to 
most logging systems where this would involve reading all the logs and 
figuring out which ones apply and which to ignore.


Most logging systems can tell you what happened; Eliot tells you why 
it happened:


Cool, this looks awesome Itamar, thanks for letting us know!

Does Eliot make use of the new logging system at all?  If not, is there 
anything that might make Eliot's Twisted backend easier to maintain if 
we added it?  Just kind of curious about adoption of those new APIs in 
the world of logging...


As a user of Eliot and Twisted, an unpleasantness I commonly encounter 
is that the logs written by trial when running tests for code that uses 
Eliot for logging are a mish-mash of text messages and mangled Eliot 
logs.  The Eliot bits often get in the way of me reading the text part 
and the text parts and mangling ensure I can't use any Eliot tools to 
interpret the Eliot bits.


My understanding is that there aren't any existing hooks in Twisted to 
customize trial's logging which would allow this to be improved.


There is some related discussion (maybe more oriented towards a 
particular solution than a full description of the problem, though) on a 
ticket soon to be a couple years old:


 https://twistedmatrix.com/trac/ticket/6939

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Twisted is removing Python 2.6 support

[exarkun]

On 10:30 am, gl...@twistedmatrix.com wrote:


I think that new hardware would still be preferable because this 
machine is old, slow, overtaxed, and unreliable.  For example, it runs 
Speed Center as well as the OS X builder.  We have redundancy in most 
of our other builders now, which is very useful especially during times 
of heavy activity, like sprints, where lots of concurrent builds are 
going on, but this is the sole OS X builder.


Another area where redundancy would be nice is when doing upgrades. 
For example, if the download ever finishes, I'm going to pull the 
trigger on this Yosemite upgrade, and ... hopefully it works?  If it 
doesn't, we will have zero Mac builders until someone in the data 
center that is donating hosting can plug a console into that machine 
and see why it didn't finish booting.  There's not really a good way to 
manage this right now.


Finally, it would be nice to actually support multiple pythons on 
multiple OS versions - all the still supported / updated ones, anyway - 
and we'd need multiple builders for that.


I agree with all of this with one minor caveat - there are currently two 
OS X slaves:


 http://buildbot.twistedmatrix.com/buildslaves/bot-exarkun-1
 http://buildbot.twistedmatrix.com/buildslaves/osuosl-osx10.6

(and incidentally if someone wants to rename the former I think that'd 
be nice).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[exarkun]

On 19 May, 09:01 pm, tom.pri...@ualberta.net wrote:

Glyph gl...@twistedmatrix.com writes:
You could also find some other way to split the argument list but + 
doesn't seem particularly obscure in this context to me.  (If there's 
really a need to pass a literal + to a plugin we could add an 
escaping syntax as well.)


I think if we are adding syntax, then we should also add escaping at 
the

same time.

On a related note, when designing this kind of syntax, I think it is
often valuable to explictly leave some of the space as an explict 
error,

to leave freedom to extend the syntax in the future.


I think this is 100% correct.  This is part of why I don't want the 
syntax added to `Options`.  If it's a feature of, say, a compose 
twistd plugin then you can always throw the whole compose twistd 
plugin in the trash and start again.  That gives you quite a lot of 
space for syntax changes. :)


(And of course, not introducing a syntax at all leaves you even more 
room... but talking this crowd out of inventing weird syntaxes is 
probably an exercise in futility.)


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Need advice on how to move further with ticket 7897

[exarkun]

On 19 May, 04:38 pm, jerryguitar...@gmail.com wrote:

Hi!

I filed a ticket (Log the log events in 
twisted.internet.protocol.Factory
using the new logging system 
https://twistedmatrix.com/trac/ticket/7897)

and I have also submitted a patch for it. However, while working on the
patch, I had some concerns that I have already summarized in a ticket
comment. I would really appreciate it if someone could please take a 
look
and give me some advice as to how to move further with this ticket so 
that

I can start working on other patches for porting to twisted.logger. :-)


I think this question may indicate a certain section of documentation 
that is missing for twisted.logger.


Perhaps some folks who have ideas about how this new logging library 
should be used could look at that ticket and then clarify some existing 
twisted.logger docs or add something to them.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[exarkun]

On 02:41 am, zadka.mo...@gmail.com wrote:
OK, so let me once again enumerate the tickets I'm going to open, 
because

it seems like we're reaching consensus:

[1] PROVIDES (pretty much as written: exarkun -- the idea is to walk up 
the

hierarchy and then back down, so I did mean IService -- go up until you
find a root, then go down and get all descendents)


Oops.  You can't get children *or* a parent from IService.  But you can 
at least get children from IServiceCollection.


Jean-Paul

[2] 3538 -- will do it with a flag to turn on .tac + plugin
[2.1] I'm assuming the deprecation dance is flag to turn on - flag 
to
turn on + warn if .tac+plugin and no flag - make flag no-op - 
warn on
flag - remove flag. If someone doesn't want this dance, please let 
me

know preferred alternatives.


Glyph referred to the alternative.  Here it is fully spelled out:

https://twistedmatrix.com/trac/wiki/CompatibilityPolicy#ProcedureforExceptionstothisPolicy

Alternatively, do it the nice way but not *quite* as you described:

 * Make a new flag like --python but that combines with plugins.
 * Deprecate --python
 * Remove --python

Three steps instead of five and only bother the user once instead of 
twice.


Thanks!

Jean-Paul

[3] MAKE -- API for createServiceFromNameAndOptions (basically,
compose-as-a-library)
[4] COMPOSE (implements twistd_compose plugin)

I think there's at least rough consensus that these four pieces are 
useful,
even if there's still some disagreement on the details, so maybe the 
next

step is to open the three to-be-opened tickets, and then to discuss the
details on the tickets. I'm going to do the ticket opening tomorrow, 
and
maybe even work on them during the SF Python Meetup (anyone in SF -- 
you

should [a] go and [b] say hi also [c] optionally, help me with that).

If anyone has serious objections to this plan, let me know!

Thanks,
Moshe Z.

On Tue, May 19, 2015 at 2:35 PM Glyph gl...@twistedmatrix.com wrote:


 On May 19, 2015, at 14:01, Tom Prince tom.pri...@ualberta.net 
wrote:


 Glyph gl...@twistedmatrix.com writes:

 You could also find some other way to split the argument list but 
+

doesn't seem particularly obscure in this context to me.  (If there's
really a need to pass a literal + to a plugin we could add an 
escaping

syntax as well.)

 I think if we are adding syntax, then we should also add escaping at 
the

 same time.

 On a related note, when designing this kind of syntax, I think it is
 often valuable to explictly leave some of the space as an explict 
error,

 to leave freedom to extend the syntax in the future.

 I wonder if there is any value in having a syntax that is nestable. 
I

 don't see any specific use case, but I can imagine wanting ot pass
 options to `compose` that are scoped to an individual plugin that it 
is
 loading. And then, maybe you want to nest things so that those 
options
 apply to a subset of plugins, which might naturally be implemented 
as

 compos of a compose.

 The suggestions in the last paragraph are perhaps somewhat 
contrived,
 and certainly not something that should be *implemnted* in a first 
(or
 even second pass). But considering the possiblity, and then picking 
a

 syntax that might allow those kinds of extension (and then explictly
 making the extension syntax an error) gives us the ability to add 
those

 features without breaking backwards compatibility.

We can un-escape '\+' as a token just fine, and if we do that, all of 
the

weird use-cases you just described are possible.  I can't think of any
options I'd want to pass to 'compose' myself, but it would be easy 
enough

to add some flags to its parser.

-g




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[exarkun]

On 07:29 pm, gl...@twistedmatrix.com wrote:



On May 20, 2015, at 04:13, exar...@twistedmatrix.com wrote:

On 19 May, 09:01 pm, tom.pri...@ualberta.net wrote:

Glyph gl...@twistedmatrix.com writes:
You could also find some other way to split the argument list but 
+ doesn't seem particularly obscure in this context to me.  (If 
there's really a need to pass a literal + to a plugin we could add 
an escaping syntax as well.)


I think if we are adding syntax, then we should also add escaping at 
the

same time.

On a related note, when designing this kind of syntax, I think it is
often valuable to explictly leave some of the space as an explict 
error,

to leave freedom to extend the syntax in the future.


I think this is 100% correct.  This is part of why I don't want the 
syntax added to `Options`.  If it's a feature of, say, a compose 
twistd plugin then you can always throw the whole compose twistd 
plugin in the trash and start again.  That gives you quite a lot of 
space for syntax changes. :)


(And of course, not introducing a syntax at all leaves you even more 
room... but talking this crowd out of inventing weird syntaxes is 
probably an exercise in futility.)


Much as I love weird syntaxes, using + as a separator for a 'compose' 
plugin seems about as straightforward as I could imagine such a thing 
working.  Do you have an alternate proposal that is less 'weird'?  I 
wouldn't promise to accept it but I wouldn't want to go with something 
unnecessarily weird just due to a failure of my imagination.


It's possible this is a case of necessary weirdness - at least, if 
you're dead-set on some kind of option-based CLI interface to this 
functionality.


Think about other CLI tools that try to offer this kind of composition. 
Only two really come to mind for me and one of those is not exactly well 
regarded for its novel syntax (hint - it rhymes with schmestreamer).


This is not to say that this kind of composition is a bad thing - or 
even that no other widely-used software supports the kind of invoke- 
several-things-of-stuff behavior that we're talking about providing 
access to here.


However, I do think it is the case that most of the world tries to solve 
this problem without trying to force everything into a world-view framed 
by GNU getopt().


With that in mind, here are some alternatives:

 (0) The null proposal

   $ twistd compose \
   web --port 80 +
   manhole --auth foo + \
   logging --path /var/run

 (1) Borrow `--` which already means stop parsing options here.  It 
makes sense if you think about sub-commands as existing on a stack and 
`--` meaning pop the stack.  Possibly not actually feasible since I 
think `Options` already knows what `--` means (by virtue of actually 
*using* getopt()) and uses it to separate options from positional 
arguments (and therefore it's not an unambiguous separator - but I 
haven't double-checked this).


   $ twistd compose \
   web --port 80 -- \
   manhole --auth foo -- \
   logging --path /var/run

 (2) Put things into strings which already have well-defined escape 
syntax:


   $ twistd compose \
   'web --port 80' \
   'manhole --auth foo' \
   'logging --path /var/run'

 (3) Load things from a file (really just a variation on (2)):

   $ cat myapp.compose
   web --port 80
   manhole --auth foo
   logging --path /var/run
   $ twistd compose myapp.compose

Note these aren't really syntax ideas.  Apart from (1), they're ideas 
for not needing a new syntax.  As soon as you actually invent a new 
syntax you invite comments like Well, maybe you should use `;` instead 
of `+` because `;` is already kind of ingrained as a separator and 
people are used to having to escape it (which, see, I put in quotes, 
because I'm totally not saying that, even if it is totally true) to 
which you can only really reply that your gut agrees or your gut 
disagrees and it's always a bummer when the best support you have for 
some programming decision is some gut feeling you had.


I might actually be somewhat attracted to (3) because:

 * I don't really want to build up giant piles of configuration on my 
command line and then rely on my shell history for being able to find it 
again.  myapp.compose is something I could even put in my version 
control system.


 * Despite shell quoting and escaping rules being perfectly well 
defined, I'd really much rather not have to deal with an automatic extra 
layer of quotes.


And a last point to consider about new syntaxes - what are the security 
considerations of a command like:


   $ twistd compose \
   web --port ${WEB_PORT} +
   manhole --auth foo + \
   logging --path /var/run

I'll grant there's a bug there (the shell variable should be quoted - 
but I only know one person who consistently gets that right).


Regardless, I think any proposal would greatly benefit from a much 
greater consideration of use-cases and examples than I've yet seen 
discussed in this thread.

Re: [Twisted-Python] trial internals help?

[exarkun]

On 10:50 am, wsanc...@wsanchez.net wrote:
 I took a stab at porting trial to use twisted.logger 
(http://tm.tl/7863) but I have a test failing:


twisted.trial.test.test_reporter.TestErrorReporting.test_hiddenException

 I don’t quite know what that test is doing.  I added some additional 
output on my branch (branches/log-trial-7863) but that didn’t make me 
any smarter.


 Is anyone familiar with this code able to explain how I may have 
broken it?


-wsv


Hi Wilfredo,

This looks like a test for the logging done in the implementation of 
`IReactorTime`.


The test runs another test, `testHiddenException`. 
`testHiddenException` uses `reactor.callLater` to run a function that 
raises an exception.  Then `testHiddenException` does a horrible thing 
to the reactor to make the delayed call run (and thus raise the 
exception which it then expects to be attributed to the running test as 
an error).  Then it *also* fails the test - I suppose because at some 
point there was a bug where failure results interfered with error 
results somehow.


I don't see how this test is particularly different from one like:

   def test_hiddenException(self):
   try:
   raise SomeException()
   except:
   log.deferr()
   self.fail(something else)

Does your branch need to port the `IReactorTime` exception handling to 
the new logging API?  Or maybe it needs to port the way Trial itself 
integrates with the logging system so that it can observe failures like 
this one?


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[exarkun]

On 12:07 pm, zadka.mo...@gmail.com wrote:

Background

Currently, twistd assumes one-run-one-plugin. It would be nice to 
load up
multiple plug-ins in Twisted, for many reasons. These include: serving 
the

same in-memory content via separate protocols, adding manhole to other
plug-ins (so the end-deployer can add it to other things, as opposed to 
the

original implementor, and a catch-all category of auxiliary services.

Auxiliary services are those which are not useful in and of themselves, 
but

add value to a service which does something else of use.

Examples of auxiliary services -- a logging service (that connects to 
some
logging protocol on start-up), a metrics service (that sends statistics 
to

a collector like statsd or riemann) or an error-sending service (to
something like Sentry).

Proposal

tl;dr: four new tickets (codenamed SUBCOMMANDS, SERVICES, MANHOLE and
PROVIDERS) and one old ticket (3538)

SUBCOMMANDS: Add '+' as a special character in t.p.usage.Options. This
behavior will be off by default, and controlled by an attribute on the
Options instance allowMultipleSubcommands.

The attribute will only be checked when the first sub-command starts, 
to
allow setting it based on global flags. When the option is on, '-+' 
will be

passed as '+' to the Options instance, to allow sending plain '+' to
sub-commands.


Having a new, weird, fragile syntax is probably the least interesting 
part of this.  I suggest not doing this part - or at the very least, not 
doing it first and not making it a general part of `Options`.


There are lots of other ways to get the service object from more than 
one service plugin.  For example, read lines from a file.  Or have a 
variation of `--python` or something else similar using the existing 
option syntax `Options` supports.


The more interesting part to get right is the underlying model which you 
discuss elsewhere.


Glyph thinks there's a ticket for it. I couldn't find it in search for
tickets in 'core' whose description mentionds 'command'. Unless anyone 
can

find it, I'll open a new ticket.


I think there is a ticket for being able to use multiple twistd plugins. 
I don't think there's a ticket for a general change to `Options`.


SERVICES (depends on SUBCOMMANDS): In twistd, set the flag aMS if
'--allow-multiple-services' is given. Add to the application all 
services.


If you skip the `SUBCOMMANDS` ticket described above, then you can skip 
this too.
3538 (depends on SERVICES): If '--allow-multiple-services' is given, 
and

'--python .tac file' is given, process subcommands as usual.


Or just process the tac file and the subcommand if they're both given - 
without requiring an extra option?  The current behavior, silently 
ignore one of the arguments, doesn't seem particularly worth keeping to 
me.

PROVIDERS: Add a function,
  providersInHierarchy(IService, IInterface) - List[IInterface]
that returns all services in the hierarchy which provide the interface.
This ticket does not depend on any other tickets.


The first argument needs to be `IServiceCollection` instead of 
`IService`.

MANHOLE (depends on PROVIDERS, SERVICES): Add a built-in twistd plugin
named manhole. The plugin will expose manhole as PB/telnet with a
namespace that includes
 {'services': providersInHierarchy(manholeService, IService)}
This ticket technically could only depend on PROVIDERS, but to be 
useful,

it also depends on SERVICES


Manhole is part of Conch now and the telnet manhole is deprecated (and 
the PB manhole really should be deprecated - using a structured protocol 
for manhole isn't a bad idea but the existing implementation is half 
broken, mostly untested, exposes tons of implementation details as part 
of the public interface, etc.  If this were a piece of widely used 
software it would probably be worth gradually renovating - but it's 
basically used by no one so starting fresh makes more sense).


So it's part of Conch.  And ... it exists already.  I'm pretty sure no 
one will object if you add a new name to the default namespace.


Thanks for taking this on.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deprecating subproject packaging

[exarkun]

On 08:04 am, hawk...@atleastfornow.net wrote:

Hi,

I think that Twisted should deprecate subproject packaging


You probably won't hear much if any disagreement with this proposal on 
this list.


The sub-projects were never for people who like and are into Twisted. 
They were a tool to help convince people who don't use Twisted yet that 
it's not an unapproachable monolith.


We could discuss whether they have been effective in this regard but if 
you want to hear from people who *might* actually benefit from the 
split, you probably need to talk to people who aren't using Twisted.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Methods to upload and download file to a sftp server using twisted conch

[exarkun]

On 14 Apr, 11:47 am, sanfo.hali...@yahoo.fr wrote:
Hi everybody,I am student and need help of any one.I am working on 
internship and I have to implement a SFTP client usign twisted conch 
module.I have program thoe code to establish th ssh tunnel using 
password authentication. I arrive to create directory with 
MakeDirectory method of FileTransfertClient class.But my objective is 
to upload and download files and the FileTransfertClient  class has no 
method [or as I konw ] to do that.
Can you help me with way to do the uploading and the downloding with 
twisted conch?

Here is my program.code'''Created on 8 avr. 2015
@author: hsanfo'''


Hi Sanfo,

You may want to be a little more careful to preserve 
ownership/authorship information.  Your presentation of this code, 
particularly with an author tag including only your name, gives a 
strong impression that you are the author and copyright holder of this 
program.


Doing this sort of thing can get you into some trouble.  You may want to 
familiarise yourself with stackoverflow's terms of service and ask 
someone where you're interning to give you a basic overview of copyright 
law as it pertains to software.


Good luck,
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Project Job Posting

[exarkun]

On 29 Mar, 02:52 pm, exar...@twistedmatrix.com wrote:

On 28 Mar, 04:22 pm, exar...@twistedmatrix.com wrote:

Hello all,

The Twisted issue tracker needs someone to pay attention to it for 
spam content and clean up that content.  I've been taking care of this 
for a while now but I'm going to step back from this position soon.


Thanks for volunteering, Hawkowl, Eeshan Garg, and Pantelis Theodosiou.


I've granted these permissions to Hawkowl and Eeshan now (I didn't see 
Pantelis' trac username on the ProjectJobs page).


https://twistedmatrix.com/trac/ticket/7837 would be a good ticket to 
start with.


Let me know if you have any questions - otherwise, I'm officially hands- 
off now.


Thanks,
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted and twisted-infra organizations on GitHub

[exarkun]

On 2 Apr, 12:37 pm, a...@roiban.ro wrote:

Hi,

I would like suggest the following changes regarding the way repos are
organized in GitHub.

Move all twisted-infra repos into Twisted organization.. and maybe
merge many twisted-infra repos into braid.

Move treq, klein, txmongo, filepath... divmod projects into a new
organization dedicated to projects which are part of Twisted
ecosystem. Any project which imports Twisted can be part of it.

What do you say?


Perhaps you can explain why someone would want to do this.  As is, it 
sounds like a lot of busy work that, at best, will confuse and disrupt 
people.  If you explain what benefit would come from making this change, 
maybe it will be easier to see why it's a good idea.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Project Job Posting

[exarkun]

On 28 Mar, 04:29 pm, hawk...@atleastfornow.net wrote:

Hi exarkun,

I'd also like to be able to reset passwords on trac -- there's been a 
few instances where people aren't able to reset their password and 
people with the power haven't been around.


I don't know how trac password reset works.  Presumably the ideal 
solution would be to fix trac's built-in self-service password reset 
feature?  As far as I know noone has managed to understand why (or even 
under what conditions) it is broken, though.


If you'd like to deal with this manually until that happens, I suggest 
adding an entry to the ProjectJobs wiki page describing the requirements 
and volunteering yourself for the job for some period of time.  If you 
can describe the trac permissions required I'm sure someone can grant 
them to you.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Project Job Posting

[exarkun]

On 28 Mar, 04:22 pm, exar...@twistedmatrix.com wrote:

Hello all,

The Twisted issue tracker needs someone to pay attention to it for spam 
content and clean up that content.  I've been taking care of this for a 
while now but I'm going to step back from this position soon.


Thanks for volunteering, Hawkowl, Eeshan Garg, and Pantelis Theodosiou.

I've updated the description of this position on the ProjectJobs wiki 
page:


   https://twistedmatrix.com/trac/wiki/ProjectJobs#TicketSpam

Please have a look and (if it still sounds good), add your trac username 
to the Assigned to field.  Since there are several of you and you each 
have different availability times, it might also make sense to record 
that information as well (so we can see what kind of coverage we have 
and know what we're losing when you've had enough).


I'll set up the necessary trac permissions in a few days.  Let me know 
if you have any questions about the job or the job description.


Thanks!
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Project Job Posting

[exarkun]

On 28 Mar, 04:38 pm, jerryguitar...@gmail.com wrote:

Hello Mr. exarkun!

I would love to help out with this. I am from India. I have been making
some beginner bitesized contributions to Twisted (you have reviewed and
merged some of them) and I am looking forward to contribute in any way 
that
I possibly can. Please let me know what you think! Also, I'll be coming 
to
PyCon in Montreal this year and I hope to see you and learn from you 
there!


Hi Eeshan,

I'll be at PyCon too.  Looking forward to meeting you. :)

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Project Job Posting

[exarkun]

On 12:22 pm, a...@roiban.ro wrote:

Hi,

Without a Trac account you can not create a new ticket or update wiki 
page.


I just tried to create a new account named 'robot-spammer' and it
looks like it would be extremely easy to automate the process.

Observing notifications on IRC I saw that many spam comments were
posted by dedicated spam accounts.
Is there any effort in improving the registration process to prevent
spam accounts being created in the first place?
Email confirmation or reCAPTCHA or linking to external providers like 
GitHub?


Rather than hunting spam comments I prefer to work on preventing those 
comments.


Feel free to work on that if you like.

My goal with this thread is only to find a replacement for myself 
performing the cleanup steps.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted Project Job Posting

[exarkun]

Hello all,

The Twisted issue tracker needs someone to pay attention to it for spam 
content and clean up that content.  I've been taking care of this for a 
while now but I'm going to step back from this position soon.


These days spam is primarily submitted as new tickets.  When a spam 
ticket is filed, it needs to be deleted.  There's a form in the trac web 
interface for deleting tickets.  Access needs to be specially granted, 
so you need a trac account to volunteer for this role.


Trac is configured with some automatic spam filtering software.  It 
catches some, but obviously not all, spam submissions.


The trac rss feed can be used to monitor new content for spam.  Total 
time commitment for this job is pretty minimal (perhaps a few minutes a 
week) but timely action on spam is preferred.


It's probably also possible for this responsibility to shared between 
multiple people (divided up however those people find works).


I'll continue covering this role for another week but after that I'll 
stop - so please volunteer before then!


Thanks,
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Branch name convention

[exarkun]

On 12:10 am, gl...@twistedmatrix.com wrote:


For what it's worth, many of those branches can be removed.  We used to 
have a script that automated this deletion, but I can't find it now 
:-\.


It's part of twisted-trac-integration.

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 15.1 feedback

[exarkun]

On 03:41 pm, chriswwo...@gmail.com wrote:

Hawkowl,

The new functionality for installing optional dependencies does not on
Python3 (ticket: https://twistedmatrix.com/trac/ticket/7807).

This is due to two things:
1. some of the optional dependencies do not support python3
2. setup3.py isn't using the optional extras built in the main setup.py
file.


Are these regressions compared to 15.0?  I don't think they are because 
these features are brand new (on Python 2) in 15.0.  #7807 should be 
fixed (and hopefully for 15.2) but it doesn't sound like it's a blocking 
issue for completing the 15.1 release because it doesn't remove 
something that was included in 15.0.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Does anyone need / make use of SSHTransportAddress introduced in 12.1 ?

[exarkun]

On 17 Mar, 01:44 pm, a...@roiban.ro wrote:


Am I the only one who fails to see the advantages of using
SSHTransportAddress over a normal IAddress?


This doesn't make any sense.  `IAddress` has no attributes or methods. 
You can't do *anything* with an object that only provides `IAddress`.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] __nonzero__ for Deferred

[exarkun]

On 18 Mar, 10:03 pm, gl...@twistedmatrix.com wrote:
I'd like to deprecate and remove __nonzero__ on Deferreds. 
Occasionally I see code where someone (maybe even me) does if d: 
where they should have been doing @d.addCallback def cb(result): if 
result: 


Anybody else have this issue?  Objections?


Er.  What?  It's a little early for april fools' day, isn't it?

Seriously, this is weird.  Am I missing something?

Can expand on what it would mean to deprecated a special method of a 
class that doesn't even define it and what makes Deferred unique among 
all user-defined classes in this regard.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Removing Python 2.6 Support after Twisted 15.1

[exarkun]

On 18 Mar, 10:45 am, hawk...@atleastfornow.net wrote:

Hi,

As Python 2.6 has been officially abandoned for a year+ (and will 
receive no further official security updates) and Python 2.7 has been 
available for almost five years, it might be time to put Python 2.6 
support to bed. The only currently supported distro that has Python 2.6 
is RHEL6, and RHEL7 with Python 2.7 has now been out for a year.


This would be in line with other major projects (eg. Django, Plone) and 
Python's porting guide (https://docs.python.org/3/howto/pyporting.html 
#drop-support-for-python-2-6-and-older). We can also remove some things 
from TestCase, and there's random smatterings of 2.6 support 
workarounds/comments that we could also remove.


As such, I think we should make 15.1 the final release which officially 
supports Python 2.6.


Any objections?


Please keep in mind that the way Twisted drops support for various 
platforms is by:


 1) announcing that it will be so
 2) removing the BuildBot configurations that demonstrate Twisted works 
on those platforms
 3) dropping the development/review requirement that the test suite be 
kept passing on those platforms


Contrast this with another possible meaning of drop support which is 
*not* what Twisted does:


 1) actively prioritize a bunch of work to make sure that Twisted 
definitely will be as broken as possible on the de-supported platform


Thanks.
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Development documentation in Trac wiki

[exarkun]

On 02:35 pm, a...@roiban.ro wrote:

Hi,

I would like to talk about moving Twisted development documentation out 
of

official Twisted doc.

They can be moved either into the Trac wiki or a separate project ...
maybe twisted-dev-tools

The problem with Twisted main repo is that (AFAIK) updates are done
only after a release... so I change might be delay quite a lot (6
months)


The coding standard really shouldn't be changing so frequently that this 
is a major concern.  Additionally, everyone who is developing Twisted 
has a Twisted checkout with the most recent version of the information 
anyway.


Also, releases *should* be happening more frequently than every six 
months.


This doesn't seem like a very compelling reason to make this change.

Jean-Paul

We already have a lot of info inside the Trac wiki..

My plan it to move the following docs into Trac and organize existing
specification and development wiki pages:

http://twistedmatrix.com/documents/current/core/specifications/index.html
http://twistedmatrix.com/documents/current/core/specifications/banana.html
http://twistedmatrix.com/documents/current/core/development/index.html
http://twistedmatrix.com/documents/current/core/development/naming.html
http://twistedmatrix.com/documents/current/core/development/philosophy.html
http://twistedmatrix.com/documents/current/core/development/security.html

I would also move the following docs to Trac so that they can be
updated much faster.

http://twistedmatrix.com/documents/current/core/development/policy/svn- 
dev.html

http://twistedmatrix.com/documents/current/core/development/policy/index.html
http://twistedmatrix.com/documents/current/core/development/policy 
/coding-standard.html
http://twistedmatrix.com/documents/current/core/development/policy 
/writing-standard.html
http://twistedmatrix.com/documents/current/core/development/policy 
/test-standard.html


http://twistedmatrix.com/documents/current/core/development/policy/doc- 
standard.html

Do we still use XHTML ?

Trac support RST format, so we can maintain those pages in RST... and
trac wiki allow to view changes as diffs.

Having docs in repo is nice as you can send them for review, but i
feel that this is counter-productive for the current case.

What do you think?

Thanks!

--
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Making sense of buildbot problem log

[exarkun]

On 6 Feb, 07:35 am, a...@roiban.ro wrote:

Hi,

I got this problem log in builbot

https://buildbot.twistedmatrix.com/builders/osx10.6-py2.6-select/builds/4868/steps/select/logs/problems

and anothher for Windows

https://buildbot.twistedmatrix.com/builders/windows7-64-py2.7/builds/4147/steps/select/logs/problems

they talk about 4 errors, but I can only see one.

The log is full of non-problem info like skip and todo, so I tried to
search for [ERROR] string


Failures and errors are always at the end of the log.

Jean-Paul

How can I find the other errors?

Is there a way to see the log only with the problems? Ignoring
skipped/toto test?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Review tag for closed tickets

[exarkun]

On 26 Jan, 08:51 pm, a...@roiban.ro wrote:


The review should be completed before the ticket is merged, so the bot
should not be doing this automatically.


but in the commit message I do add the author, why not have the bot
automatically assign the first author form the list, review merge and
close the ticket?

What is the purpose of these manual task and why you don't want to
automate them?


You jumped pretty quickly from it is not automated to you don't want 
to automate them.


Building reliable automation is work.  If that's work you want to do, 
feel free.


One challenge you'll have to contend with is that the metadata 
automatically collected by trac is often incorrect.  For example, the 
author field is populated based on the person who creates the svn 
branch.  Non-committers can never create svn branches so the tickets 
worked on by non-committers often have an incorrect author field.


I'm sure this can be overcome.  It's just one example of an extra 
difficulty you might not have immediately recognized when thinking about 
how reliable automation to take over this menial task could work.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Spurious failure on winxp32-py2.7

[exarkun]

On 02:19 pm, a...@roiban.ro wrote:

While running tests I see that the following tests fails randomly...
it it was failing  from some time


twisted.words.test.test_xpath.XPathTests.test_position


This is an unrelated test.  I'm not sure if this is a simple copy/paste 
mistake or if you're mis-reading trial output.  This is the test before 
the test that failed.


The test that failed is named at the bottom of the failure report:

twisted.trial._dist.test.test_disttrial.DistTrialRunnerTestCase.test_runUntilFailure

This test has intermittently failed on Windows since disttrial was 
merged into trunk.


The ticket tracking this problem is 
https://twistedmatrix.com/trac/ticket/6027

===
[FAIL]
Traceback (most recent call last):
 File 
C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_dist\test\test_disttrial.py,

line 371, in test_runUntilFailure
   self.assertEqual(5, len(called))
 File 
C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_synctest.py,

line 447, in assertEqual
   % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = 5
b = 2

--

On my local computer (Ubuntu 14.04) the test is always skipped.


The skip report should tell you why it's skipped.  Presumably you're 
missing one of the dependencies - though off the top of my head I'm not 
sure what third-party dependencies this code has.

Does anyone has any idea why the test is not skipped ?


The default state of a test should *not* be to skip.  It should be to 
run.  A skipped test is a useless test.


Jean-Paul

I would like to have stable results from this tests as with the
current state I always have to double check if a failure on XP is a
new one or is just this bad tests.

Thanks!
--
Adi Roiban


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twistedchecker - Names of constant which are used as configuration options

[exarkun]

On 02:09 pm, a...@roiban.ro wrote:

On 25 January 2015 at 01:01,  exar...@twistedmatrix.com wrote:

On 24 Jan, 07:51 pm, a...@roiban.ro wrote:


On 24 January 2015 at 19:23,  exar...@twistedmatrix.com wrote:


On 06:45 pm, a...@roiban.ro wrote:



Hi,

Why reviewing a patch I got into this error from twistedchecker.
This is old code, just that someone it was touched by recent 
changes,



Also: pre-existing errors like this one should not block new 
development
work being done.  It is *not* a requirement that a patch fix all 
nearby but

unrelated problems like this one.


OK. but then maybe twistededchecker and pyflakes should not be part of
the stable builders.
In my head, all stable builders should pass before merging new code.
New and changed code should conform to the standard and try not to 
introduce
new errors.  If the errors being introduced are due to bugs in a 
tool that

reports errors, the bugs should be fixed.


New code should only try not to introduce new errors?


The tools are there to help development, not hinder it.  If a tool is 
producing an error that doesn't help development, slavishly adhering to 
a policy that requires additional work that doesn't help the project is 
counter-productive.


If all the tools were perfect we could say that all new code *must* 
never introduce new errors.  Unfortunately the tools are not perfect.
I was expecting that new code is not allowed to introduce new errors. 
Period.

Existing code *near* new or changed code can and should be left alone.


Does this mean that The (boy) scout rule does not apply for Twisted 
development?


I'm not familiar with The (boy) scout rule.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twistedchecker - Names of constant which are used as configuration options

[exarkun]

On 06:45 pm, a...@roiban.ro wrote:

Hi,

Why reviewing a patch I got into this error from twistedchecker.
This is old code, just that someone it was touched by recent changes,

C0103:574,12:NetstringReceiverTests.test_buffer: Invalid name
MAX_LENGTH for type attribute (should match
((([a-z_])|([a-z]+_[a-z]))[a-zA-Z0-9]+)$)

The code is like this, where TestNetstring is a line receiver
protocol... so MAX_LENGTH act as a configuration option.

   a = TestNetstring()
   a.MAX_LENGTH = 699

What to do in this case and how I can have  this code accepted by
twistedchecker?


Fix twistedchecker.  MAX_LENGTH doesn't violate the Twisted naming 
convention.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twistedchecker - Names of constant which are used as configuration options

[exarkun]

On 24 Jan, 07:51 pm, a...@roiban.ro wrote:

On 24 January 2015 at 19:23,  exar...@twistedmatrix.com wrote:

On 06:45 pm, a...@roiban.ro wrote:


Hi,

Why reviewing a patch I got into this error from twistedchecker.
This is old code, just that someone it was touched by recent changes,


Also: pre-existing errors like this one should not block new development 
work being done.  It is *not* a requirement that a patch fix all nearby 
but unrelated problems like this one.


New and changed code should conform to the standard and try not to 
introduce new errors.  If the errors being introduced are due to bugs 
in a tool that reports errors, the bugs should be fixed.


Existing code *near* new or changed code can and should be left alone.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 12:15 pm, tobias.oberst...@tavendo.de wrote:

which POSTs structured data (though urlencoded, not JSON) to the
codespeed server.

I see. And codespeed parsest that, stores it in a database and 
produces

graphics?

It seems, reporting the results via a WAMP RPC to a backend should be
quite easy to add in speedcenter.py

WAMP?  I don't think so.  I understand you work in that space a lot 
but I think
you might want to take a step back.  The extra complexity of 
WebSockets is

pointless for this project.  Just HTTP POST some JSON.


All the extra complexity is hidden using WAMP and AutobahnJS.


Oh well.  I'm not going to spend much more effort to convince you that 
this is a bad idea.  Maybe someone else will.

This can be really simple.  Your time is almost certainly better spent
elsewhere.


No. Sinking time into Django/codespeed is lost time for me ..


If you read my emails, you'll see that I explicitly recommended 
*against* contributing to codespeed.  If you're not going to read what I 
write then I'll stop writing, I guess.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 05:00 pm, tobias.oberst...@tavendo.de wrote:

On Jan 12, 2015, at 12:49 PM, Tobias Oberstein But what is the
interface between test cases from twisted- benchmarks to
codespeed?

Codespeed runs the benchmark, and they print out this stuff:
https://bazaar.launchpad.net/~twisted-dev/twisted-
benchmarks/trunk/view/head:/benchlib.py#L12

POSTing them via JSON would be nicer, structured data is great.

Nothing parses that output.  It's just for humans.

The code you're looking for is:

  https://bazaar.launchpad.net/~twisted-dev/twisted-
benchmarks/trunk/view/head:/speedcenter.py

which POSTs structured data (though urlencoded, not JSON) to the
codespeed server.


I see. And codespeed parsest that, stores it in a database and produces 
graphics?


It seems, reporting the results via a WAMP RPC to a backend should be 
quite easy to add in speedcenter.py


WAMP?  I don't think so.  I understand you work in that space a lot but 
I think you might want to take a step back.  The extra complexity of 
WebSockets is pointless for this project.  Just HTTP POST some JSON. 
This can be really simple.  Your time is almost certainly better spent 
elsewhere.


Jean-Paul

How is Twisted speedcenter orchestrated / triggered?

I mean, a new commit to Twisted repo will trigger rerunning all speed 
tests? If so, how does that work?


/Tobias


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 03:21 am, gl...@twistedmatrix.com wrote:



On Jan 12, 2015, at 1:08 PM, exar...@twistedmatrix.com wrote:
E.g. I need latency histograms, but this seems unsupported (benchmark 
results can
only have avg/min/max/stddev). For me, this isn't nice to have, but 
essential.
Throughput is one thing. Constistent low latency a completely 
different. The latter is

much much harder.


Codespeed is terrible.  But this is not one of the ways in which it is 
terrible.  Codespeed doesn't care if you label your measurement 
latency.  I think you've just noticed that what the existing 
benchmarks measure is mostly (entirely?) throughput.  If you wanted to 
write a latency benchmark, I don't think anything's stopping you.


I believe Tobias was not saying codespeed can't have a measurement 
called 'latency' but rather codespeed can't do histograms of 
measurements, which we need for measurement of latency and you don't 
need for measurement of throughput.  Is that accurate?  I don't know 
if there's a histogram feature hidden in the UI somewhere.


It would be nice to at least try a little bit to contribute things 
(like a histogram feature) to codespeed before charging off in a 
completely different direction.


I wasn't suggesting it would be a good idea to contribute to codespeed. 
I think codespeed should be thrown in the trash.  It was a great 
demonstration of a concept and we should thank it for that.  However, as 
the basis of future development - no, it's an awful piece of 
unmaintained software.


I was just trying to say that work towards replacing it should learn 
what it can from codespeed to try to avoid creating another piece of 
awful, ultimately unmaintained software.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 03:18 am, gl...@twistedmatrix.com wrote:



On Jan 12, 2015, at 12:49 PM, Tobias Oberstein
But what is the interface between test cases from twisted- 
benchmarks to codespeed?


Codespeed runs the benchmark, and they print out this stuff: 
https://bazaar.launchpad.net/~twisted-dev/twisted- 
benchmarks/trunk/view/head:/benchlib.py#L12


POSTing them via JSON would be nicer, structured data is great.


Nothing parses that output.  It's just for humans.

The code you're looking for is:

 https://bazaar.launchpad.net/~twisted-dev/twisted- 
benchmarks/trunk/view/head:/speedcenter.py


which POSTs structured data (though urlencoded, not JSON) to the 
codespeed server.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 08:49 pm, tobias.oberst...@tavendo.de wrote:


Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be 
used

by speedcenter.twistedmatrix.com: I have issues here as well.

E.g. I need latency histograms, but this seems unsupported (benchmark 
results can
only have avg/min/max/stddev). For me, this isn't nice to have, but 
essential.
Throughput is one thing. Constistent low latency a completely 
different. The latter is

much much harder.


Codespeed is terrible.  But this is not one of the ways in which it is 
terrible.  Codespeed doesn't care if you label your measurement 
latency.  I think you've just noticed that what the existing 
benchmarks measure is mostly (entirely?) throughput.  If you wanted to 
write a latency benchmark, I don't think anything's stopping you.


I think we can find something better for that part.


I suggest more fully understanding the capabilities and limitations of 
of codespeed before embarking on a project to replace it.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 1 Jan, 10:21 am, tobias.oberst...@tavendo.de wrote:

Hi,

I am doing network performance tests using netperf on a trivial Twisted 
TCP echo server (code at the end).


One of the tests that netperf offers is throughput, and I am running 
into an issue with this.

? [snip]


Now, my suspicion is that Twisted is reading off the TCP stack from the 
kernel and buffering in userspace faster than the echo server is 
pushing out stuff to the TCP stack into the kernel. Hence, no TCP 
backpressure results, netperf happily sends more and more, and the 
memory of the Twisted process runs away.


What you said here about buffering in userspace is ambiguous.  It's 
not clear if you meant data is being buffered in userspace on the read 
side before your protocol gets a chance to handle it or if you meant 
that data being written to the transport by the protocol is being 
buffered in userspace.


The former doesn't happen.  There are no no userspace read buffers in 
Twisted between the transport and the protocol.  Bytes are read from the 
socket and then passed to dataReceived.


The latter would be addressed by using producer/consumer APIs as Glyph 
suggested.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[exarkun]

On 12:06 pm, tobias.oberst...@tavendo.de wrote:


Fact is: somehow memory runs away.

How do I track down _where_ exactly the mem is spent? Probably that 
leads to the why then ..


There are memory profiling tools for Python.  For example, 
memory_profiler:


   https://pypi.python.org/pypi/memory_profiler

As far as I know, there are none that are specific to Twisted.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Mixing Python threads and Twisted code

[exarkun]

On 06:15 pm, alexan...@quessy.net wrote:

Hello everyone,

What happens if some code starts a thread while I am using Twisted's
reactor?


Nothing special.

Here is what I need to use:
https://github.com/ekulyk/PythonPusherClient/blob/master/pusherclient/connection.py#L12

This thread polls some socket to handle received TCP messages. It 
appears

it has been very slow in production. Is it this mix of twisted and
non-twisted threading that causes this brutal slowdown?


Not likely.  Twisted doesn't have its own threading implementation. 
Twisted has a threadpool module which is a thin layer on top of Python's 
threading module.

If so, how can I
easily speed it up without having to twistify the library I need to 
use?

(and practically rewrite it)


Profile it.  Find the particular operations that are slow.  Then you'll 
be one step closer to knowing how to speed things up.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doc bloat

[exarkun]

On 12:55 am, gl...@twistedmatrix.com wrote:



On Dec 2, 2014, at 20:05, exar...@twistedmatrix.com wrote:


Are there lots of useless docstrings on nested function definitions 
purely for the sake of twistedchecker?  Or are there undocumented 
nested functions that are actually a little bit difficult to 
understand on their own?


twistedchecker does not presently require nested function definitions 
to have docstrings.  I recently merged a fix to an incongruity where it 
was requiring this of classes defined within functions: 
https://github.com/twisted/twistedchecker/commit/4af4e97f99d6e5f683b65272a8dbe7bce2087aa7. 
So this one, at least, we can cross off for the future :).


The broader context of this suggestion was that we should inspect the 
codebase to see what policy changes would improve the quality of the 
code/documentation while reducing the effort required to develop and 
maintain it.


It sounds like you have some ideas about such changes already.  Does 
that mean you'd like to suggest them (presumably in the form of issues 
filed against twistedhecker) instead of doing this investigation?


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] doc bloat

[exarkun]

On 10:02 am, a...@roiban.ro wrote:
As a Twisted API users, I don't find a big problem the fact that 
__repr__

docstring is presented by pydoctor together with the other method,

As a Twisted contributor, I don't like when I have to copy/paste 
docstring

from an interface or from an inherited class ...


I hate it when people do this, too.  I never want anyone to do this. 
When I review changes in which this appears, I ask the author to change 
the submission to include useful, non-redundant documentation instead.


The coding standard doesn't explicitly say that you should not copy the 
docstring from an implementation to an interface - but this is a good 
example of the kind of thing our documentation can't be expected to 
cover.  Copying an interface docstring to an implementation is just one 
of the many possible bad practices a developer could engage in; a list 
of all possible bad practices would make the developer documentation 
infinitely long.


To the larger issue, I think the most productive thing to do would be to 
actually look at our code base and see if there are glaring examples of 
useless documentation.  If we can generalize from these examples to a 
rule, we can add that rule to the project policy (preferably encoded in 
a tool, not as prose in the developer documentation).


I'm a little bit tempted to agree with the `__repr__` case.  People who 
are reading the API documentation might not be put off by documentation 
for these methods but are they helped in any way?  I think that most of 
the time they probably aren't.  I'm basing this on times I can recall 
having written a `__repr__` docstring and how I had to struggle to put 
useful words into that docstring instead of pointless boilerplate.  I 
still think it would be good to collect a bunch of actual `__repr__` 
docstrings from Twisted and see if such evidence supports this 
conclusion.

or too explicity state
that the type of None is NoneType... and then to add a link to it :)


It sounds like there's agreement on this point, at least.  For what it's 
worth, this seems uncontroversial to me: the type of `None` is 
`NoneType` and the only instance of `NoneType` is `None`.  Documenting 
one or the other conveys all possible information there is to convey so 
documenting one should be enough.  Even if we wanted the rendered 
version of the documentation for this case to be redundant, pydoctor can 
synthesize that redundancy - it doesn't need the source material to be 
redundant.



As a first approximation, I think we could ask twistedchecker to stop
enforcing docstring requirements for objects directly matching a 
private

naming pattern.

Is there a ticket/issue for that?
What does `stop enforcing docstring requirements` mean?

I think that docstring are important, and testing code or private 
methods
is still code which should be maintained and it should have docstrings, 
but

it would be nice not to ask for
strict @param @type @return @rtype.


Eh.  Test code has to be maintained just like anything else.  As the de 
facto specification for most features, it's perhaps (but only perhaps) 
more important to have everything clearly documented in a test suite.  I 
think that requiring parameter and return type documentation in the test 
suite is a very good thing.


However, requiring the same degree of documentation for *all* code may 
not be necessary.  If a test method defines a simple nested function, 
perhaps that doesn't need a docstring.  This is another thing we could 
inspect the current Twisted codebase for.  Are there lots of useless 
docstrings on nested function definitions purely for the sake of 
twistedchecker?  Or are there undocumented nested functions that are 
actually a little bit difficult to understand on their own?


Jean-Paul

---

+1 for getting serious about twistedcker ...

I would be happy to help with twistedchecker, as well as with the other
automated tools.

What do you say if we put all problems/issues/comments here
https://github.com/twisted/twistedchecker/issues
and have 2 or 3 people who can decide if an issues is accepted, or help
define the right behaviour or reject it and then also review the pull
requests.

Right now there are many undecided or not tagged tickets, and I don't
know which one is a real problem or not.

One important requirement to move things forward is to provide quick
feedback, both for the initial issue report and the review.
Waiting months to get feedback from a patch is not fun and is a sign 
that

core developers don't care about the newly proposed changes.

--

+1 for getting serious about the other code checkers: pyflakes and
pydoctor. Right now, is a PITA to run all these tools on your local
computer ...

Cheers
--
Adi Roiban


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] I would like to port twisted to c++(14)

[exarkun]

On 28 Nov, 10:43 am, dany...@gmail.com wrote:

Hello everyone,

I am Danny, a C++ dev,
I'm familiar with twisted from past projects and by this day I am 
enchanted

with its magic.

I would like to port twisted to c++ (or at least the base ot it) and 
have

couple of questions -
1. Is it legally ok to port it and open source it at github ?
2. Would anyone consider helping to this cause ?
3. Would you know of a code base that would benefit from such port ?


Perhaps first have a look at ACE:

   http://www.cs.wustl.edu/~schmidt/ACE.html

and Boost::ASIO:

   http://www.boost.org/doc/libs/1_57_0/doc/html/boost_asio.html

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I debug this network problem?

[exarkun]

On 18 Nov, 12:52 pm, peter.westl...@pobox.com wrote:

On Tue, 18 Nov 2014, at 01:09, exar...@twistedmatrix.com wrote:

On 12:42 am, peter.westl...@pobox.com wrote:


Okay, how about http://bugs.python.org/issue9090, Issue #9090 : 
Error

code 10035 calling socket.recv() on a socket with a timeout
(WSAEWOULDBLOCK - A non-blocking socket operation could not be
completed
immediately)?

Twisted also doesn't use sockets with timeouts - although the way 
socket
timeouts are implemented in Python, I could perhaps imagine some 
timeout

configuration leaking into Twisted's sockets due to some other code
running in the same process.

Can you check to see if your AMP connection's underlying socket has a
timeout set?  It *shouldn't*.  Twisted doesn't set one.  If some other
code is causing a timeout to be set, I guess this could be the 
culprit.


socket.gettimeout() returns 0.0, which just means non-blocking, right?


According to the Python documentation, yes.  I was expecting to see 
`None` instead, though.

Given that strace shows a recvfrom failing with an EAGAIN, followed by
a recvfrom reading the AMP reply after the missing one, I think this
must
be a kernel or hypervisor bug. In hindsight it could never have been
1038591
even if Twisted had used that bit of code, because that reads the data
in and
then loses it. So strace would have shown it being read.

I'll get some help from my local TCP experts, and let you know if I 
find

anything
out. In the meantime, thank you all for your patience and help.


Good luck!  Please let us know if you discover anything else.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] to force-builds.py or not to force-builds.py

[exarkun]

On 18 Nov, 06:40 pm, a...@roiban.ro wrote:

As commented by @glyph

Since we've added the buildbot link to trac pages, force-builds.py is
basically obsolete.  I think we should  just scrub all mentions of it,
and probably delete it.

I have created a ticket to remove it and clean documentation:

https://github.com/twisted/twisted-dev-tools/issues/5

Is there anyone else still using this tool?
Should we keep it and rather than removing it, improve its 
documentation?


Given that it already exists, I'm not sure it makes much sense to delete 
it.  Perhaps some folks prefer a workflow that is command-line oriented 
rather than web-browser based.


I don't really feel very strongly about it, though.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Permissions to trigger buildbot tests

[exarkun]

On 11:48 am, a...@roiban.ro wrote:


Is there a reason why a try scheduler is not implemented? I find it 
awesome

for TDD!

On my normal workflow I can do this, even without commiting changes:

./HELPER test some.test
./HELPER test-remote sol-10-sparc some.test

test-remote just wraps buildot try --builder --properties --wait --vc


Because it requires setting up proper credentials to access control the 
feature.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Permissions to trigger buildbot tests

[exarkun]

On 12:10 pm, a...@roiban.ro wrote:

On 17 November 2014 11:55, exar...@twistedmatrix.com wrote:

On 11:48 am, a...@roiban.ro wrote:


Is there a reason why a try scheduler is not implemented? I find it
awesome
for TDD!

On my normal workflow I can do this, even without commiting changes:

./HELPER test some.test
./HELPER test-remote sol-10-sparc some.test

test-remote just wraps buildot try --builder --properties --wait --vc


Because it requires setting up proper credentials to access control 
the

feature.



But there are already credentials for the HTTP status page... why not
configure the same credentials for buildbot try?


The credentials for the HTTP status page are there to prevent annoying 
spurious builds.  The credentials for `buildbot try` need to be serious, 
secure credentials because they prevent attackers from submitting 
arbitrary code to execute on the buildslaves.


Basically, it's more work.  If someone wanted to set it up right, I 
don't know of any reason we wouldn't want to allow it.  Though that may 
just be because we haven't ever used it so I haven't yet learned the 
reasons it's undesirable to have enabled. ;)


Jean-Paul

What I would like to hear is something like this:

* buildbot try is too simple, does not work with our workflow and for 
our

project... we don't want to spend time implementing it

or

* buildbot try could help, it still has some problems and  we don't 
have

time to fix issue 1, 2, 3, ... etc


--
Adi Roiban


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Permissions to trigger buildbot tests

[exarkun]

On 09:44 am, a...@roiban.ro wrote:

On 16 November 2014 22:44, exar...@twistedmatrix.com wrote:
Is there a wiki page describing what are the steps required for a 
regular
contributor to get permissions to run buildbot tests and get write 
access

to the repo?


I'll send you the credentials off-list.  FWIW, this is password 
protected
only because spambots found the form and kept triggering garbage 
builds.


There's no standard policy or procedure for granting commit access to 
the
repository.  Usually it goes like someone asks for it, someone gives 
it to

them.



I got them.

Is there a wiki page describing how buildbot should be used?


I just created 
https://twistedmatrix.com/trac/wiki/ContinuousIntegration/DeveloperWorkflow. 
Feel free to edit it if you think parts are unclear.  Or point out any 
questions you think it doesn't answer.  I haven't linked to it from 
anywhere yet.  Suggestions about where a good place (such that people 
will actually find it) welcome.

I found some notes hidden in the review process page... it looks like
builds are triggered from the web page with click.


I guess maybe this would be an okay place?
The review page also talks about a misterious force-builds.py script 
which
is not part of the main twisted repo.. or at least, I could not found 
it


It's part of one of the twisted dev tools repositories (used to be 
twisted-dev-tools but I think someone moved it, not sure exactly to 
where).  Perhaps you could update this page to mention where - if you 
find it.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Contributor advancement path - proposal (RFC)

[exarkun]

On 11:48 am, exar...@twistedmatrix.com wrote:


This sounds great. Could you jot it down on a wiki page?


Hmmm okay. :P


I wrote up 
https://twistedmatrix.com/trac/wiki/Proposal/ContributorAdvancementPath. 
Notice Proposal in the URL.  It would be nice to get feedback from 
some more contributors - particularly people who have recently gotten 
commit access (about whether the problems are real and what they think 
of the proposed solutions) but also from other long-time Twisted 
contributors about whether they like this new idea about how to bring 
more people in (and whether they would volunteer to do the necessary 
candidate reviews - without which effort this isn't practical).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I debug this network problem?

[exarkun]

On 02:51 pm, peter.westl...@pobox.com wrote:


Looking at the code, _SignalReactorMixin.mainLoop() takes the output of
ReactorBase.timeout() and passes it to doIteration(). The 
implementation

of doIteration() in PollReactor passes the value straight through to
poll(). If there is any kind of timed call pending (which there is), 
the

timeout value counts down. Eventually it gets very close to zero. If
there is a burst of AMP activity around that time, there's a good 
chance

that the one of the poll() calls will time out.

How this leads to the data being lost is another question, but the
coincidence of events is too much to ignore.


This is entirely normal.  It's how timed calls are implemented in almost 
all of the reactors and it's how they've been implemented for at least 
13 years. :)


It's possible there's a bug here but this sequence of poll() calls with 
these timeout values doesn't surprise me at all.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I debug this network problem?

[exarkun]

On 12:42 am, peter.westl...@pobox.com wrote:


On Mon, 17 Nov 2014, at 20:25, Glyph wrote:



On Nov 17, 2014, at 19:32, Peter Westlake
peter.westl...@pobox.com wrote:

It's a known bug in Python!http://bugs.python.org/issue1038591,
Python 2.3+ socket._fileobject handles EAGAIN with data loss.


This bug is in socket.makefile(), which Twisted never uses - not
socket.socket().


Okay, how about http://bugs.python.org/issue9090, Issue #9090 : Error
code 10035 calling socket.recv() on a socket with a timeout
(WSAEWOULDBLOCK - A non-blocking socket operation could not be 
completed

immediately)?


Twisted also doesn't use sockets with timeouts - although the way socket 
timeouts are implemented in Python, I could perhaps imagine some timeout 
configuration leaking into Twisted's sockets due to some other code 
running in the same process.


Can you check to see if your AMP connection's underlying socket has a 
timeout set?  It *shouldn't*.  Twisted doesn't set one.  If some other 
code is causing a timeout to be set, I guess this could be the culprit.


Jean-Paul

I actually found that one first, but it appeared to be Windows-only.
Further down the comments it becomes clear that it's not, and the patch
at https://hg.python.org/cpython/rev/8ec39bfd1f01 confirms that.

It would be good news if it was 9090, because it's easier to switch to
Python 2.7.8 than 3.

Peter.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Permissions to trigger buildbot tests

[exarkun]

On 10:04 am, a...@roiban.ro wrote:

Hi,

To help with the review process I would like to ask permissions for
triggering buildbot builds.

Is there a wiki page describing what are the steps required for a 
regular
contributor to get permissions to run buildbot tests and get write 
access

to the repo?


I'll send you the credentials off-list.  FWIW, this is password 
protected only because spambots found the form and kept triggering 
garbage builds.


There's no standard policy or procedure for granting commit access to 
the repository.  Usually it goes like someone asks for it, someone 
gives it to them.




In the same time, maybe we can write a few words about the steps 
required

for a new contributor to become a full reviewer. Ex.


There is no official process but the frequently discussed unofficialy 
process is just getting commit access.  The project doesn't distinguish 
between committers in any way as far as the development workflow is 
concerned.


Perhaps it should and we should discuss how it could do this.

1. Contribute a few patches (ex 10) and learn the basic review process.
  Observe how reviewers respond to your patch.

2. Start doing review as junior reviewer, without merging. Once you are 
ok

with the patch, invite another core developer to take a final view and
merge the patch

3. Once you have reviewed a few patches without errors (ex 10) you can 
ask
for full review permission or a core developer will let you know that 
you

can merge the patch without asking someone else.

This can be part of the current review process page:
https://twistedmatrix.com/trac/wiki/ReviewProcess

What do you think?


I think this process probably involves little enough learning that it 
won't make a significant difference to the quality of code reviews done 
for the project (so it will only add overhead to the process of keeping 
track of different kinds of reviewers and where in their progression 
junior reviewers are).


A modification that would help very slightly (but I think still not 
enough to be worthwhile, particularly since it adds even more overhead) 
would be to require a correct review covering each of the many relevant 
areas - for example, howto-style docs, example-style docs, api-style 
docs, unit test coverage, coding convention compliance (whitespace, 
variable naming, etc), etc.  After demonstrating competence in all areas 
the junior reviewer could advance to normal review status.


Unfortunately notice I didn't say anything about the software being 
built or changed itself.  I don't know of an easy way to test folks for 
competence at that kind of thing except to see them write a lot of code.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do I debug this network problem?

[exarkun]

On 05:38 pm, peter.westl...@pobox.com wrote:


Here is some highly suspicious-looking output from strace:

sendto(5, \0\4_ask\0\004867c\0\10_command\0\tLogOutpu..., 343, 0,
NULL, 0) = 343 recvfrom(5,
\0\7_answer\0\004867c\0\6status\0\4True\0\0, 65536, 0, NULL, NULL) =
31 sendto(5, \0\4_ask\0\004867d\0\10_command\0\tLogOutpu..., 183, 0,
NULL, 0) = 183 sendto(5, \0\4_ask\0\004867e\0\10_command\0\4Ping\0\0,
30, 0, NULL, 0) = 30 recvfrom(5, 0x7fec3005c6d4, 65536, 0, 0, 0) = -1
EAGAIN (Resource temporarily unavailable) sendto(5,
\0\4_ask\0\004867f\0\10_command\0\4Ping\0\0, 30, 0, NULL, 0) = 30
recvfrom(5, \0\7_answer\0\004867f\0\3ack\0\4True\0\0, 65536, 0, NULL,
NULL) = 28

Requests 867d and 867e never receive their callbacks.


But in this trace, it looks like those two calls also never received a 
response from the server - so it's correct that the callbacks never 
fire.


Keep in mind this is a totally valid AMP interaction.  The reason these 
things have unique identifiers is so responses can arrive out of order.


Earlier you mentioned you saw tcpdump reporting that the response data 
corresponding to the unfired Deferreds was arriving.  Can you double 
check that?  From this `strace` output I would guess that it isn't.  The 
alternative explanation is a terrible, data-losing bug in the kernel's 
TCP stack...

I see that twisted.internet.tcp.Connection.doRead() does check for
EWOULDBLOCK (=EAGAIN), and returns None. Shouldn't it try to read 
again?


It will try again - later, when select or epoll or whatever tells it 
that there's a better chance of some data coming out.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending longer messages in AMP

[exarkun]

On 02:57 pm, ga...@gromper.net wrote:

Hi,

We're using AMP and are starting to hit TooLong errors when scaling
our application. In one respect it's a sign that we should do
something like paging large requests and responses, but that's a lot
more work, and comes with its own problems. We also don't need
particularly large payloads: right now, a limit of ~500kiB would allow
us to scale as far as we need and beyond.

I've put together a fork of Twisted's AMP implementation that uses
32-bit length prefixes everywhere, though it limits the maximum
message size to 2MiB. Every other aspect of it is the same so it's a
drop-in replacement, as long as both ends of a connection use it.
However, there's no negotiation phase so it's completely incompatible
on the wire. The overhead of a few extra bytes is negligible for our
use cases, where the networks are all assumed to be low-latency
high-bandwidth LANs.

Are there any reasons that we shouldn't be doing this? Was there a
good reason for 16-bit length prefixes that still holds? Should we be
doing something else?


The short length limit is in place to encourage two things:

 * messages that can be processed in a cooperative-multitasking-friendly 
way


 * the AMP channel can reliably used to multiplex multiple operations

The limit encourages the former by limiting the total amount of data 
it's possible to receive in a single command.  Of course, you can still 
do ridiculously complicated work based on a small bit of data so this 
doesn't guarantee that no matter what you do you'll be safe.  But doing 
even something simple on a ridiculously large amount of data is probably 
guaranteed to take a while.


The limit encourages the latter by putting a limit on the data that 
needs to be transferred to complete any one command (or answer).  Again, 
this isn't a guarantee of safety (you could always have a `for i in 
range(1e10): callRemote(...)` loop and clog up the channel for ages) but 
it pushes things a bit more in that direction.


At ClusterHQ we *also* maintained a fork of AMP with this limit raised. 
Basically, it worked.  It did let us get into the kind of trouble that 
the limit was supposed to try to avoid (in particular it let us send 
around messages that would take longer and longer to be processed - in a 
system where keeping latency down was actually sort of important; 
fortunately we had *worse* problems introducing latency so this in 
particular never bit us too hard ;).

If I assume that the answers are all no, would someone find this
protocol useful if we submitted it for inclusion in Twisted itself?


There are better solutions to the problem.  The trouble is that they're 
also more work to implement. ;)  I think Twisted should hold out for the 
better solutions though, not adopt a like-AMP-but-with-different-hard- 
coded-limits solution.


What are the better solutions?  Library support for paging, basically. 
Or, to consider things more generally, library support for streaming. 
The AMP implementation in Twisted (note, not the *protocol*) should be 
extended to make it easy to pass arbitrarily large streams of data 
around - suitably broken into smaller pieces at the box level.


As of right now, the way I'd do that is by introducing a new argument 
type (or two) supporting `IProducer` and `IConsumer`.  Pass in an 
`IProducer` and the library will take the necessary steps to read data 
out of it, chunk it up into =16kB chunks, and re-assemble them on the 
receiving side (as another `IProducer`).


There are two reasons I'm not working on this right now (apart from the 
standard reasons of not having time to do so ;):


 1) IProducer / IConsumer aren't amenable to this kind of decoupling. 
You can register a producer with a consumer but you can't register a 
consumer with a producer.  By the time you give the IProducer to AMP, 
it's too late to tell it you want it to send its data into the AMP 
implementation for the necessary handling.  We worked around this in 
twisted.web.client.Agent by introducing a new IProducer-like interface. 
It solves the basic problem but it doesn't go any further to improve the 
usability of the interfaces.


 2) Tubes.  Glyph is working on a replacement for IProducer/IConsumer 
that does go a lot further to improve usability.  With this promise of a 
bright, prosperous future looming, it's hard to get excited about 
implementing for AMP a just-barely-good-enough solution like the one 
used by Agent (in particular, with the knowledge that the tubes solution 
will be API incompatible and we'll most likely want to deprecate the 
IProducer/IConsumer thing).


Jean-Paul

The code right now is a straight copy of amp.py and test_amp.py with
changes to 32-bit length prefixes everywhere, but for upstreaming we'd
probably propose instead to modify the original to have an optional
negotiation phase, and to make the maximum message size a parameter.

Thanks!

Gavin.

___

Re: [Twisted-Python] The Path to Twisted 14.1

[exarkun]

On 5 Nov, 11:57 pm, ita...@itamarst.org wrote:

On 11/04/2014 09:54 PM, Glyph wrote:



I don't follow.  Are you saying that logger fixed some python 3 stuff, 
and by reverting it we're losing that, or that log regressed after 
14.0 and logger fixed it and that by reverting it we are regressing 
with respect to its usability in 14.0?
Logger broke installation on Python 3; Twisted trunk is unusable on 
Python 3 because of logger. It's a breakage that can be easily fixed 
(one of the modules isn't on the list of things to be installed) but 
it's incorrect to say that logger didn't break existing code.


This sub-thread seems to be based on a misreading/misunderstanding of an 
earlier statement.  This one, I think:
In favor again of reverting is the fact that no code outside 
twisted.python.logger or twisted.python.log has been modified to take 
advantage of the new system, so we're not going to be breaking any 
dependencies on trunk.


I don't think it has been suggested that logger didn't break anything 
(in fact, clearly it broke many things - that's why it's being reverted 
;).


Instead, Glyph was just commenting that removing logger doesn't have 
particularly far-reaching consequences throughout Twisted because none 
of Twisted has been updated to use the new API instead of the old API.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Serializing cftp and ssh sessions

[exarkun]

On 06:02 am, axel@chaos1.de wrote:
I have adapted t.c.s.cftp and one of the ssh examples to transfer files 
to a remote host and than execute some commands there.

Both has to be done for several remote hosts in sequence.
Start of the ssh session has to wait until the cftp sessions are 
complete and after that has completed, the cftp session for the next 
host has to be started . . .


The cftp part begins so:


We might need more to go on.  Do you have a complete, minimal example? 
(See http://sscce.org/)


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Boston Area Twisted Sprint

[exarkun]

Hello all,

I'm happy to invite you to a Twisted sprint on Saturday, November 1st
in Cambridge, Massachusetts.  ClusterHQ will be hosting up to twenty
(20) Twisted sprinters at the Cambridge Innovation Center (Coronet
City room) in Kendall Square from 10AM until 11PM.  The sprint topics
are attendee choice - hacking on any part of Twisted is welcome.  A
large part of the ClusterHQ team will be taking part in the sprint.

ClusterHQ is developing Flocker, a data management tool for use with
Docker written in Python and based on Twisted.  Follow us on Github:

   https://github.com/clusterhq/flocker

ClusterHQ will provide lunch and dinner (something like delivery
pizza, comment on the meetup page if you'd like to suggest
something!).

Find details and chat about sprint topics at the Boston Twisted Python
meetup page:

   http://www.meetup.com/Boston-Twisted-Python/

Early birds, feel free to contact me off-list about meeting up for 
coffee at

around 9AM before the sprint starts.

Space at CIC is limited.  Please RSVP on the Meetup page!

See you there,
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] porting twisted.spread to Python3

[exarkun]

On 3 Oct, 11:09 pm, wolfgang@rohdewald.de wrote:

I now have a local git branch with about 70 commits, always
rebased onto current trunk.


It makes me sad to learn you're carrying so many patches to Twisted.  It 
might be useful to the project if you could share why this development 
was easier to do outside the tree rather than contributing it to the 
project as you developed it.

Those test suites pass with Python2 and Python3:

test_banana, test_jelly, test_pb, test_pbfailure, test_persisted

I also have a real app using PB (kajongg, a KDE game). Running
the server and the client with any combination of Python 2 and
Python 3 works but more testing is needed there. Especially
transferring failures with non-ascii test, so I need to inject
random errors.


For testing?  Probably not random - instead, comprehensive.

spread.banana is now fully covered by tests (was at 85% before).
There is still a lot of code in twisted.spread which has no
tests yet.
I do not think I want to add missing tests for everything,
only for those regions where I do changes or where it is not
too difficult. And of course if an application triggers a bug
yet uncovered by tests.


The coding standard only requires you to add tests for code you're 
changing.  Python 3 mixes things up a bit though - by changing 
everything.  Calling a module Python 3 compatible without changing it 
at all still changes lots of things how about it works.


I think that before the Twisted project wants to call a particular 
module ported, we want it to have test coverage that can run on Python 
3.

Can you point me to other PB applications I could use for testing?
Preferrably ones that are already mostly compatible with PY3
themselves,


I'll let other people volunteer projects they know of - but I strongly 
suspect there are few or no such applications.

but that is not a _must_, I would try porting them
unless it turns out to be too much work. And preferrably ones
developed by people with a non-ascii native language. I do not
yet trust Failures and logging with non-ascii.


Below is the current list of commits. Everything you see
there is still much in flux. This includes shuffling code
between commits, combining and splitting commits, changing order.
I want to open one ticket per commit. Simple things hopefully
not needing much discussion first.

Estimating one week per review/commit to trunk, this process
will take more than one year. Often I can only add a new ticket
after the previous one has been integrated because they
depend on each other.


If twisted.spread were completely supported on Python 3 in only another 
year I'll be very impressed. :)

So if it really seems to take that long, at some point I might
just decide to put what I have to github and not care anymore.
Which of course means I might have to distribute a patched
twisted version with my application. I really want to leave
Python2 behind, and only twisted is in the way. Latest versions
of KDE do not have Python2 bindings anymore (KDE Frameworks 5).


That's too bad.

This log of course includes things being in tickets up for review.

git log --pretty='format:%s' --reverse trunk..pb3
snip


Again, it seems unfortunate that you have all of this work ... 
somewhere.  Somewhere only you  (as far as I can tell) can see it. 
Somewhere only you can test it.  Somewhere only you can work on it and 
get it contributed to Twisted.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] my branches merged by you (exarkun)

[exarkun]

On 12:00 pm, wolfgang@rohdewald.de wrote:

That should have been exarkun, not itamar. Sorry for that.

Am Sonntag, 5. Oktober 2014, 11:59:40 schrieb Wolfgang Rohdewald:

Thanks for fixing my mistakes!

Based on your fixes I started a personal checklist, see below.
Is that list correct so far? (it will not include things I already
did right)

I noticed that you also changed 'x' to b'x' in some places. I did
not do that yet because it is not needed for those tickets, I wanted
to keep them small and focussed.


I think I only did this where new uses of 'x' was being introduced to 
represent a bytes object.  My goal is for new code being added to be as 
correct as possible so that it doesn't increase the burden of making 
these fixes later.  The time when the code is being first introduced is 
the time when it's easiest to make sure these things are correct.

Instead I am planning tickets for just that: (after minimizing the
number of places to be fixed)
twisted.spread.banana and tests: literal strings become bytes where 
appropriate
twisted.spread.jelly and tests: define *_atom consistently for all 
atoms, change them to bytes
twisted.spread.pb and tests: literal strings become bytes where 
appropriate


The current checklist:

Line length max. 79
*.rst doc: One line per sentence, no length limit
fully qualified names in doc and news, twisted.*
no __ in Twisted. Everything in a test case is private anyway.
if possible __execute__ (not __define__) only one assert per test


Hmm.  Yes?  I'm not sure.  What do you have in mind here?

document changed API like raising more or different exceptions
use new print() formatting syntax
Document and isolate test code using the private API of the module to 
be tested.


I'm also unsure about this one.

test everything with PY2.6 too
did twistedchecker add new warnings?
news: most frequent form: Qualified now does ...
news about spread belong into twisted/topfiles
doc: not pb but PB
Epydoc
docstring ends with point
break docstring lines not sooner than needed
avoid the word “test” in test docstrings
do not use the “I am” form in docstrings
test docstring: describe what happens, including exceptions
doc: avoid mentioning the private API


Apart from those two items, the rest of these look great.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] porting twisted.spread to Python3

[exarkun]

On 04:23 pm, wolfgang@rohdewald.de wrote:
Am Sonntag, 5. Oktober 2014, 13:48:12 schrieb 
exar...@twistedmatrix.com:

On 3 Oct, 11:09 pm, wolfgang@rohdewald.de wrote:
I now have a local git branch with about 70 commits, always
rebased onto current trunk.

It makes me sad to learn you're carrying so many patches to Twisted. 
It

might be useful to the project if you could share why this development
was easier to do outside the tree rather than contributing it to the
project as you developed it.


This is just my personal style of development. A bit chaotic. Sometimes
trial and error. And really a lot of git rebase -i. Whenever I find
some necessary or helpful change that could be done before porting,
put that at the top of the patch list and readjust everything.
The methodic part comes last: Look at what I changed, rethink why that
is really needed, and look for similar places I overlooked. Then 
generate

tickets for things that seem ready.

BTW what about Ticket 7628, news extension port? I could soon start
feeding porting tickets but if this extension is useful, I guess it
should be applied first.


Looks like Kevin is working on it.  I'll review it when it comes up.



I think that before the Twisted project wants to call a particular
module ported, we want it to have test coverage that can run on Python
3.


I was afraid you'd say that.
https://twistedmatrix.com/trac/wiki/Plan/Python3 does not explicitly
say so, maybe this should be changed. I can do that, but then I
have no write permission for the wiki.


Ah.  I didn't think anyone was paying any attention to the plan anymore 
(and I've run out of energy to try to get people to stick to it).


For example, following the plan, I would have expected to see perhaps 
two patches to port the banana module: one to add the missing test 
coverage and one to make any and all changes necessary to get the test 
suite to pass on Python 3.


(Note, I'm just pointing this out for the sake of clarity.  I'm not 
complaining about the tickets/patches you recently contributed.  As I 
said, I'm out of energy for that).


As far as wiki access goes, I'm not clear on how those permissions work 
anymore.  Perhaps someone else can chime in.  If not, consider asking on 
#twisted-dev.



I'll let other people volunteer projects they know of - but I strongly
suspect there are few or no such applications.


That makes me wonder if I really should have used PB at all. But
now I do and have no plans to change that.


Sorry about this.  For anyone who's reading and thinking about starting 
a project, I personally recommend you not use PB.  It's almost certainly 
not the right tool to solve the problem you have (if you think your 
problem is so unique and challenging it needs a tool like PB, maybe it 
is!  Feel free to start a discussion about the specifics on this list. 
Maybe you've got one of the rare problems where PB makes sense.)



Again, it seems unfortunate that you have all of this work ...
somewhere.  Somewhere only you  (as far as I can tell) can see it.
Somewhere only you can test it.  Somewhere only you can work on it and
get it contributed to Twisted.


I believe I saw some mailing list posts where many years ago somebody
said he has spread running with PY3 but as it seems nothing of that
got into trunk. Be assured that I want to avoid that.

What is the preferred place for Twisted public repositories?
No svn please, only git.


Anywhere public.  Collaboration can't take place when things are 
private.


But first I want to do some more cleaning and reshuffling, I cannot
really do that anymore with commits already pushed to a public
repository. Maybe 2 or 3 weeks.


Pushing something isn't a promise not to abandoning, screw it up, rebase 
it, delete it later, whatever.


I think that the notion that before something is shared it has to be 
carefully arranged and re-arranged to make it look like it was developed 
by an all-knowing, all-programming super being.


Real software doesn't get built one perfect, self-contained step at a 
time.  The construction of real software is messy.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] my branches merged by you (exarkun)

[exarkun]

On 04:26 pm, wolfgang@rohdewald.de wrote:
Am Sonntag, 5. Oktober 2014, 14:20:35 schrieb 
exar...@twistedmatrix.com:

I noticed that you also changed 'x' to b'x' in some places. I did
not do that yet because it is not needed for those tickets, I wanted
to keep them small and focussed.

I think I only did this where new uses of 'x' was being introduced to
represent a bytes object.


Then I would not have mentioned this. Three such changes in svn r43220.
b'remote' and two banana encoded strings. I knowingly left it at 
'remote'

because I wanted to change that everywhere together.


Hm.  You're right on two counts.  The lines:

   undecodable = b'\x7f' * 65 + b'\x85'
and

   decodable = b'\x01\x81'

were not related to the change.  I'm not sure why I decided to make 
those changes.  If I were doing it again now, I wouldn't.


But I'll stick by the decision to write `b'remote'` instead of 
`'remote'` on the line:


   vocab = b'remote'

My goal is for new code being added to be as
correct as possible so that it doesn't increase the burden of making
these fixes later.  The time when the code is being first introduced 
is

the time when it's easiest to make sure these things are correct.


I fully agree. Excluding cases where you just re-use an existing string
in one more place because then you have both 'remote' and b'remote'
in the source until you change them all (for that you have to grep
everything anyway). On the other hand, not adapting existing code
to changed guidelines right when they are changed always generates
a lot of such inconsistencies, so this one is really peanuts.
(I know - more manpower is always needed). This is actually a small
problem for me because I rather copy the style of existing code
instead of reading boring guidelines.


I'm glad we're mostly in agreement and I'll take that as a win. :)



if possible __execute__ (not __define__) only one assert per test

Hmm.  Yes?  I'm not sure.  What do you have in mind here?


svn r43207 Split up the test with multiple assertions into several 
tests


There you unfolded the loop. If the real reason was the different
constructions of the type names (like __builtin__.type), I would
have expected a different commit text, and of course you could have
left the loop, like

for obj, name in ((type, '__builtin__.typ'), ...)

So I assumed that you really want as few assertions executed in one
test as possible. I cannot think of any other reason than those two.


Ah.  I understand now, and you divined my intent correctly.

BTW your commit epytext fixes most but not all. There are still a
few ``xxx`` left. What markup language did you have in mind there?


sigh... reStructuredText... which Twisted uses for the other half of its 
documentation.
Document and isolate test code using the private API of the module 
to

be tested.

I'm also unsure about this one.


Cases like svn r43221

Hoist the use of this private API to a single location in a helper 
function.


which explains why we need to use the private API.


Ah.  Great.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] porting twisted.spread to Python3

[exarkun]

On 08:28 pm, wolfgang@rohdewald.de wrote:
Am Sonntag, 5. Oktober 2014, 19:58:29 schrieb 
exar...@twistedmatrix.com:

For example, following the plan, I would have expected to see perhaps
two patches to port the banana module: one to add the missing test
coverage and one to make any and all changes necessary to get the test
suite to pass on Python 3.


Impossible. test_jelly imports pb, test_pb imports jelly.


That doesn't mean it's impossible.  It just means you may also have to 
shuffle some files around.

But I can of course reduce the number of tickets related to porting and
put more than one patchfile into them.


Note that my point isn't that we need to reduce the number of tickets 
related to porting or that there should be more than one patch to 
resolve any one ticket (there shouldn't be).


The point is that the documented porting plan is to port module by 
module, not Python 3 incompatible change by Python 3 incompatible 
change.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python 3.3 buildslaves

[exarkun]

On 08:26 am, a...@roiban.ro wrote:


I think that it would be nice to have some process to initiate new
contributors to the whole review process and once one or more
senior-reviewers consider that the new contributors is good to do
reviews by itself, to give him commit privileges.


+1

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] natty64-py2.7maint twistedchecker builders

[exarkun]

Hello,

These two builders appear to be having problems.  See, for example:

https://buildbot.twistedmatrix.com/builders/natty64-py2.7maint/builds/4193
 https://buildbot.twistedmatrix.com/builders/twistedchecker/builds/2229

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Python 3.3 buildslaves

[exarkun]

Hello,

I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker
for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot
configuration/availability issues.

Who can look at why one of the Python 3.3 slaves is offline and why
the two slaves are configured differently?

Thanks,
Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] PEP3131: non-ascii identifiers

[exarkun]

On 09:19 am, wolfgang@rohdewald.de wrote:

This does not seem to be supported by Python yet.

Should that be enabled at all?
If one process with PY3 sends such identifiers to
a separate process with PY2, that will fail. I am not
sure if that would be a problem, whoever uses this must
make sure PY3 is used everywhere.


This is why we will *not* change the PB wire protocol as part of the 
porting work.  The wire protocol will remain the same whether you are 
using Python 2 or Python 3 to run your program or.


This is the point of a protocol, after all.  It is to let two programs 
communicate with each other.

If this should be forbidden, I will add a test to
test_pb for this. And of course somebody should document that
somewhere. There more PEP3131 is used, the more users will
fall into this trap.


I'm not exactly sure what you mean here.  Using unicode where only bytes 
are allowed is probably already forbidden throughout PB.


If this should be enabled (which I think is not difficult,
at least for pb):

At least the patch below will be needed (only for PY3),
maybe it is already sufficient. Given that nativeString
and networkString are always used (done that for pb).

networkString may then return bytes with the high bit set


Definitely not.

But since networkString is called in many places I want to ask and
make sure that it may really be changed this way.


https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html
does not speak against it, so I wonder why networkString has that 
limitation

to 7bit.


That is the sole purpose of `networkString`.  It is a work-around for 
the inconvenient fact that Python changed the meaning of the string 
literal syntax from bytes to unicode.


concrete banana-encoded example, from modified test_pb: (the method 
name is getSimpleä)

test_pb still passes with patched nativeString/networkString (but I
only have one test for this so far, test_refcount).

b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary'


diff --git twisted/python/compat.py twisted/python/compat.py
index 6f76c39..6919cf6 100644
--- twisted/python/compat.py
+++ twisted/python/compat.py
@@ -348,10 +348,9 @@ def nativeString(s):
raise TypeError(%r is neither bytes nor unicode % s)
if _PY3:
if isinstance(s, bytes):
-return s.decode(ascii)
+return s.decode(utf-8)
else:
-# Ensure we're limited to ASCII subset:
-s.encode(ascii)
+return s
else:
if isinstance(s, unicode):
return s.encode(ascii)
@@ -428,7 +427,7 @@ if _PY3:
def networkString(s):
if not isinstance(s, unicode):
raise TypeError(Can only convert text to bytes on Python 
3, I got %r % (s,))

-return s.encode('ascii')
+return s.encode('utf-8')

def networkChar(integer):



This change definitely won't be acceptable.  It completely removes the 
feature `networkString` exists to provide: verifying that strings that 
might be either unicode or bytes can still be implicitly combined into 
bytes.


Can you point out the specific places where you think PB needs to start 
using UTF-8 instead of ASCII?  Those are the places that need to be 
fixed, not the underlying porting helpers they happen to use.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] PEP3131: non-ascii identifiers

[exarkun]

On 11:44 am, wolfgang@rohdewald.de wrote:
Am Montag, 8. September 2014, 10:54:44 schrieb 
exar...@twistedmatrix.com:
Can you point out the specific places where you think PB needs to 
start

using UTF-8 instead of ASCII?  Those are the places that need to be
fixed, not the underlying porting helpers they happen to use.


My question is whether PEP3131 should be supported or not. After all,
it is an integral part of Python3. I have no opinion about this,
I just want to point it out as a potential problem.

So you say it should not be supported, and I have no problem with that.

Just to be clear about it:

In Python3 you can legally write

def ÜnicödeMethod€Name(self):

or see
https://mail.python.org/pipermail/python-3000/2007-June/008172.html
for a nice Russian example.

but PB cannot transfer those identifiers. Which is not what the
user would expect.


PB supports unicode perfectly well and has for many years.  This is why 
I asked which specific part of PB has a problem.  I don't see why you 
think unicode is a problem here.  This doesn't mean there isn't a 
problem - after all, you've been looking at the code and trying to make 
it work on Python 3 and I haven't - but you haven't made it clear what 
that problem is.


A failing unit test or a minimal example (http://sscce.org/) would 
communicate this most clearly, but perhaps you can just mention a 
specific API and give an incomplete example of how it will fail when it 
runs up against the changes defined by PEP 3131.


Jean-Paul


This should be clearly communicated somewhere in the docs, IMHO.

--
Wolfgang


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] PEP3131: non-ascii identifiers

[exarkun]

On 01:07 pm, wolfgang@rohdewald.de wrote:
Am Montag, 8. September 2014, 12:04:46 schrieb 
exar...@twistedmatrix.com:
PB supports unicode perfectly well and has for many years.  This is 
why

I asked which specific part of PB has a problem.


PB transfers names of methods and classes as bytes, not as unicode.


PB actually transfers *everything* as bytes. ;)  Bytes are the only 
thing you can send over a socket.  If you have anything else - integers, 
unicode, whatever - you have to encode them as bytes first.


As far as I know, the PB protocol is not specified apart from the 
implementation in Twisted (and this being the only implementation (I am 
intentionally disregarding TwistedJava) it must serve as the 
specification, I think).


What you meant above, I think, is that PB represents method names as 
bytes at the banana layer.  That is, when you want to call a remote 
method, you indicate its name by supplying bytes to the banana encoding 
layer - not unicode (which is good because banana doesn't actually 
support unicode at all, that's a jelly feature).


This does indeed mean we don't simply want to start sending unicode to 
refer to methods by name - because we can't!  At least, not unless we 
extend banana to support a new type which we probably don't want to do - 
that would be another incompatible protocol change and so not allowed 
(since it could break interoperability between different implementations 
of PB).


So, it is necessary to continue to represent method names using bytes.

This is fairly easily done. On Python 3, encode any unicode strings 
which represent method names (using a well-known encoding, probably 
UTF-8) when making the call and decode them in the same way when 
dispatching those calls.


This can almost be done at the application level:

   # Some Python 3 code
   def remote_ä(self):
   pass

   ref.callRemote(uä.encode(utf-8))

except that Python 3 has actually changed to enforce the type of the 
second argument of getattr - if it is not a unicode string then a 
TypeError is raised - so it's not possible to make the decoding step 
happen (which one might otherwise have done using `__getattr__` or by 
adding the encoded name of the method to the class dictionary).


So if it is going to be supported in Twisted's PB API then that support 
probably needs to be in Twisted's PB implementation.  The same general 
idea applies, though.  Just move the encoding into the implementation of 
`callRemote`:


   def callRemote(self, _name, ...):
   ... _name.encode(utf-8) ...

And add a corresponding decode to the other side (probably in 
`_recvMessage`).


This would make the Python 3 PB API be method names are unicode 
strings which makes sense considering the decisions that were made for 
Python 3.  Note that it does not change the wire protocol - method names 
are still bytes at the banana level.  Or does it?  These bytes were 
previously always an ASCII subset.  Is expanding out of the ASCII range 
an incompatible change?


What could this break?

Let's say you have Python 2 talking to Python 2.  It's already possible 
to construct a method call like this:


   class Foo(Referenceable):
   def remote_foo(self):
   pass
   setattr(Foo, uremote_fooä.encode(utf-8), Foo.remote_foo)

   ...

   ref.callRemote(uremote_fooä.encode(utf-8))

This actually works.  Python 2 doesn't much care about how you name your 
class attributes.  PB doesn't care that the high bit is set in one or 
more of the bytes in the method name.  It all just works.


So let's say you have Python 2 talking to Python 3 instead.  In Python 
3, you can't do that setattr() call (the language and runtime disallow 
it).  But you can have `def remote_fooä(self)` instead.  If PB on Python 
3 decodes the method name before dispatching it (using UTF-8) then again 
things work.


And if you reverse the situation and PB on Python 3 encodes the method 
name before sending it, then Python 2 is still happen because it can 
operate on that UTF-8 encoded byte string.


Finally, if Python 3 talks to Python 3 then it also works because the 
sending side encodes and the receiving side decodes.


So we get to make a judgement call here, I think.  Without a 
specification there's no objectively correct answer.  So, because the 
current implementation is actually perfectly compatible with non-ASCII 
bytes - even though the intent is clearly that you would never have 
those - combined with the point that I made above, that there are no 
other PB implementations, I suspect it's fine to expand beyond ASCII 
here because it won't actually break any real world programs.


The only case that I can think of that actually would be a problem is 
the case where someone is already sending non-ASCII, non-UTF-8 method 
names around.  These might decode wrong or might fail to decode at all. 
I don't think this is likely enough to worry about - but maybe someone 
who is doing this will speak up and prove me wrong. ;)


And to 

Re: [Twisted-Python] Python3: should paths be bytes or str?

[exarkun]

On 01:26 am, wolfgang@rohdewald.de wrote:

The porting guide says

No byte paths in sys.path.


What porting guide is that?


doc for FilePath says
   On both Python 2 and Python 3, paths can only be bytes.


I stumbled upon this while trying to find out how much work it might be
to make bin/trial run with python3

admin/run-python3-tests already passes for all twisted.spread related
tests but I still need to clean up a lot.

after adding an assert to FilePath.__init__, python3 bin/trial ... 
gives


 File /home/wr/ssdsrc/Twisted/twisted/scripts/trial.py, line 601, in 
run

   config.parseOptions()
 File /home/wr/ssdsrc/Twisted/twisted/python/usage.py, line 277, in 
parseOptions

   self.postOptions()
 File /home/wr/ssdsrc/Twisted/twisted/scripts/trial.py, line 472, in 
postOptions

   _BasicOptions.postOptions(self)
 File /home/wr/ssdsrc/Twisted/twisted/scripts/trial.py, line 382, in 
postOptions

   self['reporter'] = self._loadReporterByName(self['reporter'])
 File /home/wr/ssdsrc/Twisted/twisted/scripts/trial.py, line 369, in 
_loadReporterByName

   for p in plugin.getPlugins(itrial.IReporter):
 File /home/wr/ssdsrc/Twisted/twisted/plugin.py, line 209, in 
getPlugins

   allDropins = getCache(package)
 File /home/wr/ssdsrc/Twisted/twisted/plugin.py, line 134, in 
getCache

   mod = getModule(module.__name__)
 File /home/wr/ssdsrc/Twisted/twisted/python/modules.py, line 781, in 
getModule

   return theSystemPath[moduleName]
 File /home/wr/ssdsrc/Twisted/twisted/python/modules.py, line 702, in 
__getitem__

   self._findEntryPathString(moduleObject)),
 File /home/wr/ssdsrc/Twisted/twisted/python/modules.py, line 627, in 
_findEntryPathString

   if _isPackagePath(FilePath(topPackageObj.__file__)):
 File /home/wr/ssdsrc/Twisted/twisted/python/filepath.py, line 664, 
in __init__

   assert isinstance(path, bytes), 'path must be bytes: %r' % (path,)
AssertionError: path must be bytes: 
'/home/wr/ssdsrc/Twisted/twisted/__init__.py'


If paths are being represented using unicode somewhere and you want to 
use them with FilePath then you have to encode them (or you have to add 
unicode path support to FilePath and let FilePath encode them).


Unfortunately it's not entirely obvious how to make FilePath support 
unicode paths since not all platforms Twisted supports represent 
filesystem paths using unicode.


The choice python-dev made to bridge this gap was the creation of the 
surrogateescape error handler for the UTC-8 codec.  This lets you 
pretend that any time you need to convert between bytes and unicode the 
correct codec is UTF-8 (with this special error handler).


It's not clear this was a good choice (since the result is unicode 
strings that may contain garbage which will confuse other software) but 
it's also not clear it's possible for Twisted to try to make any other 
choice (at some point Twisted has to interoperate with the path-related 
APIs in Python itself - `sys.path`, for example).


Not sure if that helps you at all.  Maybe it outlines the problem a 
little more clearly, at least.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 06:30 am, a...@roiban.ro wrote:

On 4 September 2014 12:59,  exar...@twistedmatrix.com wrote:
[snip]
What would be really nice is if someone collected *all* of the 
complaints
about `spawnProcess` into one place and integrated solutions to them 
into a

design for a replacement. :)

Jean-Paul


Since I can not create wiki pages here are my notes:

To do:

* Always return a deferred. #2415

* Provide the option to timeout the execution... and on timeout errback 
with

 a dedicated failure #2415

* Add childs to the same process group #2415

* allways errback when failing to spawn (rather than raising OSError) 
and
 errback with a different failure when failing to spawn the executable 
#4184


* on Unix provide a method to install SIGCHLD handler, independent of 
the

 general reactor.run() signal handlers. ... maybe related to #5710

* Allow passing Unicode environment (at least on Windows) maybe needed
for Python3 #5968

* Allow passing Unicode command and arguments on Windows (current not
supported by pywin32 CreateProcess) #6470... maybe needed for Python3
#5968

* Improve fork/exec for speed and memory usage #5710  maybe by 
creating

 a auxiliary process which handles forks and child management.

* Execute as a different account in Windows

* Spawn only after reactor starts ? I am ok with forking before.

* Support client endpoint?

---

For a new design:

* I like the ProcessProtocol API


I don't. :)  ProcessProtocol means you can't re-use any of your existing 
IProtocol implementations.  Oops.


It seems like a better API would let you say Here is a protocol, hook 
its output up to fd 0 in the process, hook fd 1 in the process up to its 
input.  Here is another protocol, hook fd 2 in the process up to its 
input (this would probably be a common configuration - speak some 
regular protocol over stdin/stdout, have a little special logic 
(probably logging) for stderr).


On the other hand, it's possible to build this on top of ProcessProtocol 
and no one ever has...
* spawnProcess will always return a deferred... for backward 
compatibility

 we need to decide a name for this new method which returns a deferred.
 Maybe createProcess or connectProcess

* create spawnUnixProcess which has childFDs uid/gid

* create spawnWindowsProcess which has Windows specific options

* createProcess / connectProcess will call spawnUnixProcess or
spawnWindowsProcess
 depending on current OS

* ProcessProtocol.processEnded is similar to connectionLost..
 so maybe it can be rename it and add other methods so that 
IProcessProtocol
 can inherit from IProtocol. I guess that this will help with 
connectors.


Let's not introduce new cases where inheritance is encouraged.

--

I think that child management problem, does not affect the public API 
so I am

not discussing it here.

Timeouts are handled at protocol level and I see that we have 2 timeout 
types:

total execution time and  timeout after last child data was received.

-

Does it make sense to put it on wiki or what should be the Twisted
Enhancement Proposal process?


I think it's easier to maintain a document on the wiki than in threads 
on the mailing list.

Can I get 'create page' rights on Trac?


Hm.  I thought everyone had wiki edit rights (except for a small number 
of privileged pages) by default - so I'm not sure what's going on here 
or how to give you permissions.  Perhaps something changed in the trac 
upgrade or as part of the recent ticket permission reconfiguration.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 07:26 am, a...@roiban.ro wrote:

On 3 September 2014 18:55,  exar...@twistedmatrix.com wrote:

On 03:27 pm, a...@roiban.ro wrote:


On 3 September 2014 14:39,  exar...@twistedmatrix.com wrote:

[snip]

Do you have any suggestion for how the calls should be made?

reactor.run(installSignalHandlers=True,  installStopHandlers=False)




Also note there is an old, widely scoped ticket:

 https://twistedmatrix.com/trac/ticket/2415

with some more stuff (not necessarily directly related to your comments 
on signal handling) on it.


What would be really nice is if someone collected *all* of the 
complaints about `spawnProcess` into one place and integrated solutions 
to them into a design for a replacement. :)


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Graceful shutdown of twistd application

[exarkun]

On 10:52 am, sangiova...@nweb.it wrote:

Hello list,

I need to implement a graceful shutdown procedure for a twistd 
application.

The application is made up of two services: an internet.TCPClient and
an internet.TCPServer.
They're glued together with a MultiService instance, which is in turn 
set

to have 'application' as parent.
The server and the client work together, making a proxy (SMTP server 
and

AMQP client).

My goal is the following:
- intercept a SIGTERM signal
- 'block' on the server side: since it's SMTP I get this by setting a
variable that makes the server return tempfails (4xx) for new messages,
while keeping current sessions active
- wait until current requests are satisfied (I keep a dictionary of 
current

pending messages)
- shut the whole thing down


This is exactly what before shutdown triggers are for.  Alternatively, 
use the higher-level API and implement `stopService` on one of your 
services.


Either way, return a `Deferred` that only fires when you're satisfied it 
is time for shutdown to proceed.


You said before shutdown triggers are too late but you didn't say why. 
I think that's based on a misunderstanding - but if not, then explain 
why it doesn't work for your scenario.


Jean-Paul
What is the best solution for this use case? It's not really clear to 
me
how to catch SIGTERM and handle pending requests *before* the 
underlying

services start to shutdown (i.e. even addSystemEventTrigger('before',
'shutdown', callable) is called too late for my needs).

Thank you very much for your help!

Fabio


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Graceful shutdown of twistd application

[exarkun]

On 12:36 pm, sangiova...@nweb.it wrote:

On Thu, Sep 4, 2014 at 2:02 PM, exar...@twistedmatrix.com wrote:



You said before shutdown triggers are too late but you didn't say why. 
I
think that's based on a misunderstanding - but if not, then explain 
why it

doesn't work for your scenario.


Hi, thanks for your reply.

I've tried the following:

def sleep(secs):
   log.msg('from within trigger')
   d = defer.Deferred()
   reactor.callLater(secs, d.callback, None)
   return d

reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10)


All 'before' trigger are run concurrently.  If you're using 
`Application` then your `sleep` trigger runs concurrently with the 
application's `stopService` trigger (because `Application` has its 
stopService added as another 'before' shutdown' trigger alongside 
yours).


If you want to delay your application shutdown, you need to cooperate a 
little more closely with it.  Either attach your application shutdown 
code as a callback to the sleep Deferred or move the sleep into the 
stopService implementation of one of the services on your application 
and trigger the remaining stopService calls (eg the stopService call on 
the MultiService you mentioned) when the sleep Deferred fires there.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] High throughput database logger

[exarkun]

On 12:51 pm, adi.libot...@proatria.com wrote:

Hi,

I'm looking at various options for implementing a high throughput 
database logger that will work with Twisted.


My requirements, listed by importance:

1) small memory footprint
2) high speed
3) low garbage generation

The application I'm working on runs continuously (24/7). I've 
experimented a bit with pysqlite and Twisted to see which approach is 
better suited (see attached example).




Question 1: I noticed that all of the Twisted based versions are very 
slow compared to the plain sqlite3 test. This seems to be caused by 
atomic transaction management, namely a commit after each insert.


Not only this but in some of the Twisted versions you've introduced a 
round-trip communication from the reactor thread to a worker thread 
between each operation.  This will greatly impact throughput by adding 
lots of latency to each insert.
Would be interested to know if there is a simple way to avoid this and 
do my own transaction management (aka batch commit).


Using twisted.enterprise.adbapi?  You could probably hack something 
horrible together but it would definitely be a hack.  I suggest you take 
a look at adbapi2 instead - http://trac.calendarserver.org/wiki/twext.
One other thing is the greatly varying amounts of garbage generated 
(peak memory) and memory usage between the Twisted variants.


Garbage and peak memory are different things.  The Twisted-using 
version does a lot more - and some of your Twisted-using versions put 
the *entire* data set into memory (in a vastly expanded form, where each 
insert is represented by multiple large objects including Deferreds). 
So it's not too surprising the memory usage is greater.



Question 2: I would have expected B (Twisted ADBAPI) to behave very 
similar to C/E since I'm using a connection pool of size 1 and all 
requests are queued and handled sequentially.


Could any of you please give me some pointers as to why this is 
happening?


You didn't actually label the code with these letters. :)  I'm guessing 
B is the `adbapi` function, C is `inline_callbacks`, and E is 
`cooperator`.


Also you didn't say in what respect you expected them to behavior 
similarly.  You expected their memory usage to be the same?  You 
expected their runtime to be the same?  You expected them to put the 
same data into the database?


As far as memory usage goes, B uses lots of memory for the same reason 
`semaphore` (D?) uses lots of memory.  You queue up the entire dataset 
in memory as piles of tuples, lists, Deferreds, etc.


adbapi might be executing the operations one at a time, but the *loop* 
inside `adbapi` runs all the way to the end all in one go.  It starts 
every one of those `runOperation`s before any of them (probably) has a 
chance to execute.



Question 3: Even though objgraph lists the exact same reference count 
once the code is ran, the amount of used memory greatly differs. Any 
ideas what might be causing this?


Hopefully the above helps explain this.

Something else you might consider is batching up your inserts (not just 
only committing after a batch of inserts).  Since SQLite3 can only write 
from a single thread at a time, you're effectively limited to serialized 
inserts - so it doesn't make sense to try to start a second insert 
before the first has finished.


When the first finishes, if 50 more data points have arrived, you should 
do one insert for all 50 of those - not 50 inserts each for one piece of 
data.  This cuts off a bunch of your overhead - Python objects, round- 
trip latency for inter-thread communication, function calls, etc.


Jean-Paul
Any suggestions and/or pointers on how to improve/do this are more than 
welcome.


Thank you for your time,
Adrian


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 01:05 pm, a...@roiban.ro wrote:

On 2 September 2014 14:50,  exar...@twistedmatrix.com wrote:

On 01:38 pm, j...@editshare.com wrote:



[snip]


Without my workaround, I continue to have the problem with the
gtk2reactor.


Have you reported this bug?

Jean-Paul


In my initial use case, signal handlers were not installed since I
wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can
use my custom Unix Daemon and Windows Service adapters.

I understand why reactor.run has the installSignalHandlers argument.

Do you think it would make sense to have some kind of public API which
would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do
reactor.stop())  but would install the SIGCHLD handler?


Yes.  Providing more fine-grain control over signal handlers would be a 
fine improvement.


Another fine improvement would be making child processes work even if a 
SIGCHLD handler cannot be installed (for example, by polling children 
periodically, by using wait() in a sidecar process, or by using a better 
system-specific child process monitoring API (eg kqueue's EVFILT_PROC)).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows

[exarkun]

On 03:49 pm, a...@roiban.ro wrote:

Hi,

If I call spawnProcess with a bad executable, on Unix the process will
exit with exit code 1 while on Windows an OSError is raised.


Ideally, these two implementations of the same interface would have the 
same error behavior in this case.  In other words, this seems like a bug 
to me.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 03:27 pm, a...@roiban.ro wrote:

On 3 September 2014 14:39,  exar...@twistedmatrix.com wrote:


Yes.  Providing more fine-grain control over signal handlers would be 
a fine

improvement.


Do you have any suggestion for how the calls should be made?

reactor.run(installSignalHandlers=True,  installStopHandlers=False)


Perhaps.

or

reactor.installStopHandlers = False
reactor.run()


Probably not this one.  Setting attributes on random things is kind of 
sad. :)


It might be nice to try to be somewhat flexible - in case there's some 
reason to change what signals the reactor wants to handle in the future. 
Perhaps:


   reactor.run(installSignalHandlers={SIGCHLD})

An entirely different direction could be to make this bit of 
configuration into initialization for the reactor.


   from twisted.internet.epollreactor import install
   install(installSignalHandlers={SIGCHLD})

   from twisted.internet import reactor
   ...
   reactor.run()

By keeping these details away from `IReactorCore.run`, that method 
remains maximally useful.  For example, if you could set up the reactor 
this way, a normal `twistd` plugin would still be able to benefit from 
your choice, even with twistd's naive call of `reactor.run()` with no 
extra arguments.


Application code calling these `install` functions is already supported 
(it's how you select a specific reactor, after all).  Some of the 
install functions even accept arguments already.


This would actually eliminate another existing issue - 
`IReactorCore.run` is actually defined to take no arguments.  The 
implementations ignore this because someone thought it was important to 
be able to disable installation of signal handlers.


Another fine improvement would be making child processes work even if 
a

SIGCHLD handler cannot be installed (for example, by polling children
periodically, by using wait() in a sidecar process, or by using a 
better
system-specific child process monitoring API (eg kqueue's 
EVFILT_PROC)).


I see that GlibReactorBase inherits from PosixReactorBase so it should
install the SIGCHLD handler... this should not be the reason why gtk2
reactor don't work.


Gtk messes with signals too.  There are confusing order-of-execution 
dependencies and Gtk changes subtly from release to release, re-breaking 
things after we fix them or changing them to be broken in a different 
way.


So that's *why* it probably doesn't work with Gtk2 reactor - if not 
*how*. ;)


I think I missed the explanation of what in particular wasn't working 
with Gtk2 reactor though.

As a poor man's fix and Unix independent fix maybe we can call
reapAllProcess in a task.LoopingCall...
What are the reasons why SIGCHLD handler cannot be installed?


Either because you want to run the reactor in a non-main thread (where 
Python won't let you install signal handlers for the good reason that 
mixing signals and threads has crazy behavior) or because you want to 
use a different library that depends on having its own SIGCHLD handler 
and you're not interested in Twisted's process support.

I am asking since I hope it could help me understant where and how to
enable the poor man's fix... and how to fill the bug report.

kqueue's EVFILT_PROC would be great, but I guess that we still need a
general fix


Perhaps, perhaps not.  A general fix might be less code but having half 
a dozen specialized fixes, one for each reactor, would also still fix 
the problem.  The different reactor implementations are essentially the 
big piles of specialized fixes needed to do non-blocking I/O on 
different platforms.  This would just be a little more of the same.


The sidecar process is an example of a general fix, though.  The idea 
there is that Twisted itself runs a private child process (perhaps only 
when the first call to spawnProcess is made).  It talks to that process 
using a file descriptor.  That process can install a SIGCHLD handler 
(because Twisted owns it, application developers don't get to say they 
don't want one installed) or use another more invasive strategy for 
child process management.  When you want to spawn a process, the main 
process tells the sidecar to do it.  The sidecar relays traffic between 
the child and the original parent (or does something involving passing 
file descriptors across processes).


This removes the need to ever install a SIGCHLD handler in the main 
process.  It also probably enables some optimizations (reapProcesses is 
O(N!) on the number of child processes right now) that are very tricky 
or impossible otherwise.


Jean-Paul

-

For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles
but keep SIGCHLD I do:

# Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK 
handlers

_SignalReactorMixin._handleSignals = lambda self: None
reactor.run()


--
Adi Roiban


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 05:55 pm, exar...@twistedmatrix.com wrote:


Gtk messes with signals too.  There are confusing order-of-execution 
dependencies and Gtk changes subtly from release to release, re- 
breaking things after we fix them or changing them to be broken in a 
different way.


So that's *why* it probably doesn't work with Gtk2 reactor - if not 
*how*. ;)


I think I missed the explanation of what in particular wasn't working 
with Gtk2 reactor though.


Oh right, that was where Justin's bug was.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows

[exarkun]

On 05:17 pm, a...@roiban.ro wrote:

On 3 September 2014 16:49, Adi Roiban a...@roiban.ro wrote:

Hi,

If I call spawnProcess with a bad executable, on Unix the process will
exit with exit code 1 while on Windows an OSError is raised.

I am working on a multi-OS software and to make my life easier I ended
up with this hack inspired by _BackRelay

https://gist.github.com/adiroiban/bac493f00ce5e94738ce



Is there something already in Twisted doing this? Am I reinventing the 
wheel?


If not, do you think that it would help to update _BackRelay to
support timeout and cross-os error handling?


Searching the Twisted trac I found this ticket
http://twistedmatrix.com/trac/ticket/4184

Maybe we can leave spawnProcess as the low-level API and then create
another wrapper which will handle/raise the errors in an consistent
way.

For the case where command is not found, I prefer the behaviour from
Windows where an OSError is raise, rather than
looking at errReceived and trying to guess is execvpe call failed or 
not.


It is more convenient, certainly.  Unfortunately on POSIX the underlying 
API isn't synchronous - which is why the POSIX implementation doesn't 
deliver the exception synchronously.  So to be uniform we probably need 
to make the Windows behavior asynchronous.


I suggest we can improve the situation by reporting the error more in an 
easier-to-recognize way when we report it asynchronously, though.


For example, instead of forcing the application to parse stderr we could 
deliver a different exception type to processEnded.


A better solution would be to make `spawnProcess` properly asynchronous 
- have it return a `Deferred` that fires with an `IProcessTransport` and 
let that `Deferred` fire with a `Failure` if there is a problem creating 
the new process.


But that's probably more like introduce an API to replace 
`spawnProcess` (for which I believe there is also a ticket).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Return only when value/handle is available

[exarkun]

On 05:52 pm, vikas.c.ku...@oracle.com wrote:

Hi Jonas,

This is nice.
But still we are retrying it repeatedly at regular interval(10). Can't 
we get the notified asynchronously as and when handle is available.


You tell us.  What's a handle?  Where do you get it from?  Does that 
system produce an event when it is ready?


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures

[exarkun]

On 01:38 pm, j...@editshare.com wrote:


FWIW, looking at your original code, you may also want to call your 
Deferreds in a reactor.callLater.  I learned the hard way that if you 
callback a Deferred from processEnded and the callback spawns another 
process, your process will still be Twisted's process table even though 
it's dying.


--

I dig deeper and I found out that since I was using
reactor.run(installSignalHandlers=False)  _SIGCHLDWaker was not
installed.

I have switched to using just reactor.run() and the process is now 
killed.


Without my workaround, I continue to have the problem with the 
gtk2reactor.


Have you reported this bug?

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] jelly with new-style classes: method references are not resolved

[exarkun]

On 11:39 am, wolfgang@rohdewald.de wrote:

Hi,

while trying to port jelly to Python 3 (banana tests already pass with 
Python 3),


I found a bug in jelly. It has already been reported 3 years ago with 
ticket 4935.


And there is a fix attached to that ticket. This fix resolves the 
problem for me.


Why has this fix never been included? It is necessary for porting to 
Python 3.


Perhaps because the ticket was never given the review keyword - so no 
one ever noticed that there was a potential fix attached to it.


See https://twistedmatrix.com/trac/wiki/ReviewProcess

Jean-Paul

--
Wolfgang

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Using spawnProcess

[exarkun]

On 12:24 pm, grigorescu_cipr...@yahoo.com wrote:

Hello,
I have wrote a custom protocol(inherit from SMTPClient) and 
factory(inherit from ClientFactory) in order to send emails from 
multiple ips(multiple connections, factories).


Now I want to run this pice of code multiple times using spawnProcess 
to send emails simultaneous, each process having n connections from 
n ips.


How can i use spawnProcess? Or is another way to send emails in a 
parallel fashion with twisted? Maybe another approach?


You don't need multiple processes to send email concurrently.  If you 
were to use `twisted.mail.smtp.sendmail`, then you could do this by just 
making multiple calls to `sendmail`.  You don't have to wait for one 
email to be completely sent before you start trying to send the next 
one.  For example, to send two emails concurrently:



   from sys import stdout

   from twisted.mail.smtp import sendmail
   from twisted.internet.defer import gatherResults
   from twisted.internet.task import react
   from twisted.python.log import startLogging

   def main():
   return gatherResults([
   sendmail(...),
   sendmail(...),
   ])

   startLogging(stdout)
   react(main, [])

You should be able to do something similar with the API you've 
constructed (or perhaps you can just use `sendmail` instead).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Trac

[exarkun]

On 26 Aug, 08:09 pm, exar...@twistedmatrix.com wrote:

On 02:06 pm, alex.gay...@gmail.com wrote:
I see new tickets that are created through the IRC bot - I've deleted 
all

of these now.


Okay.  Thanks,


Here are a few more:

#7604
#7605
#7606

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Trac

[exarkun]

On 03:42 am, alex.gay...@gmail.com wrote:

exarkun at twistedmatrix.com writes:


Hi Alex,

I notice some spam has started to get through.  See the latest bug
summary email.

Jean-Paul


(Please be aware I don't read twisted regularly, please CC me to ensure 
I see

it, Glyph was kind enough to draw this to my attention)

Yes, I saw (I deleted the spam when I saw it). The solution to this is 
going to
be to get a second spam checker system (probably the Bayesian filter 
one), I'll
work on that with Glyph as soon as he returns from vacation (next 
week).


In the meantime, I'll keep manually delete any spam I see.


How are you watching for spam?  It looks like you didn't see it all:

#7558
#7564
#7565
#7567

And in the interim, #7575 - #7585.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Trac

[exarkun]

On 02:06 pm, alex.gay...@gmail.com wrote:
I see new tickets that are created through the IRC bot - I've deleted 
all

of these now.


Okay.  Thanks,

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Trac

[exarkun]

On 24 Aug, 02:40 pm, exar...@twistedmatrix.com wrote:

On 16 Jul, 06:02 pm, alex.gay...@gmail.com wrote:

Hi all,

A short while ago Twisted's trac installation was changed to reject 
new tickets

from anyone except a special white listed group.

As of yesterday I have reverted this change, (and turned the spam 
filter back
on). If you experience any issues trying to post, or notice and spam, 
please

feel free to reach out to me.


Hi Alex,

I notice some spam has started to get through.  See the latest bug 
summary email.


Ping

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Trac

[exarkun]

On 16 Jul, 06:02 pm, alex.gay...@gmail.com wrote:

Hi all,

A short while ago Twisted's trac installation was changed to reject new 
tickets

from anyone except a special white listed group.

As of yesterday I have reverted this change, (and turned the spam 
filter back
on). If you experience any issues trying to post, or notice and spam, 
please

feel free to reach out to me.


Hi Alex,

I notice some spam has started to get through.  See the latest bug 
summary email.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there pb documentation somewhere?

[exarkun]

On 03:31 am, sank.dan...@gmail.com wrote:

Dustin,
Adding what amounts to a use-after-free bug to the protocol seems like 
a

really bad idea to me.

Oh goody, a sarcastic comment which doesn't actually bother to explain 
the

bug :)


Sarcastic?  What?  Dustin *was* explaining the problem.  Without 
sarcasm, so far as I can tell.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Serving WSGIResource via SSL endpoint

[exarkun]

On 04:32 pm, piper.mas...@gmail.com wrote:

On Fri, Aug 15, 2014 at 2:15 AM, Glyph gl...@twistedmatrix.com wrote:

When I use the openssl command you gave me, I get this:

openssl s_client -connect localhost:8000
   CONNECTED(0003)
   ^C

(I had to Ctrl-C it, because it just stuck there)


You might find it instructive to run these commands against a known 
working TLS server so you can draw some conclusions about the results 
when you use them against yours.


For example, this would show you that the TLS handshake is never 
actually completing.


Are you sure you've actually set up TLS on this port?

An http://sscce.org/ would help the list diagnose the problem.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Scalability of timers

[exarkun]

On 09:38 pm, tobias.oberst...@tavendo.de wrote:


What I am after is to explicitly _control_ the maximum syscall rate to 
select() - not simply max. out the box on syscall rate.


Like: limit syscall rate to select() at 1000Hz - regardless how many 
timers I issue per second.


In other words:

If you ask Twisted to wake up N times per second, is there an API to 
make Twisted wake up M (MN) times per second instead?


Is that what you're looking for?

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there pb documentation somewhere?

[exarkun]

On 05:57 pm, gl...@twistedmatrix.com wrote:


On Aug 7, 2014, at 10:42 AM, Daniel Sank sank.dan...@gmail.com wrote:

glyph,

 I really wish we would stop calling things bad and good.

My wording of exarkun's wording. He gave a much more detailed 
description of what he think's is crazy about pb.


This was a complaint about a general trend, not about specific words. 
Clearly exarkun gave you the impression that it is bad, whether he 
specifically said so or not.


I don't understand what you're saying here.

Do you want people to not describe the shortcomings of certain pieces of 
software?


Or do you want people not to conclude from such descriptions that those 
pieces of software are not the most well suited for certain 
applications?


Or do you want people to write two pages of description every time they 
want to refer to the idea that a certain piece of software isn't the 
best choice for a certain application?


Could you clarify what you think the problem here actually is?


We're all intimately familiar with everything that's terrible about all 
of our code, and we aren't shy about sharing.  I just would like it if 
we could really lead with the details and refrain from value judgements 
:).


In this case, it seems like that's exactly what happened.  I led with 
detail.  The value judgement of PB being bad (which is a gross over- 
simplification, but a convenient shorthand) came afterwards.


Jean-Paul

 make your own decisions about how to write your own code.

Indeed, but gathering information from wiser folks is always a good 
idea, and usually best done _often_ during development :)


I might quibble with wiser but okay.  I'm happy to provide feedback 
earlier so I don't have to say what is this disaster later ;-).
 I'm happy to trade 2-for-1 - if you do two code reviews, I will 
regard it as an immediate obligation for

 me to review a ticket you direct me to ;).

Deal. However, rather than direct your attention to tickets, at this 
stage I would rather trade reviews for discussion. I'll do two reviews 
and then post a few questions to this mailing list thread. Once I 
start actually writing patches/new code we can trade reviews for 
attention to tickets. Ok?


I'm happy to do that.
 These would also be easier to land, and a couple of decades in open 
source has taught me that nothing
 motivates development activity like successful development activity 
;).


Indeed. There are one or two architectural issues I want to understand 
before moving on to real coding. I will try to get through that asap 
by reviewing tickets and trading for discussion of those architectural 
issues.


I'll try to respond to these questions regardless.  I would like to 
help.  It's just that the reviews will create a more tangible sense of 
commitment :).
 Hopefully you can make sense out of the explanations above and your 
own existing knowledge.

 Are there any other phases of the process which are confusing?

This all makes sense now. I hadn't understood the point of the cooker, 
but now that you've explained it, I understand what's going on. I will 
transform your mailing list explanation to documentation shortly.


Great, glad that helped.

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Embedding manhole interpreter into insults widget.

[exarkun]

On 6 Aug, 06:14 pm, lacrima.ma...@gmail.com wrote:

Hello,

I have been trying to create a widget that encloses manhole 
interpreter.
Here is somewhat hacky implementation that I came up with at this 
moment:


Widgets and the manhole interpreter are implemented using two pretty 
distinct approaches.


As you've figured out, these approaches can be integrated (at least in 
one direction) using a terminal emulator - 
`twisted.conch.insults.helper.TerminalBuffer`.


However, `TerminalBuffer` is an extremely low quality terminal emulator. 
Among its many flaws is that it uses a number of incredibly inefficient 
implementations for common terminal operations (many of its other flaws 
are along the lines of it has a bug in the implementation of X).


If I were going to create a manhole widget then I would do it natively 
- not by bridging the existing manhole protocol to the widget system 
with TerminalBuffer but by writing a new widget that does manhole stuff 
all by itself.


I suggest that you might want to pursue this approach instead of trying 
to optimize and fix bugs in TerminalBuffer.


Ultimately it would be nice if a widget-based manhole were *the* manhole 
and the current implementation went away.


However, that said, as far as I know, no vt-style terminals support 
efficient redrawing for the vertically scroll one portion of the 
terminal separated from another portion by a vertical divider.


You will be able to avoid the redundant work of re-rendering the Python 
buffer representing the terminal state but you'll still have to re- 
transmit the bytes representing the entire terminal display to the 
terminal on each redraw - unless you avoid vertical dividers.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] TerminalSession instance has no attribute 'windowChanged'

[exarkun]

On 10:17 am, gl...@twistedmatrix.com wrote:
On Jul 31, 2014, at 3:03 AM, Maxim Lacrima lacrima.ma...@gmail.com 
wrote:

Hello!

When size of terminal window of ssh client is changed, then twisted 
ssh server throws an exception:


exceptions.AttributeError: TerminalSession instance has no attribute 
'windowChanged'


`manhole_ssh.TerminalSession` doesn't implement this method, but 
according to ISession interface it should.


Is this a bug?


Yes, this is definitely a bug.  The interface says it should be there, 
it's called by a user of the interface, and it is not present.  Pretty 
clearly wrong.  Can you file a ticket?


Good news.  No need to file a ticket:

   https://twistedmatrix.com/trac/ticket/3303

Just need to resolve it.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.names.client.getHostByName() vs. reactor.resolve()

[exarkun]

On 02:16 pm, rut...@osu.edu wrote:

Hi,

I need to do a DNS resolution of a hostname to an IPv4 address in order 
to
then send out a UDP packet via twisted’s APIs.  There appears to be 2 
ways
to do this.  Is it better to use twisted.names.client.getHostByName(), 
or
is it better to use reactor.resolve()?  Is there a bias for one over 
the

other?  Is one deprecated and the other isn’t?


`reactor.resolve` is do something, who knows what really, to turn a 
name into an IPv4 address.


`twisted.names.client.getHostByName` is use Twisted's DNS client 
implementation to issue the necessary DNS queries to resolve a name into 
an IPv4 or IPv6 address.


If you definitely only care about IPv4 then `reactor.resolve` is going 
to work correctly in more situations.  By default it will use the system 
resolver.  Therefore, by default, you will get whatever weird 
configuration the system resolver has been given (for example: use 
/etc/hosts or use mDNS or use NIS or YP or whatever.


Note that `reactor.resolve` is a pluggable interface and application 
code is allowed to substitute an alternate resolver implementation which 
may work in some other way.


If you care specifically about DNS or you need IPv6 or you have a reason 
to want to avoid threads (which, by default, `reactor.resolve` uses), 
you might prefer to use `twisted.names.client` directly.


Most applications will probably be okay with `reactor.resolve` but I 
imagine this could change as IPv6 use gradually becomes more widespread.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Tubes!

[exarkun]

Hello all,

Some of you may have heard rumors of some work in progress on a 
replacement for Twisted's IConsumer/IProducer interfaces.


Tubes have been largely Glyph's effort (though a lot of people have 
contributed in one way or another).  And a large effort it's been. 
Development is proceeding in a Twisted branch and comes to over three 
thousand lines of additions so far.


Given the large size of the implementation and the long time that this 
effort has been underway (I remember the Twisted meetup at the Rackspace 
offices that *I* attended when I was visiting SF... a year and a half 
ago... at which point tubes wasn't exactly a brand new project) I'd like 
to re-raise the idea that the best next step for the project is to see 
some distribution in its *current* state.


Specifically, I think it would be beneficial to set up a tubes project 
on Github under the Twisted organization and try for a release in the 
very near future.


I think this has several advantages over the status quo:

 1) As an independent project, tubes will attract more attention than it 
presently gets as a relatively unknown ticket  branch of Twisted.


 2) As a separate Python package, the logistics of actually using tubes 
are simpler (just consider how you might declare a dependency on a 
branch of Twisted - keeping in mind you may want to use tubes in a 
project that already depends on some version of Twisted).  It may not 
make sense to say that it is the same quality as Twisted proper right 
off the bat (on the other hand, it may well - I suspect tubes in its 
current form actually is a lot higher quality than large sections of 
Twisted) but that doesn't mean people (not to mention the tubes project) 
can't benefit from being able to experiment with it.


 3) Decoupling tubes from Twisted frees tubes from certain of Twisted's 
policies which are more challenging to follow for the kind of non- 
trivial, brand new code base that tubes is.  Technically we could just 
say that these policies don't apply to a tubes package *in* Twisted but 
this kind of subtle distinction is often lost on users (ie application 
developers).


   a) Twisted's compatibility policy need not apply.  It could either be 
sped up or abandoned more thoroughly.  I'm generally a fan of being 
backwards compatible even when you have few users because it actually 
makes development easier, but loosening the policy to say things might 
break if it's just really inconvenient to keep them working (whereas 
Twisted goes to the inconvenience to keep them working) seems 
reasonable.


   b) tubes can undergo a faster release cycle to benefit more from user 
feedback.


 4) At this point, a normal review of the tubes branch is going to be a 
problem.  We do not have good tools or mechanisms for dealing with 
branches this large.  The code in the current tubes branch can just 
become master of a new project.  Development going forward from this 
point should continue to follow the feature-branch, small-changes, pre- 
commit-peer-review process.  But those 3k lines are written already. 
Short of an extremely expensive effort to break the work up into 
smaller, self-contained pieces there's simply never going to be a *good* 
review in the typical style.


Additionally, it may turn out that tubes can remain independent 
indefinitely.  Someday perhaps Twisted would come to depend on it to 
allow the various protocols and applications implemented in Twisted to 
benefit from the superior abstractions it provides.  Or maybe once it 
has undergone a few iterations it will make sense to bring it back to 
Twisted.  I don't think this needs to be decided now.


There are downsides, of course.  All of the boring maintenance involved 
with having a separate project - setting up CI, actually doing the 
releases, etc.  Perhaps we could find some volunteers to help out with 
these tasks, though, in exchange for getting some great code out there?


I'm curious what the folks out there who develop applications using 
Twisted would find to be the easiest path forward.  I'm also curious to 
hear what Glyph thinks about all this. ;)


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tubes! (package metadata and version constraints)

[exarkun]

On 10 Jul, 08:37 pm, gl...@twistedmatrix.com wrote:


On Jul 10, 2014, at 12:32 PM, exar...@twistedmatrix.com wrote:
2) As a separate Python package, the logistics of actually using tubes 
are simpler (just consider how you might declare a dependency on a 
branch of Twisted - keeping in mind you may want to use tubes in a 
project that already depends on some version of Twisted).  It may not 
make sense to say that it is the same quality as Twisted proper right 
off the bat (on the other hand, it may well - I suspect tubes in its 
current form actually is a lot higher quality than large sections of 
Twisted) but that doesn't mean people (not to mention the tubes 
project) can't benefit from being able to experiment with it.


I would love it if there were a way to release a package in an actually 
experimental state, and not just have the release of a package 
implicitly tell people that it's time to put it into production and 
demand long-term support for it.  Quick sanity check: go run 'pip 
freeze' in a production virtualenv you're running - what percentage of 
the version numbers that come back start with a zero?  I will bet a 
significant amount of money that it's not 0% :-).


As it stands, if you're not willing to use a random outdated branch of 
Twisted with unknown bugs that may change without warning, you're 
probably not willing to adopt Tubes yet.


I think you missed part of my point here.

I want to try tubes and see if they make my application simpler or 
better or faster or whatever.  My application depends on Twisted = 
14.0.0.  It doesn't matter how or why: that's what the metadata says and 
I can't use an older version of Twisted unless I do a bunch of stupid 
package/distribution related hacking that I don't want to do.  *So* much 
of the Python tooling has now moved to requiring and respecting explicit 
dependency declarations that trying to side-step these now is a 
significant hassle.  Separating tubes from Twisted solves this problem.


It's not at all a question of whether the code is stable or production 
ready or even works at all, it's a matter of packaging constraints.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] bringing LDAP back

[exarkun]

On 11:30 am, hawk...@atleastfornow.net wrote:
On 8 Jul 2014, at 5:26, Glyph Lefkowitz gl...@twistedmatrix.com 
wrote:

On Jul 7, 2014, at 2:21 PM, exar...@twistedmatrix.com wrote:
It might not be.  Now that so many new projects are being added to 
the github Twisted repository (and particularly, projects that have 
new contributors), perhaps it is (slightly past) time to set down 
some of these things officially?


I think we should make a list of some suggestions before we try to 
make that list into an official policy.  Date-based version numbers is 
a good suggestion to start that list with :).


Maybe we need a Request For Comment/PEP-style way of 
formalising/proposing policy? Django’s doing it as well, and it seems 
like a nice way of doing it.


I don't see this as the necessary conclusion.  We're not arguing about 
what the policy should be yet.  I'm just saying it should be written 
down in one place - not smeared out across 10 years of mailing list 
archives.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Weekly Bug Summary

[exarkun]

These stats are incorrect due to buggy interactions between the stats 
script and the new version of trac.  Correct trac stats will return once 
https://code.launchpad.net/~twisted-dev/twisted-trac-integration/fix- 
summary-with-new-trac/+merge/225813 is merged and deployed.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] bringing LDAP back

[exarkun]

On 09:02 pm, gl...@twistedmatrix.com wrote:


On Jul 7, 2014, at 4:13 AM, HawkOwl hawk...@atleastfornow.net wrote:


On 7 Jul 2014, at 18:50, Hynek Schlawack h...@ox.cx wrote:

On 7 Jul 2014, at 12:23, Bret Curtis wrote:

It is my recommendation that after this is reviewed and hopfully
commited that we make a branch and tag the release as 0.54.0. The
reason for the large jump is that antong's last semi-offical release
was 0.53 on PyPI. At this point we (myself and anyone else that 
wants

to help) should reach out to downstream projects (PyPI, Debian, and
etc.) to make them aware that Ldaptor development is again active.


Since ldaptor is a Twisted project now, may I suggest you copy its 
time-based version numbers? 14.0 has more meaning to itself than 
0.54.0.


Well, Nevow is a “Twisted Project” and it doesn’t — but I agree that 
it has more meaning.


It might not be a bad idea to do this for Nevow too, for that matter.


It might not be.  Now that so many new projects are being added to the 
github Twisted repository (and particularly, projects that have new 
contributors), perhaps it is (slightly past) time to set down some of 
these things officially?


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] dumb question on qt4reactor

[exarkun]

On 5 Jul, 08:35 pm, kebi...@gmail.com wrote:

calling reactor.callLater within qt4reactor in an attempt to async non
webkit work but not seeing expected behavior.  Webkit locks while the
callLater function executes.


`callLater` does not make a blocking function into a non-blocking one. 
It just calls a function after a specified delay.  If the function 
blocks, then the reactor will block when it gets around to calling it.


If you need to call blocking APIs you may need to call them in a 
separate thread (Twisted includes a threadpool that can help you with 
this) or a separate process (Twisted has some APIs for launching and 
interacting with other processes, too).  Of course, the ideal solution 
is often to find a non-blocking equivalent to the blocking API and use 
that instead (for example, I know that txaws is a project that exists - 
perhaps it can do the things to S3 that you want to do).


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Request.getClient

[exarkun]

Hello all,

twisted.web.http.Request.getClient has a terrible implementation.  It 
does blocking network I/O (DNS).  Fortunately it is only used in one 
place in Twisted - the CGI implementation.  Unfortunately this makes the 
CGI implementation somewhat unsuited for real-world use.


`Request.getClient` has always been allowed to return `None` under 
certain circumstances.  I propose making it always return `None` and 
deprecating it.


This is implemented in the branch linked to https://tm.tl/2252.

Chris Armstrong suggested that this change might not be strictly keeping 
with our backwards compatibility policy.


I suggest that either it is - because `None` was always a possible 
return value - or that removing the possibility of blocking I/O from 
applications that are mistakenly using this API makes it worth the not- 
strictly-compatible change.


A minor adjustment might be to make it always return the IP address 
instead, as this was another behavior it previously had.


Please comment.  Thanks.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request.getClient

[exarkun]

On 06:26 pm, dus...@v.igoro.us wrote:

Actually, the apidocs say Undocumented, so I think you could make it
return uThe  is made of green 酪  and still be in spec.

http://twistedmatrix.com/documents/14.0.0/api/twisted.web.http.Request.html#getClient


The interface is documented:

https://twistedmatrix.com/documents/14.0.0/api/twisted.web.iweb.IRequest.html#getClient

(as deprecated, already).

Also, we generally give some weight to how an implementation actually 
behaves - even if the documentation is lacking or contradictory.


If you ignore incorrect docs and figure out how to use an API so it 
works, you could conceivably have working software.  Twisted tries not 
to break such code.


If you ignore how the implementation actually behaves and blindly trust 
the documentation (sorry, wish you could do that) then your software 
probably doesn't work.  Since it's probably already broken, Twisted 
isn't as concerned making changes that might break it more or 
differently.


Jean-Paul

Dustin

On Wed, Jul 2, 2014 at 12:26 PM,  exar...@twistedmatrix.com wrote:

Hello all,

twisted.web.http.Request.getClient has a terrible implementation.  It 
does
blocking network I/O (DNS).  Fortunately it is only used in one place 
in

Twisted - the CGI implementation.  Unfortunately this makes the CGI
implementation somewhat unsuited for real-world use.

`Request.getClient` has always been allowed to return `None` under 
certain
circumstances.  I propose making it always return `None` and 
deprecating it.


This is implemented in the branch linked to https://tm.tl/2252.

Chris Armstrong suggested that this change might not be strictly 
keeping

with our backwards compatibility policy.

I suggest that either it is - because `None` was always a possible 
return
value - or that removing the possibility of blocking I/O from 
applications

that are mistakenly using this API makes it worth the not-
strictly-compatible change.

A minor adjustment might be to make it always return the IP address 
instead,

as this was another behavior it previously had.

Please comment.  Thanks.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python