[Twisted-Python] Can I post a consultant request here?

[Steve Steiner (listsin)]

Hey!

I apologize in advance if this type of request is not desired/allowed  
on this list.

I'm looking for a very short-term mentoring/consulting contract to   
help me get an Ajax/Comet type of connection working between a web  
interface and a fairly long-running, server-side process.

I want the web interface to reflect the server-side data as it  
becomes available.

I'm sure someone on this list knows exactly how to hook the pieces  
together; I don't.

It would be much more efficient to have someone who already knows  
help me hook it up instead of spending an unpredictable amount of time  
figuring it out myself.

I'd also like to have, as a primary product of this project, a  
tutorial showing a real-world application of Twisted to handle a  
modern, real-world use-case with full, open, well documented source  
code.

If you're interested, please contact me off-list at  
sstein...@gmail.com.

Thanks,

S





___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Performance

[Steve Steiner (listsin)]

Cool, please let me know off-list if you'd like a quick review before  
"going live."

S

On Oct 14, 2009, at 9:30 PM, Dave Peticolas wrote:

> Steve Steiner (listsin) wrote:
>> On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
>>> Personally I'm a big fan of this series of blog posts:
>>> http://krondo.com/blog/?page_id=1327
>>
>> Very nice, thanks for sharing!
>
> Thank you. I really meant to make a public announcement
> about that on this list, but I was trying to get to
> Deferreds first :)
>
> Anyway...I'm writing an Introduction to Twisted and
> asynchronous programming. Surprise :)
>
> dave
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Performance

[Dave Peticolas]

Steve Steiner (listsin) wrote:
> On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
>> Personally I'm a big fan of this series of blog posts:
>> http://krondo.com/blog/?page_id=1327
> 
> Very nice, thanks for sharing!

Thank you. I really meant to make a public announcement
about that on this list, but I was trying to get to
Deferreds first :)

Anyway...I'm writing an Introduction to Twisted and
asynchronous programming. Surprise :)

dave

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Performance

[Daniel Griffin]

Sounds good, I am going to try to get this up and running over the next few
days. I'll get back with some results.


Dan

On Wed, Oct 14, 2009 at 7:14 AM, Steve Steiner (listsin) <
list...@integrateddevcorp.com> wrote:

>
> On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
> > Personally I'm a big fan of this series of blog posts:
> > http://krondo.com/blog/?page_id=1327
>
> Very nice, thanks for sharing!
>
> S
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Steve Steiner (listsin)]

On Oct 14, 2009, at 8:26 AM, Tim Allen wrote:

> On Wed, Oct 14, 2009 at 07:43:41AM -0400, Steve Steiner (listsin)  
> wrote:
>> That's the thing about Twisted; sometimes it's hard to know whether
>> the stuff that has been built into standard Python since Twisted
>> 'rolled their own' is a superset, a subset, or a completely
>> different beast.  Logging is a good case in point.  Since we're
>> using Python's logging everywhere, I wasn't even sure whether there
>> would be an advantage to learning Twisted's similar system.
>> Twisted's trial is another example; we've just been using nose.
>> Seems like there's always some little extra that makes the Twisted
>> stuff worth knowing.
>
> Most of the duplicate stuff in Twisted has the excuse of being written
> before any alternatives were available, and usually having one or two
> extra little integration features that current alternatives don't
> provide.

Yes, that extra little integration feature sometimes makes a huge  
difference.  But, as you said, being as much of Twisted's stuff  
predates the 'equivalent' stdlib stuff, it's hard to know where the  
leverage is going to be.

For example, Python's standard logging has the advantage of being able  
to generate many different types of logging outputs (e.g.  
logging.handlers.SMTPHandler) whereas Twisted's is better integrated  
with Twisted but is missing many of those types of things.

Trial and deferreds vs. nose and its integration with e.g. coverage...

It's hard to know when to use which and, as you said, we want to use  
standard library stuff wherever possible.

> Once I was using Trial, which automatically sets up Twisted's logging
> system to log to a handy file for debugging purposes, I had to decide
> what to do about logging - try to reroute Twisted's logging into the
> Python logging system (which would mean doing something complicated  
> for
> tests), or just give up and use Twisted's logging everywhere. I opted
> for the latter route as the path of least resistance (and because it
> would be easier to set up with twistd later).

I'm working out a blended method having a PythonLoggingObserver that  
relay's Twisted's log messages to Python standard logging, as per  
Glyph's suggestion.  That way we don't have to mangle all of our  
working logging code, bug special notifications we can reroute to a  
special "Twisted Messages" file that we know to keep an eye on.

> I believe nose does for Python logging what Trial does for Twisted
> logging these days, and if nose can run Trial's TestCase subclasses,  
> you
> might just as well stick with what you're already using. There are
> integration advantages to using all the Twisted goodies together, but
> it's not nearly the same amazing quantum leap in understanding and
> capability that Twisted's core provides.

Again, we'll probably end up with a blended approach since we use nose  
for coverage and profiling integration as well.

We are having fun now, right?

S



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Glyph Lefkowitz]

On Wed, Oct 14, 2009 at 4:43 AM, Steve Steiner (listsin) <
list...@integrateddevcorp.com> wrote:

>
> On Oct 14, 2009, at 2:07 AM, Glyph Lefkowitz wrote:
>
> Yes, we're working on it but it's a large code base and we started with
> exactly zero tests.  While that leaves infinite room for improvement, it's a
> little overwhelming.  Oh well, at least we know where to concentrate first
> ;-0.
>

I realize it's a challenge, but, well, "Doctor, doctor, it hurts when I
don't do this!".  My immediate reaction is "go ahead and do it, then" :-).

> Trial will helpfully fail tests which cause exceptions to be logged, so you
> don't need to write any special extra test to make sure that nothing is
> blowing up; just test your error-handling case, and if it blows up you will
> see it.
>
>
> We've just been using nose; is that something Trial handles specially for
> Twisted?
>

Yes.

> If you want to handle logged exceptions specially, for example to put them
> in a separate file, or to e-mail them to somebody, consider writing a log
> observer that checks for the isError key and does something special there.
> You can find out more about writing log observers here: <
> http://twistedmatrix.com/projects/core/documentation/howto/logging.html>.
>
>
> This is an area of Twisted I haven't explored at all since the code's all
> using the standard Python logging.
> That's the thing about Twisted; sometimes it's hard to know whether the
> stuff that has been built into standard Python since Twisted 'rolled their
> own' is a superset, a subset, or a completely different beast.  Logging is a
> good case in point.  Since we're using Python's logging everywhere, I wasn't
> even sure whether there would be an advantage to learning Twisted's similar
> system.  Twisted's trial is another example; we've just been using nose.
>  Seems like there's always some little extra that makes the Twisted stuff
> worth knowing.
>

Another option here would be to set up a PythonLoggingObserver that relay's
Twisted's log messages to Python standard logging, then setup a standard log
handler (another poster suggests nose already does this) to fail the test
when appropriate.  I'm fairly sure that the information you need is being
properly relayed to the relevant parties, but if I'm wrong, it's a bug and
probably a pretty easy one to fix :).
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Tim Allen]

On Wed, Oct 14, 2009 at 07:43:41AM -0400, Steve Steiner (listsin) wrote:
> That's the thing about Twisted; sometimes it's hard to know whether
> the stuff that has been built into standard Python since Twisted
> 'rolled their own' is a superset, a subset, or a completely
> different beast.  Logging is a good case in point.  Since we're
> using Python's logging everywhere, I wasn't even sure whether there
> would be an advantage to learning Twisted's similar system.
> Twisted's trial is another example; we've just been using nose.
> Seems like there's always some little extra that makes the Twisted
> stuff worth knowing.

Most of the duplicate stuff in Twisted has the excuse of being written
before any alternatives were available, and usually having one or two
extra little integration features that current alternatives don't
provide.

A year or two ago I was planning a new project using Twisted, and (as
a reaction to the masses of horrible, legacy code at my employer) had
decreed that the project would use the Python stdlib wherever possible,
including logging and testing, and well-integrated third-party tools
where necessary.

First on the chopping block was the use of standard Python unit-tests
and nose. At the time, I don't believe it was possible to run TestCases
inheriting from twisted.trial.unittest.TestCase in anything besides
Trial - and you need to use trial's TestCase class if you want to do
anything with Deferreds, so nose and the standard unittest.TestCase had
to go.

Once I was using Trial, which automatically sets up Twisted's logging
system to log to a handy file for debugging purposes, I had to decide
what to do about logging - try to reroute Twisted's logging into the
Python logging system (which would mean doing something complicated for
tests), or just give up and use Twisted's logging everywhere. I opted
for the latter route as the path of least resistance (and because it
would be easier to set up with twistd later).

I believe nose does for Python logging what Trial does for Twisted
logging these days, and if nose can run Trial's TestCase subclasses, you
might just as well stick with what you're already using. There are
integration advantages to using all the Twisted goodies together, but
it's not nearly the same amazing quantum leap in understanding and
capability that Twisted's core provides.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Performance

[Steve Steiner (listsin)]

On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
> Personally I'm a big fan of this series of blog posts:
> http://krondo.com/blog/?page_id=1327

Very nice, thanks for sharing!

S


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Steve Steiner (listsin)]

On Oct 14, 2009, at 2:07 AM, Glyph Lefkowitz wrote:

On Tue, Oct 13, 2009 at 8:02 PM, Steve Steiner (listsin) > wrote:

I've been hunting down a problem that I've finally found the cause of
and I'd like to know what's the Twisted way to catch this "error
within the code handling the error"  type of error.

The right way to catch this is to write tests for your code and run  
them before deploying it to production :).


Yes, we're working on it but it's a large code base and we started  
with exactly zero tests.  While that leaves infinite room for  
improvement, it's a little overwhelming.  Oh well, at least we know  
where to concentrate first ;-0.


Trial will helpfully fail tests which cause exceptions to be logged,  
so you don't need to write any special extra test to make sure that  
nothing is blowing up; just test your error-handling case, and if it  
blows up you will see it.


We've just been using nose; is that something Trial handles specially  
for Twisted?



Basically, in one branch of the errBack, there was a typo.  A simple
typo that caused an unhandled NameError exception, but only once in  a
few thousand runs.

If it's a NameError, you also could have used Pyflakes to catch it :).


That's in our list of 'things to put in the commit pre-hook' as well.   
I'm not sure pyflakes would have caught this one, though because it's  
a legitimate instance variable, it's just not set to something usable  
before this particular error condition comes up.



The exception got caught and "displayed" by Twisted, but it wasn't
going anyplace anyone was looking (buried under zillions of lines of
logging) and the app continues on as if nothing went wrong.

The real lesson here is that you should be paying attention to  
logged tracebacks.


There are many ways to do this.  Many operations teams running  
Twisted servers will trawl the logs with regular expressions.  Not  
my preferred way of doing it, but I'm not really an ops person :).


I'm not much on the ops end either but I guess I'm learning...

If you want to handle logged exceptions specially, for example to  
put them in a separate file, or to e-mail them to somebody, consider  
writing a log observer that checks for the isError key and does  
something special there.  You can find out more about writing log  
observers here: .


This is an area of Twisted I haven't explored at all since the code's  
all using the standard Python logging.


That's the thing about Twisted; sometimes it's hard to know whether  
the stuff that has been built into standard Python since Twisted  
'rolled their own' is a superset, a subset, or a completely different  
beast.  Logging is a good case in point.  Since we're using Python's  
logging everywhere, I wasn't even sure whether there would be an  
advantage to learning Twisted's similar system.  Twisted's trial is  
another example; we've just been using nose.  Seems like there's  
always some little extra that makes the Twisted stuff worth knowing.



What is the best way to handle programming errors like this in
deferreds so they don't slip by, unnoticed?

I'm answering a question you didn't ask, about logged errors,  
because I think it's the one you meant to ask.  The answer to the  
question you are actually asking here, i.e. "how do I handle errors  
in an errback", is quite simple: add another errback.  This is sort  
of like asking how to handle exceptions in an 'except:' block in  
Python.  For example, if you want to catch errors from this code:


try:
  foo()
except:
  oops()

you could modify it to look like this:

try:
  foo()
except:
  try:
oops()
  except:
handleOopsOops()

which is what adding another errback is like.  But, as I said: I  
don't think this is what you want, since it will only let you handle  
un-handled errors in Deferreds (not unhandled errors in, for  
example, protocols) and you will have to attach your error-handling  
callbacks everywhere (not to mention trying to guess a sane return  
value for the error-handler-error-handler.


Right, I started thinking down that infinitely nested slippery slope  
and figured there must be a better way.  I think the logging question  
you answered that I didn't ask was the one I meant ;-).


Thanks again for another enlightening answer.

S



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Steve Steiner (listsin)]

On Oct 14, 2009, at 3:22 AM, Andrew Bennetts wrote:

> Glyph Lefkowitz wrote:
> [...]
>>   I'm answering a question you didn't ask, about logged errors,  
>> because I
>>   think it's the one you meant to ask.  The answer to the question  
>> you are
>>   actually asking here, i.e. "how do I handle errors in an  
>> errback", is
>>   quite simple: add another errback.  This is sort of like asking  
>> how to
>>   handle exceptions in an 'except:' block in Python.  For example,  
>> if you
> [...]
>
> To think about it another way, what happens when an unhandled  
> exception occurs
> in a thread?  The thread dies, and Python simply throws the  
> traceback at stderr.
> There's not much else it can reasonably do.  If you want to catch  
> (or suppress)
> that error, you add (another) try/except at the outermost layer of  
> the call
> stack.

Yah, I started thinking about how far out I'd have to start the try/ 
except and started to get a headache when I thought about how much  
code it would have to encompass or, with an alternate approach, how  
many zillions of little extra blocks I'd have to insert everywhere.

> Possibly Twisted could provide some sort of last-ditch error  
> reporting hook,
> like sys.excepthook in core Python, but I'm not sure it would offer  
> much
> advantage over writing a log observer that looks for events where  
> isError is
> set.

I'm going to use the log observer option with tail -f in a separate  
console window on a linux console that's just sitting there doing  
nothing anyway.  That way, if it shows up on that terminal, something  
bad is happening.

Thanks!

S


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Performance

[Laurens Van Houtven]

On Wed, Oct 14, 2009 at 4:44 AM, Daniel Griffin  wrote:
> Hi,
>
> I am new to twisted and have been having trouble finding out information
> about twisted's performance. I have a fairly simple setup where I need to
> open a bunch of TCP connections that last for varying amounts of time but
> dont do much. I have tried using threads(got GILed to death) and
> Processes(even worse). Now I am looking at either making a system to start
> the connection and send info to have the remote point "phone home" when its
> done, then closing the connection or using something like Twisted.
>
> my socket conversation:
> my app -> send a message that triggers an action on the other end
> other end -> recv's message does action(can take any amount of time)
> other end - > sends results back to my app

Sure, that's a blueprint for pretty much every Twisted app. You just
need to make the thing that takes a long time not block the reactor.
How you do that mostly depends on what the "long thing" is.

Usually this means using an existing library, sometimes it means
writing your own, and for some unfortunate cases it means deferring to
a thread. (Note that deferring to a thread to fake non-blocking IO is
not anywhere near as bad as doing actual _work_ in threads ;-))

Personally I'm a big fan of this series of blog posts:
http://krondo.com/blog/?page_id=1327

And you should probably read the finger tutorial and the deferreds tutorial :-)

> Can twisted handle up to several hundred connections like this? Is there a
> better approach? Is there anything I should avoid?

Depends how much work you're doing in the Twisted process or on the
same box, of course. You're probably looking at several thousand
rather than several hundred, depending on how much work needs to be
done in each one.

> Thanks,
>
> Dan

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?

[Andrew Bennetts]

Glyph Lefkowitz wrote:
[...]
>I'm answering a question you didn't ask, about logged errors, because I
>think it's the one you meant to ask.  The answer to the question you are
>actually asking here, i.e. "how do I handle errors in an errback", is
>quite simple: add another errback.  This is sort of like asking how to
>handle exceptions in an 'except:' block in Python.  For example, if you
[...]

To think about it another way, what happens when an unhandled exception occurs
in a thread?  The thread dies, and Python simply throws the traceback at stderr.
There's not much else it can reasonably do.  If you want to catch (or suppress)
that error, you add (another) try/except at the outermost layer of the call
stack.

Possibly Twisted could provide some sort of last-ditch error reporting hook,
like sys.excepthook in core Python, but I'm not sure it would offer much
advantage over writing a log observer that looks for events where isError is
set.

-Andrew.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python