Re: [Twisted-Python] logging
I'm surprised that a thread with 32 messages about logging doesn't seem to have once mentioned windows events, osx structured syslog, or systemd journal as important design points. Maybe people are thinking about such things in the background but it looks a lot like this is being designed in a vacuum when there's plenty of air around. And, no sane sysadmin should ever want a twisted-specific log file format or to write custom python log filters. That's crazy. Gimme a verbosity knob and the ability to emit structured log events to existing systems, with a fallback plain text file format. Great. The prime goal, it seems to me, should be exposing features useful for facilities present in existing log systems. And having a logging system which doesn't even support a basic log level is just silly. Hopefully the new system can at least have that.___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A new & improved IResource
On Jul 24, 2011, at 11:36 AM, Laurens Van Houtven wrote: > I've been looking at this again recently and honestly it's beginning to feel > like I'm reimplementing large parts of web2. Does that mean I'm doing it > wrong? Either that or that web2 was doing it right... Of course, you don't really need to reimplement it, just copy the existing implementation from before it was removed, and s/web2/web/. :) J ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Ways to register stuff only done for backwards compatibility
On Jul 5, 2011, at 8:32 AM, Laurens Van Houtven wrote: > In doing twisted.positioning I find my self writing a bunch of code in ways I > would ordinarily write it differently, because we have to support 2.4 still > (when is that going away? Isn't the most recent RHEL 2.6 already?). > > Is there some way to register that so that as soon as we stop supporting 2.4, > we can make a lot of code a lot prettier? For certain functions such as > any/all, perhaps a twisted.python._backports (with the explicit mention that > code in backports will go away as soon as the version it's built to work > around is no longer supported). That way, as soon as you support 2.5 (which > has any/all), you just remove it from _backports, see which tests break, > remove the imports, run tests again, commit. Woo! > > Of course, _backports is obviously not a solution for everything, since not > every language feature can be fixed by defining a class or a function > somewhere. Yes, we've done that many times before, in twisted.python.compat. (that should probably be an _ module now; it has always been treated as such, but it's not named appropriately.) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] server issues; SVN in read-only, but Trac isn't
On Jul 4, 2011, at 7:06 PM, Glyph Lefkowitz wrote: > Hello from the Twisted server operations team, > > The Twisted SVN server has run into some minor unexpected trouble during > routine system maintenance. For now, SVN is in read-only mode. However, > this shouldn't affect Trac, so feel free to keep doing reviews and submitting > patches in the meanwhile. Hopefully many of you are working from DVCS > mirrors and will hardly notice the interruption :-). > > We will send another notice as soon as things are back to normal. I believe it should be functional now. Please advise if anything seems broken. :) James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Refactoring Documentation
> On 21 January 2011 22:35, Glyph Lefkowitz wrote: >> �I don't believe we have a mirror on >> github, but maybe somebody could correct me. > > There are 79 repos on github that match the word twisted. > > This one claims to be updated every 15 minutes and seems to be pretty > up to date (has a commit from 13 hours ago) > > https://github.com/powdahound/twisted Here's what might pass for an "official" git mirror of Twisted svn: http://repo.or.cz/w/twisted.git It seems to have diverged a while back from powdahound's github one, due to what looks like a bogus commit in powdahound's. (I do need to go back and redo the "releases" branches/tags at some point though; they're not being copied at the right level.) James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Question about SerialPort
On Jan 20, 2011, at 6:32 AM, Albert Brandl wrote: > The main problem seems to be the specification of Modbus RTU. RTU > messages are separated by time gaps that are at least 3.5 characters > long. Unfortunately, this is the only reliable way for separating > messages - there is no length header or something similar. The > specification does define a trailing CRC, though. I don't think you'll ever be able to do this reliably in Python. At 9600 baud, 3.5 characters is just about 275hz (9600/10/3.5). Linux schedules process at around 500hz. So, unless you use real-time scheduling, I think you have no guarantee that you'll be able to detect such a gap at all -- some other process may be running and you may completely miss it. And you can't reasonably use real-time scheduling in python... On the other hand, are you sure you need this? http://www.libmodbus.org/ seems to have code which figures out what length packets should be, and no code to detect a 3.5char delay between packets. That seems much more reliable. It also doesn't seem to have any code to ensure a 3.5char between sent packets, which indicates to me that either it's totally brokem or else that not even the hardware end actually uses a 3.5char delay to determine packet boundaries...I'd guess the second, since even on an embedded CPU, it's much trickier to ensure the appropriate serial port timing than to simply count the correct number of bytes. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Handling of NMEA messages
On Dec 1, 2010, at 9:33 AM, Albert Brandl wrote: >> You may also want to look at >> https://code.launchpad.net/~lvh/twisted/positioning and see if it >> influences your plans at all. It would be nice if we eventually >> merged that code. :) > > It's kind of hard to understand what this repository is about. The > README file looks like a copy of the README for Twisted 8.2.0 ;-). Right: it's a branch of Twisted. You can look at the changelog and see what changed. The interesting parts of that branch are all in the twisted/positioning directory, though: http://bazaar.launchpad.net/%7Elvh/twisted/positioning/files/head%3A/twisted/positioning/ James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] signalfd
On Nov 17, 2010, at 10:40 AM, Neal Becker wrote: > One more piece seems available: > > http://code.google.com/p/python-atfork/ Nope, that's not a "real" atfork: it only monkey-patches python's os.fork() function, so it doesn't catch any fork done by C code. So it's not good enough. (but of course, as mentioned previously, pthread_atfork itself isn't good enough either). See also a comment I wrote on lwn.net recently: http://lwn.net/Articles/415684/ James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] signalfd
On Nov 3, 2010, at 10:23 PM, exar...@twistedmatrix.com wrote: > On 01:45 am, ndbeck...@gmail.com wrote: >> On modern linux, signalfd can be used to convert sigchld into an event >> on a >> file descriptor. Looks like just what is wanted for processProtocol. > > It does, indeed. I'm not sure it's really worth bothering though. Here > are the drawbacks of using signalfd(): > > - It only works on Linux > - It works differently on older versions of Linux than on newer > versions > - You still can't have another SIGCHLD handler installed that will > called (so it's not any more cooperative with other library code) Another drawback is that spawned subprocesses will start with SIGCHLD blocked (because the sigprocmask is inherited across fork/exec, like SIG_IGN is, but unlike signal handlers), unless twisted adds a pthread_atfork handler (which there is no way to do from Python, so that'd have to be more custom C code). And even with that, system() would still remain broken, because system() inexplicably doesn't call atfork handlers on linux/glibc (notabug according to glibc maintainers). Of course, the Twisted process API could unblock SIGCHLD itself before calling exec, but this would leave spawning a process with any other API (subprocess/etc) broken. Many programs will not react well to starting with SIGCHLD blocked. Since Twisted just recently managed to make spawning subprocesses with non-Twisted APIs actually work right, it'd be quite poor to break that again. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] _twistd_unix.py / daemonize() / OSX / USING_FORK_WITHOUT_EXEC_IS_NOT_SUPPORTED_BY_FILE_MANAGER
On Oct 13, 2010, at 12:05 PM, Erik van Blokland wrote:
>
> On Oct 13, 2010, at 5:52 PM, Phil Mayers wrote:
>
>> Other info indicates this is Apple trying to be clever and "protect"
>> applications from some signal handling issues surrounding fork() with
>> libraries which aren't safe in that situation; presumably these are
>> MacOS-specific libraries?
>>
>> If this really is true, then MacOS X is no longer posix-compliant in
>> those circumstances, and Twisted is going to need some work :o(
>>
>> Can you share more details about the application; the context in which
>> it is started, and which system libraries it's likely to load?
>
> Here's a compact version. The script uses OSX' Quartz to create and
> manipulate images. Twisted is used to wrap it as a webserver, images are
> served. More pointers on how to start and use it in the script itself.
You need to avoid using or importing any OSX APIs until after the daemonization
has occurred. Unfortunately, twisted executes the entire script file before
daemonizing. [that's unfortunate for other reasons besides this, too]
Here's a corrected version of your script which works properly. It defers
importing Quartz until the reactor is running, by moving it into a function
called by reactor.callWhenRunning().
#!/usr/bin/python
import objc
objc.setVerbose(True)
import os, sys
"""
This should test the interaction between a OSX Quartz call
and twisted.web.
To run, call the twistd binary with this script.
In order to establish that the image generation works, independently of twisted,
this script will generate an image first, and print some results.
Then twisted will start and do its thing.
This starts the process as a daemon, and will cause a crash on 10.6 flagged with
USING_FORK_WITHOUT_EXEC_IS_NOT_SUPPORTED_BY_FILE_MANAGER
/usr/bin/twistd -y twisted.quartz.test.py --pidfile quartztestpid.txt --logfile quartztest.log
When the script is called with the no daemon option:
/usr/bin/twistd -n -y twisted.quartz.test.py --pidfile quartztestpid.txt --logfile quartztest.log
...the server works. Images are generated.
Then talk to the server in a webbrowser:
http:127.0.0.1:8081
e...@letterror.com
"""
import sys
from random import random
from twisted.web import resource, server
from twisted.internet import reactor
from twisted.application import service, internet
import time
def getTestImagePath():
p = os.path.join(os.getcwd(), "LayoutTest1.png")
print os.path.exists(p)
return p
def _makeContext(size):
width, height = size
data = None
bitsPerComponent = 8
numComponents = 4
bestByteAlignment = 16
bytesPerRow = width * (bitsPerComponent / 8) * numComponents
bytesPerRow = ((bytesPerRow + (bestByteAlignment - 1)) & ~(bestByteAlignment - 1))
bitmapInfo = Quartz.kCGImageAlphaPremultipliedLast
data = None
context = Quartz.CGBitmapContextCreate(data, width, height, 8, bytesPerRow, rasterColorSpace, bitmapInfo)
return context
def _drawTestData(context):
steps = 10
size = 10
for x in range(steps):
for y in range(steps):
r, g, b, a = random(), random(), 0, 0.5
color = Quartz.CGColorCreate(rasterColorSpace, (r, g, b, a))
Quartz.CGContextSetFillColorWithColor(context, color)
Quartz.CGContextFillRect(context, Quartz.CGRectMake(x*size, y*size, size, size))
def _drawImage(context):
#path = u"/Users/erik/Develop/lettersetter/trunk/lettersetter/TestData/LayoutTest1.png"
#path = u"http://www.python.org/images/python-logo.gif";
path = getTestImagePath()
print "_drawImage step 1", os.getpid()
url = Quartz.CFURLCreateWithFileSystemPath(None, path, Quartz.kCFURLPOSIXPathStyle, False)
print "_drawImage step 2", url, type(url)
imageSource = Quartz.CGImageSourceCreateWithURL(url, {Quartz.kCGImageSourceTypeIdentifierHint: "png"})
print "_drawImage step 3", imageSource
image = Quartz.CGImageSourceCreateImageAtIndex(imageSource, 0, {})
w = Quartz.CGImageGetWidth(image)
h = Quartz.CGImageGetHeight(image)
x, y = (10,10)
rect = Quartz.CGRectMake(x, y, w, h)
print "_drawImage step 4", x, y, w, h
Quartz.CGContextDrawImage(context, ((x, y), (w, h)), image)
def _makeTestData(context):
data = Quartz.CFDataCreateMutable(None, 0)
dataConsumer = Quartz.CGDataConsumerCreateWithCFData(data)
fileType = "public.png"
imageDestination = Quartz.CGImageDestinationCreateWithDataConsumer(dataConsumer, fileType, 1, None)
image = Quartz.CGBitmapContextCreateImage(context)
properties = {
Quartz.kCGImagePropertyDPIWidth : 72,
Quartz.kCGImagePropertyDPIHeight : 72,
}
Quartz.CGImageDestinationAddImage(imageDestination, image, properties)
Quartz.CGImageDestinationFinalize(imageDestination)
# still a CFMutableData Reference, it needs to be a string
return str(data)
def b
Re: [Twisted-Python] reactor.spawnProcess() and file descriptors
On Jun 28, 2010, at 9:03 AM, Itamar Turner-Trauring wrote:
> On Mon, 2010-06-28 at 13:21 +0100, Carlos Valiente wrote:
>> I wanted to skip the calls to close()
>> -- 1024 of them, in my case -- because, according to strace(1),
>> that
>> takes about 100 ms (and I'd like to save those milliseconds).
>
> Ah, I see. So the issue is performance. If you do file a ticket (and
> you
> probably should), it should be about the speed issue, with suggested
> solutions attached, so people know the underlying issue you're
> trying to
> solve.
>
> It's quite likely that closing could be sufficiently sped up by
> having a
> tiny C extension do it -- less Python overhead, and you don't have to
> raise and catch 1000 exceptions (which is fairly expensive).
Or, if just listdir("/proc/%d/fd/" % getpid()). If that doesn't fail
(aka /proc/pid/fd actually exists and is mounted on this OS) then only
close the listed fds. That'll be much much faster.
James
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Review process, news fragments
On Jun 20, 2010, at 6:26 PM, exar...@twistedmatrix.com wrote: > Obviously this is English prose we're talking about here, and > there's a > lot of room for variation. I don't want to impose a *strict* style > guide here. I do want the fragments to all aim for roughly the same > level of detail, and they should all at least make sense to a reader > who > hasn't looked at the history of the relevant ticket. And where it > makes > sense, I'd like the presentation of the information (like the age of > the > deprecation on an API being removed or changed) to be uniform. I think the only real way to ensure this is to have a human edit the information after it's concatenated together, before release. Having pre-written news fragments from all the authors of the changes makes that task a lot easier, but I think you really need to see everything together in one file and have a single person apply some editorial judgement to the result to have it come out well. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Punching a firewall with (or without) Twisted - the plot thickens
On Jun 17, 2010, at 3:16 PM, Zooko O'Whielacronx wrote: > Some > people think there aren't enough routers that implement UPnP well > enough to make it worthwhile Doesn't *every* home router purchased in the last 5 years support one of UPnP or NAT-PMP? That's been my experience, at least. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Punching a firewall with Twisted?
On Jun 17, 2010, at 12:59 PM, Maarten ter Huurne wrote: > Maybe it uses UPNP to tell the router which ports to open? > > As far as I know, there are no hole-punching techniques for TCP like > there > are for UDP. So the only way to get a port forward is to configure the > router, manually or through an API like UPNP. Yes, many bittorrent clients use UPNP and/or NAT-PMP to open a hole in the firewall. This will work for most users of home routers, but not other NAT scenarios. I'm just waiting for the kernel patch which makes socket()/bind() automatically send a UPNP/NAT-PMP to the router to open the port. :) James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Test, plz ignore
test mail. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SSHSessionProcessProtocol.inConnectionLost() behavior and Git over Conch
On Mar 4, 2010, at 11:51 PM, Michael Hudson-Doyle wrote: > Maybe > it's a simple logic error and it method should be > 'outConnectionClosed' instead? I'd agree with that. Clearly it should not be sending EOF there for the server process closing its stdin. One more detail though: you actually need to wait for both stdout and stderr to close, not just stdout. [[[Sidenote: There *is* no standard SSH protocol message for closing your stdin. But if openssh 5.1 or later connects to an openssh 5.1 or later server, it will use the nonstandard e...@openssh.org command to indicate a closed stdin. Thus, on standard ssh servers/clients, the command: ssh hostname yes | true will hang forever. True closed its stdin by virtue of exiting, but that can't get propagated across the ssh link. So, yes never gets a SIGPIPE, and thus never closes its stdout and the command keeps running forever. But on openssh 5.1 or greater on both ends, it will "work properly". BTW, you can't reasonably implement this extension in conch even if you wanted to, because openssh client will only send it to servers which identify themselves as openssh servers. http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL ]]] James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On Feb 23, 2010, at 10:45 PM, Tim Allen wrote: > On 02/24/2010 02:21 PM, Glyph Lefkowitz wrote: >> On Feb 21, 2010, at 11:08 PM, Jonathan Lange wrote: >>> On Sun, Feb 21, 2010 at 10:57 PM, Darren >>> Govoni wrote: Hi, This is great! Is there a list of new features or improvements for this release? I'd like to start planning my update. >>> >>> Each of the tarballs has a NEWS file. >> >> A thought for future releases: since we'd really like folks to >> download the prereleases and try them out, perhaps we should put >> the NEWS file on the web somewhere official, too, so they can see >> all the cool stuff they can try out? >> >> Here's a not-so-official link to the 10.0.0 prerelease NEWS: >> >> http://twistedmatrix.com/trac/browser/branches/release-x-4290/NEWS?format=raw > > It seems a pity that the "other fixes" bug numbers aren't URLs to the > bugs in question, but I suppose that would make them take up an awful > lot of room in plain-text. Yeah: an auto-HTMLized version to put on the website would be nice, even if the only change was to add links for the ticket numbers. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] lore2sphinx table handling?
On Jan 4, 2010, at 8:32 PM, Tim Allen wrote: > I think the List Table format is probably the easiest to maintain in > a simple text editor, followed by the Simple Table format. CSV mode > looks like it's really designed for you to keep the table in an > external > file and edit it in a spreadsheet, or regenerate it from a database, > and > while Grid Tables look pretty, actually editing them requires an Emacs > mode, or a lot of patience. But when you *do* have an emacs mode (and, really, doesn't everynoe?) it's the easiest to use. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Coding standard question: flowing long imports with parens
On Dec 21, 2009, at 12:03 AM, Jonathan Lange wrote: > Now that we no longer support Python 2.3, may we also allow imports in > the following style? > > from very.long.package import ( >bar, >foo, >) Separating "from X import" and "Y" over a newline totally breaks grepping for imported modules. I'd not use it myself, purely because of that, except perhaps when the "Y"s are functions in a module, rather than modules. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] web vs web2 clarification
On Dec 10, 2009, at 5:29 PM, Mikhail Terekhov wrote: > That is all true but it is very close to Joel's reasoning, kind of a > manager's point of view. It is too business/money oriented and doesn't > exhaust all the reasons why people write software in open source world in > particular. And what is more important it doesn't explain why they still > rewrite it from scratch sometimes and succeed? ;) > > But this became completely unrelated to this list, sorry for dragging > discussion so far. I started the rewrite because I wanted to, and believed it was the most reasonable way to achive the goal of a better HTTP implementation and API. And let's not even call it "rewrite from scratch": it was branched from the twisted.web code and used ideas previosuly explored in Nevow. And there is even backwards compatibility in there for old twisted.web.Resource classes, which worked fine for most resources (but is not absolutely 100% compatible). There were certainly some issues, both with functionality and with compatibility. (Especially with Nevow: Nevow is basically its own reimplementation of half of twisted.web, so the twisted.web2 backwards-compat code needed special-casing to work with it, and that was never completed). But where the project really went wrong is when I stopped working on it, and nobody else was interested in finishing that job. There were many reasons I stopped working on it, but I don't really want to get into it. The people actually doing the work now want to keep working within the old framework. And that's the open source way: he who does the work calls the shots. Tada. I do think twisted.web2 could have succeeded in a form substantially similar to its present state, but alas, it was not meant to be. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
On Dec 9, 2009, at 4:18 PM, Valeriy Zamarayev wrote: > I want to second Jared's point. In my case, the responses from web servers, > including the body, often end up in log files. HTML looks pretty ugly there. > Though this is a minor point for me in the otherwise great Twisted software! Essentially every web server on the internet responds with HTML to error conditions, so you should probably modify your client. As I said, I'm all for customizability, but returning HTML bodies for error pages is unambiguously the correct default. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
On Dec 9, 2009, at 11:52 AM, Jared Gisin wrote: > I’m writing a HTTP server that exposes various resources as an API. Unless > I’m missing something, twisted’s HTTP protocol implementation is in > twisted.web.http. > > The problem with this package is that it’s inexplicably wrapped up in HTML. > HTML has nothing to do with HTTP as a whole. Sure, HTML is often what HTTP > requests return, but there’s no reason why it should nor is there any RFC > that says it should. HTTP request can return anything. > > The software I’m writing is a programmatic interface. One never uses a web > browser, so things such as displaying tracebacks > (twisted.web.util.formatFailure) in HTML format is completely wrong. When > implementing HTTP, why assume the client always wants HTML. It seem > completely wrong from these modules and libraries to be so full of HTML > output. HTML output should be provided as a separate config or option for > twisted.web. In this case, why not just dump the traceback directly to the > HTTP entity-body? As a consumer of twisted.web, I should not have to battle > with the hard-coded HTML output of this library. If I want the library to > dump things in HTML output, I should have an option to tell it to do that > (and I should be able to better customize the HTML), but I should not get > HTML by default. Customizability is great, and certainly error pages should really not be hard-coded at all. However, the default of returning HTML descriptions of errors is a good default, and should remain thus. There's usually two cases of interest: 1) A program is responding to the error. In that case, it will use the HTTP result code, and not care what the content is at all. 2) A human is reading the error message. At least 99% of the time, this will be happening through a web browser. They understand HTML. I'd expect your program to be in case #1, although I don't know if you have some need to parse out the particular tracebacks for some reason? James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted webserver performance issues
On Nov 8, 2009, at 1:08 PM, James Y Knight wrote: >> >> Can we attach gdb to the process and trigger an all-threads stack >> dump when it happens? > > That's a good idea. Unfortunately, currently it seems to be working > quite smoothly. :) The intermittent long pauses were happening again, so I got a gdb "thread apply all bt". I discovered that there were two threads not in sem_wait waiting for GIL to be released. And both were waiting in nanosleep. One had the GIL held, the other had released it. The one with the GIL held was sleeping inside of sqlite3_prepare, called from pysqlite_statement_create. We were using the pysqlite shipped with python 2.5. Fortunately, the latest version of pysqlite standalone has already corrected that function to release the GIL around the call to sqlite3, so I downloaded and installed it. Hopefully that'll fix the remaining issues. I also added an Expires header to the static content (of 24hours), although I think web browsers are pretty good at not reloading referenced content frequently even without an explicit expires, so I doubt that it really matters. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Daemon processes on windows
On Nov 8, 2009, at 10:54 PM, Brian Granger wrote: Bummer, then I can't use this approach. My "server" uses reactor.spawnProcess which needs the signal handlers to be installed (SIGCHLD specifically) to work properly... do you know if it can be done without the dual thread trick. ...On windows? I don't think Windows has SIGCHLD. James___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted webserver performance issues
On Nov 8, 2009, at 9:02 AM, exar...@twistedmatrix.com wrote: > On 04:26 am, f...@fuhm.net wrote: >> When I last looked into the performance issues, I found that >> sometimes >> trac appears to block for long periods of time without releasing the >> GIL. That seems to be the core of the performance issues, currently. >> When it's responding normally, it's perfectly snappy. But, sometimes, >> it blocks for 10sec at a time. > > How were you detecting these times? I first saw it while reloading the website and watching the resource loading timeline in Safari. And then after I pulled out the static resources from Trac's threads (which seems to have helped substantially for the /usual/ case), sometimes even those would hang for 10 seconds unresponsive. So then I had a loop running curl; sleep 2 on Cube, and noticed it happening periodically. I don't know for sure it's a GIL-releasing issue, it might be that something actually hung up the twisted mainloop thread, but I'd bet 5 cookies it was one of the trac threads blocking doing something with the GIL held. I didn't get farther in the investigation than this. > Can we attach gdb to the process and trigger an all-threads stack > dump when it happens? That's a good idea. Unfortunately, currently it seems to be working quite smoothly. :) James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted webserver performance issues
When I last looked into the performance issues, I found that sometimes trac appears to block for long periods of time without releasing the GIL. That seems to be the core of the performance issues, currently. When it's responding normally, it's perfectly snappy. But, sometimes, it blocks for 10sec at a time. My next attempt to improve performance was going to be to run trac in subprocesses instead of threads. That would hopefully substantially fix the delay problem. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Phasing out old-style classes
On Oct 27, 2009, at 10:32 AM, Steve Steiner (listsin) wrote:
> Is this something that can be handled by just running the tests with
> the right switch so we can see everything that passes without the
> switch but doesn't with?
>
> Is there a "new-style" switch or just -3?
Download:
http://twistedmatrix.com/trac/browser/sandbox/foom/newstyler.py?format=txt
And, before importing anything from twisted, run:
import newstyler
newstyler.newstyle_prefix('twisted')
Or (perhaps easier), you could put those statements in your copy of
twisted/__init__.py before it does anything else.
I haven't run the tests to see what breaks; I'll leave that to you. :)
James
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Phasing out old-style classes
On Oct 25, 2009, at 1:38 AM, Glyph Lefkowitz wrote: What do you do? Well, the obvious upgrade path here is to make a class which (A) inherits from "Library" to get Twisted functionality and (B) inherits from "object" to get new-style-ness. So you go ahead and write: # in your application class Application(object, Library): pass Maybe the answer "why not" is obvious and I should already know it, but wouldn't it be more obvious to write: class Application(Library, object): pass instead. That at least has the advantage of not blowing up if Library later becomes newstyle, right? James___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Phasing out old-style classes
On Oct 8, 2009, at 7:59 PM, Mark Visser wrote: > I've been bitten a couple times by twisted's use of old-style classes. > Now that Jython is finally off the 2.2 branch, is there any real > reason > to stay backwards compatible? I don't see any reason to make this change for twisted running on the 2.x branch of CPython. Switching all the classes in twisted to newstyle before then could break some user's code or tests, and doesn't really provide any benefits as far as I can tell. For Python 3.x, oldstyle classes don't exist, so that'll happen automatically during the conversion. Apparently in PyPy, oldstyle classes are significantly slower than newstyle classes. So it might be interesting for them to force everything in twisted.* to be newstyle classes and see what breaks. :) James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using reactor with multiple processes
On Sep 16, 2009, at 12:33 PM, David Yoakley wrote: > Thanks Phil for the reference to ampoule. We will look at that > next. We are still hoping to get the parent set up in such a way > that whatever the forking :-) shared state is, it does not get > established until after the children are spawed. I'll post back > later with what we find. Phil your thoughts on this would also be > appreciated. It think it would be a nice for this to work. Importing (but not running) twisted should ideally not cause strange side effects. So if this turns out to be a problem with some twisted-internal code rather than your code, a bug report would be a good idea, and a patch causing such problematic setup to be done at reactor startup rather than import time even better. That should at least fix the case of forking before starting the reactor. For people who want to fork while the reactor is running, there would need to be API for clearing out all the timed events, open sockets, etc and starting over. Which is probably more trouble than it's worth. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] os x installer, multiple source locations
On Sep 5, 2009, at 10:08 PM, Phil Christensen wrote: > This is a frustrating partial install of Twisted that ships with OS X Not true. It was fixed soon after 10.5's release. I think in the very first patch, 10.5.1, but I could be wrong about that. > Interestingly enough, it's gone > from Snow Leopard It's not gone from Snow Leopard -- Twisted is still installed by default. Along with a bunch of other software. See: /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/ python/twisted/ James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Conch/SSH & Cisco IOS
On Sep 3, 2009, at 5:38 AM, Phil Mayers wrote: > All, > > I've been having some problems using Conch/SSH to talk to the SSH > server > on Cisco IOS (specifically the netconf subsystem) > > It seems that the IOS SSH server reacts badly to the following: > > c: syn > s: syn,ack > c: ack > c: PSH , > s: PSH > > > i.e. IOS doesn't like being bombarded with either the version string > or > KEX before it's sent its own banner. I'm surprised to hear that, given that other users have posted programs using conch that run commands against multiple Cisco routers -- and apparently those programs worked. Do you have a particularly old IOS? (Or maybe particularly new?) But if that's the case, it is clearly a bug in their ssh implementation. From http://www.ietf.org/rfc/rfc4253.txt: >Since the new client MAY immediately send additional data after its >identification string (before receiving the server's identification >string), the old protocol may already be corrupt when the client >learns that the server is old. When this happens, the client > SHOULD >close the connection to the server, and reconnect using the old >protocol. But anyhow, a patch to add a "broken-server-bug-workaround" option seems reasonable. Once you've reported the bug to Cisco, so they'll fix it at some point, that is. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] (Mis)indented Posts
On Jul 3, 2009, at 10:32 AM, Gerrat Rickert wrote:
> ...ok, I've created a new post twice on this list twice, and both
> times
> my post has been indented and placed under a completely
> different/irrelevant post
>
> Here:
> http://twistedmatrix.com/pipermail/twisted-python/2009-June/
> 019859.html
> And Here:
> http://twistedmatrix.com/pipermail/twisted-python/2009-July/
> 019902.html
>
> ...if this post hijacks a thread as well, then it will be 3 times.
>
> ...this seems to be something specific to the twisted python mailing
> list:
> http://www.mail-archive.com/mailman-users%40python.org/msg53996.html
I really don't know how that could be specific to the twisted python
list. I notice in the referenced message, Mark Sapiro says that
rebuilding the archives from the mbox file generated correct threading
for him. Well, I just did that, ("mmarch --wipe twisted-python") and
it generated the same incorrect bad threading.
Perhaps it's a bug in mailman 2.1.9-9ubuntu1 which has been fixed
already upstream?
James
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] python properties in twisted don't work
On Dec 18, 2008, at 3:06 PM, Tristan Seligmann wrote: * Alec Flett [2008-12-18 11:03:48 -0800]: On Dec 18, 2008, at 6:04 AM, Drew Smathers wrote: Protocol is an old-style class - doesn't inherit from object - so property won't work in that context. This won't be a problem in python 3 - old-style/new-style classes are consolidated. Is there any interest in adding 'object' to some of the core classes like Deferred, in a future release of Twisted? It would help with some debugging/logging (because there's better introspection) The proble with doing this is that it is a backwards-incompatible change; doing this in a way that would comply with Twisted's backwards compatibility policy would be somewhat awkward: http://twistedmatrix.com/trac/wiki/CompatibilityPolicy The only way this could be done would be to do so in an "opt-in" fashion. E.g. something like: import twisted twisted.useNewStyleClasses(True) But even that would be global for a given instantiation of the python interpreter which isn't really very good either if you need to mix new- style-expecting and old-style-expecting classes. So it really doesn't esem worth it, given that people can mixin object themselves if they desire. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Deferred question
On Dec 9, 2008, at 4:09 AM, Gabriel Rossetti wrote: Forget it, I monkey-patched defer.Deferred to add __del__ to see if it was called, and it was, so they are getting destroyed... BTW, that is not a good way to tell if objects are being deleted. Simply the act of adding a __del__ to an object can *cause* it to be not collected, due to the way python's GC works. Luckily that was not the case here. :) A better method to tell if objects are being collected is to create a list of weakrefs (with callbacks to print a message, say) to the objects. James ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
