Re: [Twisted-Python] Forking after starting AsyncioSelectorReactor: Supported?
On Mon, Aug 27, 2018 at 09:12:57PM -0700, David Foster wrote: > So my question is, does Twisted support being forked after starting a > reactor or not? I haven't used Twisted with the AsyncioSelectorReactor on macOS myself, but the `asyncio` docs suggest[1] that the default macOS event loop uses the `kqueue` system call. Searching for "macos fork kqueue" finds a report[2] of the same behaviour in the C++ Boost asyncio library. Apple doesn't seem to publish manpages publically anymore, but the `kqueue` system call was borrowed from FreeBSD which does[3]: # The kqueue() system call creates a new kernel event queue and returns a # descriptor. The queue is not inherited by a child created with fork(2). So, my guess is that `kqueue` just can't be used with `fork` in that way. If you really need to set up a reactor and then fork, perhaps you can configure `asyncio` to use the `selectors.SelectSelector` or `selectors.PollSelector` event loops instead; they're less efficient, but they should work after a fork. This behaviour doesn't occur on Ubuntu because Linux provides the `epoll` system call instead of `kqueue`, which behaves differently. [1]: https://docs.python.org/3.6/library/asyncio-eventloops.html#mac-os-x [2]: https://svn.boost.org/trac10/ticket/3238 [3]: https://www.freebsd.org/cgi/man.cgi?query=kqueue=FreeBSD+11.2-RELEASE+and+Ports ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 16.7.0rc1 Release Candidate Announcement
On Thu, Dec 29, 2016 at 05:27:44AM -0800, Glyph Lefkowitz wrote: > Increasingly, we are assuming a pip-like packaging toolchain for > dealing with Twisted's dependencies, so getting familiar with this > stuff - pip, requirements.txt, pinning, virtualenv - is worthwhile. > (It'll make your life easier in more ways than one.) I wasn't going to mention it (because we found a workaround), but since the topic has come up... My current employer has a fully pip-based packaging toolchain for all our internal Python apps, where we build wheels for our apps and all their dependencies, and then deploy them into a virtualenv in production. For reproducability reasons, we have a PyPI mirror inside the corporate firewall, and our wheel-building Docker image has a `pip.conf` configured to point at it. Twisted 16.6 broke this system when it added a package named "incremental" to the `setup_requires` list in setup.py. Because `setup_requires` is a feature provided by setuptools, packages listed there are installed by (the same machinery as) setuptools' `ez_install` command, which does *not* respect `pip.conf`, and so it tried to download the package from upstream PyPI and timed out banging its head against the firewall. As a workaround, we messed with our build-scripts to manually "pip install" incremental before installing Twisted, or anything that depends on it, so everything's fine again. The Python Packaging Authority already has plans for replacing `setup_requires` with something more Pip-friendly (see PEP 518), so over time this won't be an issue. For the short term, though, I hope there won't be too many more things added to `setup_requires` without announcement. Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Problem with installing twisted 13.0 with Python 2.7 on 64 bit Win 7
On Tue, Jun 18, 2013 at 04:12:17PM +1000, Weikai (Victor) Xie wrote: import twisted twisted.web Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'web' I believe that's just how packages work in Python. On a Linux machine I have handy: import twisted twisted.web Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'web' import twisted.web twisted.web module 'twisted.web' from '/usr/lib64/python2.6/site-packages/twisted/web/__init__.pyc' ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] question : How to listen to USB-HID
On Fri, May 17, 2013 at 02:54:47PM +0700, b...@indoakses-online.com wrote: But I plan to replace arduino with digispark ( http://digistump.com/products/1 ) Problem is I just realize that this device will recognized as USB-HID From this page: http://digistump.com/wiki/digispark/tutorials/linuxtroubleshooting ...it sounds like the device implements the USB serial protocol (because it mentions 'KERNEL==ttyACM*' and 'tty' is what Linux calls a serial port). However, this page: http://digistump.com/wiki/digispark/tutorials/basics says The Digispark does not have a hardware serial port nor a hardware serial to USB converter. An example library (DigiUSB) is provided, as well as some example code and a serial monitor like program, and the wiki-page for the DigiUSB driver suggests you can use it to implement any USB protocol you like, including USB Serial, USB-HID, USB Mass Storage, etc. There's no (easy) way to hook a Twisted program up to an arbitrary USB-HID device; the best you could do would be to write a Twisted program that communicates with stdin/stdout, then run that program in a window and remember not to Alt-Tab to another program while your Digistump is doing stuff. It sounds like a better solution would be to make sure you use the DigiUSB library to implement the standard USB Serial protocol and then continue using your existing code. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] So... Python 3.4 is getting its own async I/O system
In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here: https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx ...while this is the PEP he's talking about: http://www.python.org/dev/peps/pep-3156/ At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators. I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Effects of backlog parameter on listenTCP
On Wed, Jul 18, 2012 at 04:43:01PM -0700, Tobias Oberstein wrote: Could someone shortly elaborate on the semantics / effect of the parameter backlog with listenTCP? It exactly corresponds to the backlog parameter of the BSD socket API function listen(int sockfd, int backlog). On my Debian Testing machine, the listen(2) manpage says: The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow. If a connection request arrives when the queue is full, the client may receive an error with an indication of ECONNREFUSED or, if the underlying protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds. ..and later: The behavior of the backlog argument on TCP sockets changed with Linux 2.2. Now it specifies the queue length for completely established sockets waiting to be accepted, instead of the number of incomplete connection requests. The maximum length of the queue for incomplete sockets can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog. When syncookies are enabled there is no logical maximum length and this setting is ignored. See tcp(7) for more information. If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. In kernels before 2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128. Obviously the Twisted documentation can't go into all this detail because Twisted runs on more operating systems than just Linux. Every OS that supports the BSD socket API should have documentation explaining the precise definition and meaning of the backlog parameter on that platform. For example, here's the MSDN docs for the listen() function in WinSock: http://msdn.microsoft.com/en-us/library/windows/desktop/ms739168%28v=vs.85%29.aspx ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] questions about twisted usage
On Thu, Mar 22, 2012 at 12:20:37PM +0200, Uri Okrent wrote: 1. Is deferToThread running the function in a real python thread? Should this be used (rather than a standard deferred) for any function that might block? Yes, deferToThread() runs things in a real Python thread. If you have code that runs very quickly, or is written in Twisted's asynchronous style (with Deferreds), you should be fine; if you have something that takes a while to run you should use deferToThread(). 2. I understand that deferreds run later. However, once a deferred (or a deferToThread) is picked up and run, does it run from start to finish? Can it be interrupted in the middle of the function? How about its callback/errback? Can another deferred jump in for processing in between a deferred and it's callback/errback, or in the middle of processing another deferred? A Deferred can wait on the result of other Deferreds; while one Deferred is waiting (say, waiting for a timer to go off, or waiting for network activity), others may be running. Each individual callback/errback function is run in its entirety, though. 3. Are there any guarantees regarding the order of execution of deferreds? (I.e., are deferreds processed in the order in which they are created?) Maybe, but whenever a Deferred waits on the result of another Deferred, you're at the mercy of whatever they're waiting for. For example, say you use Twisted to retrieve the contents of two web-pages: getPage(http://a.example.com;).addCallback(process_data) getPage(http://b.example.com;).addCallback(process_data) The request for http://a.example.com; will be launched first, but if that server takes longer to respond, process_data() might receive the response from server B first. 4. Related to #2, and #3, does it make sense to use twisted when requests that are serviced may depend on one another. For example, a client makes a request 'add-A' which is deferred (so that the server can keep processing requests), and immediately afterwards makes a request 'modify-A' (which is also run as a deferred). Can I count on add-A being done so that modify-A doesn't attempt to work on something which hasn't been created yet? In general, no. For example, if you get a request from the network that causes you to send insert into table to the database, and meanwhile get another request from the network that causes you to send update table to the database, you have no way of knowing whether one will complete before the other. Twisted's asynchronous database wrapper uses a connection pool, so the two requests might be sent down separate connections in parallel, so the database could actively prevent the update from seeing the results of the insert. If you're getting stateful requests without any kind of stateful framing (say, 'begin transaction'/'commit transaction' messages, or some kind of session ID, or something like that), you have a problem Twisted cannot help with. If you *do* have some kind of session, you can set up your server so that you have a DeferredSemaphore per session, which will ensure that the next Deferred until you've finished with the previous one. For example: ds = DeferredSemaphore() def got_message(msg): newDefer = ds.acquire() newDefer.addCallback(lambda _: processMsg(msg)) newDefer.addBoth(lambda _: ds.release()) 5. Related to all of the above. What If I want to modify a database inside a deferred? Is that incorrect usage? Specifically, if all my requests run as deferred, and they all start a transaction and the beginning, and commit the transaction at the end, will I run into problems due to context switching in the middle of deferreds? (such as one request committing for both requests, starting two transactions in a row, committing twice in a row, and so on.) As somebody else mentioned, what you want is twisted.enterprise.adbapi. It maintains a connection pool, and every database call you make (usually via the .runQuery() or .runOperation() methods) will be run in a separate connection, so .commit() or .rollback() will be run on the correct connection, and you won't have problems with cross-talk between concurrent requests. Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] questions about twisted usage
On Thu, Mar 22, 2012 at 02:34:41PM +0200, Uri Okrent wrote: Maybe I should clarify my question: thing = 0 def a(): thing += 1 thing += 1 thing += 1 thing += 1 thing += 1 def b(): thing = 0 def show_thing(): print thing a.addCallback(show_thing) b.addCallback(show_thing) given the two deferreds 'a' and 'b', with 'a' being called first, it is possible for twisted to jump to 'b' in the middle of 'a' correct? No. The situation above doesn't actually work because a and b don't return Deferreds, but during the execution of a, it's the only thing running (unless you have other threads running... but since Twisted is largely a way of avoiding thread-programming, that's not often the case). In the real world, things aren't so simple, because nobody writes Deferreds for code that can just execute in a straight line. People use Deferreds when they want to wait for some external event before going further, and you generally don't have control over the order those events occur in. For example, here we have a function that does some work, then does a deferred operation, then does some more work in the callback: thing = 0 def a(): global thing # Do some work. thing += 1 # Cause some_db to do some work on another machine, or in # another thread, or something. d = some_db.do_work(a) def add_more(res): global thing thing += 1 thing += 1 thing += 1 return res # When some_db is done, call this callback. d.addBoth(add_more) # Do more work. thing += 1 return d def b(): d = some_db.do_work(b) def clear_thing(res): global thing thing = 0 return res d.addBoth(clear_thing) return d The pattern of events would go like this: - Initially, thing is 0. - Somebody calls a(), which increments thing twice (under the comment do some work and do more work, and launches a database call. - thing is now 2. - Somebody calls b(), which launches a different database call. - Sometime later, the database gets back to us with an answer, and either a or b's callback will be fired first. - If a's callback is fired before b's callback, thing will be incremented three more times then (some time later) set to 0. - IF b's callback is fired before a's callback, thing will be set to 0, then (some time later) incremented 3 more times, resulting in 3. So the only possible results are 0 and 3, because the 'synchronisation points' are at the edges of callbacks. There's no way to get a result of 1 or 2, because a callback can't be interrupted while it's running (again, unless threads are involved). Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Conch text attribute flattening quirks
On Wed, Feb 22, 2012 at 01:47:46PM +0200, Jonathan Jacobs wrote: from twisted.conch.insults.helper import CharacterAttribute from twisted.conch.insults.text import flatten, attributes as A flatten(A.normal['hello', A.bold[' world '], 'quux'], CharacterAttribute()) 'hello\x1b[1m world quux' My expectations are that only world will be marked up with bold attributes, since it is the only piece of content in the bold attribute, while hello and quux both appear in normal text, i.e. without any additional markup. Looking at the output you can see that hello appears as normal text and then world quux appears in bold. Since it's in twisted.conch, I'm guessing that this character-attribute stuff is designed to model the VT100 character attribute system, rather than some generic tree-of-strings-and-attributes. For example, somebody used to the way HTML works might want to nest bold and italics like this: ihello bworld/b quux/i However, to achieve the same result on a traditional terminal (and using the tput(1) command to produce the formatting codes), you'd have to do something like this: tput sitm # enable italics echo -n hello tput bold # enable bold echo -n world tput sgr0 # disable all special attributes tput sitm # enable italics again echo quux ...that is, there's no code for 'end bold' or 'end italics' (or blink, dim, underline, invisible, any kind of colouring, etc.) just an 'end all special attributes' code. Therefore it's reasonable for conch's helper library to not handle nested formatting, since no terminal program will produce such a thing, because it's impossible to represent in the VT100/VT200 formatting language. I guess an argument could be made that the helper function should track which attributes are enabled at any particular point in the string, and calculate the correct sequence of disable-everything/re-enable-the- remaining-attributes codes, but evidently nobody's needed such a thing before. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Question : Combining wokel XMPP and twisted.internet.serial
On Fri, Dec 23, 2011 at 11:09:32AM +0700, bino oetomo wrote: the print self.writethis do as told but the self.serial.write(self.writethis) line raise an exceptions --START- File ./mytac01.tac, line 44, in onMessage self.serial.write(self.writethis) File /usr/lib/python2.6/dist-packages/twisted/internet/abstract.py, line 191, in write self._tempDataLen += len(data) exceptions.TypeError: object of type 'Element' has no len() -- It looks like self.writethis is an Element, not a string. Python's print statement will automatically call str() on things before it prints them, but Twisted's .write() methods do not. You'll have to change your code to something like this: self.serial.write(str(self.writethis)) Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Log output formatting (was Re: logging question)
On Wed, Nov 30, 2011 at 01:04:27PM -, exar...@twistedmatrix.com wrote: On 04:07 am, screwt...@froup.com wrote: If the standard Twisted logging functions automatically constructed LogMessage instances from dict instances, it should be easy enough for future ILogObserver implementations to do the right thing (by just calling str(msg)). Of course, they could also do more sophisticated things like pulling particular keys out of the message to set observer-specific message properties (like syslog channel and severity, etc.) Oooo there could be a function that takes a dict intended to represent a text message and returns that message as a string. As mentioned, we already have one of those in the form of textFromEventDict(). The trouble is that everybody who writes a LogObserver needs to know that it exists, and remember to call it. If it were the __str__() method of a LogMessage object, they'd have to go out of their way to *not* do the right thing. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Log output formatting (was Re: logging question)
On Tue, Nov 29, 2011 at 10:44:21AM -0800, Don Dwiggins wrote: Looking at the source of log.py, I'm at a bit of a loss to reconcile these two forces. Certainly, the two log observers implemented there use textFromEventDict, but one could create a different observer that does things entirely differently. The best idea I can come up with is, in the documentation for msg, refer to the documentation for the chosen log observer; then, in each log observer's documentation, describe how the formatting is done, either explicitly, or by reference to textFromEventDict. (And maybe in the documentation for ILogObserver, recommend strongly that each implementation be explicit about message formatting.) There should probably also be something in the logging howto. Any better suggestions? While not every log observer needs to flatten a log event dict into a string, that particular approach is probably common enough that it deserves a simple solution. I think, in my ideal world, there would be a LogMessage class that inherits from dict, with a __str__ method that looks something like: def __str__(self): if msg in self: return self[msg] elif format in self: return self[format] % self else return dict.__str__(self) If the standard Twisted logging functions automatically constructed LogMessage instances from dict instances, it should be easy enough for future ILogObserver implementations to do the right thing (by just calling str(msg)). Of course, they could also do more sophisticated things like pulling particular keys out of the message to set observer-specific message properties (like syslog channel and severity, etc.) The documentation for the standard Twisted logging functions could then just point to the LogMessage documentation to describe what goes into a log event and how it gets interpreted, with a footnote to the effect that different log observers can handle the same message in different ways. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Mon, Nov 14, 2011 at 01:33:45PM -0500, Glyph wrote: Really the most important thing here though is just to get the automatic mirroring initially set up, not the never-ending ambassadorial work. That way git users wouldn't _need_ elaborate instructions as to what to clone and how; if we just say get twisted from github and have that automatically updated it would be easier for everyone. I've done some experimentation, and (as best I can tell) it's impossible to use git-svn to create a sensible SVN → Git conversion of the Twisted repository. This is because of the various, inconsistent branch-naming schemes that have been used over Twisted's long history. As best I can tell, git-svn can import branches that exist at some specific depth in the tree... so if your branches look like this: root | +- branches | +- add-a-feature-1234 +- fix-a-bug-2345 +- release-v1.0.x +- release-v1.1.x ...you can say git svn clone --branches='branches/*' and they'll be imported as Git branches. Likewise, if you group your branches in some way: root | +- branches | +- topic-branches | | | +- add-a-feature-1234 | +- fix-a-bug-2345 | +- release-branches | +- v1.0.x +- v1.1.x ...then you can say git svn clone --branches='branches/*/*' and they'll all be imported cleanly. Unfortunately, the Twisted repository's branches directory contains subdirectories representing branches *and* subdirectories representing groups of branches... and *those* contain both branch directories and branch-group directories. Here are the branch-group directories I've discovered so far: branches/ branches/releases/ branches/releases/conch/ branches/releases/mail/ branches/releases/names/ branches/releases/words/ When I import a range of revisions that happens to include a change to a branch in branches/releases (I've been using the range 33049:33062 for testing), git-svn winds up importing the entire releases subtree as a branch, which is... suboptimal. If I use the --ignore-paths option to ignore everything under branches/releases, then it still generates the branch and faithfully records all the commits affecting it... but ignores all the files, so when you check out that branch it deletes everything in your working directory. What's really annoying about this is that git itself doesn't care about branch naming, and it's perfectly happy to have add-a-feature-1234 and releases/v1.0.x as branch names. It's just the git-svn tool not being flexible enough. A quick Google doesn't reveal much in the way of other people who have dealt with repository layouts like this, except for one report which basically went use 'svnadmin dump' to export the repository, use sed to rewrite all the paths, load the dump into a new svn repository and import from there. That sounds like a lot of work to me, and not the sort of thing that could be reliably repeated to keep the two repositories in sync. Perhaps someone should file a bug with the git-svn maintainer, if only they had a bug-tracker. So, would a repository with a huge, bogus releases branch be acceptable in an Official Mirror? It's not strictly needed for developing patches, but it *will* create lots of puzzled frowns and annoyance, perhaps from people who want a Git mirror to integrate with their automatically download new releases code. Also in the spirit of writing things down so future contributors can figure things out, I note the Git FAQ on the Git Wiki has a How do I mirror a SVN repository to git? question: https://git.wiki.kernel.org/articles/g/i/t/GitFaq_ebc3.html ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Wed, Nov 16, 2011 at 08:17:51AM -0600, Jeffrey Ollie wrote: The solution that I've come up with is to use git-svn to create a separate Git repository for each branch, and then git-push that branch into a central Git repository. This does burn *a lot* of disk space, but only one person needs to do it - everyone else can clone the central repository. The initial setup takes a while, but subsequent runs should be relatively fast. You can see the converted repository I'm building at: https://github.com/jcollie/twisted It's still in the process of building up so not all the branches are there yet. Oooh. That's looking pretty good. As you say, not all the branches are there, and none of the tags, but I can see you've already successfully imported some of the tricky release branches. I've put the script that I'm using here: https://github.com/jcollie/twisted-svn-convert One feature that I'd like to add before calling this final is converting SVN usernames to proper names/email addresses like Git prefers to use. I see your script repository already has the beginnings of such a mapping file; unfortunately you've already filled in most of the names I recognise, and I don't know if there's any other authoritative list of such information (and of course, it would need to be updated as Twisted adds new committers). The how to mirror Git to SVN section of the Git FAQ includes information on how to take a cloned git repository and add git-svn magic so you can commit to the original upstream repository; I guess that wouldn't really work with your custom import script. I think this would be a much better base for an Official Twisted Git Mirror than anything I've come up with so far. How difficult would it be for somebody unfamiliar with Git (such as, say, the Twisted core devs) to keep running? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Sun, Nov 13, 2011 at 09:20:46PM -0500, Glyph Lefkowitz wrote: On Nov 13, 2011, at 1:13 AM, Tim Allen wrote: Should I update the GitMirror page to suggest people run git svn clone -rHEAD instead of using the no-longer-updated official mirror? Perhaps the page should also be renamed? Sounds like this is a better idea than what we're advising people to do now, GitMirror page updated. I discovered a wrinkle, though: it turns out that git svn clone -rHEAD only works if the HEAD of the SVN repo happens to be a commit to trunk. If the HEAD happens to be a commit on a branch, tag, or anything else, it only downloads the contents of the branch, tries to set up the trunk and can't find it, then collapses in a heap. I've added a step that uses svn info to find the last trunk commit, so the user can substitute that revision number into the git svn clone command. I'd appreciate it if anyone with a vague familiarity with Git could read over the page and tell me if there's anything I can improve. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Mon, Nov 14, 2011 at 12:22:59AM -0500, Glyph wrote: Tim: you're right that there isn't much interest among the current Twisted core development team. But the whole point of having a Git mirror (and a presence on Github, which I think is the next step) is to attract new developers to this community. That's why the topic of this thread is Twisted Project Jobs Volunteer. We're asking for someone with expertise in this area to step up, volunteer to help maintain this mirror, which should help us attract more people who are interested in the same going forward. There is a thing here that worries me, although I'm not sure I can express it clearly without sounding crazy. Back in the days when CVS and SVN were king, the only way you could interact with a project was by submitting patches, and since only commiters interacted directly with the master repository, it didn't really matter much how branches and merges were handled. As long as you kept things in a way that made sense to you, that was OK - nobody else would see it. Since Git is a distributed VCS, suddenly everybody has access to everybody else's repositories, up close and personal, and suddenly (like when any subjective matter is discussed by a large group of people), people have *opinions* about other people's repositories, and there are *fashions* and *best practices* and all that stuff. I'm not a Github user myself, so I don't know this first hand, but I as I understand it the Github community is has even more social expectations, like pull requests and the Github issue tracker. It's very well to say make git mirror, push to Github, get new contributors, but I think there's a social impedance mismatch here that's going to cause problems, or at least make people wary because Twisted's Github project behaves weirdly and differently from other Github projects they're used to. Some examples that I can think of: - The most obvious example: in Git, when you merge a branch to trunk, you get a merge commit; in SVN (at least the way Twisted uses it) you just get an ordinary commit that squashes together all the branch commits. That's not a technical problem, but the difference between the Twisted commit graph and an ordinary commit graph is a something's not right warning. - In Git, the cultural expectation is submit a sequence of patches; Twisted generally wants all your changes in a single patch, since the divisions will be lost on merge anyway. - In Git, if I develop on a branch, then submit that branch as a patch sequence that gets applied by the repository owner, there's a good chance that when I 'git pull', git will recognise those patches as 'my' patches and can rebase my branch intelligently. When those patches have gone via git svn, they seem to have changed enough that rebasing my branch causes horrible merge conflicts. - In Git, you'd generally only create one branch for a patch series, and occasionally rebase it to keep up with trunk changes. The Twisted process seems to favour creating completely new branches (foo-1234, foo-1234-2, foo-1234-3, etc.) to handle trunk changes. - As mentioned, Github merge requests and the Github issue tracker. None of these are show stoppers, they're annoyances at worst, and could be easily explained in documentation. But who will write that documentation? Who will tirelessly explain the differences between the Git world and the Twisted world on the mailing list and #twisted, poll for merge requests and redirect them to Trac, forever? TL;DR: Github users have expectations of how to interact with projects. Twisted can make it easy for Github users to contribute by meeting those expectations, switching from SVN to Git, and changing the UQDS to match. Keeping things the way they are and adding a Git mirror is probably not going to have the desired effect without a lot of effort on somebody's part. Is that person you? You're already maintaining the wiki page - and your participation in this thread has probably taken more time than the setup would have, if you have the relevant knowledge :). Is there something you're missing? Some administrative credential, perhaps? Information about where the relevant code lives? If you're up for continuing on this, I'm sure we can get you whatever you need :). Updating a wiki-page is not terribly onerous because it's bounded: solve a problem by reading some documentation, write some more documentation, done. Being the liaison between Twisted core developers and every Git user who might want to contribute is unbounded... or at least, very large and always growing as the number of Twisted devs and Git users increases. I've not forgotten that I have/had Twisted commit access, and coming back to help on a more regular basis is definitely on my list of things to do, although it's pushed down a fair way at the moment. However, even volunteer for Twisted was right at the top of the list, I'd be a mug to sign up for such an
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Wed, Nov 09, 2011 at 12:50:23AM -0800, Glyph Lefkowitz wrote: On Nov 8, 2011, at 7:39 PM, Tim Allen wrote: As far as I know (having written most of the documentation in the linked wiki page, and from a brief skim through the git-svn manpage) it's impossible to make a shallow clone with git-svn (something like an ordinary svn checkout, or git clone --depth N), so anyone who wants to contribute to Twisted via git needs to clone the repository from scratch (potentially overloading the SVN server, although nobody seemed to notice or complain when I was doing my git-svn clone), or just copy a tarball of somebody's comprehensive, elaborate, automatic mirroring setup. This is the part I don't understand. Why doesn't 'git clone' work right in the face of svn metadata? Git has a .git directory in each repository, and expects certain files to be present within it. If there's more stuff that it doesn't know about, it just ignores it. git-svn keeps its metadata in other files that git-clone doesn't know or care about, and hence they don't get cloned. I guess there's an argument for not cloning them - just because person X has particular access rights to an SVN server doesn't mean that person Y should have them just because they cloned a repository from X. bzr-svn has some metadata it caches about the svn repository which doesn't stick around in the bzr repo or branch, but (from the performance of using it, at least) it doesn't need to stop the world and grab all of that data for most operations. In a git-svn-cloned repository, ordinary git operations are as speedy as you'd expect. git svn fetch is slow because (I think) it has to separately request each changed file from each SVN revision, and I think there's some other operations that are slow because they involve talking to the SVN server (like 'tell me what SVN properties are attached to this file'). It would be better (for most users) to point to a canonical way to get access to a git-svn clone than to document how to make one, if making one takes 24 hours :). Don't worry, it doesn't take 24 hours! It's *much* longer than that! ;) (as the wiki page states, in early 2010 and from the other side of the world it took about a week; I don't have a feel for how it changes over time) I'm not sure why that would be, except that possibly they found a tarball of somebody else's git-svn clone and forgot to update it, or they're confused about the best way to get cloned from some unofficial, no-longer updated mirror. Fixing this probably depends on having the canonical, correct, convenient instructions and advertising them widely. You edited GitMirror before, you can do it again :). I'm editing it to include what we've discussed here, as well as a few other things that I researched this afternoon and put into a reply that my MUA ate before it was sent. I believe the next step should be that somebody with the required permissions should connect to a machine on the same physical network as the SVN server and run: time git svn clone --stdlayout --prefix=svn/ \ svn+ssh://svn.twistedmatrix.com/svn/Twisted ...then check the load on the SVN server and see if it's going to be an issue to let the clone complete. If the clone completes successfully, then (a) we know about how long it takes, and (b) we have a seed repository we can potentially put up for people to download. I'd be happy to download it, check it, and write up some documentation about how people should update it. If it doesn't complete successfully, we should have some helpful error messages, adjust the clone command line and try again. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Wed, Nov 09, 2011 at 01:22:19PM -, exar...@twistedmatrix.com wrote: Why do we need to do this again? There's already http://svn.twistedmatrix.com/git/Twisted, http://svn.twistedmatrix.com/git/Twisted2.git, and http://svn.twistedmatrix.com/git/Twisted3.git. The wiki mentions the first one a couple of times, although strictly speaking you need to add /.git to the end to get a URL you can clone from. Apart from being out-of-date, it seems to be missing a lot of branches (there's only 56, and it stopped updating in 2011-03; my personal git-svn hasn't been updated since 2010-09 and has 2062 branches), it hasn't correctly imported the branches under releases, and the only tags are last_vfs_and_web2 and releases. I guess it's not strictly necessary to have tags or releases imported if the goal is to let people develop patches against trunk, but the fact that branches are missing worries me - if a Twisted committer adds your patch to a branch, you want to be confident that the new branch will actually show up when you update your repository. As for the other two, this is the first I've heard of them. Is there any documentation about how they differ, are there records of how these repositories were created and/or updated that I could add to the GitMirror wiki-page? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Wed, Nov 09, 2011 at 09:30:57AM -0500, Tom Davis wrote: It's worth noting that you aren't *required* to clone all eight trillion Twisted revisions in order to get a working clone. Doing `git svn clone -rN:HEAD url` where `N` is some revision number will only clone from that revision. Yet another useful thing I've missed in the git-svn manpage. Thanks! In fact, it seems you can just specify -rHEAD and you'll get a repo with only the latest trunk revision in it. If the resulting repository can gracefully handle updates that add new branches, and new commits on old branches that haven't been imported, this would probably be the best approach for Git-using potential contributors. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Project Jobs Volunteer
On Wed, Nov 09, 2011 at 02:49:50PM -, exar...@twistedmatrix.com wrote: On 02:18 pm, screwt...@froup.com wrote: On Wed, Nov 09, 2011 at 01:22:19PM -, exar...@twistedmatrix.com wrote: Why do we need to do this again? There's already http://svn.twistedmatrix.com/git/Twisted, http://svn.twistedmatrix.com/git/Twisted2.git, and http://svn.twistedmatrix.com/git/Twisted3.git. The wiki mentions the first one a couple of times, although strictly speaking you need to add /.git to the end to get a URL you can clone from. Apart from being out-of-date, it seems to be missing a lot of branches (there's only 56, and it stopped updating in 2011-03; my personal git-svn hasn't been updated since 2010-09 and has 2062 branches), it hasn't correctly imported the branches under releases, and the only tags are last_vfs_and_web2 and releases. It's out of date because no one who knows how to keep it updated has taken the necessary steps for that to happen. Are there other problems with it which can only be fixed by starting over? I suspect that the other problems I mentioned can only be fixed by starting over, yes. The point I am trying to convey is that performing some one-off import operation has been done before, repeatedly. What's going to be different this time that causes the clone to stay up to date? I fully expect that whatever broke previous attempts will break this new attempt too. The difference will be that since there's a record of how the new repository was created, and assuming there'll be a record of what happened when the new repository broke, this time we'll be able to reproduce the problem and work out a solution. I guess it's not strictly necessary to have tags or releases imported if the goal is to let people develop patches against trunk, but the fact that branches are missing worries me - if a Twisted committer adds your patch to a branch, you want to be confident that the new branch will actually show up when you update your repository. Indeed. Is this something that's addressed by the git command in your earlier email? The --stdlayout parameter causes git-svn to import branches in the standard SVN branches directory, which is where Twisted seems to keep its feature branches (the most important ones for people working on patches). With some effort it might be possible to import release branches in branches/releases, but that's obviously a lesser priority. As for the other two, this is the first I've heard of them. Is there any documentation about how they differ, are there records of how these repositories were created and/or updated that I could add to the GitMirror wiki-page? I have no idea where they came from or how they differ. It turns out Twisted2 is the newest (July 2011) and has about the right number of branches. It also seems to have imported the various ctrl+backslash-2371+2123 branches that I recall causing issues before. On the other hand, it still imports branches/releases as a single branch, and has no tags. Twisted3 might have been created for one of those reasons... or possibly for some other reason that's not immediately obvious. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Global reactor unit tests in the Twisted test suite
On Tue, Nov 01, 2011 at 02:48:02PM -0400, Glyph wrote: If you can think of a better solution that addresses all of these concerns simultaneously somehow, please share, I'd love to hear it :-). I'm not sure if it addresses all your concerns, but twisted.python.context will let you set a particular value for things you call and all their descendants (unless one of them sets a new value in the context). I can imagine interleaving code that passes a reactor parameter explicitly and code that grabs a reactor from the current context without much hassle. Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Running the same trial test suite against a live server and a mock server
On Mon, Sep 26, 2011 at 01:37:16PM +0200, Laurens Van Houtven wrote: The obvious solution is to have a makeClient function defined in module scope for the given test module. I could then have a bin/liveTest.py that takes a username, password and cert, and monkeypatches that module. I'm assuming it can't be hard to tell trial to just take a test module and run it... Are there any better ways to pass test data to trial? If you have a particular set of tests you want to run against a mock server and a real server, I'd be highly tempted to use inheritance: class ServerTestMixin: def _get_server_details(self): raise NotImplementedError def test_foo(self): server = self._get_server_details() ... class MockServerTest(ServerTestMixin, unittest.TestCase): def setUp(self): # start mock server def tearDown(self): # shutdown mock server. def _get_server_details(self): return MOCK_SERVER_DETAILS class LiveServerTest(ServerTestMixin, unittest.TestCase): def _get_server_details(self): return LIVE_SERVER_DETAILS The other trick I've used in a similar situation is to define ServerTestMixin and MockServerTest in a module matching test_*.py so that Trial will find it by default, and define LiveServerTest in a module like integration_tests.py that won't be automatically run. That way, you can test your code with trial and random fluctuations in the live server won't cause spurious test failures, but you can pretty easily run your tests against the live server whenever you want. If you're using continuous integration, you could have a script that fails if the unit tests break, but just warns if the integration tests break, etc. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Async-pep (again)
On Wed, Jul 13, 2011 at 02:03:03PM +0200, Laurens Van Houtven wrote: So, some of you might remember my async-pep post a while ago. Some people correctly complained there was no code or text. There's some code and quite a bit of text now. In fact, it even has a PEP number (3153)! So I'm soliciting feedback again. The idea of Protocols implementing Transports is vaguely gestured at as a Useful Thing, but not much detail is given. I think it would be useful for the final PEP to address that topic more rigorously - partially because it's good to have a firm basis on which to model SOCKS and SSH libraries, but mostly because figuring out how SSL should interact with TCP is going to give people headaches. Twisted, so far as I can see, just sort of punts and says Yeah, SSL is just another transport like TCP, but then you have to make the SSL transport support all the same options that the TCP transport supports (socket options? IPv6?), but then what if you want to run SSL over a serial port or a SOCKS connection... A! In practice, it might be simpler because SSL means whatever subset of TCP functionality we can coax OpenSSL into providing rather than a fully stackable protocol-providing-a-transport. The thing with Consumers and Producers seems... very abstract. If I'm sitting down to retrieve email via POP3 (to pick a random protocol), 'transports' and 'protocols' are tools that nestle very comfortably in my mental model of the task in front of me; consumers and producers are not. Are they concepts that should be handled by transport implementors? Protocol implementors? Protocol users? Should they be mapped onto XON/XOFF or RTS/CTS by serial transports? At least in Twisted, transports and protocols do not exist in a vacuum; they have to be hooked up via the reactor. Will this PEP define a (skeletal) API to be implemented by potential reactors, or is that going to left entirely unspecified, like WSGI? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Async-pep (again)
On Thu, Jul 14, 2011 at 10:05:00AM +0200, Laurens Van Houtven wrote: On Thu, Jul 14, 2011 at 8:48 AM, Tim Allen screwt...@froup.com wrote: The idea of Protocols implementing Transports is vaguely gestured at as a Useful Thing, but not much detail is given. I think it would be useful for the final PEP to address that topic more rigorously - partially because it's good to have a firm basis on which to model SOCKS and SSH libraries, but mostly because figuring out how SSL should interact with TCP is going to give people headaches. Twisted, so far as I can see, just sort of punts and says Yeah, SSL is just another transport like TCP, but then you have to make the SSL transport support all the same options that the TCP transport supports (socket options? IPv6?), but then what if you want to run SSL over a serial port or a SOCKS connection... A! In practice, it might be simpler because SSL means whatever subset of TCP functionality we can coax OpenSSL into providing rather than a fully stackable protocol-providing-a-transport. Cool. Can I shove those 2 paragraphs into a ticket or will the copyright monster haunt me? Go right ahead! I guess most of these things should be tickets, but I don't have a GitHub account and I'm not particularly looking to register on more websites at the moment. The thing with Consumers and Producers seems... very abstract. If I'm sitting down to retrieve email via POP3 (to pick a random protocol), 'transports' and 'protocols' are tools that nestle very comfortably in my mental model of the task in front of me; consumers and producers are not. Are they concepts that should be handled by transport implementors? Protocol implementors? Protocol users? Should they be mapped onto XON/XOFF or RTS/CTS by serial transports? Yes, Consumers and Producers are about flow control, and most Transports probably are producers. Having looked at the issues list after sending that message, I see this is basically issue 13, Why are producers/consumers important, how are they different from protocols/transports? If your PEP includes producers and consumers (and I note that the current example code doesn't, it just has a FlowControl class), you'll want to have an example Protocol that uses producers and consumers in some useful, illustrative fashion. At least in Twisted, transports and protocols do not exist in a vacuum; they have to be hooked up via the reactor. Will this PEP define a (skeletal) API to be implemented by potential reactors, or is that going to left entirely unspecified, like WSGI? Entirely unspecified, because different implementations have to do pretty different things. I guess the selection of available Transports is up to the hosting event-loop, too - it might be worth noting that in the section on Transports. Unless, of course, the Transport in question is implemented by another Protocol, in which case I guess it's anybody's guess how you might hook your Protocol up. I almost think that, for pedagogy's sake, there should be an additional Encapsulator or Framer abstract class, that inherits from Protocol, but adds a .connectProtocol() method that takes another Protocol instance, and hooks itself up as that Protocol's transport. Sure, anyone who understands what's going on should be able to figure out what's going on, but I think an extra class would make it blindingly obvious, and I like APIs that save me from having to think too hard. While there's still people listening to my half-formed opinions: - Issue 7 seems to have settled on removing support for half_closing transports. I seem to recall somebody mentioning support for half-close as being one of those weird-corner cases that nobody thinks they need until they're trying to figure out why their SSH sessions always die with broken pipe errors. While it probably would complicate the documentation to include it, I'd hope that many frameworks that implement this PEP would want to include support for half-closing transports, and it'd be nice if there was a standard API for it instead of everybody adding their own methods with their own semantics. Perhaps there could be an HalfClosableTransport(Transport) ABC, that's optional in the same way that, say, DB-API's connection.rollback() method is defined but optional. - For issue 6 (Scatter/gather IO API doesn't make sense yet), I can't see much of a use for readv/scatter, because I imagine the benefits come from having a bunch of pre-allocated buffers lying around, and even if the Python VM had such buffers, they probably wouldn't be visible or useful to running Python code. On the other hand, I can easily imagine Python code having a bunch of independently-generated buffers that need to be written in a particular order (framing data and framed data, for example), and being able to avoid ''.join() could be a big win. Again, perhaps this could be an optional extension
Re: [Twisted-Python] Async-pep (again)
On Thu, Jul 14, 2011 at 07:05:38PM +1000, Tim Allen wrote: - You might also want to create an optional Transport method to wrap the sendfile(2) and/or splice(2) functions. I suggested this not knowing whether Python would ever grow support for sendfile(), since it seemed like the sort of thing that performance-oriented async-io frameworks might want to set up with ctypes or similar. However, I've just discovered that os.sendfile() will be in Python 3.3: http://docs.python.org/dev/library/os.html#os.sendfile Since your PEP has a 3000-series number anyway, os.sendfile() might potentially be quite useful. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Moving Twisted off Trac and SVN to somewhere nicer
On Tue, Jul 05, 2011 at 09:41:12AM +0300, anatoly techtonik wrote: To know if Github is terrible or not, you need some data - examples, use cases. The first step in planning is to look at the current workflow and gather a list of ways current Trac+SVN is used and see where Github has advantages and where it suxx. Usually, people realize the latter when it's too late. As has been mentioned in earlier in this thread: http://twistedmatrix.com/trac/wiki/WorkflowRequirements (which I have updated with some of the website requirements that Glyph mentioned in one of his posts). ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Moving Twisted off Trac and SVN to somewhere nicer
On Fri, Jul 01, 2011 at 12:29:01PM +0200, Laurens Van Houtven wrote: There's a few existing hosting solutions: 1. Launchpad (+ Bazaar as the default vcs) 2. Bitbucket (+ Mercurial as the default vcs) 3. Github (+ Git as the default vcs) As a very-occasional Twisted contributor (but a long-time fan!) I'll vote for option 3. I'm not particularly a fan of Github[1], but Git is the DVCS I know best, and I'd be more than happy for Twisted Labs to make it easier for me (and people like me) to contribute. On the other hand, using git would probably complicate the build/review process: since Github repositories are (as far as I know) owned by individuals, you might not be able to set up access for multiple people, and hence the current scheme of push your changes to a branch on the central server, tell the buildbots to build it might not be possible. Tim. [1] In fact, I'm vaguely uneasy about distributed version control being so centralised on a single piece of commercially-owned infrastructure. On the other hand, the only hosted alternative would be Gitorious, and it doesn't have a lot of the shiny features Github has, like bug-tracking. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Moving Twisted off Trac and SVN to somewhere nicer
On Fri, Jul 01, 2011 at 12:48:37PM +0200, Tristan Seligmann wrote: On Fri, Jul 1, 2011 at 12:29 PM, Laurens Van Houtven _...@lvh.cc wrote: Although I've hated git for a long while (and I still don't like it very much), I firmly believe Github is the right thing for Twisted. My incredibly unscientific poll amongst people who like Twisted but aren't devs is that they all love or at least like Github, and a surprising number has a distaste for Launchpad (unfamiliarity with Bazaar, perceived developer-unfriendly UI, slowness). I'm not sure I understand what you mean by aren't devs. Do you mean aren't Twisted developers? I don't see why someone who isn't a developer would particularly care what development tools Twisted uses. Perhaps potential Twisted developers who haven't decided to contribute yet? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Conch Testing server and client
On Fri, Jul 01, 2011 at 05:44:31PM +0530, Anshul Singhle wrote: What I'm trying is : class TestAccountStatus(unittest.TestCase): Is that the Python standard library's unittest.TestCase, or twisted.trial.unittest.TestCase? As Jean-Paul says, Twisted's TestCase should automatically handle the reactor for you; that's why it exists. :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SURVEY: Have you submitted a patch to Twisted and it never got in?
On Fri, Jul 01, 2011 at 09:11:34PM +0200, Laurens Van Houtven wrote: On Fri, Jul 1, 2011 at 8:59 PM, Itamar Turner-Trauring ita...@itamarst.orgwrote: Or for that matter, you can include e.g. an github URL in the ticket instead of attaching the patch. Only if there's a decent Github mirror to fork from, otherwise you're asking people to do a multi-hour operation (I know, because I'm doing it right now) to get a decent git repo, Last time I tried (perhaps a year ago), a git-svn clone of the Twisted SVN repo took the better part of a week. I seem to recall somebody preparing a tarball of a git-svn-clone'd repository to help people bootstrap, but my clone was already completed at that point so I didn't investigate further. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Asynchronous code PEP
On Thu, Jun 02, 2011 at 12:50:33AM +0200, Laurens Van Houtven wrote: See people? Feedback works, now give me more of it please ;-) Perhaps I'm looking at the wrong place; the pep-.rst I'm looking at via the web interface contains only an short abstract and a rationale that won't be controversial to anyone subscribed to this list. There doesn't seem to be much to give feedback *on*. I guess that means my feedback is 'please write more text'. :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Plugins - Implementation Discussion
On Thu, Apr 07, 2011 at 03:24:57PM +0900, David wrote: On 04/07/2011 02:08 PM, Tim Allen wrote: If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps. Having had experience with JSON for configuration: it is a terrible format for configuration, if only because it does not support comments. The syntax is also a bit too strict: enough to be annoying in something you want to edit all the time and easily in my experience. Well, that's pretty depressing. The only other candidate I can even think of is YAML, and that's not in the standard library (as far as I know). Who'd have guessed it'd be so complicated to associate keys with values? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Persisted protocol?
On Wed, Apr 06, 2011 at 12:41:08PM +0200, Laurens Van Houtven wrote: Whoa hang on. Without trying to hijack the thread, this is the entire premise of infobarb, the IRC bot I'm building for #python-*, except s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I build it. I'm not familiar with the reasons Twistd moved from .tap to .tac (it's rather before my time), but as a user I'd be worried about things like what happens if, due to a bug, the server winds up choking on some unexpected input, or deadlocked, or just mis-files some piece of state? Restarting the process in question is a sledgehammer approach, but that's often what you want if the alternative is unscheduled downtime. The difference between Pickle and SQLite is that a SQLite database has probably had some thought put into its schema, and is much less likely to accidentally scoop up random other objects by reference. Also, if your database *does* pick up some unwanted state, you have the option of tinkering with the database manually; something that's much more difficult to do with pickles. I'd be interested in hearing from Twisted greybeards why .tap was deprecated, but I don't think your IRC bot has a fatal design flaw. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Plugins - Implementation Discussion
On Thu, Apr 07, 2011 at 10:35:18AM +1000, Stephen Thorne wrote: So the goal of my post to this mailing list is: * I would like glyph's goal of having less arbitary code executed at twistd launch time to become a realisation, * I would like the process of creating a twisted plugin to be less of a cut+paste+fill-in-blanks hassle. I notice that Tarek Ziadé's distutils2 is moving away from load a Python module and probe for well-known attributes toward define all metadata in a static file. It sounds like you want a similar thing for twistd plugins. Perhaps an implementation might look something like this: - At startup, twistd scans twisted/plugin directories on sys.path looking for files whose filenames end with '.twistd'. - Each such file is loaded with Python's ConfigParser module. - Each section in the ConfigParser module represents a plugin whose 'tapname' is the section name. - Each section has a 'description' option, whose value is a human-readable string describing the plugin. - Each section has a 'module' option, whose value is a string that can be passed to Python's __import__ builtin to get a Python module. ...where the module defined by 'module' exposes a 'make_service(options)' function, and an 'options' global variable that is an instance of t.p.usage.Options. I know you said you didn't like t.p.usage.Options, but I'd be sad to lose the ability for twistd to support twistd $PLUGIN --help, and for that kind of introspection to work, the options data needs to be in *some* known format. Maybe this might be the time to move to the stdlib's optparse - or maybe not, now that optparse is (presumably) deprecated in favour of argparse. Maybe twistd could examine a number of different well-known variable-names, for different option-parsing libraries. Just tossing this out as a strawman for people to point and laugh at. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Plugins - Implementation Discussion
On Thu, Apr 07, 2011 at 12:54:45AM -0400, Glyph Lefkowitz wrote: If we invent our own file extension which has to be separately installed, we have to teach distutils, and setuptools, and distribute, and pip, and distutils2, and 'packaging' (as I'm sure that will eventually be incompatible with distutils2 for some silly reason), and easy_install, and dpkg, and rpm, and yum, and apt, and probably five other horrible Python packaging things that I don't even know about yet, how to deal with it. So I am strongly in favor of keeping everything in .py files and just making a minor tweak to what's stored in dropin.cache (and perhaps allowing dropin.cache to be stored in some location more likely to be writable by individual users, in case the installation process doesn't update it). My understanding was that .py files have to be installed into twisted/plugins as binary blobs, not as ordinary Python modules, because of special rules like twisted/plugins must not be a Python package. If distutils/setuptools/etc. can handle a binary blob with a .py extension, I figured it could handle a binary blob with any other extension. If that's wrong, then yeah, I guess that would be a problem. - Each such file is loaded with Python's ConfigParser module. The first rule of the Twisted cabal is of course don't talk about the Twisted cabal, but the second and possibly even more important rule is no '.ini' files. I'd seriously much rather we use XML. And you can ask Stephen how he feels about XML configuration files. (Although I'd strongly recommend standing well clear of him when you do that, and making sure that no sharp or otherwise dangerous objects are within easy reach.) Well, the nice thing about ConfigParser is that it's in the stdlib, and people already know how to create them, and rolling yet-another-config-file-format seems crazy in this day and age. If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps. I don't want a solution that is hard-coded to deal with the metadata that 'twistd' specifically needs, as Twisted plugins are already used for more than just twistd plugins, and I'd like them to be used for even more. I've never actually come across anything that used Twisted plugins besides twistd, so I'd forgotten they weren't twistd-specific. So as not to make this message too long, I'll defer a description my own preferred implementation strategy for a future post to this thread. I'll look forward to it. :) Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] running several services from a single app
On Tue, Mar 29, 2011 at 09:46:11AM +0200, Aljoša Mohorović wrote: is there a way to tell reactor to run application and services defined so i can skip .tac file and just execute python file? maybe something similar to this: application = service.Application(Services) factory1 = protocol.ServerFactory() factory1.protocol = Protocol1 internet.TCPServer(8000, factory1).setServiceParent(application) factory2 = protocol.ServerFactory() factory2.protocol = Protocol2 internet.TCPServer(9000, factory2).setServiceParent(application) # howto make reactor do something like this: reactor.run(application) It's not clear from your message whether you're using internet.TCPServer() in your actual production code, or if that's just something you picked to make a simple example for the mailing-list. If you really are using internet.TCPServer(), you can just skip the Application object entirely, and call reactor.listenTCP() instead of internet.TCPServer(), then reactor.run() at the bottom. If you've written your own Server or Application subclass that you want to run, Kevin Horn's answer is probably the one you want. Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] prerelease preview predocumentation
On Sun, Mar 27, 2011 at 09:57:28PM -0400, Glyph Lefkowitz wrote: On Mar 23, 2011, at 9:34 PM, Glyph Lefkowitz wrote: http://twistedmatrix.com/~glyph/sphinx-preview-11.0pre1/ Anyone have comments about this? With all the recent excitement about the docs, I thought there would be a much more active thread here! I think it looks lovely - but then, even Kevin's first drafts looked lovely, so I didn't think that was particularly comment worthy. I hope to be able to help out with the markup and spacing nitpicking, once whoever's responsbile says Yes, this is good enough, it can be our official doc system from now on. I'm not sure if it's not finished yet, or I did something stupid, but I tried to test out the search system by picking a word I figured would be common in Twisted code (addCallbacks), and... got no hits. Even Twisted gets no hits. At least, I assume that's what's going on; there's no No results found message, but no results are displayed. Thoughts about whether we should link it from the front page? I'm not sure if a try our new documentation system link should be on the website's front page, but it's definitely the sort of thing you'd put on a mailing-list, or blog-post, or Planet Python, or your microblogging venue of choice. You know, just to get the message out that the Twisted project cares about its documentation and Things are Actually Being Done. :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Announcing Twisted 11.0.0pre1!
On Tue, Mar 22, 2011 at 08:17:25PM -0400, Jessica McKellar wrote: * a new templating system in Twisted Web, twisted.web.template, derived from Divmod Nevow. Oh boy! Are there pre-release API or other docs for this feature, or will I have to download the tarball to read the code and/or wait for 11.0.0 to be released to find out about it? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] UDP Logging Server
On Sat, Mar 12, 2011 at 02:33:45PM +0200, Pandelis Theodosiou wrote: On Sat, Mar 12, 2011 at 3:32 AM, Tim Allen screwt...@froup.com wrote: Of course, if you flush after every disk read, your program will run a bit more slowly and with more I/O... for an application where reliability is more important than performance (like logging) that's probably acceptable. You may also setup a timer that flushes files every, say, 5 minutes. One other thing I've read in the Python.org site is that flush() is not 100% sure to work immediately and should be used in combination with os.fsync(). Is there someone that can explain if that is correct? Depends what you mean by 'work'. The standard library (Python's or C's) buffers reads and writes because calling into the kernel is expensive. The kernel buffers reads and writes because disk I/O is even more expensive. flush() tells the standard library send buffered data to the kernel right now which means your data should survive if your process crashes. fsync() tells the kernel send buffered data to the disk right now, which means your data should survive if the entire machine crashes. Whether you call nothing, just flush(), or both flush() and fsync() depends on how your software balances performance versus reliability. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] UDP Logging Server
On Fri, Mar 11, 2011 at 01:15:47PM -0600, SIC FS LIST wrote:
So far I have a working implementation ... but I'm noticing that if I do
the following:
-- log when a message is received
-- that for that message it might show up in the file a pretty lengthy
period of time later
Assuming the objects stored in DISKINFO[1] etc. are file objects, you
seem to be writing to the files but never calling flush(). If you don't
call flush(), Python (well, the C standard library) won't send the data
on to the OS until its buffer is full, or the file handle is closed. If
you're not getting that many log lines, it can take a while for that to
happen.
Of course, if you flush after every disk read, your program will run
a bit more slowly and with more I/O... for an application where
reliability is more important than performance (like logging) that's
probably acceptable.
The actual UDP protocol:
class VocsLogger(DatagramProtocol):
def datagramReceived(self, data, (host, port)):
_proc_msg(self.transport, data, (host,
port))._new_msg().addCallback(handler)
_proc_msg doesn't seem to be complicated enough to need its own class,
why not just do what _proc_msg does in VocsLogger?
The _proc_msg class:
class _proc_msg:
def __init__(self, sck, data, (host, port)):
self._sck = sck
self._data = data
self._host = host
self._port = port
def _new_msg(self):
d, _ = LogMsg().ParseSocketMsg(self._data)
if d.type.upper() == DISKINFO[0]:
DISKINFO[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
elif d.type.upper() == LOADAVG[0]:
LOADAVG[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
elif d.type.upper() == MEMINFO[0]:
MEMINFO[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
elif d.type.upper() == NETDEV[0]:
NETDEV[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
elif d.type.upper() == PSAUX[0]:
PSAUX[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
elif d.type.upper() == WHOINFO[0]:
WHOINFO[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
else:
DEFAULT[1].write(d.ToString() + \n%s\n % (LOG_DELIM))
It depends on what DISKINFO[0] and DISKINFO[1] actually are, but
assuming they're a string and a file-handle, this code would look more
Pythonic as something like this:
LOGSINKS = {
DISKINFO: open(/var/log/diskinfo, w),
LOADAVG: open(/var/log/loadavg, w),
MEMINFO: open(/var/log/meminfo, w),
NETDEV: open(/var/log/netdev, w),
PSAUX: open(/var/log/psaux, w),
WHOINFO: open(/var/log/whoinfo, w),
DEFAULT: open(/var/log/default, w),
}
def _new_msg(self, data):
d, _ = LogMsg().ParseSocketMsg(data)
type = d.type.upper()
sink = LOGSINKS.get(type, LOGSINKS['DEFAULT'])
sink.write(%s\n%s\n % (d.ToString(), LOG_DELIM))
Hope that helps!
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Refactoring Documentation
On Thu, Jan 20, 2011 at 05:22:55PM -0600, Kevin Horn wrote: Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs. I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC). You mean these DTDs? twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Refactoring Documentation
On Thu, Jan 20, 2011 at 06:34:17PM -0600, Kevin Horn wrote: On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen screwt...@froup.com wrote: You mean these DTDs? twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time. Huh. Never even knew that was there. It probably could, and the reason it downloads from the internet was because that's the default way of doing it in lxml. I've since figured out how to override that behavior (which is how the caching works) so maybe that wouldn't even be hard. The easiest/fastest fix for the moment though would probably be to pre-populate the cache as I mentioned before, since IIRC, this would just involve adding the file to my hg repo. I'll have to look into it though, it may be just as easy to do it the other way, though I don't want to depend necessarily on having Twisted's code available (remember, this is supposed to work on the various divmod projects, and anything else that uses Lore, too). Well, you wouldn't necessarily be depending on Twisted, just depending on Lore - and I imagine any not-Twisted project whose documentation depends on Lore has already made peace with the idea of depending on Lore. :) If it's easier to just copy these well-known DTDs into the lore2sphinx repository, I guess that's a good plan too - it's not like the W3C is going to suddenly issue updated XHTML DTDs. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted and PEP 3148 support
On Tue, Jan 11, 2011 at 02:15:59AM -, exar...@twistedmatrix.com wrote: On 01:54 am, screwt...@froup.com wrote: - Futures support cancellation; Twisted finally managed to get rid of cancellation support in Deferreds. We only got rid of Deferred.setTimeout. In exchange, we added generalized cancellation support. Ah, yes, I misremembered. Thank you for the correction. I haven't used Deferred's cancellation support; I don't know how compatible it is with Future's cancellation support. Futures may call their callbacks in any thread. So the line: d.fail(e) must instead be something like: reactor.callFromThread(d.errback, e) PEP 3148 says: # Added callables are called in the order that they were added and are # always called in a thread belonging to the process that added them. I assumed that implied some kind of internal .callFromThread() magic; I could be wrong. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [newb] another process question
On Fri, Nov 05, 2010 at 07:28:24PM -0400, Neal Becker wrote: The status is passed in the form of a Failure instance, created with a .value that either holds a ProcessDone object if the process terminated normally (it died of natural causes instead of receiving a signal, and if the exit code was 0), or a ProcessTerminated object (with an .exitCode attribute) if something went wrong. The deal is, there are multiple kinds of 'went wrong' in the Unix process model. One kind (described the documentation you quote above) is a non-zero exit code; you can test this by running /bin/false rather than sleep 15 as your test command. Another kind is 'killed by a signal', in which case the value returned by the process encodes the number of the signal that killed it, rather than a process-defined error code. Looking at the source of the ProcessTerminated class mentioned: http://twistedmatrix.com/trac/browser/trunk/twisted/internet/error.py#L268 ...it looks like killing a process with a signal puts 'None' in the .exitCode property, and the signal number in the .signal property. The instance also has a useful str() representation if you just want to log a sensible error message. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Don't Thread On Me t-shirt proposal
On Thu, Oct 21, 2010 at 08:49:37AM +0200, Marco Giusti wrote: On Wed, Oct 20, 2010 at 08:00:54PM -0400, David Ripton wrote: On 10/20/10 17:10, Laurens Van Houtven wrote: Feedback is welcome: do you think the layout or relative sizes should be changed? Is the apostrophe being missing okay? I think the missing apostrophe looks bad. Stupid question from a non anglophone: why the apostrophe is missing? It's missing from Laurens Van Houtven's artwork because it's missing from the Gadsden's Flag artwork that is being parodied. As to why the original flag doesn't have an apostrophe... it's hard to say: http://wiki.answers.com/Q/In_the_Gadsden_and_Navy_Jack_%27Dont_Tread_on_Me%27_flags_does_the_%27don%27t%27_have_an_apostrophe ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Don't Thread On Me t-shirt proposal
On Thu, Oct 21, 2010 at 11:54:59AM +0200, Laurens Van Houtven wrote: Yep, sorry, I realise the flag is based on an exclusively American symbol. OTOH, I'm not (very) American so it still rings bell for me :-) I'm not American at all, and I think it's hilarious. About the apostrophe: yes, the reason it is currently missing is because it is missing on the Gadsen flag. That's also the reason the t-shirt is yellow. I'm getting mixed feedback on how important mimicing the Gadsen flag is, in terms of: - typeface (this is Cardo 99 SIL. Printing on T-shirts occasionally has problems with small, thin serifs on serif fonts.) I notice that Wikipedia's Gadsden Flag page includes a scan from an 1885 school-book that includes a reproduction of said flag, and it definitely uses a sans-serif typeface. Like most visual designs that have been drawn and redrawn thousands of times by hundreds of people over the centuries, I suspect a little variance is permissible. - color (although I like it because it's a Python-related shirt, and yellow isn't my favorite color) The common ingredients of a Gadsden Flag reference seem to be yellow, snake and the slogan; given how much the Twisted Matrix logo resembles the traditional rattlesnake (i.e. not very much) keeping the colour and the slogan close to the original seems important. - apostrophe (authenticity vs correctness - fortunately it's just a lexer problem in don't and not the far more expensive parser problems of your/you're/there/their/they're) I see somebody tried to add an apostrophe to the SVG version of the flag on Wikipedia, which was quickly reverted (sadly without a rationale or citation). A Google Image Search for DONT TREAD ON ME shows that both with-apostrophe and without-apostrophe variants are common, but the ones without seem more... authentic somehow. Enough bike-shedding? :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Strange error: SQL-Server tries to rollback
On Fri, Sep 10, 2010 at 06:54:34PM +0300, Pantelis Theodosiou wrote: sql = ''.join( [ INSERT INTO fts_data VALUES ( , ' , serialNumber , ' , , , ' , customerAccount , ' , , , ' , dateAndTime , ' , , , camcount , , , fpv , , , ' , reff , ' , , , ' , restOfFrame , ' , ) ] ) This is unrelated to your Twisted problem, but *please* tell me you are not building SQL like this in production code. If you need a reminder, the best possible way to do this in Python[1] is: sqlPattern = INSERT INTO fts_data VALUES ( %s, %s, %s, %s, %s, %s, %s ); # if you're using a DB-API module directly: cursor.execute(sqlPattern, [serialNumber, customerAccount, dateAndTime, camcount, fpv, reff, restOfFrame]) # if you're using ADBAPI: deferred = pool.runOperation(sqlPattern, [serialNumber, customerAccount, dateAndTime, camcount, fpv, reff, restOfFrame]) Note that there's no % formatting operator between sqlPattern and the list of values to be substituted in; the substitution is done with all the proper quoting and escaping, not with Python's naïve formatting operator. Apologies if you already knew this, but apart from correctness it's one of the few security issues where doing things the safe way is actually easier than doing it the unsafe way - for a start, the SQL is all together in one string. :) [1]: The examples here use what DB-API calls the format quoting style. Not all DB-API modules use it, but pymssql and psycopg2 do. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to bisect Twisted?
On Wed, Aug 11, 2010 at 05:30:36PM +0100, Peter Westlake wrote: In the Git version of the Twisted source code, there are tags for releases, e.g. the revision with SHA id 85a6a0f9e3c4ea22d1d72d4ca8eb0c08ee99ca85 corresponds to 8.0.0. When you say git version of the Twisted source code, do you mean the official git described here? http://twistedmatrix.com/trac/wiki/GitMirror Do tags for the various Twisted releases show up if you run: git tag -l ? This is good, but there aren't any intermediate changes on that branch (tag?), just the releases! So there isn't much for git bisect to work with. I'm trying to trace a bug that wasn't in 2.5.0 but is in 8.0.0; what arguments should I give to bisect? Apologies if this is obvious to anyone who knows how to use git, but I'm just a beginner. I haven't used the official git mirror myself, but I have my own mirror created with git-svn. It looks like git-svn got very confused about Twisted's release-tagging system; of all the tags I see listed at: http://twistedmatrix.com/trac/browser/tags ...the only thing in .git/refs/remotes/tags is the releases subdirectory, which isn't actually a proper tag at all. So, uh, yeah, I'm stumped. If you *do* figure out what to do, though, feel free to add it to the GitMirror wiki-page - or I'd be happy to write it up for you. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Email problem
On 07/13/2010 07:32 PM, Alvin Wang wrote: # that's all falks writer.lastpart() m = message.getvalue() message.close() Here, you set m to be a string containing the message contents. factory = ESMTPSenderFactory(sender, senderpw, sender, to, m, result, contextFactory=contextFactory) The documentation for ESMTPSenderFactory[1] says that the parameter after to is: file A file-like object containing the message to send. Try passing the message variable instead of your m variable. [1] http://twistedmatrix.com/documents/10.1.0/api/twisted.mail.smtp.ESMTPSenderFactory.html#__init__ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Lore to Sphinx Conversion Progress Report 6
On Sat, Jul 10, 2010 at 02:58:40AM -, exar...@twistedmatrix.com wrote: At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here: http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/ (or with a different revision number for different revisions; the containing directory is browseable). It would be great if everyone could go look around and report any problems you see. - In the Documentation Table Of Contents page, Historical Documents is listed between Twisted Core and Twisted Lore (presumably because of alphabetical order) but this seems a strange place for it. Could it be moved to the bottom, or an appendix, or something? - The bottom of every page has a Download in other formats: Plain Text link that doesn't work; presumably scooped up from the Trac templates. It should be removed (the Sphinx Show Source link is in the right-hand sidebar anyway). - When the breadcrumb navigation at the top of the page gets too long[1], it wraps to a second line, which looks kind of odd because the nice gradient background image loops. Changing the CSS background colour to be the same as the bottom of the gradient image would probably help. - Firefox's Error Console reports: - a bogus % on line 56 of twistedtrac.css, - bogus // comments on lines 559, 577, 588, 598 of trac.css (CSS only allows /* winged comments */) - Chromium's error console reports: - A 404 error for /trac/chrome/common/js/jquery.js - A 404 error for /trac/chrome/common/js/trac.js - A 404 error for /trac/chrome/common/js/search.js - A 404 error for /builds/sphinx-html-15615/_static/dots.gif - The Twisted.Conch tutorial[1] has a number of things surrounded with double-backticks like ``Deferred`` or ``ClientTransport``. I'm not sure if that's deliberate or if it's markup gone horribly wrong. - On the same page, the text :api:` twisted.internet.interface.Transport twisted.internet.interface.Transport` appears, which also looks wrong. - The Conch code examples (as linked from the code examples page[2]) are sent with a Python mime-type (which Firefox tries to download) while the .tac files are sent as text/html; this may be an artifact of the docs being hosted on the buildbot machine rather than Sphinx, but I think it would be nice if by default they were displayed in the browser, syntax-highlighted, with an option to download the original files (much like Trac does). - The Twisted Split FAQ[3] has more visible markup: twisted.protocols:superscript::ref:`[1] core-upgrades-2.0-split-protocols `. The footnote target is kind of messed up, too. - The Twisted Split FAQ[3] has a heading Why are arr the packages still named twisted.*subproject*?, but it's not obvious whether the author was trying to use italics (in which case it's a lore2sphinx bug) or globbing (in which case it's not). - The Twisted Zope Interfaces FAQ[4] has inline Python source examples that don't appear to be marked up properly (although this is possibly a problem with the original Lore source). The answer to question How can I update my own code? is one example. - The Banana Protocol Specifications[5], under the heading Examples mentions the type bytes are marked in bold but actually they're surrounded by double-asterisks. - The Twisted Coding Standard[6], mentions the complete test suite in tr...@head - and tr...@head is converted into a mailto: link. - Links followed by punctuation seem to leave a space between the link text and the punctuation. See test-driven . and Test Standard . in the Twisted Coding Standard[6], but I've observed the same problem on multiple pages. - The Twisted Coding Standard[6] mentions If you modify, or write a new, HOWTO, please read the Lore documentation to learn how to format the docs; if there's not already a follow-up ticket for the lore-sphinx conversion titled Remove references to Lore from Twisted's non-Lore-related documentation, there should be, and this should be in it. - The Twisted Coding Standard[6], under the heading Modules says Use this template: and then ../listings/new_module_template.py.. and *then* includes the content of said template inline. Presumably it should either link to it or include it, but not both. - The HTML Documentation Standard for Twisted[7] has more visible markup: in the list of allowable markup, most tags are rendered properly except for ``tr``. - It seems that the HTML Documentation Standard for Twisted[7] is actually how to write Lore documentation, despite the name. Maybe this file should be scheduled for the
Re: [Twisted-Python] Lore to Sphinx Conversion Progress Report 6
On Sat, Jul 10, 2010 at 03:17:03PM -0500, Kevin Horn wrote: On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen screwt...@froup.com wrote: That's probably enough feedback to be getting on with; most of the problems appear to be from normalising \n in Lore docs to instead of , and also from adding whitespace after things. It is generally looking pretty great, though! Yeah, that's pretty much it. As I said above though, if you fix it one place, it breaks in another, so I tried to balance things in such a way that the least broken markup appears in the output. Almost all of the remaining problems will need to be fixed manually. It sounds like a lot of the manual fixes will involve rewriting things so that inline markup does not appear at the end of a sentence next to the punctuation. That seems like a recipe for awkward prose. :/ Some extra thoughts: - The ReviewingDocumentation wiki page has a section called Editing man pages that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around? Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this. So you're suggesting convert the man pages to Lore format - use lore2sphinx to convert the resulting Lore docs to rst - build as part of the Sphinx process, yes? Yes. At least for trial, there's a bunch of stuff that's *only* in the man page and not the online docs, and a bunch of stuff that's *only* in the online docs and not the man page (and stuff that's *only* in the core/development/policy section of the docs, and not in the Trial section...). Hopefully if they're all part of the same doc system, it'll be easier to have everything in the one place and easy to find. I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket). lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around. Something to add to the open ticket for: anything else??? section of your transition plan, then? :) Thanks for the fantastic (and nicely detailed) feedback, Tim! Please take a look at the transition plan. In a few days (maybe sooner), I should have the base docs in a branch, and the chunk tickets referenced in the transition plan created. This is pretty much _exactly_ what I'd like to see in those chunk tickets. Having done pretty much the first two of your suggested chunks in my previous mail, they look to be about the right size. Hopefully you haven't already burned up your brain staring at markup issues. :) We could really use this kind of help throughout the process. I'm looking forward to learning a little more about Sphinx and ReST. :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] 10.1 release follow-up
On Tue, Jul 06, 2010 at 02:09:47PM +0100, Jonathan Lange wrote: The delay on waiting for the ticket to be reviewed happened last time also. I don't really have any ideas as to what to do about it, but I'd like to avoid the experience of sitting around for hours killing time at my computer on a weekend waiting for a review that may never come. This might be an argument for always having two people go through the release process together: firstly, so that one can write the ticket and the other can review it, and secondly to keep the release process Bus Factor above 1: http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.1.0/twisted/python/_release.py#L29 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] making Screwtape (Tim Allen) a committer
On Wed, Jun 09, 2010 at 06:25:05PM -0400, Glyph Lefkowitz wrote: I think we should give screwtape an SVN account and make him an official reviewer. He's been doing a lot of work on the distutils code which would be made easier by that, and he's been doing pretty decent reviews. Well, this is a bit of a boost to the ol' ego. :D The reason I've resisted asking for commit access before this is that most of my efforts for Twisted have been either (a) things directly useful for $EMPLOYER's Twisted deployments (such as the distutils changes you mention), or (b) things indirectly useful for the purposes of (a), such as cutting down the review queue to make it more likely that my patches will be reviewed. While my non-work-related contributions are non-zero, they're a lot less than my work-related contributions. That said, I've been following the Twisted project since (checks mailing-list archive) October 2002, and I fully intend to keep contributing reviews at the very least no matter who my employer is at the time. If you still think I'm a good candidate for committership, I'll happily accept. Tim Screwtape Allen ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Seeking advice: Coping with heavy log file loads
On Fri, Apr 09, 2010 at 05:35:51PM +0900, Paul Goins wrote: We use DailyLogFiles for file rotation along with the PythonLoggingObserver to allow for logLevel support. (Of course, this gives us significant overhead from Python's complex logging architecture; maybe we should drop this...) I don't know if this will help you, but why do you need to route through Python's logging framework for log-level support? For example, Twisted's SyslogObserver support allows the use of syslog log levels without involving Python's logging system: http://twistedmatrix.com/documents/current/api/twisted.python.syslog.SyslogObserver.html If Python's logging framework doesn't show up in your profile output as an appreciable percentage of the time spent in Twisted's logging framework, I guess you might as well leave it alone - but if it is contributing, it shouldn't be too hard to figure out a way to work around it. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] My twisted application could not start after reboot be cause of twistd.pid
On 03/16/2010 06:07 PM, Peter Cai wrote: I think an init.d script could resolve this problem. But I am curious about why turning off power without shutdown twisted server before could cause such a problem. Is it a bug or it has some more fundamental reason? When twistd starts up, it creates a .pid file as a signal that it's running. If you try and start it a second time, it notices that the .pid file exists, and refuses to run a second time. When twistd is shut down cleanly, it removes the .pid file so that it can be run again. Unfortunately, this means that if twistd is shutdown *uncleanly* (by a power failure, or SIGKILL, or some other means), the .pid file must be removed before twistd will start again. This isn't really a bug in twistd as such; almost all Unix daemons use the same fundamental system. Try running ls /var/run/*.pid on a Linux system, for example. It's possible to add refinements to the basic .pid system, to make it more resistant to this kind of problem (off the top of my head, checking that the named PID exists, that it's a process of the right kind, checking for stale .pid files with exclusive file locks), but all of them require more code to implement, reduce portability, and still have corner-cases that would require an admin to manually delete the .pid file anyway. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] http://twistedmatrix.com/ trac installation updated
On Mon, Mar 15, 2010 at 09:30:05AM -0500, Mikhail Terekhov wrote: On Sun, Mar 14, 2010 at 1:44 AM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Mar 11, 2010, at 3:05 PM, exar...@twistedmatrix.com wrote: If you notice anything broken about the website now, please point it out. Also, the spacing on the revision log and repository browser seems wrong; many table rows are double height for some reason. Still, I'm not complaining - just the fact that things paginate now has made the experience of doing triage much more pleasant! In case of FF-3.6 the rows are triple height and filled in a funny ragged way. There's a span class=expandnbsp;/span in the markup at the beginning of the first cell in each row, followed by an a tag that actually links to the directory in question. For some reason, a tags in that table have display: block (/trac/chrome/common/css/browser.css, line 50) which among other things forces them to start on a new line, after the span. If the span was removed (putting the a at the beginning of the cell), or the display:block was removed (making the a follow the span on the same line), or the nbsp; was removed (collapsing the span to zero height so it wouldn't leave a gap), things should be prettier. In addition to that the menu is overlapped in FF but this was the same way before update. BTW in IE6 menu is OK. Which menu is that? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 9.0.0?
On Thu, Mar 04, 2010 at 01:55:50PM -0500, Glyph Lefkowitz wrote: Now that our release process is better documented (THANK YOU JML), you might be able to have a look at http://twistedmatrix.com/trac/wiki/ReleaseProcess and figure out where 'setup.py upload' could fit in without uploading a tarball that is _not_ actually our official release. If setup.py upload involves setup.py sdist then I guess this is ticket #4138 again. I still plan to tackle it sometime soon, but totally would not object if anybody else wanted to dive in. :D ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On 02/24/2010 02:21 PM, Glyph Lefkowitz wrote: On Feb 21, 2010, at 11:08 PM, Jonathan Lange wrote: On Sun, Feb 21, 2010 at 10:57 PM, Darren Govonidar...@ontrenet.com wrote: Hi, This is great! Is there a list of new features or improvements for this release? I'd like to start planning my update. Each of the tarballs has a NEWS file. A thought for future releases: since we'd really like folks to download the prereleases and try them out, perhaps we should put the NEWS file on the web somewhere official, too, so they can see all the cool stuff they can try out? Here's a not-so-official link to the 10.0.0 prerelease NEWS: http://twistedmatrix.com/trac/browser/branches/release-x-4290/NEWS?format=raw It seems a pity that the other fixes bug numbers aren't URLs to the bugs in question, but I suppose that would make them take up an awful lot of room in plain-text. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
On 02/24/2010 02:56 PM, James Y Knight wrote: On Feb 23, 2010, at 10:45 PM, Tim Allen wrote: On 02/24/2010 02:21 PM, Glyph Lefkowitz wrote: Here's a not-so-official link to the 10.0.0 prerelease NEWS: http://twistedmatrix.com/trac/browser/branches/release-x-4290/NEWS?format=raw It seems a pity that the other fixes bug numbers aren't URLs to the bugs in question, but I suppose that would make them take up an awful lot of room in plain-text. Yeah: an auto-HTMLized version to put on the website would be nice, even if the only change was to add links for the ticket numbers. If the text version were Trac-markup, presumably it would be possible to automate creating a /trac/Releases/x.y.z/ReleaseNotes wiki page, and all the bug numbers would become links as per the usual Trac magic. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] lore2sphinx table handling?
On Mon, Jan 04, 2010 at 06:08:17PM -0600, Kevin Horn wrote: Any of these formats should work fine, but I'm curious as to what people (in particular the core devs) think as to which should be the preferred method in the Sphinx documentation. I'm not a core dev, but I'll chime in so that at least you'll have some feedback if nobody else does. :) I think the List Table format is probably the easiest to maintain in a simple text editor, followed by the Simple Table format. CSV mode looks like it's really designed for you to keep the table in an external file and edit it in a spreadsheet, or regenerate it from a database, and while Grid Tables look pretty, actually editing them requires an Emacs mode, or a lot of patience. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] lore2sphinx table handling?
On Mon, Jan 04, 2010 at 11:23:37PM -0500, James Y Knight wrote: On Jan 4, 2010, at 8:32 PM, Tim Allen wrote: while Grid Tables look pretty, actually editing them requires an Emacs mode, or a lot of patience. But when you *do* have an emacs mode (and, really, doesn't everynoe?) Well, no, no I don't. Does that mean I don't have to supply documentation changes with any patches I supply in future? ;) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Some news about currently-pending code-reviews
On Thu, Dec 24, 2009 at 09:41:11AM -0500, Gerrat Rickert wrote: [snip] http://twistedmatrix.com/trac/ticket/3956 Add arraysize option to runQuery in adbapi Well, as the guy who initiated this ticket, I'm certainly using adbapi.ConnectionPool with cx_Oracle. I'm not currently using any placeholders named arraysize or cp_arraysize. But you are using the keyword-parameters-as-query-parameters extension that cx_Oracle provides? This kind of informal poll *might* help us rule out using these (if someone says they're currently using them) - but it won't be very definitive; and this change will still technically be backwards-incompatible. Perhaps the only reasonable backwards-compatible change that could be made would be adding either an attribute or method to the ConnectionPool to set the arraysize. I think the two positions here would be: a: adbapi.ConnectionPool is designed to wrap DBAPI2 modules; keyword parameters to cursor.execute() are not allowed in DBAPI2; therefore adbapi.ConnectionPool can use keyword parameters for itself. b: adbapi.ConnectionPool has never really enforced DBAPI2 compliance, so people have been using it with all kinds of crazy DBAPI2 extensions and we should allow people to keep doing so as much as possible. My cunning plan (which has somewhat backfired) was that one of these alternatives would seem sane, and one would seem ridiculous, and once the mailing list decided which was which I could go back to the ticket with that decision. The way things are at the moment, I'm leaning towards (b), but I believe the developer who's worked on the patch leans towards (a) and I don't feel I have the authority to demand a change of approach. I left the ticket awaiting review, in the hope that somebody with more authority or firmer opinions would come along to review it (it's a pretty small change!), but the ticket's been sitting there for weeks now - I felt I needed to do something more drastic to help it make progress. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Decide on and document Python version support policy
On 12/24/2009 02:26 PM, exar...@twistedmatrix.com wrote: * No reasonably current releases of mainstream platforms that I'm aware of are still bound to Python 2.3. http://www.redhat.com/security/updates/errata/ RHEL4 is still bound to Python 2.3, and only enters the production 3 phase (also known as please, please upgrade) in 2009Q4, but is supported for qualified security errata of important or critical impact and selected mission critical bug fixes until February 29, 2012. I suspect most potential and current Twisted users on RHEL4 or CentOS4 are either using an existing version of Twisted that still works with Python 2.3, or already planning on upgrading to RHEL5, so Twisted drops support for old versions of Python when the associated RHEL release enters Production 3 seems a pretty reasonable policy. RHEL5 with Python 2.4 enters Production 3 no earlier than Q1 of 2012. If we're lucky, RHEL6 will come with Python 2.6 or even 2.7 rather than 2.5. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Some news about currently-pending code-reviews
First, a public service announcement about code-reviews: I'm not a Twisted Developer, I'm just a guy who submitted a patch and got talked into helping out by reviewing other people's Twisted patches while I was waiting for other people to review mine. You don't have to be a Twisted expert to help out, and don't worry about whether you're experienced enough to spot every possible problem - most patches get reviewed *at least* two or three times by different people, and every problem you pick up and the developer addresses is one less issue the next reviewer has to think about. As I've started reviewing over the past week or two, the developers whose code I've reviewed and the regulars in the #twisted IRC channel have all been super-helpful, and I highly recommend code-reviewing as a pleasant way to spend an idle afternoon. Everything you need to know to get started should be listed here: http://twistedmatrix.com/trac/wiki/ReviewProcess -- As I mentioned above, I've been doing Twisted code-reviews recently, and I've seen a few tickets in particular I thought the general Twisted community might like to know about, if not help out with. :) http://twistedmatrix.com/trac/ticket/3956 Add arraysize option to runQuery in adbapi - The arraysize property is the only writable cursor property defined by DBAPI2. Setting it can make the .fetchall() method faster with some DB modules, but Twisted's adbapi module offers no way to set it. - This ticket has a patch that adds an cp_arraysize keyword argument to ConnectionPool.runQuery() to set the .arraysize property, but this could interfere with DB modules that accept a cp_arraysize keyword parameter to cursor.execute(). - I looked up the API docs for as many Python DB modules as I could think of, and none of them attach any significance to a cp_arraysize keyword parameter. - However, at least cx_Oracle supports using keyword parameters for populating parameterised queries, like this: cursor.execute(select :name from dual;, name=Fred) So, my questions to the Twisted community are: - How many of you are using adbapi.ConnectionPool with cx_Oracle, or another DB module that supports arbitrary keyword arguments to cursor.execute? - Of those, how many of you are using parameterised queries or prepared statements with placeholders named arraysize or cp_arraysize? http://twistedmatrix.com/trac/ticket/4138 A fresh Twisted checkout should support setup.py sdist - Occasionally people ask about building RPMs of Twisted with ./setup.py bdist_rpm; this ticket is the first step in getting that working - the rest of the process is #1696. - I'm not sure if it helps with other kinds of bdist, like bdist_wininst or bdist_msi - if it does, and you've been waiting for that, then you might want to help review the code and check it works properly on your platform! http://twistedmatrix.com/trac/ticket/4004 subunit reporter. FTW. - This ticket adds another test-reporter to Twisted Trial, one that outputs results in a format that can be parsed by the tools in the third-party subunit project, https://launchpad.net/subunit - Among other things, this would help integrate Python tests with test results from other systems (tap2subunit, subunit2junitxml), compare the results of different test-runs (subunit-diff), and enable cute GUI-based test-runners (subunit2gtk). - If subunit sounds like a tool you would find useful, you might want to help review the code! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] zope.interface
On Mon, Dec 14, 2009 at 11:48:48AM +, Phil Mayers wrote: Jared Gisin wrote: Is there a good document that explains why zope.interface is necessary to the twisted project? A counter-question: Is there a good document that explains why people care? It's just another dependency isn't it? Trying to remember back to when I was a Twisted newbie, I seem to recall a pang of dismay at seeing the word zope, until I realised that zope.interface was very small and self-contained and didn't have further dependencies on the rest of Zope. Another possibility might be that the worry comes from people using systems that don't have good built-in dependency management. I don't know where pip/easy_install are at these days, but Debian's apt and Fedora's yum have numbed the painful memories of downloading some interesting code off the 'net and discovering yet further dependencies I would have to track down and install to get things working. I'm not saying these are well-researched or compelling reasons, just speculating about what might be going on. It occurs to me that recent discussions about splitting Deferreds off into their own library might consider zope.interface as a model. Has it prospered, as separate library? Does anyone besides Zope and Twisted use it? I don't know, but it might be worth finding out. Tim. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] subprocess.Popen
On Tue, Dec 15, 2009 at 02:32:40AM -0200, Aníbal Pacheco wrote: Is it possible to attach a callback to a process launched using subprocess.Popen?, I want to receive a notification on process termination. I don't believe Twisted has any special support for subprocess.Popen, but you can do much the same things with Twisted's reactor.spawnProcess(). ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Retrying function calls
On Sun, Nov 01, 2009 at 05:53:31PM +0100, Terry Jones wrote: def simpleBackoffIterator(maxResults=10, maxDelay=5.0, now=True, initDelay=0.01, incFunc=None): assert maxResults 0 remaining = maxResults delay = initDelay incFunc = incFunc or partial(mul, 2.0) if now: yield 0.0 remaining -= 1 while True: if remaining == 0: raise StopIteration yield (delay if delay maxDelay else maxDelay) delay = incFunc(delay) remaining -= 1 Since this is a generator function, it will automatically raise StopIteration once control-flow falls off the end of the function, so your while-loop could just be written: while remaining 0: yield (delay if delay maxDelay else maxDelay) delay = incFunc(delay) remaining -= 1 ...making the function of the remaining counter just a little more explicit. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to find out if exceptions are being raised in your errBack?
On Wed, Oct 14, 2009 at 07:43:41AM -0400, Steve Steiner (listsin) wrote: That's the thing about Twisted; sometimes it's hard to know whether the stuff that has been built into standard Python since Twisted 'rolled their own' is a superset, a subset, or a completely different beast. Logging is a good case in point. Since we're using Python's logging everywhere, I wasn't even sure whether there would be an advantage to learning Twisted's similar system. Twisted's trial is another example; we've just been using nose. Seems like there's always some little extra that makes the Twisted stuff worth knowing. Most of the duplicate stuff in Twisted has the excuse of being written before any alternatives were available, and usually having one or two extra little integration features that current alternatives don't provide. A year or two ago I was planning a new project using Twisted, and (as a reaction to the masses of horrible, legacy code at my employer) had decreed that the project would use the Python stdlib wherever possible, including logging and testing, and well-integrated third-party tools where necessary. First on the chopping block was the use of standard Python unit-tests and nose. At the time, I don't believe it was possible to run TestCases inheriting from twisted.trial.unittest.TestCase in anything besides Trial - and you need to use trial's TestCase class if you want to do anything with Deferreds, so nose and the standard unittest.TestCase had to go. Once I was using Trial, which automatically sets up Twisted's logging system to log to a handy file for debugging purposes, I had to decide what to do about logging - try to reroute Twisted's logging into the Python logging system (which would mean doing something complicated for tests), or just give up and use Twisted's logging everywhere. I opted for the latter route as the path of least resistance (and because it would be easier to set up with twistd later). I believe nose does for Python logging what Trial does for Twisted logging these days, and if nose can run Trial's TestCase subclasses, you might just as well stick with what you're already using. There are integration advantages to using all the Twisted goodies together, but it's not nearly the same amazing quantum leap in understanding and capability that Twisted's core provides. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] simple global counter
On Tue, Sep 15, 2009 at 04:54:09PM +0700, Artem Bokhan wrote: Tim Allen пишет: On Mon, Sep 14, 2009 at 10:09:38PM +0700, Artem Bokhan wrote: May somebody give a sample code of simple global counter which could be used with twisted (non-web) enviroment? I'm not sure what you mean. I want find out if it's possible to count things in context of the whole twisted server, not just one tcp session. Well, of course - you can create global variables in Python, and nothing in Twisted affects that. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] simple global counter
On Mon, Sep 14, 2009 at 10:09:38PM +0700, Artem Bokhan wrote: May somebody give a sample code of simple global counter which could be used with twisted (non-web) enviroment? I'm not sure what you mean. If you want to keep a count of something, you can just store it in a variable like anything else. event_counter = 0 def handle_event(event): global event_counter event.do_something() event_counter += 1 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to detect when there is nothing waiting in the event queue
On Mon, Sep 14, 2009 at 12:11:46PM -0400, Mark Visser wrote: I've added a signal handler to catch SIGHUP and call stop() on the looping call and stopListening() on the perspective broker Root. How can I detect when all remaining Deferreds have fired so I can stop the reactor safely, without stranding any running processes? I was looking at something like this for our production system a while back; so far as I can tell, the only thing you can do is to maintain a list of all 'live' deferreds as your process runs. When you get the 'stop' signal, stop accepting incoming requests, put all the deferreds into a DeferredList and don't exit until the DeferredList fires. Our system makes it rather difficult to gather all possible deferreds, and our processes only run for less than a minute, so we decided to just put up with occasional lost processes. As well as catching SIGHUP, you might want to investigate the reactor's 'addSystemEventTrigger' method, where you can queue calls to be called at shutdown, and return a deferred that will cause the shutdown to be delayed until the deferred fires. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Is there an equivalent of tap2deb?
On Fri, Sep 04, 2009 at 02:53:37PM +0100, Peter Westlake wrote: Now that .tap files are obsolete, is there a supported way of making Debian packages like tap2dep? .tap files (Pickle-based config files) may be obsolete, but I don't believe the other variants (.tac for Python-based config files, and I think there's an XML serialisation too) are obsolete. I've never used tap2deb, but twistd and tap2rpm both seem to be quite happy with .tac files in our production systems here. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Some way for Trial to allow selective running of tests?
Don Dwiggins ddwigg...@advpubtech.com wrote: Phil Christensen wrote: trial takes an argument at the command-line where you can specify a package or test case. for example: trial twisted trial twisted.test trial twisted.test.test_explorer trial twisted.test.test_explorer.TestBrowser trial twisted.test.test_explorer.TestBrowser.test_chain lets you get more and more specific. -phil Sorry, I'm feeling dense. I currently run it as trial MyServerTestFile.py; it's not a package or part of one. In the file are several classes, each with several test cases. How would I adapt the above to my needs? trial takes either a path and filename, or a Python import name (I'm not sure what the exact term here is). If you ran python in the same directory where you ran trial in your example, you could say: import MyServerTestFile MyServerTestFile.MyServerTestCase.test_my_server ...to refer to a particular test method, and trial will accept the same syntax: trial MyServerTestFile.MyServerTestCase.test_my_server In general though, a lot of distribution and packaging and deployment things become easier if your project is laid out in Python packages. I usually follow these rules and everything turns out pretty well: http://jcalderone.livejournal.com/39794.html signature.asc Description: PGP signature ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] stop conditions and deferToThread()
I'm not sure this is the problem you're facing, but... On Wed, Jun 10, 2009 at 12:04:16PM +0200, Thomas Jakobsen wrote: import time from twisted.internet import reactor from twisted.internet.threads import deferToThread running = True def foo(): while running: print working time.sleep(1) def bar(): global running time.sleep(3) print stopping running = False d1 = deferToThread(foo) d2 = deferToThread(bar) reactor.run() ...does that make things run better? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] xmlrpc - Possable failure to understand async concept
Tim Hughes thug...@thegoldfish.org wrote: Am I completly missing the point here or is there something incorrect with my code. Yes, it seems you have missed the point somewhere. def blocking_method(self, duration=5): block the instance for a specified duration started = time.asctime() time.sleep(duration) data = I started at %s and i slept %d seconds then woke at %s % (started, duration, time.asctime()) return data def blocking_method_fixed(self, duration=5): d = Deferred() d.callback(self.blocking_method(duration)) return d Your blocking_method_fixed() wraps the result of blocking_method() in a Deferred, but it doesn't actually do anything to prevent blocking_method() from blocking. Deferreds have no magical ability to prevent things from blocking on their own, they are just a tool you can use to handle some of the control-flow issues that arise when writing non-blocking code. For example, here's a method that does more or less what blocking_method() does, but in a non-blocking manner: def non_blocking_method(self, duration=5): started = time.asctime() d = Deferred() reactor.callLater(duration, d.callback, None) def do_stuff_after_timeout(result): # Here, 'result' will contain the None we passed to # callLater data = I started at %s and I slept %d seconds then woke \ at %s % (started, duration, time.asctime()) return data d.addCallback(do_stuff_after_timeout) return d See how the method does some initial preparation, then schedules a callback to be run after the long-running operation has completed. If you have a long-running network call to do instead of a simple sleep, there's probably a Twisted API or addon that will perform the call and give you back a Deferred, rather than blocking until it has an answer. If you're trying to use a third-party client library that doesn't offer a non-blocking API, about the only thing you can do is call that API in a thread, using the reactor.callInThread() method (but note that Twisted is not thread-safe, so the thing you call in a thread can't use any of Twisted's functionality except via reactor.callFromThread()) signature.asc Description: PGP signature ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] official packaging
Esteve Fernandez est...@sindominio.net wrote: Anyway, it would be great to have recent packages of Twisted for distributions that offer some kind of long term support contracts (Ubuntu, RHEL, etc.), but that tend to get a bit outdated. It's worth mentioning that someone has recently stepped up to fix ticket 1696 after I'd ignored it for Far Too Long, and very soon it should be possible to build Twisted RPMs for RHEL just by running ./setup.py bdist_rpm in a fresh checkout. For Debian-based distros, I hear there's a tool called debhelper that apparently has some useful tricks for making .deb packages from a distutils setup.py; I'm not sure how it interacts with or without python-central, and I don't know how well it would deal with Twisted's rather... customised setup.py. signature.asc Description: PGP signature ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
