Re: [Zope3-dev] Re: Re-revisiting IResult (for in-Zope pipelining)
Lennart Regebro wrote: > On 7/13/07, Tres Seaver <[EMAIL PROTECTED]> wrote: >> You can't ask "upstream" to produce a thousand different pipelineables; >> the interface there needs to be dirt simple, and *always the same.*. >> In particular, you can't return unicode *or* a pipelineable: that puts >> the policy choice in the wrong place. > > Right. Also, after further discussion of the issues, which I'll try to > put down in a sprint-report, we have concluded that the correct place > to do the theming-pipeline is most likely in the BrowserPublication, > because there you have access to the context as well as the published > object typically (being a view class). > Doesn't the published object, being a view class, have context and request as instance variables? object.context and object.request should work to access these things. Or am I missing something? -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: revisiting IResult (for in-Zope pipelining)
> I'm more concretely +1 on (a) now that I've spelled out these > options. Since no one has given a true -1 on it, I will proceed with > that, unless we get further discussion. Thanks, Gary I am looking forward to using some of this! -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Dieter Maurer wrote: > Jim Washington wrote at 2007-3-27 16:28 -0400: > >> ... >> Yes, I think so, at least in the implementation/algorithm I am using. >> There may be other implementations that do not need this. Note, >> however, that the canonical list does not have to be complex objects. >> The canonical list is just a representation of the "unsorted" state. It >> can, for example, be a proxy list of iids, list indexes, or OOBTree >> keys. The algorithm does not care what it is reordering. >> > > If you need to keep the "canonical list" around, then sort them > and then keep the sorted result around (i.e. cache the sorted list). > > This way, you could avoid the "factoradic index". > > > Yes, if there is only one ordering that will be used. Factoradic indices may be obtained and used for any arbitrary ordering of the canonical list. It may be useful, for example, to keep an index that has books sorted by title, another one sorted by author and title, and another one in publication date order. - Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Paul Winkler wrote: > On Tue, Mar 27, 2007 at 03:25:00PM -0400, Jim Washington wrote: > >> A factoradic index is representable as a long integer. Given that >> integer and the canonical list, you can regenerate the permutation >> represented by that integer. So, instead of caching the sorted list >> itself, you find and keep this integer, which is all the information >> needed to algorithmically re-obtain the sorted list. >> >> So, this would be slower than caching, but (I think) faster than re-sorting. >> > > Doesn't it mean you need the entire canonical list in memory? > > Yes, I think so, at least in the implementation/algorithm I am using. There may be other implementations that do not need this. Note, however, that the canonical list does not have to be complex objects. The canonical list is just a representation of the "unsorted" state. It can, for example, be a proxy list of iids, list indexes, or OOBTree keys. The algorithm does not care what it is reordering. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Dieter Maurer wrote: > Jim Washington wrote at 2007-3-27 08:24 -0400: > >> ... >> If you see a sort order as one permutation of a list, the factoradic >> technique provides a key to that permutation. So, in theory, one would >> sort the list, and store the factoradic index for that permutation. The >> next time the particular sort order is requested, it's a simple matter >> of unpacking the factoradic. I have not done any tests to see whether >> unpacking a factoradic is significantly less expensive than re-sorting. >> Intuitively, it should be. In practice, I am not so sure. >> > > In what way is it different from caching? > The "packed factoradic" seems like a cache to me. > > > Hi, Dieter A factoradic index is representable as a long integer. Given that integer and the canonical list, you can regenerate the permutation represented by that integer. So, instead of caching the sorted list itself, you find and keep this integer, which is all the information needed to algorithmically re-obtain the sorted list. So, this would be slower than caching, but (I think) faster than re-sorting. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Hi, Martijn I have a suggestion, only because I have played around with the idea a bit. Google for "python factoradics", and you will get my blog entry about factoradics. I see the problem statement as "How to obtain batching without re-sorting multiple times". If you see a sort order as one permutation of a list, the factoradic technique provides a key to that permutation. So, in theory, one would sort the list, and store the factoradic index for that permutation. The next time the particular sort order is requested, it's a simple matter of unpacking the factoradic. I have not done any tests to see whether unpacking a factoradic is significantly less expensive than re-sorting. Intuitively, it should be. In practice, I am not so sure. Anyway, this is FWIW. :) -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: [SpringCleaning07]
Roger Ineichen wrote: Hi Jim, It's premature to announce (we plan to have eggs on pypi soon) , but take a look at zif.xtemplate at zif.sourceforge.net . It's pretty alpha at the moment, but it uses a DTD and some xpath to get around the "tags that shouldn't be minimized" issue, and it includes a first stab at an HTML sanitizer, to use when snippets of untrusted HTML are to be included on a page. In addition, the entire page DOM is available for postprocessing right up until serialization. Of course, those with better lxml knowledge are encouraged to point out issues with the implementation. What's up with jsonserver? Did you move the package to a new repository? Can you offer a mailinglist for the zif packages? I'm very interested in observing the state of jsonserver since we use it most projects. Can you give a short statement about the zif and jsonserver repo? Hi, Roger There has been interest in a namespace for the packages I have been offering. And a public repository. And eggs. David Pratt and I have been working on this on sourceforge for less than a week, so I was definitely a bit premature in pointng anyone to the SourceForge site. Unfortunately, I just could not help saying something about xtemplate while templating was being discussed here. So, a short statement about zif and jsonserver: zif.jsonserver on sourceforge is the same as jsonserver on codespeak, except for a bit of an update for a zif namespace. There will be some kind of announcement after we are satisfied with an upcoming release. Additional contributors will, of course, be welcome. Roger, I acknowledge your interest in a mailing list. I think it would be a good idea, though I am a bit hesitant ATM to start YAFML. I think it may be just a matter of throwing a switch on SF, so it is certainly doable. For the moment, the Zif Collective is all "alpha" and experimental on SourceForge. Nothing released. Nothing to be alarmed about. Nothing decided about other repositories. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: [SpringCleaning07]
Martijn Faassen wrote: Jim Washington wrote: Martijn Faassen wrote: http://genshi.edgewall.org Inspired by Kid (in turn among others inspired by ZPT), the main template language of TurboGears, written by the people who also created Trac, and it seems to be getting traction. TurboGears among others is going to adopt it, but also things like the creator of SQLAlchemy (and Myghthy) spending time optimizing it, etc. It's close enough to ZPT to be palatable to me, and has some nice features for reuse. If we're going to get out of the server business we could also consider getting out of the template language business. :) I'm a big fan of using lxml.etree for templating. Very pythonic, very easy to refactor, very explicit. Cool! It's premature to announce (we plan to have eggs on pypi soon) , but take a look at zif.xtemplate at zif.sourceforge.net . It's pretty alpha at the moment, but it uses a DTD and some xpath to get around the "tags that shouldn't be minimized" issue, and it includes a first stab at an HTML sanitizer, to use when snippets of untrusted HTML are to be included on a page. In addition, the entire page DOM is available for postprocessing right up until serialization. Of course, those with better lxml knowledge are encouraged to point out issues with the implementation. I just took a brief look at this. Do I understand that this templating solution basically generates the entire template from Python? There is no actual textual template present at all, right? I understand you could add them back and use XPath, the elementtree API and even XSLT to generate templates, but in the default there is no template? I have used this approach when I needed to generate very particular XML, but for web templating I generally expect there to be a textual template. This way it becomes more easy to take a template created by a designer and integrate it into ones system. What are your thoughts about this? The current implementation allows you to start with an HTML file. That's the "template" class parameter, which is the full path to an initial (well-formed) document, which is parsed on __init__ into an ElementTree. If it has "id" attributes in tags, all the better for hooking-in dynamic content. At the moment, there is a bit of an issue with namespaces / namespaced elements and attributes, but a file with standard, non-namespaced HTML will work just fine. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: [SpringCleaning07]
Martijn Faassen wrote: http://genshi.edgewall.org Inspired by Kid (in turn among others inspired by ZPT), the main template language of TurboGears, written by the people who also created Trac, and it seems to be getting traction. TurboGears among others is going to adopt it, but also things like the creator of SQLAlchemy (and Myghthy) spending time optimizing it, etc. It's close enough to ZPT to be palatable to me, and has some nice features for reuse. If we're going to get out of the server business we could also consider getting out of the template language business. :) Martijn I'm a big fan of using lxml.etree for templating. Very pythonic, very easy to refactor, very explicit. It's premature to announce (we plan to have eggs on pypi soon) , but take a look at zif.xtemplate at zif.sourceforge.net . It's pretty alpha at the moment, but it uses a DTD and some xpath to get around the "tags that shouldn't be minimized" issue, and it includes a first stab at an HTML sanitizer, to use when snippets of untrusted HTML are to be included on a page. In addition, the entire page DOM is available for postprocessing right up until serialization. Of course, those with better lxml knowledge are encouraged to point out issues with the implementation. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Content-types in ResourceDirectory
I wrote:
> A small but important caveat: Paste.urlmap (0.9) does not seem to like
> :port in urls at the moment, so no http://localhost:8180 for now. Zope3
> can be listening at 8180, but you need a non-port alias for everything
> to work properly. Of course, it is possible I have a local
> misconfiguration somewhere that makes this break.
>
It turns out that twisted and Paste both want to handle the :port from
the browser url, so, when they are in combination like here, you get
host, port = host.split(':', 1)[0]
exceptions.ValueError: too many values to unpack
when there is a :port involved and the procedure is repeated. At least,
I think that is what is happening here. The easy work-around is to put
the development instance behind pound (or apache, squid, etc.) and use
http://localhost/ urls.
I do not want to leave an impression that Paste is not working
properly. I think this falls properly under unforeseen use.
-Jim Washington
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Content-types in ResourceDirectory
Jim Washington wrote: > Still playing with Dojo. Very Nice! :) > > I'm having a problem with content-type for items served from a > resourceDirectory. > > In particular, one piece of dojo, a file with a .html extension begins > with . The DOCTYPE says XHTML 1.0 > Strict, but Zope3 returns it as text/plain (X-Content-Type-Warning: > guessed from content). > > Unfortunately, that is a problem, because the client looks at > content-type to decide how to parse. > > If I set the content-type to text/html, it works OK. > > So, is this a Dojo bug or a Zope3 bug? > Hearing none, the answer is: (drumroll...) Neither. :) Purely static files can be served just fine through zope.paste. No need to bother Zope 3 at all to do this. Quick recipe: 1. install zope.paste, Paste, PasteDeploy 2. in the [Zope3 instance]/etc folder, paste.ini can look like (this somewhat complicated example uses my gzipper as a middleware filter): [composite:Paste.Main] use = egg:Paste#urlmap / = zope3 /@@/dojo = dojo [pipeline:zope3] pipeline = gzipper main [app:main] paste.app_factory = zope.paste.application:zope_publisher_app_factory [app:dojo] use=egg:Paste#static document_root=/local/filesystem/path/to/dojo/folder filter-with = gzipper [filter:gzipper] paste.filter_factory=gzipper.gzipper:filter_factory nocompress=jp gz zip tempfile=0 exclude=test Something similar should work for any static or third-party content. Maybe a /static/images directory? This may be old hat to some, but I thought I would share. A small but important caveat: Paste.urlmap (0.9) does not seem to like :port in urls at the moment, so no http://localhost:8180 for now. Zope3 can be listening at 8180, but you need a non-port alias for everything to work properly. Of course, it is possible I have a local misconfiguration somewhere that makes this break. wsgi+zope=very cool! -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Content-types in ResourceDirectory
Still playing with Dojo. Very Nice! :) I'm having a problem with content-type for items served from a resourceDirectory. In particular, one piece of dojo, a file with a .html extension begins with . The DOCTYPE says XHTML 1.0 Strict, but Zope3 returns it as text/plain (X-Content-Type-Warning: guessed from content). Unfortunately, that is a problem, because the client looks at content-type to decide how to parse. If I set the content-type to text/html, it works OK. So, is this a Dojo bug or a Zope3 bug? I've done a wsgi filter trick to get around this for now. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Zope 3 lacks Ajax capability?
Jeff Rush wrote: > Benji York wrote: >> Jeff Rush wrote: >> >>> Just checking if I'm missing something -- with the removal of HTTP >>> streaming/chunking in 3.2, this means that the async bi-directional >>> persistent socket communications associated with Ajax is NOT >>> possible at this time? >> >> Don't know if it is or isn't, but that's a novel definition of the >> "A" in AJAX that I haven't seen before. > > I got the definition from Wikipedia: > http://en.wikipedia.org/wiki/AJAX > "Ajax, shorthand for Asynchronous JavaScript+CSS+DOM+XMLHttpRequest, > is a Web development technique for creating interactive web > applications." > > From the page of the guy that coined the name: > http://www.adaptivepath.com/publications/essays/archives/000385.php > "The name is shorthand for Asynchronous JavaScript + XML". > > The key points re Zope3 are: > > (1) a piece of Javascript downloaded with a page request turns around > and issues an HTTP GET back to Zope3 such that that connection is kept > open indefinitely, and the thread within Zope3 that is servicing that > request hangs around and periodically shoves bits of data (XML, JSON, > whatever) down that pipe that get acted upon by Javascript within the > client's page. This provides server => client communications via a > brief REQUEST and a neverending RESPONSE chunked up into pieces > representing operations run on the client. > > (2) another piece of Javascript downloaded with a page request > similarly turns around and issues an HTTP GET/PUT back to Zope3 with a > header to keep the connection open. Over this pipe the client can > issue conventional multiple REQUEST/RESPONSE cycles to invoke > functions within the server, providing client => server > communications. However the trick re Zope3 is that the URL/view to > which that request connects must be one that can rendevous with the > thread or a piece of state associated with (1) above, so that the > server and client have a meaningful basis on which to talk. > > So far as I can see, Zope3 doesn't provide any way to (1) hang onto a > thread that can send further response data at any time or (2) manage a > rendevous with a thread/context of another HTTP request from the same > browser. I think I can implement (2) but (1) is harder until > streaming is added back in. #1 could also be implemented as some > piece of context that is handed to an arbitrary request-servicing > thread each time data becomes available to send to the client but that > approach still needs some element that does computation within the > server and then signals it has data ready to send to the client. > In my recent experiments with wsgi middleware, the twisted publisher in Zope 3 will do "chunked" transfer-coding when the published object is an iterator that is specifically not a list or tuple. hth, -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Preview - gzipping (improved) as wsgi middleware
Gary Poster wrote: > > ...ok, and reading further, "go Paste" and "go Zope 3 paste > integrator". :-) If I go further I'll start thanking Guido, Tim > Berners-Lee, and so on. But anyway, zope.paste looks great > too--thanks Sidnei! > Yup. Go Team! :) I'm really excited about the possibilities this is a small example of. I recall Gary saying something about zc.resourcelibrary possibly being wsgi-middleware-able, and that would be really cool! And with wsgi-friendly zope2, (does zope.paste work in Five?), well, I could go on and on and on... :) -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Preview - gzipping (improved) as wsgi middleware
Back in March, there was a (tiny) bit of discussion here about using wsgi middleware for gzipping zope3 responses. I have now had time to take a look at the concept, and it seems good. So, for a preview of another gzipping implementation for zope3, please take a look at http://zif.hill-street.net/gzipper For use with zope3, it requires zope.paste (Thanks, Sidnei!) features: iterable-friendly - outputs as iterable at the same time it receives from the application iterable memory-friendly? - can use a tempfile buffer to perhaps reduce memory usage configurable excludes - easily exclude any content-types you want from gzipping At the same URL, I'm also previewing a similar package to minimize javascript. As usual, installation and configuration instructions are in the README (http://zif.hill-street.net/gzipper/README.txt). development status: "works for me" -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] zc.datetimewidget and ITZInfo request adapter
Bernd Dorn wrote: hello guys from zc is there an implementation of the adapter from request to ITZInfo you use in zc.datetimewidget for example in http://svn.zope.org/zc.datetimewidget/trunk/src/zc/datetimewidget/datetimewidget.py?rev=41765&view=auto ... tzinfo = ITZInfo(request, None) ... if not, what would be the prefered way to implement such an adapter I found something that seems to work for me, mostly: http://mail.zope.org/pipermail/zope3-users/2005-October/001165.html I recall a typographical error somewhere in the code, but you should be able to figure that out. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] httpgz in zope.conf?
Gary Poster wrote: On Mar 20, 2006, at 9:39 AM, Stephan Richter wrote: On Monday 20 March 2006 09:35, Jim Washington wrote: [...] BTW, I would be glad to see a proposal to add this to the core. I think httpgz is general very interesting to a lot of people. +1 If it goes into the core, httpgz can be done much more elegantly. Perhaps another proposal could suggest a method for registering and sequencing post-processing utilities? I can think of several possible post-content-generation operations that might take advantage of such a hook. - javascript minimization / obfuscation - watermarking or tagging pdf's and images - inserting 'this is the development site, not the real site' or copyright statements - tidying - wow the folks that 'view source' - httpgz - zc.resourcelibrary -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: httpgz in zope.conf?
Tres Seaver wrote: Zope3 could add a similar abstract section type to its top-level schema, for use by third-party extensions. Such a change would *not* require settling the config-file-format debate. Thanks, Tres So, it's not an easy thing to do right now because something in core has to be updated first. I'll maybe look into that when I have time. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] httpgz in zope.conf?
I announced release of httpgz, http://zif.hill-street.net/httpgz on Zope3-Users this past weekend. It does very nice things to bandwidth. One particular page (tabular and very repetitious) went from around 400K to 25K for the client. MochiKit.js went from 94254 bytes to 24854 bytes. This makes web pages seem much faster, particularly on slower connections. I am wondering what the best mechanism for turning this on and off would be. I suppose it is simple enough to insert and remove the httpgz-configure.zcml file in etc/package-includes, but is there a place for third-party hooks in zope.conf? Or would that be a bad idea? -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Additional address(es) in access.log
I have attached a diff against zope.twisted.web2.log that puts
additional ip address(es) in access.log.
With this patch, the ip address section in the log may become a list of
addresses delimited with '>'.
It uses the 'x-forwarded-for' headers, so the info may not be trustable,
but it is much more informative than '192.168.1.1' for every address if
you run z3 behind a proxy like pound.
Please pardon if the patch is not-quite-right. I did not read the
directions.
-Jim Washington
*** log.py Sat Jan 28 19:20:33 2006
--- /usr/lib/zope-3.2.0/lib/python/twisted/web2/log.py Sat Jan 28 20:46:34 2006
***
*** 148,158
request.method,
request.uri,
'.'.join([str(x) for x in request.clientproto]))
!
self.logMessage(
'%s - %s [%s] "%s" %s %d "%s" "%s"' %(
! request.chanRequest.getRemoteHost().host,
# XXX: Where to get user from?
"-",
self.logDateString(
response.headers.getHeader('date', 0)),
--- 148,165
request.method,
request.uri,
'.'.join([str(x) for x in request.clientproto]))
! proxyAddresses = request.headers.getRawHeaders('x-forwarded-for',None)
! singleHost = request.chanRequest.getRemoteHost().host
! if proxyAddresses:
! proxyAddresses.append(singleHost)
! logAddress = '>'.join(proxyAddresses)
! else:
! logAddress = singleHost
self.logMessage(
'%s - %s [%s] "%s" %s %d "%s" "%s"' %(
! #request.chanRequest.getRemoteHost().host,
# XXX: Where to get user from?
+ logAddress,
"-",
self.logDateString(
response.headers.getHeader('date', 0)),
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: xmlrpc:view (and jsonrpc:view)
Paul Winkler wrote: On Sun, Jan 29, 2006 at 12:19:24PM +0100, Balazs Ree wrote: So this is where I see the importance of this: allow access to a method/template via RPC only, and disallow the method to be called directly from the browser. I'm curious. How do you distinguish between a jsonrpc request and a direct browser request? There are a few attributes that distinguish a jsonrpc request in Zope3 from a direct browser request. First, the request must be a POST with a content-type of "application/json-rpc". That is the listening handshake registered by jsonserver so that the request is handled as a JSONRPCRequest. The URL for object traversal is the URL of the context object. The method to be performed is identified in the body of the request, and has a specific syntax for method and parameters. The methods on the server side are provided in a view class for the context object, and are protected by permissions registered in zcml. When jsonserver gets the request, the appropriate method of the view class is invoked in the context of the context object, and the results are returned in jsonrpc format. The methods of the view class are not very different from methods of any other browser view class. In a page template, they would likely be accessible as "view/method_name". But they are also not accessible by direct URL traversal, unless you do extra work to allow that. In Zope2, with ttw object methods, the distinction between a jsonrpc request and a direct browser request can be a bit blurred. Such a method would have a URL and would be ordinarily traversable in a direct browser request. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] xmlrpc:view (and jsonrpc:view)
I have been watching with some interest the thread about zcml simplification (ZCML bad et.al.), and of course I looked at Philipp's list of things that could be dead chickens in ZCML. I agree with most, and even with the directive. My question starts with the fact that for the jsonserver package, I copied the pattern for JSON-RPC. Should xmlrpc views be browser:views, and more specifically for me, should jsonrpc views be browser:views? Do we really need other types of views than browser:views? It seems that the different uses are entirely up to the developer, and not specifically controlled by Zope3, anyway. I have discovered, for example, that a browser:view can be accessed through jsonrpc just fine; it just gets wrapped in JSON-RPC syntax, which is actually pretty nice if you want to have multi-use of a page snippet for page composition then later for AJAXish insertion by the client. Anyway, I am considering removing the jsonrpc:view directive from my third-party project (to be replaced by the already available browser:view), and am wondering if there is a risk I have not considered. The main risk I see is angry customers, but I think I can calm them by explaining the added simplicity it brings. -Jim Washington it-seemed-to-be-the-right-thing-to-do-at-the-moment ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Twisted? We lost the username in the access.log
Jim Washington wrote: Any word on this? This afternoon, I put together a hack that involves pushing the response's authuser in a header (zope.publisher.http.HTTPResponse.getHeaders) to twisted's response, then using that in the logger (twisted.web2.log.BaseCommonAccessLoggingObserver.emit). I am mainly interested in getting x-forwarded-for in the log (my machines are all behind pound, and 127.0.0.1 as ip address is pretty useless), but I found the XXX about user while I was spelunking there. FYI, pushing a new header into response to get username in the twisted log fails a lot of tests that assume a particular set of response headers. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Twisted? We lost the username in the access.log
Jim Fulton wrote: This is fairly serious, but I don't want it to delay 3.2.1 final. I suggest we try to fix this the week after the final release. I'd like to shoot for getting a 3.2.1 release out around the end of January. I'd like to see a beta release around the middle of the month. Any word on this? This afternoon, I put together a hack that involves pushing the response's authuser in a header (zope.publisher.http.HTTPResponse.getHeaders) to twisted's response, then using that in the logger (twisted.web2.log.BaseCommonAccessLoggingObserver.emit). I am mainly interested in getting x-forwarded-for in the log (my machines are all behind pound, and 127.0.0.1 as ip address is pretty useless), but I found the XXX about user while I was spelunking there. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] problem with multiple instances of zope3 as daemon via init script on gentoo
Bernd Dorn wrote:
hi all
i have two init scripts (see below) which start a zope3 instance
this works fine if i start one of them, but if i try to start the
second, the following message appears
* Starting Zope in /home/zope1/timetables ...
WARNING! zdrun is managing a different program!
our program = ['/home/zope1/timetables/bin/runzope']
daemon's args = ['/home/zope1/screens/bin/runzope']
daemon process already running; pid=10839
seems that the pid of the other instance is taken, does anybody know
how to solve this
or is there another way to start zope3 as an unprivileged user?
as far as i know there is no effective-user directive in zope.conf as
in zope2
my zope version is 3.2 final, but it seems not version specific
my python:
Python 2.4.2 (#1, Dec 6 2005, 15:16:51)
[GCC 3.3.5-20050130 (Gentoo 3.3.5.20050130-r1, ssp-3.3.5.20050130-1,
pie-8.7.7. on linux2
my system:
Linux flow 2.6.14-gentoo-r2 #1 SMP Mon Dec 5 14:23:57 CET 2005 i686
Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux
thx, bernd
init script listing:
INSTANCE_HOME=/home/zope1/screens
ZOPE_USER=zope1
opts="start stop restart kill quit status help"
depend() {
need net
}
start() {
ebegin "Starting Zope in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl start"
eend $?
}
stop() {
ebegin "Stopping Zope in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl stop"
eend $?
}
restart() {
ebegin "Restarting Zope in $INSTANCE_HOME"
svc_stop
svc_start
eend $?
}
kill() {
ebegin "Killing Zope in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl kill"
eend $?
}
quit() {
ebegin "Quitting Zope Daemon Manager $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl quit"
eend $?
}
reload() {
# throws an error, dunno why
ebegin "Reloading Zope config in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl reload"
eend $?
}
status() {
ebegin "Status of Zope in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl status"
eend $?
}
help() {
ebegin "Help for Zope Daemon Manager in $INSTANCE_HOME"
su - $ZOPE_USER -c "$INSTANCE_HOME/bin/zopectl help"
eend $?
}
Does this script put a zdsock file in /etc/init.d? I have noticed that
the zdsock file is created in the directory where zopectl is called. If
this is what you are seeing, one solution might be to do some cd
statements (e.g., cd $INSTANCE_HOME) so that multiple zdsock files are
created, one for each instance.
-Jim Washington.
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: rdb: Disappearing Connection
Dieter Maurer wrote:
Jim Washington wrote at 2005-12-13 21:40 -0500:
...
Now, looking closer at the code, a ping like this might be not too bad,
because isConnected() is only called when a connection is requested, not
for every SQL statement executed. So, it might not be so onerous as
originally thought. Still not ideal, though.
Be careful what you do when you find that "isConnected()" returns false.
Note, that, in general, it would be a bug to just reopen a new connection
in this case: previous operations against the connection might have
been lost without notice when you simply reopen a new connection
and do as if nothing bad had happened.
Instead, you should reopen the connection and then raise an exception
similar to "ZODB.POSException.ConflictError". Hopefully, the publisher
will handle this kind of exception correctly (by retrying the complete
request).
Thanks, Dieter.
The situation I wish to handle is the case where MySQL server silently
closes its connection with Zope after 8 hours idle, which likely will
happen at least once a day.
What seems to work for me now is the following as
mysqldbda.adapter.MySQLdbAdapter.isConnected()
def isConnected(self):
try:
self._v_connection.ping()
except:
# not connected or ping did not restore MySQL connection
if self._v_connection is not None:
self._v_connection.close()
self._v_connection = None
return False
return True
I am not well-versed in the intricacies of conflictError, so what else
should happen here? I do not see how there would be unresolved
transactions remaining, and the connection I need is now available or
soon will be established.
In the current rdb implementation, isConnected() is not called for each
SQL statement. It is only called when the utility is called.
e.g.,
conn = zapi.getUtility(IZopeDatabaseAdapter, 'spam') #isConnected has
not been called yet
db = conn() #here, isConnected() gets called
c = db.cursor() #isConnected() is not called
c.execute('SELECT 1') #isConnected() is not called
-Jim Washington
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: rdb: Disappearing Connection
jürgen Kartnaller wrote: Without going much deeper into the way zope uses database connections. Wouldn't MySQL:Ping solve the reconnect Problem. That's what I used in my old C++ projects. Some kind of "ensureConnected" at the right place. Jürgen Thanks! I'll look into it. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] rdb: Disappearing Connection
Christian Theune wrote:
Am Dienstag, den 13.12.2005, 16:32 -0500 schrieb Jim Washington:
Christian Theune wrote:
Well. First, the error would be something like an OperationalError (or
similar). Hmm. Not too distinguishable. Ideally we could:
- differentiate this kind of error from other OperationalErrors (like a
syntax error in your query)
The error in the traceback is
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
- delete the _v_connection
- restart the transaction
Otherwise you might start over within a transaction that is out ouf sync
with the state of the mysql database.
Good point. Though, this particular error is about idleness, so I would
think it unlikely that one would be resuming an 8-hour-old transaction.
Could that happen?
Sure. If I'd force it. But we can eventually ignore that with some good
faith for a moment.
That's a valid point. Additionally, IIRC this is a client message, not a
server message. The server closes the connection much earlier but the
client doesn't notice, as the client libraries are very very lazy.
That means you could just purge the connection and create a new one.
Still, we're not 100% sure that nothing was written to the connection
that is now silently assumed of having been saved whereas it isn't.
Yup. Let's not kill it unless really necessary.
What seems to work for me for
mysqldbda.adapter.MySQLdbAdapter.isConnected() looks like
def isConnected(self):
"""exercise the connection to assure it is really there"""
try:
self._v_connection.conn.cursor().execute('SELECT 1')
except:
return False
return True
It's a bare except, but the method returns False in any case where
_v_connection is not listening as an SQL connection, which is exactly
what we want to check. Asking for conn.cursor() instead of
_v_connection.cursor() gets around transactionality that should not be
invoked here.
I'll declare success for now, and move on. It's hopefully a temporary
work-around.
-Jim Washington
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: rdb: Disappearing Connection
Jim Washington wrote: Florent Guillaume wrote: Jim Washington wrote: Spelunking a bit in the code, zope.app.rdb.ZopeDatabaseAdapter.isConnected() looks at whether _v_connection is present, not whether the connection is actually alive. If we fix this here, isConnected() perhaps should handle the case where the connection might be reaped by the back end db. Frankly anything that uses _v_ attributes is asking for trouble because it relies on low level behaviour of the ZODB and persistence mechanisms. They can disappear at unpredictable times. Avoid them. Hi, Florent In this case, it is not particulary a problem if the _v_ attribute is not there. It gets recreated if it does not exist. It is designed this way so that different zope threads get separate rdb connections, which is good. The problem here is false positives, where the _v_ attribute exists, but the connection it promises has been silently expired and disconnected by the MySQL server. I was thinking of a check in isConnected() by making a cursor then "try"ing executing "select 1" and return True if it does not raise the exception. But that would be bad because it ~doubles the number of requests to the SQL server. Perhaps not smart. Now, looking closer at the code, a ping like this might be not too bad, because isConnected() is only called when a connection is requested, not for every SQL statement executed. So, it might not be so onerous as originally thought. Still not ideal, though. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: rdb: Disappearing Connection
Florent Guillaume wrote: Jim Washington wrote: Spelunking a bit in the code, zope.app.rdb.ZopeDatabaseAdapter.isConnected() looks at whether _v_connection is present, not whether the connection is actually alive. If we fix this here, isConnected() perhaps should handle the case where the connection might be reaped by the back end db. Frankly anything that uses _v_ attributes is asking for trouble because it relies on low level behaviour of the ZODB and persistence mechanisms. They can disappear at unpredictable times. Avoid them. Hi, Florent In this case, it is not particulary a problem if the _v_ attribute is not there. It gets recreated if it does not exist. It is designed this way so that different zope threads get separate rdb connections, which is good. The problem here is false positives, where the _v_ attribute exists, but the connection it promises has been silently expired and disconnected by the MySQL server. I was thinking of a check in isConnected() by making a cursor then "try"ing executing "select 1" and return True if it does not raise the exception. But that would be bad because it ~doubles the number of requests to the SQL server. Perhaps not smart. And, I've thought about a cron-like utility to ping the connection, but now that I mention threads, that might be a non-solution if a broken connection is in a different thread than the one contacted. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] rdb: Disappearing Connection
Christian Theune wrote: Well. First, the error would be something like an OperationalError (or similar). Hmm. Not too distinguishable. Ideally we could: - differentiate this kind of error from other OperationalErrors (like a syntax error in your query) The error in the traceback is _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away') - delete the _v_connection - restart the transaction Otherwise you might start over within a transaction that is out ouf sync with the state of the mysql database. Good point. Though, this particular error is about idleness, so I would think it unlikely that one would be resuming an 8-hour-old transaction. Could that happen? -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] rdb: Disappearing Connection
I have an rdb connection that keeps disappearing. I know the real problem is that the default wait_timeout and interactive_timeout for MySql 5.0 is set at 8 hours. The site I am working on for development waits 8 hours overnight for me to start working again, then I get "Mysql server has gone away" or something like that. The easy way to refresh the connection is to restart the zope3 instance. The question is Where should this be handled? Hooking a connection-check/reconnect-if-down in each method contacting the sql server seems to violate DRY. So, should the connection check be in zope.app.rdb.ZopeCursor, or further up the chain in ZopeDatabaseAdapter or MySQLdb? I'm thinking that connections disappearing after 8 hours idle is probably a good thing, so than any process that forgets to conn.close() eventually releases the connection back to the pool (zope is not the only thing that connects to this db). I think that at the moment, this is specific to MySQL 5.0. But it would not surprise me if other databases might adopt this sort of policy to reduce "out of handles" errors. Spelunking a bit in the code, zope.app.rdb.ZopeDatabaseAdapter.isConnected() looks at whether _v_connection is present, not whether the connection is actually alive. If we fix this here, isConnected() perhaps should handle the case where the connection might be reaped by the back end db. I know that I can set wait_timeout and interactive_timeout to higher numbers in my.cnf. But let's say I have authentication looking at MySQL for data and I guess a too-small number for the timeout (what would be a good default, anyway? My users are often at schools that take breaks for weeks at a time). Then, several users attempt to log in and get the "contact Jim" page I provide for errors like this. Then, of course, my telephone starts ringing. Not good. Possibly, I could restart the instance every morning at 3AM. But this is not my favorite solution. Thoughts, anyone? Collector search for 'rdb' did not turn up anything, so I'll submit a collector issue if I see consensus that zope.app.rdb is the place to remedy this. -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: JSON
Philipp von Weitershausen wrote: > Stephan Richter wrote: > >> Here is how to do it for the current trunk. >> >> 1. Create your custom HTTPPublicationRequestFactory implementation, >> basically extending the __call__() method. >> >> 2. Create a server type using your own factory; see >> zope.app.server.http. >> >> 3. Register you custom server type as a utility, see >> zope.app.server/configure.zcml. >> >> 4. In zope.conf use your custom server type. >> >> 5. Implement the protocol similar to XML-RPC. > > > If you're looking for an example of how an external package provides > another server type, see the davuseragent package I wrote once with > Paul Everitt (http://codespeak.net/svn/z3/davuseragent/trunk/). It's > written for Zope X3.0 but could easily be adapted to 3.1. > Hi, Philipp I got JSON-RPC working this weekend with Stephan's instructions and a bit of additional spelunking in the main source. I will, however, take a look at davuseragent to see if you did anything more elegantly. :) Thanks, -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] JSON
Hi, all I am working on a web app in zope3 that sends and receives data by xmlrpc. The problem is the client side; vcxmlrpc.js does not seem to work properly in my app on anything but gecko and ie6+ browsers. This is currently not a problem because I can specify browsers for my clients. But I would like to try to get it to work on recent konquerors, particularly safari so that it works on the default browser on most operating systems. One idea I am toying with is JSON. I read about it recently following a comment to a Slashdot article about AJAX. JSON is javascript object notation. It is very lightweight compared to xml-rpc. In principle, a request receives javascript code that gets exec'ed in the client. No xml parsing or browser sniffing particularly required. Since my app deals with simple python dictionaries of simple python types converted to javascript "objects", JSON seems a good fit. json-py https://sourceforge.net/projects/json-py/ has most of what would be necessary on the server side, and jsolait http://jsolait.net/ has the javascript for the client side. Both libraries are LGPL. In theory, I think it should be pretty easy to get zope3 to do JSON. Implementation would follow the pattern of xmlrpc in zope.app.publisher, then some code in zope.app.publication.httpfactory to get something different happening if the content type is "text/x-json". Now, the question: I would rather not alter and maintain the foundation-level code in zope.app.pub*. Would it be possible to do this as a product? Presuming it is bad form to monkey-patch zope.app.publication.httpfactory, is there a more component-ish way to do the same thing? Or would it be best to set-up a collaboration and put it in the core? -Jim Washington ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
