Re: [Zope-dev] Serving large files
On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). Why would you ignore that? ... b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. OK, so? c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. I wasn't aware that it would break RESPONSE.write, but, in any case, this will cause a tempirary file to be created. ... I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? You could create an entirely new server with different thread-management semantics. You could probably do this with the normal Zope publisher using one of the standard threading WSGI servers, Or you could use a temporary file or wait for blobs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I wonder what Any connection handed out after the connection means. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Jim Fulton wrote: On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I wonder what Any connection handed out after the connection means. Argh. Obviously this was a broken sentence. :) I wonder again: Would the ACID compliance break if we hand out any ZODB connection after the worker thread is completed. Sidnei already mentioned to re-create a connection for the iterator to be passed back with the tid that represents the right state as of the end of the transaction in the worker thread. -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: OpenPGP digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
FileCacheManager does the copy-to-file-and-serve-as-iterator dance for you. http://www.dataflake.org/software/filecachemanager Stefan On 17. Sep 2006, at 21:09, Sidnei da Silva wrote: One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. -- It doesn't necessarily do it in chronological order, though. --Douglas Adams ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Sep 18, 2006, at 8:16 AM, Christian Theune wrote: Jim Fulton wrote: On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I wonder what Any connection handed out after the connection means. Argh. Obviously this was a broken sentence. :) I wonder again: Would the ACID compliance break if we hand out any ZODB connection after the worker thread is completed. And perhaps, implicitly, after the worker thread has closed the connection. It is likely that all sorts of bad things would happen, as the connection might already have been haded to another thread, Sidnei already mentioned to re-create a connection for the iterator to be passed back with the tid that represents the right state as of the end of the transaction in the worker thread. Of course, ot would be a different instance of the iterator, Something like that could certainly work. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Sidnei da Silva wrote at 2006-9-17 16:09 -0300: ... c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. If you know the size and set it in the Content-Length response header, I cannot see a reason why it should break some applications. ... I would like to be able to return an iterator that can read from the ZODB. Give the iterator a new ZODB (read only) connection; close it when the iterator comes to its end. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Could you return an iterator to the server that knows enough to open its own database connection? Provide it with a callback that opens the connection and finds and chunks the relevant object? - C On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Mon, Sep 18, 2006 at 04:44:17PM -0400, Chris McDonough wrote: | Could you return an iterator to the server that knows enough to open | its own database connection? Provide it with a callback that opens | the connection and finds and chunks the relevant object? I could certainly do that if I wanted to. What I'm trying to figure out is how/if Zope should do this for me. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Oops.. should have read further on there... On Sep 18, 2006, at 4:44 PM, Chris McDonough wrote: Could you return an iterator to the server that knows enough to open its own database connection? Provide it with a callback that opens the connection and finds and chunks the relevant object? - C On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Serving large files
I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Sidnei da Silva schrieb: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. ... I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? It should be possible to feed a fifo (at least on unix) and reopen the connection each time via publisher or something alike (similar to the retry mechanism on database conflicts) and fill the buffer again. Regards Tino ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Hi, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I think this is because: When finishing the actual work we have to either commit or abort the running transaction. In the time until we set up another transaction, there already might have been a change to the database by a third transaction, making the request neither isolated nor atomic. My statement is probably only true for all solutions not paying attention to the statement. ;) Christian -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: OpenPGP digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Sun, Sep 17, 2006 at 09:24:40PM +0200, Christian Theune wrote: | I wonder whether this statement is true: | |Any connection handed out after the connection breaks the ACID |compliance of the request. | | I think this is because: | |When finishing the actual work we have to either commit or abort the |running transaction. | |In the time until we set up another transaction, there already might |have been a change to the database by a third transaction, making the |request neither isolated nor atomic. Suppose the request finishes with 'tid=X' and that it should be possible to open a read-only connection which would be a snaphsot of the database at 'tid=X'? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Sidnei da Silva wrote: On Sun, Sep 17, 2006 at 09:24:40PM +0200, Christian Theune wrote: | I wonder whether this statement is true: | |Any connection handed out after the connection breaks the ACID |compliance of the request. | | I think this is because: | |When finishing the actual work we have to either commit or abort the |running transaction. | |In the time until we set up another transaction, there already might |have been a change to the database by a third transaction, making the |request neither isolated nor atomic. Suppose the request finishes with 'tid=X' and that it should be possible to open a read-only connection which would be a snaphsot of the database at 'tid=X'? Jup. That's what I was thinking about. But what then. Then you end up again with application code that needs to do work. (In the iterator.) The thing about the iterator is that currently (I think) it is assumed that all resources are cleaned up by GC. If you do this however, I suspect you have to do work to get rid of the resources you allocated (a transaction, a connection, ...) Christian -- gocept gmbh co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: OpenPGP digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
