Re: [Zope] Serve large files efficiently from a pipe
Roman Suzi schrieb: Hi! Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ). What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content... Does Zope has anything for this? Thank you for any ideas on this topic! Have a look at: http://www.dataflake.org/software/filecachemanager it implements what you are asking for. HTH Tino ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
On Tue, 8 Nov 2005, Tino Wildenhain wrote: Roman Suzi schrieb: Hi! Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ). What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content... Does Zope has anything for this? Thank you for any ideas on this topic! Have a look at: http://www.dataflake.org/software/filecachemanager it implements what you are asking for. Thank you for the link, however, it seems that it is not what I am looking for. I do not want the file contents be read into Python program as a whole but piped to the HTTP client. This piece of FileCacheManager code give me doubts that this is how it works: """ cache_entry = FileCacheEntry(self, ob, fname, self._tempfile_path) cache_entry.write(data) """ I want something which works similar to Unix pipe but for HTTP client: this_generates_output | process_output >> http_client I do not want to store it anywhere on the FS and/or ZODB! And I certainly do not want to read the resulting file as a Python string into RAM. So, I cant understand how RAMCache or FILECacheManager is going to help me. I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later... HTH Tino Sincerely yours, Roman Suzi -- [EMAIL PROTECTED] =\= My AI powered by GNU/Linux RedHat 7.3 ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
- Original Message - From: "Roman Suzi" <[EMAIL PROTECTED]> To: Sent: Tuesday, November 08, 2005 8:47 AM Subject: Re: [Zope] Serve large files efficiently from a pipe On Tue, 8 Nov 2005, Tino Wildenhain wrote: Roman Suzi schrieb: Hi! Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ). What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content... Does Zope has anything for this? Thank you for any ideas on this topic! Have a look at: http://www.dataflake.org/software/filecachemanager it implements what you are asking for. Thank you for the link, however, it seems that it is not what I am looking for. I do not want the file contents be read into Python program as a whole but piped to the HTTP client. This piece of FileCacheManager code give me doubts that this is how it works: """ cache_entry = FileCacheEntry(self, ob, fname, self._tempfile_path) cache_entry.write(data) """ I want something which works similar to Unix pipe but for HTTP client: this_generates_output | process_output >> http_client I do not want to store it anywhere on the FS and/or ZODB! And I certainly do not want to read the resulting file as a Python string into RAM. So, I cant understand how RAMCache or FILECacheManager is going to help me. I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later... It sounds like you are looking for a streaming type of solution. You can google zope and streaming to get more info. Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
On 11/8/05, Roman Suzi <[EMAIL PROTECTED]> wrote:
> I don't know if writing to response one chunk at a time is proper
> solution? Will Zope store response body or sent it right away? I am not
> sure that it is the later...
The classic way to do streaming data in Zope is like so:
RESPONSE.setHeader('Content-length', N)
for chunk in some_iterable_yielding_data:
RESPONSE.write(chunk)
... where N is the size in bytes of the data you are going to write.
However, you can also publish any method that returns an
implementation of IStreamIterator as shown in
http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=markup
The advantage is that this has less overhead, as once your method has
returned, the Zope app and ZODB are no longer involved, and your
iterator and ZPublisher do all the work.
This can become quite significant when you are streaming large data,
more than a few hundred kB. However, your iterator *must* be able to
do its work without a ZODB connection.
You haven't said where the data you want to stream is going to come from,
so I can't guess whether the IStreamIterator technique will work for you...
--
http://www.slinkp.com
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
Roman Suzi wrote at 2005-11-8 13:05 +0200: > ... >What about serving large files given file(-like) handler? It could be >very beneficial sometimes, especially with dynamically generated >content... You must observe the restrictions required by HTTP! HTTP 1.1 requires that if a response contains a "Content-Length" header, then it *MUST* correctly specify the size of the response entity. If it does not contain a "Content-Length", then its end *MUST* be indicated by closing the connection. Zope is a bit buggy in this respect. It always adds a "Content-Length" header, if the application did not do so. Of course, this automatically added header has (almost surely) the wrong value for dynamically created content. Currently, Zope does not support the "no Content-Length; response end by connection close" possibility. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
On Tue, 8 Nov 2005, Paul Winkler wrote:
On 11/8/05, Roman Suzi <[EMAIL PROTECTED]> wrote:
> I don't know if writing to response one chunk at a time is proper
solution? Will Zope store response body or sent it right away? I am not
sure that it is the later...
The classic way to do streaming data in Zope is like so:
RESPONSE.setHeader('Content-length', N)
for chunk in some_iterable_yielding_data:
RESPONSE.write(chunk)
... where N is the size in bytes of the data you are going to write.
OK... Then streaming from Unix pipe can't be done in Zope...
However, you can also publish any method that returns an
implementation of IStreamIterator as shown in
http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=markup
The advantage is that this has less overhead, as once your method has
returned, the Zope app and ZODB are no longer involved, and your
iterator and ZPublisher do all the work.
This can become quite significant when you are streaming large data,
more than a few hundred kB. However, your iterator *must* be able to
do its work without a ZODB connection.
You haven't said where the data you want to stream is going to come from,
Sorry if it was not clear: data comes from a pipe (that is, from another
application, called by popen2...
so I can't guess whether the IStreamIterator technique will work for you...
- I do not have named file... only file handler. But maybe named pipe
will do the trick... However, missing content-length (and unfortunately,
resulting content-length is not known) is the show-stopper.
--
http://www.slinkp.com
Sincerely yours, Roman Suzi
--
[EMAIL PROTECTED] =\= My AI powered by GNU/Linux RedHat 7.3
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
On Tue, 8 Nov 2005, Dieter Maurer wrote: Roman Suzi wrote at 2005-11-8 13:05 +0200: ... What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content... You must observe the restrictions required by HTTP! HTTP 1.1 requires that if a response contains a "Content-Length" header, then it *MUST* correctly specify the size of the response entity. If it does not contain a "Content-Length", then its end *MUST* be indicated by closing the connection. Zope is a bit buggy in this respect. It always adds a "Content-Length" header, if the application did not do so. Of course, this automatically added header has (almost surely) the wrong value for dynamically created content. Currently, Zope does not support the "no Content-Length; response end by connection close" possibility. Thank you all for explanations! While I do not necessary agree that Zope is buggy in this respect, it could still be good for the purpose of serving large files to have some by side technology, which doesnt require ZODB access. Otherwise there is a need for Apache or whatever. Sincerely yours, Roman Suzi -- [EMAIL PROTECTED] =\= My AI powered by GNU/Linux RedHat 7.3 ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
Am Dienstag, den 08.11.2005, 22:34 +0200 schrieb Roman Suzi:
> On Tue, 8 Nov 2005, Paul Winkler wrote:
>
> > On 11/8/05, Roman Suzi <[EMAIL PROTECTED]> wrote:
> > > I don't know if writing to response one chunk at a time is proper
> >> solution? Will Zope store response body or sent it right away? I am not
> >> sure that it is the later...
> >
> > The classic way to do streaming data in Zope is like so:
> >
> > RESPONSE.setHeader('Content-length', N)
> > for chunk in some_iterable_yielding_data:
> >RESPONSE.write(chunk)
> >
> > ... where N is the size in bytes of the data you are going to write.
>
> OK... Then streaming from Unix pipe can't be done in Zope...
Sure it can.
> > However, you can also publish any method that returns an
> > implementation of IStreamIterator as shown in
> > http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=markup
> >
> > The advantage is that this has less overhead, as once your method has
> > returned, the Zope app and ZODB are no longer involved, and your
> > iterator and ZPublisher do all the work.
> > This can become quite significant when you are streaming large data,
> > more than a few hundred kB. However, your iterator *must* be able to
> > do its work without a ZODB connection.
> >
> > You haven't said where the data you want to stream is going to come from,
>
> Sorry if it was not clear: data comes from a pipe (that is, from another
> application, called by popen2...
>
> > so I can't guess whether the IStreamIterator technique will work for you...
>
> - I do not have named file... only file handler. But maybe named pipe
> will do the trick... However, missing content-length (and unfortunately,
> resulting content-length is not known) is the show-stopper.
What makes you think a file is always a "named file"? After all,
what you get from popen2 is just a file. The only hurdle you
need to take is the __len__ method on this object. So you
might need a little wrapper and/or investigate a bit what
ZPublisher does with it and how to prevent it (since you dont have
the size, do you?)
Regards
Tino
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Serve large files efficiently from a pipe
On Wed, 9 Nov 2005, Tino Wildenhain wrote: Am Dienstag, den 08.11.2005, 22:34 +0200 schrieb Roman Suzi: On Tue, 8 Nov 2005, Paul Winkler wrote: - I do not have named file... only file handler. But maybe named pipe will do the trick... However, missing content-length (and unfortunately, resulting content-length is not known) is the show-stopper. What makes you think a file is always a "named file"? After all, what you get from popen2 is just a file. The only hurdle you need to take is the __len__ method on this object. So you might need a little wrapper and/or investigate a bit what ZPublisher does with it and how to prevent it (since you dont have the size, do you?) Ok... While it doesn't seem like "right" solution, anyway thanks! That was my last option: to look how ZPublisher really do it. Regards Tino Sincerely yours, Roman Suzi -- [EMAIL PROTECTED] =\= My AI powered by GNU/Linux RedHat 7.3 ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
