Re: [cgiapp] Streaming Large Files using PSGI
On 10/12/2012 09:06 AM, Mike Tonks wrote: > I've been doing some more testing and it seems to work really well. > Serving up big files quite happily, and fast! > > A few additional thoughts... > > The PSGI spec allows 2 useful options: > > 1) The subref callback as originally suggested > > 2) Return a filehandle and the server hadles the streaming automatically. > > The filehandle option is very neat if all you need to do is send a > file to the browser. > > The callback option seems useful as well because it's possible to do > stuff during or after the file streaming. I like to write a log > message and delete temp files etc at the end so this is handy. > > Also I wonder if simply returning the filehandle / subref would be > more elegant that my initial $self->psgi_streaming_callback getter / > setter idea. > > I tried to think of a way to do this without hacking into > Application.pm & Stream.pm, but in the end figured it was the (best / > easiest) way to go. Do you think we can kick this about and get in > accepted into the upstream packages? I have written some tests and > pod documentation. I'm interested in having PSGI steaming well-supported in CGI::Application. Have you looked around at some other frameworks to see how they handle PSGI streaming? You might find a nice "prior art" design that you like, or find some lessons-learned that we don't need to relearn here. > Is Purdy on this list? Best to CC him explicitly to be sure, or even contact him directly Even if someone is subscribed, they have it filtered out of their inbox. Mark # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Large Files using PSGI
I've been doing some more testing and it seems to work really well. Serving up big files quite happily, and fast! A few additional thoughts... The PSGI spec allows 2 useful options: 1) The subref callback as originally suggested 2) Return a filehandle and the server hadles the streaming automatically. The filehandle option is very neat if all you need to do is send a file to the browser. The callback option seems useful as well because it's possible to do stuff during or after the file streaming. I like to write a log message and delete temp files etc at the end so this is handy. Also I wonder if simply returning the filehandle / subref would be more elegant that my initial $self->psgi_streaming_callback getter / setter idea. I tried to think of a way to do this without hacking into Application.pm & Stream.pm, but in the end figured it was the (best / easiest) way to go. Do you think we can kick this about and get in accepted into the upstream packages? I have written some tests and pod documentation. Is Purdy on this list? Regards, Mike On 8 October 2012 16:55, Mike Tonks wrote: > Hi Mark, > > Thanks for the feedback. OK I'll have a go then. > > Here's my initial attempts... > > https://github.com/miketonks/CGI--Application > > https://github.com/miketonks/CGI-Application-Plugin-Stream > > Does this approach seem ok? > > mike > > > On 5 October 2012 14:03, Mark Stosberg wrote: > >> Mike, >> >> My reading of the spec is that you can return a callback instead of a >> body, and the Plack server should handle the rest: >> >> https://metacpan.org/module/PSGI#Delayed-Response-and-Streaming-Body >> >> If the server doesn't suppport streaming, a MiddleWare can provide it: >> >> https://metacpan.org/module/Plack::Middleware::BufferedStreaming >> >> I have not needed to do this yet myself with PSGI (but will eventually). >> >>Mark # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Large Files using PSGI
Hi Mark, Thanks for the feedback. OK I'll have a go then. Here's my initial attempts... https://github.com/miketonks/CGI--Application https://github.com/miketonks/CGI-Application-Plugin-Stream Does this approach seem ok? mike On 5 October 2012 14:03, Mark Stosberg wrote: > Mike, > > My reading of the spec is that you can return a callback instead of a > body, and the Plack server should handle the rest: > > https://metacpan.org/module/PSGI#Delayed-Response-and-Streaming-Body > > If the server doesn't suppport streaming, a MiddleWare can provide it: > > https://metacpan.org/module/Plack::Middleware::BufferedStreaming > > I have not needed to do this yet myself with PSGI (but will eventually). > >Mark # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
Re: [cgiapp] Streaming Large Files using PSGI
> This is as far as I've got. Can anyone help / advise? Maybe patching > CAP::Stream is the way to go - I suppose it should be possible to > detect the running environment and 'do the right thing'. Mike, My reading of the spec is that you can return a callback instead of a body, and the Plack server should handle the rest: https://metacpan.org/module/PSGI#Delayed-Response-and-Streaming-Body If the server doesn't suppport streaming, a MiddleWare can provide it: https://metacpan.org/module/Plack::Middleware::BufferedStreaming I have not needed to do this yet myself with PSGI (but will eventually). Mark # CGI::Application community mailing list #### ## To unsubscribe, or change your message delivery options, ## ## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp## #### ## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ## ## Wiki: http://cgiapp.erlbaum.net/ ## ####
[cgiapp] Streaming Large Files using PSGI
For years I've used the handy CAP::Stream
return $self->stream_file( $file );
method and this does a lovely job of chunking the data so large files
are handled without memory issues. Files could be anything from 10Mb
up to 800Mb ish.
I'm now looking to go 'production' with a PSGI, Starman & Nginx setup.
In PSGI mode we have to set $ENV{'CGI_APP_RETURN_ONLY'} to make
streaming work, because CAP::Stream uses a print direct to STDOUT
(which is a no no for the PSGI spec afaik).
This is bypassed by the env setting, but digging into the internals it
is simply loading the file contents into $output and returning that?
Digging into plack a little, it does support streaming by the return
of a subref:
http://cpansearch.perl.org/src/MIYAGAWA/Plack-1.0004/eg/dot-psgi/echo-stream-sync.psgi
where we can $w->write(..); to the response object inside the chunking
loop (example is not for binary file but I'm assuming it's the same
basic thing).
This is as far as I've got. Can anyone help / advise? Maybe patching
CAP::Stream is the way to go - I suppose it should be possible to
detect the running environment and 'do the right thing'.
Regards,
Mike
# CGI::Application community mailing list
####
## To unsubscribe, or change your message delivery options, ##
## visit: http://www.erlbaum.net/mailman/listinfo/cgiapp##
####
## Web archive: http://www.erlbaum.net/pipermail/cgiapp/ ##
## Wiki: http://cgiapp.erlbaum.net/ ##
####
