So there is a lot here and there's definitely easier ways to do it but just to start, Mojo::UserAgent can handle the streaming response in a more async manner so you may want to start by replacing the shell out to curl with that. Then see https://metacpan.org/pod/Mojolicious::Guides::Cookbook#Streaming-response for a way to deal with the response as it is received, and https://metacpan.org/pod/Mojolicious::Guides::Rendering#Streaming for how to write it to the response.
-Dan On Mon, May 4, 2020 at 3:05 AM Joseph Fridy <jmfr...@gmail.com> wrote: > I am attempting to stream the output of a curl command with Mojolicious. > The curl command looks (roughly) like this: > > curl -H "x-amz-server-side-encryption-customer-algorithm:AES256" -H > "x-amz-server-side-encryption-cust\ > > omer-key:secretKey=" -H "x-amz-server-side-encryption-customer-key-M\ > > D5:HashOfASecret==" > "https://mybucket.s3.amazonaws.com/obscureFileLocation?AWSAccessKeyId=secretStuff&Expires=1588568911&Signature=moreSecrets"; > --\ > > dump-header header.461 --silent > > > The curl command references a pre-signed URL of a file in AWS stored with > Server Side Encryption with Client Side Keys (SSE-C), and supplies the > necessary key information via HTTP headers (the -H command options). The > curl command works - but I don't want my users to have to have a system > with curl on it to access their files. The plan is to open the curl > command as input to a pipe, and stream its output to the user's browser > with Mojolicious. The curl command also dumps out the HTTP headers from > Amazon, so they can be used by Mojolicious. They look like this: > > > x-amz-id-2: > sgMzHD2FJEGJrcbvzQwdhZK6mxUW+ePd6xdghTfgSlV45lMhliIw4prfk4cZMTHbS4fJN8N7xio= > > x-amz-request-id: 99B9CA56083DD9ED > > Date: Mon, 04 May 2020 04:57:22 GMT > > Last-Modified: Sat, 02 May 2020 03:47:35 GMT > > ETag: "b3a11409be2705e4581119fa59af79d3-1025" > > x-amz-server-side-encryption-customer-algorithm: AES256 > > x-amz-server-side-encryption-customer-key-MD5: HashOfSecretKey== > > Content-Disposition: attachment; filename = "fiveGigFile" > > Accept-Ranges: bytes > > Content-Type: application/octet-stream; charset=UTF-8 > > Content-Length: 5368709125 > > Server: AmazonS3 > > > Note that the file is 5Gig. > > > > This is my stab at streaming with Mojolicious: > > > use strict; > use Mojolicious::Lite; > use FileHandle; > use Digest::MD5; > > any '/' => sub { > my $c = shift; > $c->render(template => "test"); > }; > > any 'pickup' => sub { > my $c = shift; > my $nGigs = 0; > my $nMegs = 0; > $| = 1; > open(CURLCMD,"curlCmd"); > my $curlCmd = <CURLCMD>; > if ($curlCmd =~ /dump-header\s*(\S+)\s+/) { > > my $headerFile = $1; > open(my $curl,"$curlCmd |"); > binmode $curl; > my $initialized = 0; > my $digester = Digest::MD5->new; > > my $transferLength = 0; > > my $drain; > > $drain = sub { > > my $c = shift; > > my $chunk; > > sysread($curl,$chunk,1024*1024); > > if (!$initialized) { > # read the headers, and set up the transfer... > > open(HEADERS,$headerFile); > > while(my $line = <HEADERS>) { > > $c->res->headers->parse($line); > > } > close(HEADERS); > > $initialized = 1; > > print "header initialization completed for the following > headers\n"; > > print join("\n",@{$c->res->headers->names}),"\n"; > > } > > if ($initialized) { > > while (length($chunk)) { > > $digester->add($chunk); > > $transferLength += length($chunk); > > $c->write($chunk,$drain); > > my $currentMegs = int($transferLength/(1024*1024)); > > if (($currentMegs > $nMegs) && ($currentMegs < 1024)) { > > print "TransferLength: $transferLength\n"; > > $nMegs = $currentMegs; > > } > > my $currentGigs = int($transferLength/(1024*1024*1024)); > > if ($currentGigs > $nGigs) { > > print "TransferLength: $transferLength\n"; > > $nGigs = $currentGigs; > > } > > } > > if (length($chunk) <= 1) { > > if ($chunk == 0) { > > print "End of file found on curl pipe."; > > print "$transferLength bytes transmitted\n"; > > print "with an MD5 hash of ",$digester->hexdigest,"\n"; > > $drain = undef; > > } > > if (!defined $chunk) { > > print "Transfer error encountered on curl pipe.\n"; > > print "Error:",$!,"\n"; > > $drain = undef; > > } > > } > > } > > }; > > $c->$drain; > > } > > }; > > app->start; > > __DATA__ > > > @@ test.html.ep > > <!DOCTYPE html> > > <html> > > <body> > > <a href="/pickup" >Test of curl streaming... </a> > > </body> > > </html> > > > > When I ran this the first time, it read about 606MB of data, and the > server crashed with an "Out of memory!". Subsequent runs failed at about > 139MB, with a server crash and no "Out of memory!" message. > > > Obviously, I am an idiot. Some guidance in the precise way I am being an > idiot would be greatly appreciated. > > > Regards, > > > Joe Fridy > > -- > You received this message because you are subscribed to the Google Groups > "Mojolicious" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to mojolicious+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/mojolicious/86c878ad-8bd2-4a40-94fd-cc16fbe39201%40googlegroups.com > <https://groups.google.com/d/msgid/mojolicious/86c878ad-8bd2-4a40-94fd-cc16fbe39201%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/CABMkAVWxcEGpd1ZHDNUWZJoRddvb7iE18RAZ%2BTCx7uqkhMj4KQ%40mail.gmail.com.
