Re: [Catalyst] Setting file handle as the response body generates warnings.

[John Napiorkowski]

On Wednesday, November 27, 2013 8:03 AM, neil.lunn  wrote:

On 27/11/2013 4:28 AM, John Napiorkowski wrote:

Awesome, send me a pull request :)
>
>
>https://github.com/perl-catalyst/catalyst-runtime
>
John.
The pull is there along with another doc patch I noticed when
playing around with downstream PSGI invocations. I think the change
is sane as it worked for me in a trivial test.

What I mentioned regarding subclassing ( or rather what I tested was
more of a Moose delegation ) for such a class as
IO::Compress::Gunzip and applying an overload seems rather trivial,
though perhaps something people may come accross. Perhaps a doc
patch to Catalyst::Manual or an Advent entry just for documentation
sake. 

Also passing $c->res->body is something I have used on a few
occasions as something providing a "write" for streaming responses
(and possibly non blocking). So keen to share any thoughts re doing
this non-blocking for "read" on a future release. 

There's a few approaches, you can also access the $response->write_fh which 
returns the psgi $writer (see the PSGI docs for delayed and streaming 
responses. 


Should we really be documenting some of the newer catalyst features
in Cookbook? Or at least putting recipes in Advent . Event loop
things, response from PSGI and how about Pocket::IO seem relevant.

Advent will have a bunch of articles related to new features, and some stuff 
around web sockets. nonblocking.

If someone wants to port any of the web sockets examples to pocket.io or 
similar, for running on older browsers, that would be great.  If you know about 
pocket.io feel free to take a look over at the catalyst a-synch example 
repository on github : https://github.com/jjn1056/Perl-Catalyst-AsyncExample

Thanks!

john





   This email is free from viruses and malware because avast! Antivirus 
protection is active.  

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 27/11/2013 4:28 AM, John Napiorkowski wrote:

Awesome, send me a pull request :)

https://github.com/perl-catalyst/catalyst-runtime

John.
The pull is there along with another doc patch I noticed when playing 
around with downstream PSGI invocations. I think the change is sane as 
it worked for me in a trivial test.


What I mentioned regarding subclassing ( or rather what I tested was 
more of a Moose delegation ) for such a class as IO::Compress::Gunzip 
and applying an overload seems rather trivial, though perhaps something 
people may come accross. Perhaps a doc patch to Catalyst::Manual or an 
Advent entry just for documentation sake.


Also passing $c->res->body is something I have used on a few occasions 
as something providing a "write" for streaming responses (and possibly 
non blocking). So keen to share any thoughts re doing this non-blocking 
for "read" on a future release.


Should we really be documenting some of the newer catalyst features in 
Cookbook? Or at least putting recipes in Advent . Event loop things, 
response from PSGI and how about Pocket::IO seem relevant.



---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[John Napiorkowski]

This would make an awesome Advent article.  If you started it I'd be sure to 
proofread and help out

https://github.com/perl-catalyst/2013-Advent-Staging


There's a few in there you can crib to help you get started.  Something like

what is the problem

show the solution

example

very easy to do!

john



On Tuesday, November 26, 2013 3:14 AM, Craig Chant 
 wrote:
 
I’ve been using this for while….
 
    # check for ok status
    if($result->ok)
    {    
 
    # output header
    $c->response->content_type('application/vnd.ms-excel');
    $c->response->content_length(length($xls));    
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');
  
# create an IO::File for Catalyst
    use IO::File;
    my $iof = IO::File->new;
    
# open XLS
    $iof->open(\$xls, "r");  
  
# output XLS data  
$c->response->body($iof);
    
# close file
    undef $iof;
    }
 
It resolves the out of memory issue if you try to print direct to 
$c->response->body without putting the content in an IO::File object.
 
 
From:John Napiorkowski [mailto:jjn1...@yahoo.com] 
Sent: 26 November 2013 01:54
To: The elegant MVC web framework
Subject: Re: [Catalyst] Setting file handle as the response body generates 
warnings.
 
We should probably document what it means by $c->response->body can accept a 
file handle.  Like plack specifies the interface it must deal with.  Anyone up 
to that?
 
This FH in body thing is something I've been thinking about a lot lately, in 
regards to when Catalyst is running under an event loop for example, should 
Catalyst stream the Filehandle a non blocking manner.  But it would be great to 
document this.  Also, I am thinking setting the default size when its missing 
could really belong to middleware.  Plack has some default middleware that does 
this:
 
https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Middleware/ContentLength.pm
 
the means it works looks like this
 
https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Util.pm#L68
 
There's a few things in Catalyst.pm that could profitable just use 
Plack::Middleware counterparts.
 
just a thought.
 
johnnap
 
On Thursday, November 21, 2013 9:20 AM, neil.lunn  wrote:
On 22/11/2013 12:33 AM, Bill Moseley wrote:
 
> 
>On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn  wrote:
> 
>This approach may work for you is the compressed data is actually in a scalar 
>and not too large. And not too small. YMMV.
>
>my $z = read_file "product.json.gz";
>
>my $io = IO::Scalar->new( \$z );
>$io->seek( -4, 2 );
>$io->read( my $buf, 4);
>
>my $uncompressed_size = unpack( 'V', $buf );
> 
> 
>This indeed does work in my tests.   Thanks for all the help, Neil.   I really 
>appreciate the time you spent on this.
> 
No prob. Part of my thing on list reponses is that there is a seed for others 
to learn from what was discussed. 
I do wonder if there is a sane way to test for an in memory file handle and 
give a more appropriate warning, but that is for further discussion.
 
 
> 
>-- 
>Bill Moseley
>mose...@hank.org 
>
>
>
>___
>List: Catalyst@lists.scsys.co.uk
>Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>Dev site: http://dev.catalyst.perl.org/
> 




 
 This email is free from viruses and malware because avast! Antivirus 
protection is active.  
 
 
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
 
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied 

Re: [Catalyst] Setting file handle as the response body generates warnings.

[John Napiorkowski]

Awesome, send me a pull request :)

https://github.com/perl-catalyst/catalyst-runtime




On Tuesday, November 26, 2013 1:24 AM, neil.lunn  wrote:
 
On 26/11/2013 12:54 PM, John Napiorkowski wrote:

We should probably document what it means by $c->response->body can accept a 
file handle.  Like plack specifies the interface it must deal with.  Anyone up 
to that?
Something along these lines maybe?

Catalyst can accept a file handle type of object as the response
body in the sense that the handle reasonably has a "read" method
that will return content. In such a case and where no content length
has been already set in the response headers catalyst will make a
reasonable attempt to determine the size of the file. Depending on
the implementation of your file handle object setting the content
length may fail. If it is at all possible for you to determine the
content length of your file handle, it is recomended that you set
the content length in the response headers yourself, which will be
respected and sent by catalyst in the response.


>
>This FH in body thing is something I've been thinking about a lot lately, in 
>regards to when Catalyst is running under an event loop for example, should 
>Catalyst stream the Filehandle a non blocking manner.  But it would be great 
>to document this.  Also, I am thinking setting the default size when its 
>missing could really belong to middleware.  Plack has some default middleware 
>that does this:
>
>
>https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Middleware/ContentLength.pm
>
Which interestingly does not take a too different approach to what
the Catalyst code is attempting. Also noting that in a case such as
Bill presented it too would fail, which is only to be expected.

Noting such IO magic modules like IO::Uncompress::* and IO::Scalar
etc and speaking of cat skinning, another approach could be
subclassing these modues and overloading the file test operators to
do something sane when used, and thus return a "correct" response
when the -s test is used. Or try and convince said authors to have
these classes do this natively.



>
>the means it works looks like this
>
>
>https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Util.pm#L68
>
>
>
>There's a few things in Catalyst.pm that could profitable just use 
>Plack::Middleware counterparts.
>
>
>just a thought.
>
>
>johnnap
>
>
>
>On Thursday, November 21, 2013 9:20 AM, neil.lunn  wrote:
> 
>On 22/11/2013 12:33 AM, Bill Moseley wrote:
>
>
>>
>>
>>
>>
>>On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn  wrote:
>>
>>
>>>
This approach may work for you is the compressed data is actually in a scalar 
and not too large. And not too small. YMMV.
>>>
>>>my $z = read_file "product.json.gz";
>>>
>>>my $io = IO::Scalar->new( \$z );
>>>$io->seek( -4, 2 );
>>>$io->read( my $buf, 4);
>>>
>>>my $uncompressed_size = unpack( 'V',
$buf );
>>
>>
>>
>>
>>This indeed does work in my tests.   Thanks for all the help, Neil.   I 
>>really appreciate the time you spent on this.
>>
>>
No prob. Part of my thing on list reponses is that there is a seed for others 
to learn from what was discussed. 
>I do wonder if there is a sane way to test for an in
memory file handle and give a more appropriate
warning, but that is for further discussion. 
>
>
>
>
>>
>>
>>
-- 
>>Bill Moseley
>>mose...@hank.org 
>>
>>
>>___
List: Catalyst@lists.scsys.co.uk Listinfo: 
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: 
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: 
http://dev.catalyst.perl.org/
>
>
>>
> 
>   This email is free from viruses and malware because avast! Antivirus 
> protection is active.  
>
>
>
>___
>List: Catalyst@lists.scsys.co.uk
>Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>Dev site: http://dev.catalyst.perl.org/
>
>
>
>
>
>
>
>
>___
List: Catalyst@lists.scsys.co.uk Listinfo: 
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: 
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: 
http://dev.catalyst.perl.org/ 




 
   This email is free from viruses and malware because avast! Antivirus 
protection is active.  ___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] Setting file handle as the response body generates warnings.

[Craig Chant]

I've been using this for while

# check for ok status
if($result->ok)
{

# output header
$c->response->content_type('application/vnd.ms-excel');
$c->response->content_length(length($xls));
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

# create an IO::File for Catalyst
use IO::File;
my $iof = IO::File->new;

# open XLS
$iof->open(\$xls, "r");

# output XLS data
$c->response->body($iof);

# close file
undef $iof;
}

It resolves the out of memory issue if you try to print direct to 
$c->response->body without putting the content in an IO::File object.


From: John Napiorkowski [mailto:jjn1...@yahoo.com]
Sent: 26 November 2013 01:54
To: The elegant MVC web framework
Subject: Re: [Catalyst] Setting file handle as the response body generates 
warnings.

We should probably document what it means by $c->response->body can accept a 
file handle.  Like plack specifies the interface it must deal with.  Anyone up 
to that?

This FH in body thing is something I've been thinking about a lot lately, in 
regards to when Catalyst is running under an event loop for example, should 
Catalyst stream the Filehandle a non blocking manner.  But it would be great to 
document this.  Also, I am thinking setting the default size when its missing 
could really belong to middleware.  Plack has some default middleware that does 
this:

https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Middleware/ContentLength.pm

the means it works looks like this

https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Util.pm#L68

There's a few things in Catalyst.pm that could profitable just use 
Plack::Middleware counterparts.

just a thought.

johnnap

On Thursday, November 21, 2013 9:20 AM, neil.lunn 
mailto:n...@mylunn.id.au>> wrote:
On 22/11/2013 12:33 AM, Bill Moseley wrote:


On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn 
mailto:n...@mylunn.id.au>> wrote:

This approach may work for you is the compressed data is actually in a scalar 
and not too large. And not too small. YMMV.

my $z = read_file "product.json.gz";

my $io = IO::Scalar->new( \$z );
$io->seek( -4, 2 );
$io->read( my $buf, 4);

my $uncompressed_size = unpack( 'V', $buf );


This indeed does work in my tests.   Thanks for all the help, Neil.   I really 
appreciate the time you spent on this.

No prob. Part of my thing on list reponses is that there is a seed for others 
to learn from what was discussed.
I do wonder if there is a sane way to test for an in memory file handle and 
give a more appropriate warning, but that is for further discussion.



--
Bill Moseley
mose...@hank.org<mailto:mose...@hank.org>



___

List: Catalyst@lists.scsys.co.uk<mailto:Catalyst@lists.scsys.co.uk>

Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst

Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/

Dev site: http://dev.catalyst.perl.org/





[http://static.avast.com/emails/avast-mail-stamp.png]<http://www.avast.com/>

This email is free from viruses and malware because avast! 
Antivirus<http://www.avast.com/> protection is active.



___
List: Catalyst@lists.scsys.co.uk<mailto:Catalyst@lists.scsys.co.uk>
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
registered in England and Wales with Registration Number 50117

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 26/11/2013 12:54 PM, John Napiorkowski wrote:
We should probably document what it means by $c->response->body can 
accept a file handle.  Like plack specifies the interface it must deal 
with.  Anyone up to that?


Something along these lines maybe?

Catalyst can accept a file handle type of object as the response body in 
the sense that the handle reasonably has a "read" method that will 
return content. In such a case and where no content length has been 
already set in the response headers catalyst will make a reasonable 
attempt to determine the size of the file. Depending on the 
implementation of your file handle object setting the content length may 
fail. If it is at all possible for you to determine the content length 
of your file handle, it is recomended that you set the content length in 
the response headers yourself, which will be respected and sent by 
catalyst in the response.


This FH in body thing is something I've been thinking about a lot 
lately, in regards to when Catalyst is running under an event loop for 
example, should Catalyst stream the Filehandle a non blocking manner. 
 But it would be great to document this.  Also, I am thinking setting 
the default size when its missing could really belong to middleware. 
 Plack has some default middleware that does this:


https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Middleware/ContentLength.pm


Which interestingly does not take a too different approach to what the 
Catalyst code is attempting. Also noting that in a case such as Bill 
presented it too would fail, which is only to be expected.


Noting such IO magic modules like IO::Uncompress::* and IO::Scalar etc 
and speaking of cat skinning, another approach could be subclassing 
these modues and overloading the file test operators to do something 
sane when used, and thus return a "correct" response when the -s test is 
used. Or try and convince said authors to have these classes do this 
natively.




the means it works looks like this

https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Util.pm#L68

There's a few things in Catalyst.pm that could profitable just use 
Plack::Middleware counterparts.


just a thought.

johnnap


On Thursday, November 21, 2013 9:20 AM, neil.lunn  
wrote:

On 22/11/2013 12:33 AM, Bill Moseley wrote:




On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn > wrote:



This approach may work for you is the compressed data is actually
in a scalar and not too large. And not too small. YMMV.

my $z = read_file "product.json.gz";

my $io = IO::Scalar->new( \$z );
$io->seek( -4, 2 );
$io->read( my $buf, 4);

my $uncompressed_size = unpack( 'V', $buf );



This indeed does work in my tests. Thanks for all the help, Neil.   I 
really appreciate the time you spent on this.


No prob. Part of my thing on list reponses is that there is a seed for 
others to learn from what was discussed.
I do wonder if there is a sane way to test for an in memory file 
handle and give a more appropriate warning, but that is for further 
discussion.






--
Bill Moseley
mose...@hank.org 


___ List: 
Catalyst@lists.scsys.co.uk  
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst 
Searchable archive: 
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: 
http://dev.catalyst.perl.org/







   
This email is free from viruses and malware because avast! Antivirus 
 protection is active.




___
List: Catalyst@lists.scsys.co.uk 
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: 
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/

Dev site: http://dev.catalyst.perl.org/




___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[John Napiorkowski]

We should probably document what it means by $c->response->body can accept a 
file handle.  Like plack specifies the interface it must deal with.  Anyone up 
to that?

This FH in body thing is something I've been thinking about a lot lately, in 
regards to when Catalyst is running under an event loop for example, should 
Catalyst stream the Filehandle a non blocking manner.  But it would be great to 
document this.  Also, I am thinking setting the default size when its missing 
could really belong to middleware.  Plack has some default middleware that does 
this:

https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Middleware/ContentLength.pm


the means it works looks like this

https://metacpan.org/source/MIYAGAWA/Plack-1.0030/lib/Plack/Util.pm#L68


There's a few things in Catalyst.pm that could profitable just use 
Plack::Middleware counterparts.

just a thought.

johnnap



On Thursday, November 21, 2013 9:20 AM, neil.lunn  wrote:
 
On 22/11/2013 12:33 AM, Bill Moseley wrote:


>
>
>
>
>On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn  wrote:
>
>
>>
This approach may work for you is the compressed data is actually in a scalar 
and not too large. And not too small. YMMV.
>>
>>my $z = read_file "product.json.gz";
>>
>>my $io = IO::Scalar->new( \$z );
>>$io->seek( -4, 2 );
>>$io->read( my $buf, 4);
>>
>>my $uncompressed_size = unpack( 'V', $buf );
>
>
>
>
>This indeed does work in my tests.   Thanks for all the help, Neil.   I really 
>appreciate the time you spent on this.
>
>
No prob. Part of my thing on list reponses is that there is a seed for others 
to learn from what was discussed. 
I do wonder if there is a sane way to test for an in memory file
handle and give a more appropriate warning, but that is for further
discussion.



>
>
>
-- 
>Bill Moseley
>mose...@hank.org 
>
>
>___
List: Catalyst@lists.scsys.co.uk Listinfo: 
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: 
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: 
http://dev.catalyst.perl.org/



 
   This email is free from viruses and malware because avast! Antivirus 
protection is active.  


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 22/11/2013 12:33 AM, Bill Moseley wrote:




On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn > wrote:



This approach may work for you is the compressed data is actually
in a scalar and not too large. And not too small. YMMV.

my $z = read_file "product.json.gz";

my $io = IO::Scalar->new( \$z );
$io->seek( -4, 2 );
$io->read( my $buf, 4);

my $uncompressed_size = unpack( 'V', $buf );



This indeed does work in my tests.   Thanks for all the help, Neil.   
I really appreciate the time you spent on this.


No prob. Part of my thing on list reponses is that there is a seed for 
others to learn from what was discussed.
I do wonder if there is a sane way to test for an in memory file handle 
and give a more appropriate warning, but that is for further discussion.





--
Bill Moseley
mose...@hank.org 


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[Bill Moseley]

On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn  wrote:

>
> This approach may work for you is the compressed data is actually in a
> scalar and not too large. And not too small. YMMV.
>
> my $z = read_file "product.json.gz";
>
> my $io = IO::Scalar->new( \$z );
> $io->seek( -4, 2 );
> $io->read( my $buf, 4);
>
> my $uncompressed_size = unpack( 'V', $buf );
>


This indeed does work in my tests.   Thanks for all the help, Neil.   I
really appreciate the time you spent on this.



-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 21/11/2013 11:48 AM, Bill Moseley wrote:


$ perl -MIO::Uncompress::Gunzip -le 'use Data::Dumper; print Dumper 
+IO::Uncompress::Gunzip->new( "Catalyst-Runtime-5.90051.tar.gz" 
)->getHeaderInfo'


Actually with more reading on this getting the uncompressed size 
reliably can be a real pain, and thus is not in the getHeaderInfo over 
certain size lmits.


This approach may work for you is the compressed data is actually in a 
scalar and not too large. And not too small. YMMV.


my $z = read_file "product.json.gz";

my $io = IO::Scalar->new( \$z );
$io->seek( -4, 2 );
$io->read( my $buf, 4);

my $uncompressed_size = unpack( 'V', $buf );




___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 21/11/2013 1:04 PM, neil.lunn wrote:

On 21/11/2013 11:48 AM, Bill Moseley wrote:


$ perl -MIO::Uncompress::Gunzip -le 'use Data::Dumper; print Dumper 
+IO::Uncompress::Gunzip->new( "Catalyst-Runtime-5.90051.tar.gz" 
)->getHeaderInfo'

Oh, and a test case outside of catalyst, noting the -s operator fail:


echo "1234567890ABCDEFG" | gzip -c >x.gz

#!/usr/bin/env perl
use Modern::Perl;

use File::Slurp;
use IO::Uncompress::Gunzip;
use Data::Dumper;

my $z = read_file "x.gz";
my $fh = IO::Uncompress::Gunzip->new( \$z );

my $size = -s $fh;

print Dumper( $fh->getHeaderInfo );
print "Size: ", $fh->getHeaderInfo->{ISIZE}, "\n";
print $fh->getline();
$fh->close();

-s on unopened filehandle GEN0 at test.pl line 11.
$VAR1 = {
  'NameFlag' => 0,
  'TrailerLength' => 8,
  'Header' => '??R',
  'CRC32' => 3714309751,
  'ISIZE' => 18,
  'ExtraFlag' => 0,
  'Flags' => 0,
  'Comment' => undef,
  'isMinimalHeader' => 0,
  'ExtraField' => [],
  'HeaderLength' => 10,
  'OsID' => 3,
  'MethodID' => 8,
  'MethodName' => 'Deflated',
  'HeaderCRC' => undef,
  'TextFlag' => 0,
  'Name' => undef,
  'CommentFlag' => 0,
  'HeaderCRCFlag' => 0,
  'Type' => 'rfc1952',
  'FingerprintLength' => 2,
  'Time' => 1384950479,
  'ExtraFlags' => 0,
  'ExtraFieldRaw' => undef,
  'OsName' => 'Unix'
};
Size: 18
1234567890ABCDEFG


$VAR1 = {
  'Time' => 1383843952,
  'Flags' => 8,
  'TextFlag' => 0,
  'MethodID' => 8,
  'ExtraField' => [],
  'CommentFlag' => 0,
  'Type' => 'rfc1952',
  'NameFlag' => 1,
  'ExtraFlags' => 2,
  'HeaderCRC' => undef,
  'isMinimalHeader' => 0,
  'MethodName' => 'Deflated',
  'ExtraFlag' => 0,
  'HeaderLength' => 39,
  'ExtraFieldRaw' => undef,
  'Comment' => undef,
  'OsName' => 'Unix',
  'FingerprintLength' => 2,
  'HeaderCRCFlag' => 0,
  'OsID' => 3,
  'TrailerLength' => 8,
  'Name' => 'Catalyst-Runtime-5.90051.tar',
  'Header' => p?{RCatalyst-Runtime-5.90051.tar'
};


--
Bill Moseley
mose...@hank.org 


___
List:Catalyst@lists.scsys.co.uk
Listinfo:http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive:http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site:http://dev.catalyst.perl.org/





   

This email is free from viruses and malware because avast! Antivirus 
 protection is active.





___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 21/11/2013 11:48 AM, Bill Moseley wrote:


Seems noncompliance may be rampant.

Anyway, sounds like Catalyst isn't quite supporting this kind of file 
handle as expected.   John, is there anything you would want me to try?




Hi Bill. Back to my original response, trying to get the size of this 
handle (or any other type of in memory handle) with a file stat operator 
will not work. What catalyst is trying to do:


$size = -s $response->body;

Because it reasonably expects you have provided a handle that can read 
and most importantly you have not provided a content length. Being the 
earlier conditional.


Thinking this through, if the file was on disk then -s would not return 
the uncompressed size. Which is what you want.


As for that not working for this type of handle ( aslo see something 
like IO::Scalar ), I can't see why Catalyst should be expected to do 
that for you. Which is why there is the provision to provide a 
content_length and avioding this fallback condition.


Your issue as I see it, is that something is borked on your setup with 
the Gzip implementation that is stopping you from getting the 
uncompressed size from the content. Therefore I suggest you start 
looking there.


Again I cannot see how or why Catalyst could or should be expected to 
work this out for you, and the best method is to set the correct 
content_length once you have sorted out the Gzip issue. Or failing that 
do you have some other way in your implementation to know the 
content_length when uncompressed.


As below just works, which is as near as I can see to what you are 
basically doing.



package Gzip::Web::Controller::Root;
use Moose;
use namespace::autoclean;

BEGIN { extends 'Catalyst::Controller' }

use IO::Compress::Gzip qw/gzip $GzipError/;
use IO::Uncompress::Gunzip;
use Data::Dumper;

__PACKAGE__->config(namespace => '');

sub index :Path :Args(0) {
  my ( $self, $c ) = @_;

  my $data = "123456890ABCDEFGHIGQWERRTYYUIO";
  my ($comp, $body);

  gzip(\$data, \$comp) || die $GzipError;

  $c->res->content_type('text/plain');

  if ( $c->req->header('accept-encoding') =~ /gzip/ ) {
$c->log->debug( 'Sending compressed' );
$c->res->content_encoding('gzip');
$body = $comp;
  } else {
$c->log->debug( 'Sending uncompressed' );
$body = IO::Uncompress::Gunzip->new( \$comp );
$c->res->content_length( $body->getHeaderInfo->{ISIZE} );
$c->log->debug( Dumper( $body->getHeaderInfo ) );
  }

  $c->res->body( $body );

}






$ perl -MIO::Uncompress::Gunzip -le 'use Data::Dumper; print Dumper 
+IO::Uncompress::Gunzip->new( "Catalyst-Runtime-5.90051.tar.gz" 
)->getHeaderInfo'

$VAR1 = {
  'Time' => 1383843952,
  'Flags' => 8,
  'TextFlag' => 0,
  'MethodID' => 8,
  'ExtraField' => [],
  'CommentFlag' => 0,
  'Type' => 'rfc1952',
  'NameFlag' => 1,
  'ExtraFlags' => 2,
  'HeaderCRC' => undef,
  'isMinimalHeader' => 0,
  'MethodName' => 'Deflated',
  'ExtraFlag' => 0,
  'HeaderLength' => 39,
  'ExtraFieldRaw' => undef,
  'Comment' => undef,
  'OsName' => 'Unix',
  'FingerprintLength' => 2,
  'HeaderCRCFlag' => 0,
  'OsID' => 3,
  'TrailerLength' => 8,
  'Name' => 'Catalyst-Runtime-5.90051.tar',
  'Header' => p?{RCatalyst-Runtime-5.90051.tar'
};


--
Bill Moseley
mose...@hank.org 


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[Bill Moseley]

On Wed, Nov 20, 2013 at 3:45 PM, neil.lunn  wrote:

>   2.3.1.2. Compliance
>
>
>  A compliant compressor must produce files with correct ID1,
>  ID2, CM, CRC32, and ISIZE, but may set all the other fields in
>  the fixed-length part of the header to default values (255 for
>  OS, 0 for all others).  The compressor must set all reserved
>  bits to zero.
>
> 
>

Seems noncompliance may be rampant.

Anyway, sounds like Catalyst isn't quite supporting this kind of file
handle as expected.   John, is there anything you would want me to try?




$ perl -MIO::Uncompress::Gunzip -le 'use Data::Dumper; print Dumper
+IO::Uncompress::Gunzip->new( "Catalyst-Runtime-5.90051.tar.gz"
)->getHeaderInfo'
$VAR1 = {
  'Time' => 1383843952,
  'Flags' => 8,
  'TextFlag' => 0,
  'MethodID' => 8,
  'ExtraField' => [],
  'CommentFlag' => 0,
  'Type' => 'rfc1952',
  'NameFlag' => 1,
  'ExtraFlags' => 2,
  'HeaderCRC' => undef,
  'isMinimalHeader' => 0,
  'MethodName' => 'Deflated',
  'ExtraFlag' => 0,
  'HeaderLength' => 39,
  'ExtraFieldRaw' => undef,
  'Comment' => undef,
  'OsName' => 'Unix',
  'FingerprintLength' => 2,
  'HeaderCRCFlag' => 0,
  'OsID' => 3,
  'TrailerLength' => 8,
  'Name' => 'Catalyst-Runtime-5.90051.tar',
  'Header' => p�{RCatalyst-Runtime-5.90051.tar'
};



-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 21/11/2013 4:08 AM, Bill Moseley wrote:


On Wed, Nov 20, 2013 at 7:37 AM, Bill Moseley > wrote:



On Wed, Nov 20, 2013 at 4:08 AM, neil.lunn mailto:n...@mylunn.id.au>> wrote:

my $length = $body->getHeaderInfo


Well, that's helpful.  Thanks.   Completely missed that in the docs.


Well, except for the actual files I have no ISIZE:


Don't know. I would suspect you have a version issue somewhere. But not 
a Catalyst thing. From the Gzip RFC:




 2.3.1.2. Compliance

 A compliant compressor must produce files with correct ID1,
 ID2, CM, CRC32, and ISIZE, but may set all the other fields in
 the fixed-length part of the header to default values (255 for
 OS, 0 for all others).  The compressor must set all reserved
 bits to zero.





$VAR1 = {
  'Time' => 0,
  'Flags' => 0,
  'TextFlag' => 0,
  'MethodID' => 8,
  'ExtraField' => [],
  'CommentFlag' => 0,
  'Type' => 'rfc1952',
  'NameFlag' => 0,
  'ExtraFlags' => 0,
  'HeaderCRC' => undef,
  'isMinimalHeader' => 0,
  'MethodName' => 'Deflated',
  'ExtraFlag' => 0,
  'HeaderLength' => 10,
  'ExtraFieldRaw' => undef,
  'Comment' => undef,
  'OsName' => 'Unix',
  'FingerprintLength' => 2,
  'HeaderCRCFlag' => 0,
  'OsID' => 3,
  'TrailerLength' => 8,
  'Name' => undef,
  'Header' => ''
};

And even if I gzip a file (e.g. $ gzip foo.txt) then the resulting 
foo.txt.gz has 'ISIZE' => 0,




--
Bill Moseley
mose...@hank.org 


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[Bill Moseley]

On Wed, Nov 20, 2013 at 7:37 AM, Bill Moseley  wrote:

>
> On Wed, Nov 20, 2013 at 4:08 AM, neil.lunn  wrote:
>
>> my $length = $body->getHeaderInfo
>
>
> Well, that's helpful.  Thanks.   Completely missed that in the docs.
>

Well, except for the actual files I have no ISIZE:

$VAR1 = {
  'Time' => 0,
  'Flags' => 0,
  'TextFlag' => 0,
  'MethodID' => 8,
  'ExtraField' => [],
  'CommentFlag' => 0,
  'Type' => 'rfc1952',
  'NameFlag' => 0,
  'ExtraFlags' => 0,
  'HeaderCRC' => undef,
  'isMinimalHeader' => 0,
  'MethodName' => 'Deflated',
  'ExtraFlag' => 0,
  'HeaderLength' => 10,
  'ExtraFieldRaw' => undef,
  'Comment' => undef,
  'OsName' => 'Unix',
  'FingerprintLength' => 2,
  'HeaderCRCFlag' => 0,
  'OsID' => 3,
  'TrailerLength' => 8,
  'Name' => undef,
  'Header' => ''
};

And even if I gzip a file (e.g. $ gzip foo.txt) then the resulting
foo.txt.gz has 'ISIZE' => 0,



-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[Bill Moseley]

On Wed, Nov 20, 2013 at 4:08 AM, neil.lunn  wrote:

> my $length = $body->getHeaderInfo


Well, that's helpful.  Thanks.   Completely missed that in the docs.


-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[neil.lunn]

On 20/11/2013 3:01 PM, Bill Moseley wrote:


I just updated:

$ perl -MCatalyst -le 'print Catalyst->VERSION'
5.90051

 I can debug more in the morning -- anything specific I should look at?
I think the stat operators have this problem on any type of in memory 
filehandle. That is pretty easy to reproduce.
To me anyhow this wouldn't seem to make much sense to me to test the 
length of a yet uncompressed filehandle, even if it did just work somehow.


For my money I would do the following:

my $body = IO::Uncompress::Gunzip->new( \$gzipped_data); #Assuming scalar
my $length = $body->getHeaderInfo->{ISIZE};
$c->res->content_length( $length );
$c->res->body( $body );

So basically getting the length of the uncompressed data from the handle 
by the object method. So now Catalyst won't try to set the size because 
you already have. Oh, and the content type and well of course :)






It's pretty easy to reproduce:

package Zip::Controller::Root;
...

use IO::Uncompress::Gunzip;
use IO::Compress::Gzip qw/ gzip $GzipError /;

sub uncompress : Local {
my ( $self, $c ) = @_;

my $uncompressed = "This is some text that can be
compressed.\n" x 5;
my $compressed;
gzip( \$uncompressed, \$compressed ) || die $GzipError;

$c->res->body( IO::Uncompress::Gunzip->new( \$compressed ) );
return;
}



$ script/zip_test.pl  /uncompress

-s on unopened filehandle GEN2 at 
/home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst.pm 
line 1948,  line 1000.

[warn] Serving filehandle without a content-length
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.


--
Bill Moseley
mose...@hank.org 


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[Bill Moseley]

On Tue, Nov 19, 2013 at 6:34 PM, John Napiorkowski wrote:

>
> This looks familiar, but I thought we fixed this... what version of
> Catalyst are you using here?
>

I just updated:


$ perl -MCatalyst -le 'print Catalyst->VERSION'
5.90051

 I can debug more in the morning -- anything specific I should look at?

It's pretty easy to reproduce:

package Zip::Controller::Root;
...

use IO::Uncompress::Gunzip;
use IO::Compress::Gzip qw/ gzip $GzipError /;

sub uncompress : Local {
my ( $self, $c ) = @_;

my $uncompressed = "This is some text that can be compressed.\n" x 5;
my $compressed;
gzip( \$uncompressed, \$compressed ) || die $GzipError;

$c->res->body( IO::Uncompress::Gunzip->new( \$compressed ) );

return;
}



$ script/zip_test.pl /uncompress

-s on unopened filehandle GEN2 at
/home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst.pm
line 1948,  line 1000.
[warn] Serving filehandle without a content-length
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.


-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] Setting file handle as the response body generates warnings.

[John Napiorkowski]

This works, but Catalyst in finalize_headers issues two warnings:

-s on unopened filehandle GEN10 at 
/home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst.pm 
line 1893.
>
>
>[warn] Serving filehandle without a content-length


Are those warning a problem with how Catalyst is handling this, or something 
wrong with how IO::Uncompress::Gunzip is working?


Bill,

This looks familiar, but I thought we fixed this... what version of Catalyst 
are you using here?

John


-- 
Bill Moseley
mose...@hank.org 
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/