Re: [pylons-discuss] Pyramid Response with File-like .write() interface

2022-08-10 Thread Mikko Ohtamaa
Hi Michael, others,

On Tue, 9 Aug 2022 at 20:28, Michael Merickel  wrote:

> This harkens back to the discouraged write() callable in WSGI PEP 
> returned by the start_response() invocation. The PEP as well as Pyramid as
> a framework would encourage you to map the logic into an app_iter as Bert
> suggested.
>
> I think you'll want to define a file-like object that you can write to and
> set as the app_iter. The question will be whether you try to do this by
> writing from a separate thread or in some other way because once you return
> control to the WSGI server to iterate on your app_iter then you are no
> longer in control - you'll need some buffer between where you're generating
> your parquet file and what you're returning from the app_iter. I don't
> think a simple tempfile is good enough because you want the app_iter to
> wait instead of stopping when it hits an EOF unless you know that you've
> reached the end of the buffer.
>
>
Thank you for the good commentary.

This is very good insight, I didn't think about the threading issue. I
think I will attempt an approach where I just use a buffer like Theron
mentioned and then have a library to write from the buffer from another
thread.

Br,
Mikko

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUvKj%3DUAeESmWmauVnhBzSObH3jhKeori7GwOJkKkvaPKA%40mail.gmail.com.


Re: [pylons-discuss] Pyramid Response with File-like .write() interface

2022-08-09 Thread Theron Luhn
Could probably jury rig it together with a Queue.  (Will need to run Parquet in 
a separate thread.)

class FileLikeIter:
def __init__(self):
self.q = Queue(1)

def write(self, data):
self.q.put(data)

def __iter__(self):
try:
while True:
yield self.q.get()
except Empty:
...

— Theron



> On Aug 9, 2022, at 11:27 AM, Michael Merickel  wrote:
> 
> This harkens back to the discouraged write() callable in WSGI PEP  
> returned by the start_response() invocation. The PEP as well as Pyramid as a 
> framework would encourage you to map the logic into an app_iter as Bert 
> suggested.
> 
> I think you'll want to define a file-like object that you can write to and 
> set as the app_iter. The question will be whether you try to do this by 
> writing from a separate thread or in some other way because once you return 
> control to the WSGI server to iterate on your app_iter then you are no longer 
> in control - you'll need some buffer between where you're generating your 
> parquet file and what you're returning from the app_iter. I don't think a 
> simple tempfile is good enough because you want the app_iter to wait instead 
> of stopping when it hits an EOF unless you know that you've reached the end 
> of the buffer.
> 
> On Tue, Aug 9, 2022 at 12:59 PM Bert JW Regeer  > wrote:
> Some WSGI servers pass you the raw file descriptor as wsgi.input, but this is 
> not guaranteed (wsgiref does for instance). Instead you should return an 
> iterator that can be read incrementally so that your WSGI server can chunk 
> responses.
> 
> 
> 
>> On Aug 9, 2022, at 10:38, Mikko Ohtamaa > > wrote:
>> 
>> Hi all,
>> 
>> I'd like to stream dynamically generated Parquet-files from Pyramid server. 
>> Parquet library itself offers writing to any file-like object. I am aware of 
>> app_iter and FileResponse interfaces in Pyramid. However, does Pyramid (or 
>> any example or utility class) offer a Python file-like interface, where I 
>> could just dynamically .write() stuff?
>> 
>> Br,
>> Mikko
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discuss+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUudomfEGYnqAUk57XgOvatDLtFGGk%2Be5tFzu0w81Ez4Lg%40mail.gmail.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/2DB1D478-20A5-4F02-96BE-8F0EA792AFBF%400x58.com
>  
> .
> 
> 
> -- 
> 
> Michael
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwHF%2BJjGydJ28bHneP1_GXvN_P%3DyN3W2FnSAQ_Cea-xbjw%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/E70BB1DE-7B0B-4A0B-AE57-DB5F9F7DE15A%40luhn.com.


Re: [pylons-discuss] Pyramid Response with File-like .write() interface

2022-08-09 Thread Michael Merickel
This harkens back to the discouraged write() callable in WSGI PEP 
returned by the start_response() invocation. The PEP as well as Pyramid as
a framework would encourage you to map the logic into an app_iter as Bert
suggested.

I think you'll want to define a file-like object that you can write to and
set as the app_iter. The question will be whether you try to do this by
writing from a separate thread or in some other way because once you return
control to the WSGI server to iterate on your app_iter then you are no
longer in control - you'll need some buffer between where you're generating
your parquet file and what you're returning from the app_iter. I don't
think a simple tempfile is good enough because you want the app_iter to
wait instead of stopping when it hits an EOF unless you know that you've
reached the end of the buffer.

On Tue, Aug 9, 2022 at 12:59 PM Bert JW Regeer  wrote:

> Some WSGI servers pass you the raw file descriptor as wsgi.input, but this
> is not guaranteed (wsgiref does for instance). Instead you should return an
> iterator that can be read incrementally so that your WSGI server can chunk
> responses.
>
>
>
> On Aug 9, 2022, at 10:38, Mikko Ohtamaa  wrote:
>
> Hi all,
>
> I'd like to stream dynamically generated Parquet-files from Pyramid
> server. Parquet library itself offers writing to any file-like object. I am
> aware of app_iter and FileResponse interfaces in Pyramid. However, does
> Pyramid (or any example or utility class) offer a Python file-like
> interface, where I could just dynamically .write() stuff?
>
> Br,
> Mikko
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUudomfEGYnqAUk57XgOvatDLtFGGk%2Be5tFzu0w81Ez4Lg%40mail.gmail.com
> 
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/2DB1D478-20A5-4F02-96BE-8F0EA792AFBF%400x58.com
> 
> .
>


-- 

Michael

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwHF%2BJjGydJ28bHneP1_GXvN_P%3DyN3W2FnSAQ_Cea-xbjw%40mail.gmail.com.


Re: [pylons-discuss] Pyramid Response with File-like .write() interface

2022-08-09 Thread Bert JW Regeer
Some WSGI servers pass you the raw file descriptor as wsgi.input, but this is 
not guaranteed (wsgiref does for instance). Instead you should return an 
iterator that can be read incrementally so that your WSGI server can chunk 
responses.



> On Aug 9, 2022, at 10:38, Mikko Ohtamaa  wrote:
> 
> Hi all,
> 
> I'd like to stream dynamically generated Parquet-files from Pyramid server. 
> Parquet library itself offers writing to any file-like object. I am aware of 
> app_iter and FileResponse interfaces in Pyramid. However, does Pyramid (or 
> any example or utility class) offer a Python file-like interface, where I 
> could just dynamically .write() stuff?
> 
> Br,
> Mikko
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUudomfEGYnqAUk57XgOvatDLtFGGk%2Be5tFzu0w81Ez4Lg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/2DB1D478-20A5-4F02-96BE-8F0EA792AFBF%400x58.com.


signature.asc
Description: Message signed with OpenPGP