Re: python - handling HTTP requests asynchronously

[edgargdcv]

Could you please help me showing me your code ? I'm in a situation similar 
greetings .
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python - handling HTTP requests asynchronously

[Michael Selik]

On Sat, May 7, 2016 at 4:46 PM  wrote:

> Il giorno sabato 7 maggio 2016 21:04:47 UTC+2, Michael Selik ha scritto:
> > On Fri, May 6, 2016 at 3:01 AM  wrote:
> >
> > > The PDF is generated through an external API. Since currently is
> generated
> > > on demand, this is handled synchronously via an HTTP request/response.
> >
> >
> > Are you sending the request or are you receiving the request?
> > If you are sending, you can just use threads as you are only doing IO.
> > If you are receiving the requests and generating PDFs, you may want to
> use
> > subprocesses if the PDF-generation is compute-intensive.
> >
> >
> > > 3) multiprocessing module, with e.g. 10 as workers limit.
> > >
> >
> > multiprocessing.Pool is an easy way to use subprocesses
> > multiprocessing.pool.ThreadPool is an easy way to use threads. It's not
> > well documented, but has the exact same interface as Pool.
> >
> > the goal is to use the API concurrently (e.g. send 10 or 100 http
> requests
> > > simultaneously, depending on how many concurrent requests the API can
> > > handle).
> > >
> >
> > Sounds like you want to use threads. How does the API let you know you're
> > hitting it too frequently? Perhaps you want to code an exponential
> backoff
> > and retry wrapper for your API requests.
>
> Thanks for the reply.
> Currently the django view that does the job does three http request:
> - the first does a POST and send the payload used during the PDF
> rendering, the response contains a url to check the pdf generation progress;
> - the second loop over that url, checking the progress of pdf generation.
> Once the response contains the keyword 'status': 'complete', then it give
> also a url for the file retrieval;
> - the third one is a GET to retrieve the file, the reponse contains the
> binary content of the file, then this content is read and wrapped as
> attachment of a django http response, and then returned to the user.
>
> the goal is to reuse the existing code as much as possible, possibly doing
> concurrently, and saving the file instead on a folder.
>

I'm not a Django expert. Does the Django View require all that stuff to
happen before Django can send an HTTP Response back to the user? If so, I
suggest you respond to the user immediately: "now generating a bunch of
PDFs..." and take care of the rest in the background. Perhaps just write to
a file, database, or message queue the info for the PDFs to generate. Have
a separate program periodically read the file, database, or message queue
to do that work and then maybe email the user when it's completed.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python - handling HTTP requests asynchronously

[lordluke80]

Il giorno sabato 7 maggio 2016 21:04:47 UTC+2, Michael Selik ha scritto:
> On Fri, May 6, 2016 at 3:01 AM  wrote:
> 
> > The PDF is generated through an external API. Since currently is generated
> > on demand, this is handled synchronously via an HTTP request/response.
> 
> 
> Are you sending the request or are you receiving the request?
> If you are sending, you can just use threads as you are only doing IO.
> If you are receiving the requests and generating PDFs, you may want to use
> subprocesses if the PDF-generation is compute-intensive.
> 
> 
> > 3) multiprocessing module, with e.g. 10 as workers limit.
> >
> 
> multiprocessing.Pool is an easy way to use subprocesses
> multiprocessing.pool.ThreadPool is an easy way to use threads. It's not
> well documented, but has the exact same interface as Pool.
> 
> the goal is to use the API concurrently (e.g. send 10 or 100 http requests
> > simultaneously, depending on how many concurrent requests the API can
> > handle).
> >
> 
> Sounds like you want to use threads. How does the API let you know you're
> hitting it too frequently? Perhaps you want to code an exponential backoff
> and retry wrapper for your API requests.

Thanks for the reply.
Currently the django view that does the job does three http request:
- the first does a POST and send the payload used during the PDF rendering, the 
response contains a url to check the pdf generation progress;
- the second loop over that url, checking the progress of pdf generation. Once 
the response contains the keyword 'status': 'complete', then it give also a url 
for the file retrieval;
- the third one is a GET to retrieve the file, the reponse contains the binary 
content of the file, then this content is read and wrapped as attachment of a 
django http response, and then returned to the user.

the goal is to reuse the existing code as much as possible, possibly doing 
concurrently, and saving the file instead on a folder.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python - handling HTTP requests asynchronously

[Michael Selik]

On Fri, May 6, 2016 at 3:01 AM  wrote:

> The PDF is generated through an external API. Since currently is generated
> on demand, this is handled synchronously via an HTTP request/response.


Are you sending the request or are you receiving the request?
If you are sending, you can just use threads as you are only doing IO.
If you are receiving the requests and generating PDFs, you may want to use
subprocesses if the PDF-generation is compute-intensive.


> 3) multiprocessing module, with e.g. 10 as workers limit.
>

multiprocessing.Pool is an easy way to use subprocesses
multiprocessing.pool.ThreadPool is an easy way to use threads. It's not
well documented, but has the exact same interface as Pool.

the goal is to use the API concurrently (e.g. send 10 or 100 http requests
> simultaneously, depending on how many concurrent requests the API can
> handle).
>

Sounds like you want to use threads. How does the API let you know you're
hitting it too frequently? Perhaps you want to code an exponential backoff
and retry wrapper for your API requests.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python - handling HTTP requests asynchronously

[justin walters]

On Thu, May 5, 2016 at 11:56 PM,  wrote:

> Hi everyone,
> I need to generate a PDF report for each entry of a django queryset.
> There'll be between between 30k and 40k entries.
>
> The PDF is generated through an external API. Since currently is generated
> on demand, this is handled synchronously via an HTTP request/response. That
> will be different for this task, since I think I'll use a django management
> command to loop through the queryset and perform the PDF generation.
>
> Which approach should I follow for this task? I thought about 3 possibile
> solutions, although are technologies that I never used:
>
> 1) Celery: assign a task (http request with a different payload) to a
> worker, then retrieve it once it's done.
>
> 2) request-futures: using requests in a non-blocking way.
>
> 3) multiprocessing module, with e.g. 10 as workers limit.
>
>
> the goal is to use the API concurrently (e.g. send 10 or 100 http requests
> simultaneously, depending on how many concurrent requests the API can
> handle).
>
> Anybody here that handled a similar task and can give advices on how to
> proceed on this?
> --
> https://mail.python.org/mailman/listinfo/python-list
>



Have you tried channels: https://github.com/andrewgodwin/channels ? If it's
an asyncronous request/response cycle you're looking for, it should work
well. Essentially, you have worker processes that receive a message over a
websocket connection and then send the new message back to the "group". I
would recommend using the redis in memory transaction layer if you go this
route as it is the fastest and most efficient. The best part about channels
is that you can still run a normal django app alongside it. You can have
only one app use websockets while the rest use standard http..
-- 
https://mail.python.org/mailman/listinfo/python-list

python - handling HTTP requests asynchronously

[lordluke80]

Hi everyone, 
I need to generate a PDF report for each entry of a django queryset. There'll 
be between between 30k and 40k entries.

The PDF is generated through an external API. Since currently is generated on 
demand, this is handled synchronously via an HTTP request/response. That will 
be different for this task, since I think I'll use a django management command 
to loop through the queryset and perform the PDF generation.

Which approach should I follow for this task? I thought about 3 possibile 
solutions, although are technologies that I never used:

1) Celery: assign a task (http request with a different payload) to a worker, 
then retrieve it once it's done.

2) request-futures: using requests in a non-blocking way.

3) multiprocessing module, with e.g. 10 as workers limit.


the goal is to use the API concurrently (e.g. send 10 or 100 http requests 
simultaneously, depending on how many concurrent requests the API can handle).

Anybody here that handled a similar task and can give advices on how to proceed 
on this?
-- 
https://mail.python.org/mailman/listinfo/python-list