Re: [yocto] [layerindex-web][PATCH] layerindex-web : adding asynchronous task execution

[Paul Eggleton]

On Wednesday, 23 August 2017 12:06:09 PM NZST Paul Eggleton wrote:
> Hi Diana
> 
> I did finally manage to get to it today - sorry about the delay. I've
> sorted out the settings issue and dropped the related item from the todo list.

Oops, I neglected to mention I pushed these minor changes to the same
paule/celery branch.

> Assuming that didn't break anything on your end, could you please squash all 
> of
> these into one commit with you as the author and an appropriate shortlog /
> commit message and then send a new patch? We'd probably want to mention that
> the mechanism this establishes will be used later for executing other
> asynchronous tasks.
> 
> Thanks,
> Paul
> 
> On Saturday, 12 August 2017 7:56:15 AM NZST Diana Thayer wrote:
> > Hello! Did you get a chance to look at the PR
> > ?
> > 
> > On Wed, Jul 26, 2017 at 6:02 PM, Diana Thayer  wrote:
> > 
> > > Great! Thanks for testing it out and identifying that issue.
> > >
> > > I've put together a PR on GitHub
> > >  that includes some
> > > changes that appear to fix the issue you're describing. It also updates 
> > > the
> > > readme to document how to configure the connection to the RabbitMQ server,
> > > and how to start the Celery worker. You can see all the file changes here
> > > .
> > >
> > > On Mon, Jul 24, 2017 at 2:49 AM, Paul Eggleton <
> > > paul.eggle...@linux.intel.com> wrote:
> > >
> > >> On Friday, 21 July 2017 4:15:26 PM CEST Paul Eggleton wrote:
> > >> > On Tuesday, 27 June 2017 1:59:23 AM CEST Diana Thayer wrote:
> > >> > > This is a work-in-progress solution to issue 11197 within
> > >> layerindex-web:
> > >> > > adding asynchronous task execution.
> > >> > > I used RabbitMQ and Celery. I'm still working with Paul Eggleton on
> > >> adding
> > >> > > install instructions for RabbitMQ to
> > >> > > the site's Dockerfile.
> > >> > >
> > >> > > Previously, when a user submitted a layer, Django would synchronously
> > >> send
> > >> > > emails to users able to publish layers,
> > >> > > such that if an email bounced or otherwise failed, the user
> > >> submitting the
> > >> > > layer would see an error.
> > >> > > This patch sends those emails as asynchronous tasks, so any sending
> > >> errors
> > >> > > don't reach the user submitting a layer.
> > >> >
> > >> > I had a chance to try your patch out today, although I had to apply it
> > >> > partially - I figured a good test would just be the email part. I
> > >> installed
> > >> > rabbitmq and ensured it got started. I started the web interface and
> > >> > submitted  a dummy layer, and the email sent as expected. I then tried
> > >> > killing my local  SMTP server and the layer submission returned an 
> > >> > error
> > >> > page, which is what  happened previously - it looks to me as if the
> > >> sending
> > >> > is still being done synchronously. The celery worker isn't reporting
> > >> > anything so I can only assume  the django app is not trying to connect
> > >> to
> > >> > it.
> > >> >
> > >> > I've pushed just the changes I applied to a new branch here:
> > >> >
> > >> > http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log
> > >> /?h=paule/celery
> > >> >
> > >> > Any idea what's going wrong? Did I mess up in applying the changes, or
> > >> > misconfigure something else?
> > >>
> > >> I figured out the issue - in order to have the task executed
> > >> asynchronously
> > >> you need to call .delay() on the task object rather than the task
> > >> function itself. That leads to a "django.core.exceptions.Improp
> > >> erlyConfigured"
> > >> error reported by the worker since in my configuration the "import
> > >> settings"
> > >> succeeds but Django isn't set up properly, but at least it's actually
> > >> executing the task via the worker now.
> > >>
> > >> Cheers,
> > >> Paul
> > >>
> 
> 
> 
> 


-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH] layerindex-web : adding asynchronous task execution

[Diana Thayer]

Hello! Did you get a chance to look at the PR
?

On Wed, Jul 26, 2017 at 6:02 PM, Diana Thayer  wrote:

> Great! Thanks for testing it out and identifying that issue.
>
> I've put together a PR on GitHub
>  that includes some
> changes that appear to fix the issue you're describing. It also updates the
> readme to document how to configure the connection to the RabbitMQ server,
> and how to start the Celery worker. You can see all the file changes here
> .
>
> On Mon, Jul 24, 2017 at 2:49 AM, Paul Eggleton <
> paul.eggle...@linux.intel.com> wrote:
>
>> On Friday, 21 July 2017 4:15:26 PM CEST Paul Eggleton wrote:
>> > On Tuesday, 27 June 2017 1:59:23 AM CEST Diana Thayer wrote:
>> > > This is a work-in-progress solution to issue 11197 within
>> layerindex-web:
>> > > adding asynchronous task execution.
>> > > I used RabbitMQ and Celery. I'm still working with Paul Eggleton on
>> adding
>> > > install instructions for RabbitMQ to
>> > > the site's Dockerfile.
>> > >
>> > > Previously, when a user submitted a layer, Django would synchronously
>> send
>> > > emails to users able to publish layers,
>> > > such that if an email bounced or otherwise failed, the user
>> submitting the
>> > > layer would see an error.
>> > > This patch sends those emails as asynchronous tasks, so any sending
>> errors
>> > > don't reach the user submitting a layer.
>> >
>> > I had a chance to try your patch out today, although I had to apply it
>> > partially - I figured a good test would just be the email part. I
>> installed
>> > rabbitmq and ensured it got started. I started the web interface and
>> > submitted  a dummy layer, and the email sent as expected. I then tried
>> > killing my local  SMTP server and the layer submission returned an error
>> > page, which is what  happened previously - it looks to me as if the
>> sending
>> > is still being done synchronously. The celery worker isn't reporting
>> > anything so I can only assume  the django app is not trying to connect
>> to
>> > it.
>> >
>> > I've pushed just the changes I applied to a new branch here:
>> >
>> > http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log
>> /?h=paule/celery
>> >
>> > Any idea what's going wrong? Did I mess up in applying the changes, or
>> > misconfigure something else?
>>
>> I figured out the issue - in order to have the task executed
>> asynchronously
>> you need to call .delay() on the task object rather than the task
>> function itself. That leads to a "django.core.exceptions.Improp
>> erlyConfigured"
>> error reported by the worker since in my configuration the "import
>> settings"
>> succeeds but Django isn't set up properly, but at least it's actually
>> executing the task via the worker now.
>>
>> Cheers,
>> Paul
>>
>> --
>>
>> Paul Eggleton
>> Intel Open Source Technology Centre
>>
>
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH] layerindex-web : adding asynchronous task execution

[Diana Thayer]

Great! Thanks for testing it out and identifying that issue.

I've put together a PR on GitHub
 that includes some
changes that appear to fix the issue you're describing. It also updates the
readme to document how to configure the connection to the RabbitMQ server,
and how to start the Celery worker. You can see all the file changes here
.

On Mon, Jul 24, 2017 at 2:49 AM, Paul Eggleton <
paul.eggle...@linux.intel.com> wrote:

> On Friday, 21 July 2017 4:15:26 PM CEST Paul Eggleton wrote:
> > On Tuesday, 27 June 2017 1:59:23 AM CEST Diana Thayer wrote:
> > > This is a work-in-progress solution to issue 11197 within
> layerindex-web:
> > > adding asynchronous task execution.
> > > I used RabbitMQ and Celery. I'm still working with Paul Eggleton on
> adding
> > > install instructions for RabbitMQ to
> > > the site's Dockerfile.
> > >
> > > Previously, when a user submitted a layer, Django would synchronously
> send
> > > emails to users able to publish layers,
> > > such that if an email bounced or otherwise failed, the user submitting
> the
> > > layer would see an error.
> > > This patch sends those emails as asynchronous tasks, so any sending
> errors
> > > don't reach the user submitting a layer.
> >
> > I had a chance to try your patch out today, although I had to apply it
> > partially - I figured a good test would just be the email part. I
> installed
> > rabbitmq and ensured it got started. I started the web interface and
> > submitted  a dummy layer, and the email sent as expected. I then tried
> > killing my local  SMTP server and the layer submission returned an error
> > page, which is what  happened previously - it looks to me as if the
> sending
> > is still being done synchronously. The celery worker isn't reporting
> > anything so I can only assume  the django app is not trying to connect to
> > it.
> >
> > I've pushed just the changes I applied to a new branch here:
> >
> > http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/
> log/?h=paule/celery
> >
> > Any idea what's going wrong? Did I mess up in applying the changes, or
> > misconfigure something else?
>
> I figured out the issue - in order to have the task executed asynchronously
> you need to call .delay() on the task object rather than the task
> function itself. That leads to a "django.core.exceptions.
> ImproperlyConfigured"
> error reported by the worker since in my configuration the "import
> settings"
> succeeds but Django isn't set up properly, but at least it's actually
> executing the task via the worker now.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH] layerindex-web : adding asynchronous task execution

[Paul Eggleton]

On Friday, 21 July 2017 4:15:26 PM CEST Paul Eggleton wrote:
> On Tuesday, 27 June 2017 1:59:23 AM CEST Diana Thayer wrote:
> > This is a work-in-progress solution to issue 11197 within layerindex-web: 
> > adding asynchronous task execution.
> > I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding
> > install instructions for RabbitMQ to
> > the site's Dockerfile.
> > 
> > Previously, when a user submitted a layer, Django would synchronously send
> > emails to users able to publish layers,
> > such that if an email bounced or otherwise failed, the user submitting the
> > layer would see an error.
> > This patch sends those emails as asynchronous tasks, so any sending errors
> > don't reach the user submitting a layer.
> 
> I had a chance to try your patch out today, although I had to apply it 
> partially - I figured a good test would just be the email part. I installed 
> rabbitmq and ensured it got started. I started the web interface and
> submitted  a dummy layer, and the email sent as expected. I then tried
> killing my local  SMTP server and the layer submission returned an error 
> page, which is what  happened previously - it looks to me as if the sending 
> is still being done synchronously. The celery worker isn't reporting
> anything so I can only assume  the django app is not trying to connect to
> it.
> 
> I've pushed just the changes I applied to a new branch here:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery
> 
> Any idea what's going wrong? Did I mess up in applying the changes, or 
> misconfigure something else? 

I figured out the issue - in order to have the task executed asynchronously 
you need to call .delay() on the task object rather than the task 
function itself. That leads to a "django.core.exceptions.ImproperlyConfigured" 
error reported by the worker since in my configuration the "import settings" 
succeeds but Django isn't set up properly, but at least it's actually 
executing the task via the worker now.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH] layerindex-web : adding asynchronous task execution

[Paul Eggleton]

Hi Diana,

Sorry it's taken me a ridiculous amount of time to get back to you on this!

On Tuesday, 27 June 2017 1:59:23 AM CEST Diana Thayer wrote:
> This is a work-in-progress solution to issue 11197 within layerindex-web: 
> adding asynchronous task execution.
> I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding
> install instructions for RabbitMQ to
> the site's Dockerfile.
> 
> Previously, when a user submitted a layer, Django would synchronously send
> emails to users able to publish layers,
> such that if an email bounced or otherwise failed, the user submitting the
> layer would see an error.
> This patch sends those emails as asynchronous tasks, so any sending errors
> don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it 
partially - I figured a good test would just be the email part. I installed 
rabbitmq and ensured it got started. I started the web interface and submitted 
a dummy layer, and the email sent as expected. I then tried killing my local 
SMTP server and the layer submission returned an error page, which is what 
happened previously - it looks to me as if the sending is still being done 
synchronously. The celery worker isn't reporting anything so I can only assume 
the django app is not trying to connect to it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery

Any idea what's going wrong? Did I mess up in applying the changes, or 
misconfigure something else? 

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto