Celery integration for next

2013-03-19 Thread Lars Holm Nielsen 

Hi,

I've finished initial integration of Invenio in Celery for next:
http://invenio-software.org/repo/personal/invenio-lnielsen/commit/?h=next-celery&id=6d09ef545f03edfa6d7c77cd3a2447873b16c87e

It basically follows what we discussed in DevForum ( 
https://invenio-software.org/wiki/Tools/Celery/InvenioIntegration). Take 
a look if you have a minute and let me know if there's issues.


Currently this is the barebones integration - more user documentation is 
needed eg, and easy management of the workers I haven't looked into. 
Part of the configuration is taken from invenio.config (installation 
specific), rest is defined in miscutil/lib/celeryconfig.py (invenio 
specific). The integration consists of a special Celery loader for 
Invenio (e.g. Django and the other frameworks also have special 
loaders), which takes care of loading *_tasks.py files as well as 
initializing Flask application/request context so that you can use 
SQLAlchemy modes etc. in your tasks, and that's basically it.


*bibsomemodule_tasks.py**:*
# Flask _application context_ available here: e.g. import SQLAlchemy 
models will work.

from invenio.celery import celery

@celery.task(ignore_result=True)
def mytask(x):
#  do something (flask test _request context_ available here - 
1 test request context per task)


*othermodule.py:*
from invenio.bibsomemodule_tasks import mytask
mytask.delay(1)

*Starting Celery worker:*
celery worker --app=invenio ...

Things missing:
- Define a nice task example with everything included.
- Management: Easy start/stop of workers (define log dirs, pid files 
etc, default arguments), integration of monitoring etc.
- Backport to master: only requires to remove code from the loader that 
takes care of initializing flask.

- How to easily test tasks :-)
- FIx a nasty import issues.

Cheers,
Lars

--
Lars Holm Nielsen
Software Engineer

CERN, IT Department, Digital Library Technology Section
Office 513/1-014
Tel: +41 22 76 79182
Cel: +41 76 672 8927



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Invenio Developer Forum - Web APIs - today at 16:45 CET

2013-03-19 Thread Jay Luker 
I'm sorry for missing the forum yesterday. (It simply slipped my mind.)

When designing our new search API I went with the custom header
approach (and avoided a version indicated in the resource path) for
the reasons mentioned in the post Jan linked to, and also from reading
http://www.mnot.net/blog/2012/12/04/api-evolution.

Since our application is still sporting the "alpha" label,  we're not
worried yet about different versions of the API yet, but the
accepting/sending of the version header is already there for when we
eventually need it.

{{{

@api_blueprint.before_request
def get_api_version_header():
g.api_version = request.headers.get('X-API-Version',
config.API_CURRENT_VERSION)

@api_blueprint.after_request
def add_api_version_header(response):
"""
Add default api version header if not already set
"""
response.headers.setdefault('X-API-Version', g.api_version)
return response

}}}

That said, there doesn't seem to be much consensus on a best practice
here. There are good arguments for not doing it this way. e.g., this
guy makes sense too:
http://www.notmessenger.com/rest/3-tenets-for-implementing-a-rest-api/.

--jay

On Tue, Mar 19, 2013 at 5:44 AM, Jan Åge Lavik  wrote:
> Nice summary.
>
> I just have one small comment regarding the API versioning. I like the
> approach to use the headers, allowing clients to not specify the API version
> directly in the URIs. Or if no indication is given at all, it will use the
> latest version possible by default for the service/verb. Perhaps even
> something like this guy propose
> (http://thereisnorightway.blogspot.ch/2011/02/versioning-and-types-in-resthttp-api.html)
> where the API version is part of the request type.
>
> Cheers,
> Jan
>
>
> On Mon, Mar 18, 2013 at 1:19 PM, Tibor Simko  wrote:
>>
>> On Mon, 11 Mar 2013, Tibor Simko wrote:
>> > Today, we thought of musing about Web API standardisation process,
>> > including issues such as: (a) centralising all the various existing
>> > APIs to use leading `/api' URL space; (b) API versioning; (c) plugin
>> > system for modules; (d) API job and export task queue manager,
>> > possibly using Celery.
>>
>> I've updated the wiki page summarising the API musings:
>>
>>
>>
>> Still a work in progress, still containing undecided issues, but shows
>> the overall direction we thought of taking.  Please share any comments
>> you may have.
>>
>> Best regards
>> --
>> Tibor Simko
>
>



-- 
**
Jay Luker   Astrophysics Data System (ADS)
jlu...@cfa.harvard.edu  Center for Astrophysics
617-495-458860 Garden Street  MS 67
617-495-7356 faxCambridge, MA  02138
**

Re: Invenio Developer Forum - Web APIs - today at 16:45 CET

2013-03-19 Thread Jan Åge Lavik 
Nice summary.

I just have one small comment regarding the API versioning. I like the
approach to use the headers, allowing clients to not specify the API
version directly in the URIs. Or if no indication is given at all, it will
use the latest version possible by default for the service/verb. Perhaps
even something like this guy propose (
http://thereisnorightway.blogspot.ch/2011/02/versioning-and-types-in-resthttp-api.html)
where the API version is part of the request type.

Cheers,
Jan


On Mon, Mar 18, 2013 at 1:19 PM, Tibor Simko  wrote:

> On Mon, 11 Mar 2013, Tibor Simko wrote:
> > Today, we thought of musing about Web API standardisation process,
> > including issues such as: (a) centralising all the various existing
> > APIs to use leading `/api' URL space; (b) API versioning; (c) plugin
> > system for modules; (d) API job and export task queue manager,
> > possibly using Celery.
>
> I've updated the wiki page summarising the API musings:
>
>
>
> Still a work in progress, still containing undecided issues, but shows
> the overall direction we thought of taking.  Please share any comments
> you may have.
>
> Best regards
> --
> Tibor Simko
>