On Sun, Nov 27, 2011 at 11:21 PM, Nathan McCorkle <nmz...@gmail.com> wrote:
> Yeah I've seen the djcelery solution before... I've tried implementing
> it, but I'm getting an error on the celeryd console:
> TypeError: processXLS1() got an unexpected keyword argument 'task_is_eager'
I couldn't find anything on this error via google... but once I
removed the manual transaction stuff (I was trying to speed things up
by doing transactions only after I did a bunch of database operations)
including the decorator above the function, it started working.
>
> when I try running processXLS1.delay(dataObjectID=someInteger) from a
> function in views.py
>
> here's what's in my tasks.py:
> "
> from celery.decorators import task
> from enzymeFilter.models import *
> from django.db import transaction
>
> @task
> def printGene(y):
> print "ppppppppppppppppppp"
> fil=open('/var/www/media/testFile','w')
> fil.write('coming from background')
> fil.close()
>
> print Gene.objects.get(id=y+1)
> return True
>
> @task
> @transaction.commit_manually
> def processXLS1(datasetObjectID):
> print "processing XLS as task"
> datasetObject = Dataset.objects.get(id=datasetObjectID)
> try:
> ... more processing code
> "
>
> thanks
> -Nathan
>
> On Sun, Nov 27, 2011 at 9:52 PM, Brian Schott <bfsch...@gmail.com> wrote:
>> You really should look at django-celery and RabbitMQ. The upload submit can
>> initiate a task that is defined in tasks.py. There are separate worker
>> processes that pick up the task from the message queue and do the actual
>> work. These workers don't even have to be on the same machine with rabbitMQ
>> so you get instant scalability. Then your AJAX job status view can poll a
>> job status table that is updated by the task and you don't have to worry
>> about threads.
>> https://github.com/ask/django-celery
>>
>> Brian Schott
>> bfsch...@gmail.com
>>
>>
>>
>> On Nov 27, 2011, at 8:54 PM, Nathan McCorkle wrote:
>>
>>> I'm using Django 1.3 and am processing 3 files into genes, proteins,
>>> and genomes for a tool I built... this processing usually takes
>>> between 5 minutes to a few hours, depending on the genome size.
>>>
>>> After uploading the files (10-100 MB), the upload view begins
>>> processing the files, without returning for a long time (causing my
>>> browser to ask me if I want to kill the plupload script).
>>>
>>> For the user of this app, they don't know if their upload failed or is
>>> processing, etc... so I'd think forking the processing function (or
>>> making it a new thread) is what needs to happen, then the view can
>>> return to the user, and the upload page can start doing AJAX requests
>>> to check to processing progress.
>>>
>>> I feel like I want to do this:
>>> http://www.artfulcode.net/articles/threading-django/
>>> but am scared of crashing because I've heard Django isn't threadsafe
>>> with transactions. (and my processing function is making DB
>>> transactions)
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Django users" group.
>>> To post to this group, send email to django-users@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> django-users+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/django-users?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To post to this group, send email to django-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> django-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/django-users?hl=en.
>>
>>
>
>
>
> --
> Nathan McCorkle
> Rochester Institute of Technology
> College of Science, Biotechnology/Bioinformatics
>
--
Nathan McCorkle
Rochester Institute of Technology
College of Science, Biotechnology/Bioinformatics
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
