using guest on rabbitMQ, installed django-celery using pip... my task
returns true/false if it succeeds/fails... otherwise it does a lot of
database R/W
Its currently processing a file via the worker, and I'm able to
refresh the web browser on a page that shows how many genes have been
processed for a dataset... every refresh the number of genes is
increasing... so I guess there aren't any DB clash problems!
(hopefully)
now to implement a javascript/AJAX progress bar to start after the
upload finishes!
On Sun, Nov 27, 2011 at 11:49 PM, Brian Schott <bfsch...@gmail.com> wrote:
> What are your settings? Using carrot? Kombu? RabbitMQ?
>
> Does your task try to return a value?
>
> Sent from my iPhone
>
> On Nov 27, 2011, at 11:22 PM, Nathan McCorkle <nmz...@gmail.com> wrote:
>
>> P.S. the printGene function works... printing the messages on the
>> celeryd console terminal... the processXLS1 function doesn't even
>> print anything at all
>>
>> On Sun, Nov 27, 2011 at 11:21 PM, Nathan McCorkle <nmz...@gmail.com> wrote:
>>> Yeah I've seen the djcelery solution before... I've tried implementing
>>> it, but I'm getting an error on the celeryd console:
>>> TypeError: processXLS1() got an unexpected keyword argument 'task_is_eager'
>>>
>>> when I try running processXLS1.delay(dataObjectID=someInteger) from a
>>> function in views.py
>>>
>>> here's what's in my tasks.py:
>>> "
>>> from celery.decorators import task
>>> from enzymeFilter.models import *
>>> from django.db import transaction
>>>
>>> @task
>>> def printGene(y):
>>> print "ppppppppppppppppppp"
>>> fil=open('/var/www/media/testFile','w')
>>> fil.write('coming from background')
>>> fil.close()
>>>
>>> print Gene.objects.get(id=y+1)
>>> return True
>>>
>>> @task
>>> @transaction.commit_manually
>>> def processXLS1(datasetObjectID):
>>> print "processing XLS as task"
>>> datasetObject = Dataset.objects.get(id=datasetObjectID)
>>> try:
>>> ... more processing code
>>> "
>>>
>>> thanks
>>> -Nathan
>>>
>>> On Sun, Nov 27, 2011 at 9:52 PM, Brian Schott <bfsch...@gmail.com> wrote:
>>>> You really should look at django-celery and RabbitMQ. The upload submit
>>>> can initiate a task that is defined in tasks.py. There are separate
>>>> worker processes that pick up the task from the message queue and do the
>>>> actual work. These workers don't even have to be on the same machine with
>>>> rabbitMQ so you get instant scalability. Then your AJAX job status view
>>>> can poll a job status table that is updated by the task and you don't have
>>>> to worry about threads.
>>>> https://github.com/ask/django-celery
>>>>
>>>> Brian Schott
>>>> bfsch...@gmail.com
>>>>
>>>>
>>>>
>>>> On Nov 27, 2011, at 8:54 PM, Nathan McCorkle wrote:
>>>>
>>>>> I'm using Django 1.3 and am processing 3 files into genes, proteins,
>>>>> and genomes for a tool I built... this processing usually takes
>>>>> between 5 minutes to a few hours, depending on the genome size.
>>>>>
>>>>> After uploading the files (10-100 MB), the upload view begins
>>>>> processing the files, without returning for a long time (causing my
>>>>> browser to ask me if I want to kill the plupload script).
>>>>>
>>>>> For the user of this app, they don't know if their upload failed or is
>>>>> processing, etc... so I'd think forking the processing function (or
>>>>> making it a new thread) is what needs to happen, then the view can
>>>>> return to the user, and the upload page can start doing AJAX requests
>>>>> to check to processing progress.
>>>>>
>>>>> I feel like I want to do this:
>>>>> http://www.artfulcode.net/articles/threading-django/
>>>>> but am scared of crashing because I've heard Django isn't threadsafe
>>>>> with transactions. (and my processing function is making DB
>>>>> transactions)
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Django users" group.
>>>>> To post to this group, send email to django-users@googlegroups.com.
>>>>> To unsubscribe from this group, send email to
>>>>> django-users+unsubscr...@googlegroups.com.
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/django-users?hl=en.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "Django users" group.
>>>> To post to this group, send email to django-users@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> django-users+unsubscr...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/django-users?hl=en.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nathan McCorkle
>>> Rochester Institute of Technology
>>> College of Science, Biotechnology/Bioinformatics
>>>
>>
>>
>>
>> --
>> Nathan McCorkle
>> Rochester Institute of Technology
>> College of Science, Biotechnology/Bioinformatics
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To post to this group, send email to django-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> django-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/django-users?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>
--
Nathan McCorkle
Rochester Institute of Technology
College of Science, Biotechnology/Bioinformatics
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
