Re: Performance: backfill --mark_success

2016-12-04 Thread Vikas Malhotra
Hello Harish, Based on our understanding of Python Multiprocessing, a task instance gets a record in underlying database after there is an explicit call to airflow from that library (using Local Executor). So, I might be wrong, but you won't find a record in database until and unless that task ins

Airflow RabbitMQ with Celery across EC2 Instances

2016-10-07 Thread Vikas Malhotra
Hello Everyone, I was wondering if anyone has done setup of Airflow with Celery Executor with RabbitMQ as backend across EC2 instances and if they are willing to share their experiences and any pitfalls that I can avoid? If anyone has some basic steps or guidelines, that you would be great as wel

Re: Custom Hook development guideline

2016-09-23 Thread Vikas Malhotra
Thanks Andrew and Max, I explored the docs and various code snippets shared by other developers. I have managed to use a custom hook in my dag. Regarding contributing back to the project, I would love to, I have explored the basic part of Adobe Analytics API as of now. I would add more functionali

Custom Hook development guideline

2016-09-23 Thread Vikas Malhotra
Hello Everyone, I am not sure if this is the right place for this question. I am in the process of developing a custom hook for Adobe Analytics (Omniture) and AWS (boto3). I followed the following steps to create the hook: 1. Use the following link to create the hook class: https://pythonhoste