Jason, We've considered SQS as the messaging backend for Celery, mostly due to its serverless nature. The disadvantage is that we'd depend on AWS for local development.
Given this, we're leaning on Redis for messaging because we can use AWS's ElasticCache help manage it. This gives us the reliability of AWS while not depending on it during development. Redis can also be used as the result store for Celery, but since we're already using postgres/RDS for the metadata db, I assume we can have a separate schema for Celery's result store in the same database instance. What we don't want to do is have use different containers (using docker) in both dev and production, i.e. relying on Redis or Rabbit locally, while using SQS in production. We're still investing this heavily, but would love to keep a tight loop on the subject. If you're in NYC, we're hosting an Airflow meetup in 2 day to discuss some of our upcoming implementation ideas. I'm going to reannouce it later on mailing list. Cheers On Mon, Jan 30, 2017 at 9:59 AM, Jeremiah Lowin <jlo...@apache.org> wrote: > Jason, > > I don't believe Airflow cares about Celery's backend as long as the task > API remains the same. You should be OK (though I haven't tested to > confirm). > > J > > On Sat, Jan 28, 2017 at 5:09 PM Jason Chen <chingchien.c...@gmail.com> > wrote: > > > Hi Airflow team, > > > > Celery 4 supports AWS SQS > > http://docs.celeryproject.org/en/latest/getting-started/ > brokers/sqs.html > > > > We are using Airflow 1.7.1.3 > > Is there any problem, if we change config to use SQS for CeleryExecutor ? > > > > Thanks. > > > > Jason > > > -- *Joe Napolitano *| Sr. Data Engineer www.blueapron.com | 5 Crosby Street, New York, NY 10013
