Joseph, Thanks for sharing the ideas and I am looking forward to your announcement and results.
We do use AWS ElasticCache for Celery and lean on SQS mainly due to the cost! -Jason On Mon, Jan 30, 2017 at 11:01 AM, Joseph Napolitano < joseph.napolit...@blueapron.com.invalid> wrote: > Jason, > > We've considered SQS as the messaging backend for Celery, mostly due to its > serverless nature. The disadvantage is that we'd depend on AWS for local > development. > > Given this, we're leaning on Redis for messaging because we can use AWS's > ElasticCache help manage it. This gives us the reliability of AWS while > not depending on it during development. Redis can also be used as the > result store for Celery, but since we're already using postgres/RDS for the > metadata db, I assume we can have a separate schema for Celery's result > store in the same database instance. > > What we don't want to do is have use different containers (using docker) in > both dev and production, i.e. relying on Redis or Rabbit locally, while > using SQS in production. > > We're still investing this heavily, but would love to keep a tight loop on > the subject. If you're in NYC, we're hosting an Airflow meetup in 2 day to > discuss some of our upcoming implementation ideas. I'm going to reannouce > it later on mailing list. > > Cheers > > On Mon, Jan 30, 2017 at 9:59 AM, Jeremiah Lowin <jlo...@apache.org> wrote: > > > Jason, > > > > I don't believe Airflow cares about Celery's backend as long as the task > > API remains the same. You should be OK (though I haven't tested to > > confirm). > > > > J > > > > On Sat, Jan 28, 2017 at 5:09 PM Jason Chen <chingchien.c...@gmail.com> > > wrote: > > > > > Hi Airflow team, > > > > > > Celery 4 supports AWS SQS > > > http://docs.celeryproject.org/en/latest/getting-started/ > > brokers/sqs.html > > > > > > We are using Airflow 1.7.1.3 > > > Is there any problem, if we change config to use SQS for > CeleryExecutor ? > > > > > > Thanks. > > > > > > Jason > > > > > > > > > -- > *Joe Napolitano *| Sr. Data Engineer > www.blueapron.com | 5 Crosby Street, New York, NY 10013 >
