o-nikolas commented on PR #24019: URL: https://github.com/apache/airflow/pull/24019#issuecomment-1211353124
> > I feel like the situation here is slightly different as it is [officially supported by Celery](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html#broker-overview), and presumably properly tested there. I'm not sure we want to exhaustively test our dependencies. That said, this is a pretty critical piece of the puzzle shrug > > Actually that might change my view. I did not realize SQS is on-par with RabbitMQ/Redis for Celery. On the other Hand MAYBE this is a GREAT opportunity to add a test or two using Moto ? > > https://qxf2.com/blog/mocking-out-aws-sqs-using-moto/ > > Then we would be completely covered and it shoudl be no problem to make it official. @ferruzzi @o-nikolas @vincbeck ? WDYT> I have a love/hate relationship with using SQS as a celery broker :) (maybe this is for a blog post...) As a tl;dr: it works, but has a lot of gotchas, and I wouldn't necessarily recommend it generally to airflow users. Celery is serverless, scales great, and is relatively fast. But it doesn't actually have the same level of support as RabbitMQ and Redis have as a Celery broker. Namely the Monitor and Remote Control APIs are not supported, so while it is stable, it has far less support. Also SQS has the visibility timeout architecturally built into it, which means you can't have tasks that run longer than that timeout without that causing issues (max value for that timeout is 12 hours). I still support adding automated testing for SQS though, I think that's a worthwhile task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
