villebro commented on issue #44174: URL: https://github.com/apache/superset/issues/44174#issuecomment-5646832133
@Diveyam-Mishra that' is an **excellent** question, and one that doesn't have a single simple answer. But let me try. Firstly, we use Flask-Caching as the caching abstraction, which supports some cache [implementations](https://flask-caching.readthedocs.io/en/latest/backends/) beyond Redis/Valkey (hereafter referred to as Valkey only for simplicity). You can also [create your own](https://flask-caching.readthedocs.io/en/latest/contrib-backends/), as some have done. However, Valkey ships with a few extra features that go beyond just caching, namely [distributed locks](https://valkey.io/commands/setnx/), [Pub/Sub](https://valkey.io/topics/pubsub/) and [Streams](https://valkey.io/topics/streams-intro/), which make it possible to introduce functionality that would usually require running additional infrastructure like [Kafka](https://kafka.apache.org/). As the majority of major deployments already use Valkey, we decided to keep things simple by leveraging Valkey for all the functionality it offers. The same applies to Celery: despite showing some age, Celery is still widely used, and we've already been using Celery for long enough for it to make sense to double down on its abstractions and harden those. Having said all this, I we do want to keep the door open to open up Superset to other backends than Valkey and Celery. For this reason we're doing this in stages: 1. We've abstracted realtime coordination behind a centralized `CoordinationService`. While this currently uses Valkey under the hood, we've done our best to keep the abstractions as general as possible. Once the API of the service stabilizes, we'll know exactly what kind of functionality we need, after which it should be easy to make it possible to hook up other backends to it. 2. The same applies to the async task framework: We've guilt GTF in a way which in theory is independent of its underlying engine. While this design is to some degree influenced by current Celery abstractions, we hope to one day make it possible to swap out Celery for something else, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
