betodealmeida commented on issue #35003: URL: https://github.com/apache/superset/issues/35003#issuecomment-3716354482
@barakalon thanks for your comment! I do think we should standardize the Superset semantic layer and make it more powerful. Here's what I've been thinking for the roadmap (this is just my personal opinion, and would need discussion with the community): ### Phase 1 (this SIP). We add support for external semantic layers - In this stage we have two paths for chart building. Explore generates a `QueryObject`, which gets passed to either a `SqlaTable` (dataset), a `Query`, or a `SemanticView`. - The `QueryObject` is really complex today. It's built from 22 arguments + `kwargs`, and some of those arguments are dictionaries with multiple keys. I didn't want to bring this complexity to semantic layers — I want it to be easy to add new semantic layers — so I built an interface that converts the `QueryObject` to an isomorphic `SemanticRequest` object with a limited number of fields: metrics, dimensions, filters, order, limit, offset. ### Phase 2 (we probably want this). We add write support for external semantic layers. [Malloy](https://www.malloydata.dev/) has some cool demos exploring this — and they don't even use SQL! This could increase the iteration speed. A user in Explore would be able to define a metric, chart it, and save it back to the semantic layer. ### Phase 3 (this is just in my head right now). We formalize the Superset semantic layer - We have the simplied `SemanticRequest`, so why not use it for Superset as well? Given a `SqlaTable` and a `SemanticRequest` we can easily build the SQL necessary to fetch data, and get rid of the current mess we have (`get_sqla_query`, which generates SQL from the `QueryObject`, is function with 700+ lines of code, most of it not covered by tests). - We would implement a new path for charts: users would still create Superset datasets and explore them. The changes would be just in how the SQL is generated and fetched. There would be the current path (complex, via `get_sqla_query`) and the current path (more simple, using the `SemanticRequest`). ### Phase 4. We get rid of the old path and `QueryObject` - Here we get rid of the old path. Charts would generate the much simpler `SemanticRequest` object, and would pass it directly to the semantic views. Code would be smaller and easier to understand. - More importantly, the Superset semantic layer would be just one next to others (Minerva, DJ, Snowflake, Cube, Mallow, OSI). It would have clear interfaces, and it would be easier to extend: we could add a way of defining relationships with dimensional tables, for example. - It could also lead to Superset having more than one native semantic layer. Instead of replacing the current semantic layer with Minerva, like you want, you could add Minerva and have it side-by-side with the original one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
