villebro commented on PR #32520: URL: https://github.com/apache/superset/pull/32520#issuecomment-2734215385
On Michael's point about having thousands of metrics and columns, I think it touches on an important design requirement to always aim for being able to handle considerable scale, even if it's not obvious at first. Some examples: - We take full use of indexes in the data model, and typically have all logical models in dedicated tables with link tables for efficient RDBMS handling. - In migrations we've established patterns for paginating updates. - For listings we paginate whenever possible (both on frontend and API endpoints). - For the FE bundle, we minimize the bundle size maximally, and split everything into chunks whenever possible. - We always defer chart rendering to only the visible charts. - We've invested extensively into reducing FE overrendering. - For rendering chart queries, we've replaced `sqlparse` with `sqlglot` due to wasteful handling of the query AST. - etc In isolation, all of these would likely have seemed like overkill initially, but for each, their absence proved highly damaging as things grew. The point I'm trying to make is I think we need to always consider how something works when taken to the limit, even if initial use cases don't require it. As Airbnb's requirement would likely be solved using the already battle tested pattern of having dedicated entity tables with efficient indexing + paginated APIs, I feel we should design this from the start with high scalability in mind, so as to make sure we don't have to do significant refactoring to the feature later on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
