etr2460 commented on issue #15694: URL: https://github.com/apache/superset/issues/15694#issuecomment-881578321
>One thing that we can do to mitigate these risks is to use the new Select component. It will allow us to fetch data in a paginated manner and also remove the limit on the number of results. We can submit a PR and test in Airbnb's environment to check if performance is acceptable 😉. @michael-s-molina Pagination could help in some cases, but not in all of them. This would be especially difficult for virtual datasets, since the limit for pagination would come after the full virtual dataset query completes in Presto (for example if there's an order by in the virtual dataset query). Honestly, this is something really hard, and possibly would be better as a cached/pre-warmed set of metadata on the dataset itself. Something like `common_column_values` in the metric/column metadata that is warmed on a cadence set by the user/admin. idk, just spitballing here, but this feature in general becomes a larger problem the larger a deployment gets. It probably doesn't scale to Airbnb's deployment in the current state (which is why it's defaulted to off :P ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
