colemandsmith opened a new issue, #40131: URL: https://github.com/apache/superset/issues/40131
### Bug description [This line](https://github.com/apache/superset/blob/c83fb2bb1dcfac41ac51bcebd82471f4a7180d18/superset/db_engine_specs/bigquery.py#L419) in the bigquery engine spec uses a two-part table name for the bigquery table. Because a project is not provided, the default Bigquery connector will default to the client's default project. To reproduce: - Run the docker image for superset 6.1.0 and connect to it from the web browser - Create a BigQuery data connection where the BigQuery project is different than the source credentials. The credentials must have access to the dataset of interest in the BigQuery project. - Test connection, verify it works - Go to the `Datasets` tab and click the `+ Dataset` button - Select the new BigQuery connection as the database, and select a schema the client has access to. - Select a table the client has access to. An error message appears noting something went wrong trying to populate columns for the table. Superset server logs then show the following message: ``` Traceback (most recent call last): File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request rv = self.dispatch_request() File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/app/.venv/lib/python3.10/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps return f(self, *args, **kwargs) File "/app/superset/views/base_api.py", line 120, in wraps duration, response = time_function(f, self, *args, **kwargs) File "/app/superset/utils/core.py", line 1559, in time_function response = func(*args, **kwargs) File "/app/superset/utils/log.py", line 301, in wrapper value = f(*args, **kwargs) File "/app/superset/databases/api.py", line 1090, in table_metadata payload = database.db_engine_spec.get_table_metadata(database, table) File "/app/superset/db_engine_specs/base.py", line 1320, in get_table_metadata return get_table_metadata(database, table) File "/app/superset/databases/utils.py", line 100, in get_table_metadata "selectStar": database.select_star( File "/app/superset/models/core.py", line 836, in select_star return self.db_engine_spec.select_star( File "/app/superset/db_engine_specs/bigquery.py", line 840, in select_star return super().select_star( File "/app/superset/db_engine_specs/base.py", line 1936, in select_star partition_query = cls.where_latest_partition( File "/app/superset/db_engine_specs/bigquery.py", line 357, in where_latest_partition if partition_column := cls.get_time_partition_column(database, table): File "/app/superset/db_engine_specs/bigquery.py", line 424, in get_time_partition_column bq_table = client.get_table(table_ref) File "/app/.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 1210, in get_table api_response = self._call_api( File "/app/.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 861, in _call_api return call() File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func return retry_target( File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target next_sleep = _retry_error_helper( File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_base.py", line 216, in _retry_error_helper raise final_exc from source_exc File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target result = target() File "/app/.venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 494, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.NotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/{CLIENT_PROJECT_ID}/datasets/{CROSS_PROJECT_DATASET}/tables/{CROSS_PROJECT_TABLE}?prettyPrint=false: Not found: Dataset {CLIENT_PROJECT_ID}:{CROSS_PROJECT_DATASET} ``` where CLIENT_PROJECT_ID is the project ID of the client credentials, and the CROSS_PROJECT_* fields refer to a dataset and table in a separate project that the client has access to. Since it's trying to find the dataset name from a separate project in the client's project, it 404s. Expected behavior: - Dataset creation does not error out when using a database referencing a different project than the source credentials' project Encountered in the 6.1.0 release, though unlikely new. ### Screenshots/recordings _No response_ ### Superset version master / latest-dev ### Python version 3.11 ### Node version 18 or greater ### Browser Chrome ### Additional context Superset server logs show the following message: ``` Traceback (most recent call last): File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request rv = self.dispatch_request() File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/app/.venv/lib/python3.10/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps return f(self, *args, **kwargs) File "/app/superset/views/base_api.py", line 120, in wraps duration, response = time_function(f, self, *args, **kwargs) File "/app/superset/utils/core.py", line 1559, in time_function response = func(*args, **kwargs) File "/app/superset/utils/log.py", line 301, in wrapper value = f(*args, **kwargs) File "/app/superset/databases/api.py", line 1090, in table_metadata payload = database.db_engine_spec.get_table_metadata(database, table) File "/app/superset/db_engine_specs/base.py", line 1320, in get_table_metadata return get_table_metadata(database, table) File "/app/superset/databases/utils.py", line 100, in get_table_metadata "selectStar": database.select_star( File "/app/superset/models/core.py", line 836, in select_star return self.db_engine_spec.select_star( File "/app/superset/db_engine_specs/bigquery.py", line 840, in select_star return super().select_star( File "/app/superset/db_engine_specs/base.py", line 1936, in select_star partition_query = cls.where_latest_partition( File "/app/superset/db_engine_specs/bigquery.py", line 357, in where_latest_partition if partition_column := cls.get_time_partition_column(database, table): File "/app/superset/db_engine_specs/bigquery.py", line 424, in get_time_partition_column bq_table = client.get_table(table_ref) File "/app/.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 1210, in get_table api_response = self._call_api( File "/app/.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 861, in _call_api return call() File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func return retry_target( File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target next_sleep = _retry_error_helper( File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_base.py", line 216, in _retry_error_helper raise final_exc from source_exc File "/app/.venv/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target result = target() File "/app/.venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 494, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.NotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/{CLIENT_PROJECT_ID}/datasets/{CROSS_PROJECT_DATASET}/tables/{CROSS_PROJECT_TABLE}?prettyPrint=false: Not found: Dataset {CLIENT_PROJECT_ID}:{CROSS_PROJECT_DATASET} ``` where CLIENT_PROJECT_ID is the project ID of the client credentials, and the CROSS_PROJECT_* fields refer to a dataset and table in a separate project that the client has access to. ### Checklist - [x] I have searched Superset docs and Slack and didn't find a solution to my problem. - [x] I have searched the GitHub issue tracker and didn't find a similar bug report. - [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
