FreakingMind opened a new issue, #14865:
URL: https://github.com/apache/gravitino/issues/14865
### Version
main branch
Observed on 1.3.0 (commit 40fdf6ab). The same code paths are present on
`main` (b6406f6fac).
### Describe what's wrong
We use a `lakehouse-iceberg` catalog with `catalog-backend=rest`, pointing
to an Iceberg REST server. When a table is dropped directly through the Iceberg
REST endpoint (for example from Spark with a plain Iceberg REST catalog,
`purge=false`), Gravitino is not notified. Its `TableEntity` for that name
stays live in the entity store (`deleted_at = 0`).
If a table with the same name is then created again through Gravitino:
1. `createTable` creates the table in the external catalog and writes a new
uid into the `gravitino.identifier` table property.
2. `store.put` fails with `EntityAlreadyExistsException` on the `(schema_id,
table_name, deleted_at)` unique key, because the stale row is still there. On
1.3.0 this is only logged and `createTable` returns success. On `main` it now
throws `TableAlreadyExistsException`, but the table already exists in the
external catalog at that point.
3. Every following `loadTable` fails:
- `internalLoadTable` finds the stale entity by name, but its id does not
match the uid in `gravitino.identifier` (Case 4 in
`OperationDispatcher.operateOnEntity`), so the table is not imported;
- `importTable` tries to put an entity with the new uid and hits the same
unique key;
- `loadTable` handles `EntityAlreadyExistsException` as an HA race,
reloads, the table is still not imported, and it throws
`UnsupportedOperationException("Table managed by multiple catalogs ...")`.
Only one catalog manages the table, so the message points in the wrong
direction. The suggested fix (remove all catalogs and recreate one) is not an
option for a production catalog. The table stays unreadable through Gravitino,
and owner/role operations on it fail, until it is fixed by hand.
Workaround: remove the `gravitino.identifier` property from the Iceberg
table, e.g. `ALTER TABLE db1.t1 UNSET TBLPROPERTIES ('gravitino.identifier')`
through the Iceberg REST catalog, or a `remove-properties` table update.
`internalLoadTable` then takes the "no string identifier" path, finds the
existing entity by name and uses it. The owner and role bindings of that entity
are kept. We used this on 17 tables.
### Error message and/or stacktrace
Response of `GET
/api/metalakes/<metalake>/catalogs/<catalog>/schemas/db1/tables/t1`:
```
{"code":1006,"type":"UnsupportedOperationException","message":"Failed to
operate table(s) [t1] operation [LOAD] under schema [db1], reason [Table
managed by multiple catalogs. This may cause unexpected issues such as
privilege conflicts. To resolve: Remove all catalogs managing this table, then
recreate one catalog to ensure single-catalog management.]"}
```
Server log when the table is created again (1.3.0):
```
INFO IcebergCatalogOperations:599 - Created Iceberg table t1
ERROR TableOperationDispatcher:691 - Failed to put entity for
<metalake>.<catalog>.db1.t1 in Gravitino, with this situation the returned
object will not contain the metadata from Gravitino.
org.apache.gravitino.EntityAlreadyExistsException: The TABLE entity:
<metalake>.<catalog>.db1.t1 already exists.
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "table_meta_schema_id_table_name_deleted_at_key"
Detail: Key (schema_id, table_name, deleted_at)=(<schema_id>, t1, 0)
already exists.
INFO TableOperations:153 - Table created: <metalake>.<catalog>.db1.t1
```
Top of the stack trace on load:
```
java.lang.UnsupportedOperationException: Table managed by multiple catalogs.
...
at
org.apache.gravitino.catalog.TableOperationDispatcher.loadTable(TableOperationDispatcher.java:160)
at
org.apache.gravitino.catalog.TableNormalizeDispatcher.loadTable(TableNormalizeDispatcher.java:63)
at
org.apache.gravitino.listener.TableEventDispatcher.loadTable(TableEventDispatcher.java:103)
at
org.apache.gravitino.hook.TableHookDispatcher.loadTable(TableHookDispatcher.java:65)
at
org.apache.gravitino.server.web.rest.TableOperations.lambda$loadTable$2(TableOperations.java:189)
```
### How to reproduce
Gravitino 1.3.0 (commit 40fdf6ab), relational entity store on PostgreSQL.
1. Create a `lakehouse-iceberg` catalog `cat` with `catalog-backend=rest`,
pointing to an Iceberg REST server (in our case another Gravitino Iceberg REST
server with a JDBC backend).
2. Create `db1.t1` through the Gravitino REST API. Loading it through
Gravitino works.
3. Drop the table without going through Gravitino, directly on the Iceberg
REST server, e.g. from Spark with `org.apache.iceberg.spark.SparkCatalog`,
`type=rest`: `DROP TABLE db1.t1`.
4. Create `db1.t1` again through the Gravitino REST API. On 1.3.0 the
request succeeds and the server logs the `EntityAlreadyExistsException` above.
5. `GET /api/metalakes/<metalake>/catalogs/cat/schemas/db1/tables/t1`
returns 405 with "Table managed by multiple catalogs".
6. Remove `gravitino.identifier` from the table properties through the
Iceberg REST server. The same GET now returns the table.
### Additional context
We saw this in production: tables were dropped from Spark through the
Iceberg REST endpoint and recreated by an automated deployment through
Gravitino. The steps above are taken from those logs; I have not run them in an
isolated setup.
Related:
- #13173 / #13197: refusing an import whose `StringIdentifier` belongs to
another live entity. It covers copied identifiers (`CREATE TABLE ... LIKE`);
the PR was closed without merge. It also proposes replacing this message with
one that names the table, the id and the `gravitino.identifier` property.
- #11040: HA race handling for concurrent imports. This is the `catch
(EntityAlreadyExistsException)` branch that produces the misleading message
here.
- #13730: remaining entity-store concurrency work.
Possible improvements:
1. When the entity stored under a name has a different id than the external
table's `gravitino.identifier`, and the external table registered under the old
id no longer exists, treat the stored entity as stale: rebind it to the new id
or replace it, instead of failing the load.
2. At least, report this case with an actionable message (table, both ids,
the `gravitino.identifier` property) instead of "managed by multiple catalogs".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]