FreakingMind opened a new issue, #14865:
URL: https://github.com/apache/gravitino/issues/14865

   ### Version
   
   main branch
   
   Observed on 1.3.0 (commit 40fdf6ab). The same code paths are present on 
`main` (b6406f6fac).
   
   ### Describe what's wrong
   
   We use a `lakehouse-iceberg` catalog with `catalog-backend=rest`, pointing 
to an Iceberg REST server. When a table is dropped directly through the Iceberg 
REST endpoint (for example from Spark with a plain Iceberg REST catalog, 
`purge=false`), Gravitino is not notified. Its `TableEntity` for that name 
stays live in the entity store (`deleted_at = 0`).
   
   If a table with the same name is then created again through Gravitino:
   
   1. `createTable` creates the table in the external catalog and writes a new 
uid into the `gravitino.identifier` table property.
   2. `store.put` fails with `EntityAlreadyExistsException` on the `(schema_id, 
table_name, deleted_at)` unique key, because the stale row is still there. On 
1.3.0 this is only logged and `createTable` returns success. On `main` it now 
throws `TableAlreadyExistsException`, but the table already exists in the 
external catalog at that point.
   3. Every following `loadTable` fails:
      - `internalLoadTable` finds the stale entity by name, but its id does not 
match the uid in `gravitino.identifier` (Case 4 in 
`OperationDispatcher.operateOnEntity`), so the table is not imported;
      - `importTable` tries to put an entity with the new uid and hits the same 
unique key;
      - `loadTable` handles `EntityAlreadyExistsException` as an HA race, 
reloads, the table is still not imported, and it throws 
`UnsupportedOperationException("Table managed by multiple catalogs ...")`.
   
   Only one catalog manages the table, so the message points in the wrong 
direction. The suggested fix (remove all catalogs and recreate one) is not an 
option for a production catalog. The table stays unreadable through Gravitino, 
and owner/role operations on it fail, until it is fixed by hand.
   
   Workaround: remove the `gravitino.identifier` property from the Iceberg 
table, e.g. `ALTER TABLE db1.t1 UNSET TBLPROPERTIES ('gravitino.identifier')` 
through the Iceberg REST catalog, or a `remove-properties` table update. 
`internalLoadTable` then takes the "no string identifier" path, finds the 
existing entity by name and uses it. The owner and role bindings of that entity 
are kept. We used this on 17 tables.
   
   ### Error message and/or stacktrace
   
   Response of `GET 
/api/metalakes/<metalake>/catalogs/<catalog>/schemas/db1/tables/t1`:
   
   ```
   {"code":1006,"type":"UnsupportedOperationException","message":"Failed to 
operate table(s) [t1] operation [LOAD] under schema [db1], reason [Table 
managed by multiple catalogs. This may cause unexpected issues such as 
privilege conflicts. To resolve: Remove all catalogs managing this table, then 
recreate one catalog to ensure single-catalog management.]"}
   ```
   
   Server log when the table is created again (1.3.0):
   
   ```
   INFO  IcebergCatalogOperations:599 - Created Iceberg table t1
   ERROR TableOperationDispatcher:691 - Failed to put entity for 
<metalake>.<catalog>.db1.t1 in Gravitino, with this situation the returned 
object will not contain the metadata from Gravitino.
   org.apache.gravitino.EntityAlreadyExistsException: The TABLE entity: 
<metalake>.<catalog>.db1.t1 already exists.
   Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value 
violates unique constraint "table_meta_schema_id_table_name_deleted_at_key"
     Detail: Key (schema_id, table_name, deleted_at)=(<schema_id>, t1, 0) 
already exists.
   INFO  TableOperations:153 - Table created: <metalake>.<catalog>.db1.t1
   ```
   
   Top of the stack trace on load:
   
   ```
   java.lang.UnsupportedOperationException: Table managed by multiple catalogs. 
...
        at 
org.apache.gravitino.catalog.TableOperationDispatcher.loadTable(TableOperationDispatcher.java:160)
        at 
org.apache.gravitino.catalog.TableNormalizeDispatcher.loadTable(TableNormalizeDispatcher.java:63)
        at 
org.apache.gravitino.listener.TableEventDispatcher.loadTable(TableEventDispatcher.java:103)
        at 
org.apache.gravitino.hook.TableHookDispatcher.loadTable(TableHookDispatcher.java:65)
        at 
org.apache.gravitino.server.web.rest.TableOperations.lambda$loadTable$2(TableOperations.java:189)
   ```
   
   ### How to reproduce
   
   Gravitino 1.3.0 (commit 40fdf6ab), relational entity store on PostgreSQL.
   
   1. Create a `lakehouse-iceberg` catalog `cat` with `catalog-backend=rest`, 
pointing to an Iceberg REST server (in our case another Gravitino Iceberg REST 
server with a JDBC backend).
   2. Create `db1.t1` through the Gravitino REST API. Loading it through 
Gravitino works.
   3. Drop the table without going through Gravitino, directly on the Iceberg 
REST server, e.g. from Spark with `org.apache.iceberg.spark.SparkCatalog`, 
`type=rest`: `DROP TABLE db1.t1`.
   4. Create `db1.t1` again through the Gravitino REST API. On 1.3.0 the 
request succeeds and the server logs the `EntityAlreadyExistsException` above.
   5. `GET /api/metalakes/<metalake>/catalogs/cat/schemas/db1/tables/t1` 
returns 405 with "Table managed by multiple catalogs".
   6. Remove `gravitino.identifier` from the table properties through the 
Iceberg REST server. The same GET now returns the table.
   
   ### Additional context
   
   We saw this in production: tables were dropped from Spark through the 
Iceberg REST endpoint and recreated by an automated deployment through 
Gravitino. The steps above are taken from those logs; I have not run them in an 
isolated setup.
   
   Related:
   - #13173 / #13197: refusing an import whose `StringIdentifier` belongs to 
another live entity. It covers copied identifiers (`CREATE TABLE ... LIKE`); 
the PR was closed without merge. It also proposes replacing this message with 
one that names the table, the id and the `gravitino.identifier` property.
   - #11040: HA race handling for concurrent imports. This is the `catch 
(EntityAlreadyExistsException)` branch that produces the misleading message 
here.
   - #13730: remaining entity-store concurrency work.
   
   Possible improvements:
   1. When the entity stored under a name has a different id than the external 
table's `gravitino.identifier`, and the external table registered under the old 
id no longer exists, treat the stored entity as stale: rebind it to the new id 
or replace it, instead of failing the load.
   2. At least, report this case with an actionable message (table, both ids, 
the `gravitino.identifier` property) instead of "managed by multiple catalogs".
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to