HonahX commented on code in PR #2223:
URL: https://github.com/apache/polaris/pull/2223#discussion_r2373214921
##########
runtime/service/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -1737,6 +1761,80 @@ public boolean revokePrivilegeOnNamespaceFromRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic namespace entities for
external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param namespace the expected fully resolved namespace to be created.
+ * @param existingPath the partially resolved path currently stored in the
metastore.
+ * @return the fully resolved path wrapper.
+ */
+ private PolarisResolvedPathWrapper createSyntheticNamespaceEntities(
+ CatalogEntity catalogEntity, Namespace namespace,
PolarisResolvedPathWrapper existingPath) {
+
+ if (existingPath == null) {
+ throw new IllegalStateException(
+ String.format("Catalog entity %s does not exist.",
catalogEntity.getName()));
+ }
+
+ List<PolarisEntity> completePath = new
ArrayList<>(existingPath.getRawFullPath());
+ PolarisEntity currentParent = existingPath.getRawLeafEntity();
+
+ String[] allNamespaceLevels = namespace.levels();
+ int numMatchingLevels = 0;
+ // Find parts of the complete path that match the namespace levels.
+ // We skip index 0 because it is the CatalogEntity.
+ for (PolarisEntity entity : completePath.subList(1, completePath.size())) {
+ if (!entity.getName().equals(allNamespaceLevels[numMatchingLevels])) {
+ break;
+ }
+ numMatchingLevels++;
+ }
+
+ for (int i = numMatchingLevels; i < allNamespaceLevels.length; i++) {
+ String[] namespacePart = Arrays.copyOfRange(allNamespaceLevels, 0, i +
1);
+ String leafNamespace = namespacePart[namespacePart.length - 1];
+ Namespace currentNamespace = Namespace.of(namespacePart);
+
+ // TODO: Instead of creating synthetic entitties, rely on external
catalog mediated backfill.
+ PolarisEntity syntheticNamespace =
+ new NamespaceEntity.Builder(currentNamespace)
+
.setId(metaStoreManager.generateNewEntityId(getCurrentPolarisContext()).getId())
+ .setCatalogId(catalogEntity.getId())
+ .setParentId(currentParent.getId())
+ .setCreateTimestamp(System.currentTimeMillis())
+ .build();
+
+ EntityResult result =
+ metaStoreManager.createEntityIfNotExists(
+ getCurrentPolarisContext(),
+ PolarisEntity.toCoreList(completePath),
+ syntheticNamespace);
+
+ if (result.isSuccess()) {
+ syntheticNamespace = PolarisEntity.of(result.getEntity());
+ } else {
+ Namespace partialNamespace =
Namespace.of(Arrays.copyOf(allNamespaceLevels, i + 1));
+ PolarisResolvedPathWrapper partialPath =
+ resolutionManifest.getResolvedPath(partialNamespace);
Review Comment:
In case of nested namespace, this will fail at
https://github.com/apache/polaris/blob/06d523116fd5f5b61f95d8660166e76c00fd2a3e/polaris-core/src/main/java/org/apache/polaris/core/persistence/resolver/PolarisResolutionManifest.java#L181-L186
since we never added the `partialNamespace` key to the (passthrough) paths.
Only the full namespace in the path.
We could just use the full namespace as the identifier and get a newer
version of the partial path. If the synthetic namespace entity still not appear
in the leaf, we throw the error. If the new partial path contains even more
child entities than expected (e.g. Try to create `ns1.ns1a.ns1aa`, expect
`ns1.ns1a` but get `ns1.ns1a.ns1aa`), we could either validate and reconcile or
throw an error.
##########
runtime/service/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -2041,6 +2156,77 @@ private boolean grantPrivilegeOnTableLikeToRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic table-like entity and its
parent namespace entities
+ * for external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param identifier the path of the table-like entity(including the
namespace).
+ * @param subTypes the expected subtypes of the table-like entity
+ * @param existingPathWrapper the partially resolved path currently stored
in the metastore.
+ * @return the resolved path wrapper
+ */
+ private PolarisResolvedPathWrapper createSyntheticTableLikeEntities(
+ CatalogEntity catalogEntity,
+ TableIdentifier identifier,
+ List<PolarisEntitySubType> subTypes,
+ PolarisResolvedPathWrapper existingPathWrapper) {
+
+ Namespace namespace = identifier.namespace();
+ PolarisResolvedPathWrapper resolvedNamespacePathWrapper =
+ !namespace.isEmpty()
+ ? createSyntheticNamespaceEntities(catalogEntity, namespace,
existingPathWrapper)
+ : existingPathWrapper;
+
+ if (resolvedNamespacePathWrapper == null
+ || (!namespace.isEmpty()
+ && !resolvedNamespacePathWrapper.isFullyResolvedNamespace(
+ catalogEntity.getName(), namespace))) {
+ throw new RuntimeException(
+ String.format(
+ "Failed to create synthetic namespace entities for namespace %s
in catalog %s",
+ namespace.toString(), catalogEntity.getName()));
+ }
+
+ PolarisEntity parentNamespaceEntity =
resolvedNamespacePathWrapper.getRawLeafEntity();
+
+ // TODO: Once we support GENERIC_TABLE federation, select the intended
type depending on the
+ // callsite; if it is instantiated via an Iceberg RESTCatalog factory or a
different factory
+ // for GenericCatalogs.
+ PolarisEntitySubType syntheticEntitySubType =
selectEntitySubType(subTypes);
+
+ // TODO: Instead of creating a synthetic table-like entity, rely on
external catalog mediated
+ // backfill and use the metadata location from the external catalog.
+ PolarisEntity syntheticTableEntity =
+ new IcebergTableLikeEntity.Builder(identifier, "")
+
.setId(metaStoreManager.generateNewEntityId(getCurrentPolarisContext()).getId())
+ .setCatalogId(parentNamespaceEntity.getCatalogId())
+ .setSubType(syntheticEntitySubType)
+ .setCreateTimestamp(System.currentTimeMillis())
Review Comment:
```suggestion
.setCreateTimestamp(System.currentTimeMillis())
.setParentId(parentNamespaceEntity.getId())
```
We need to setParentId here too to make it resolvable later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
