There is no way to drop a Hadoop catalog table without removing the directory so I’m not sure what the alternative would be
Sent from my iPhone On Aug 29, 2023, at 10:10 PM, Manu Zhang <owenzhang1...@gmail.com> wrote:
Hi all,
The current behavior of dropping a table with HadoopCatalog looks inconsistent to me. When stored at the default location under table path, metadata and data will be deleted regardless of the "purge" flag. When stored elsewhere, they will be left there if not "purge". Is this by design?
@Override
public boolean dropTable(TableIdentifier identifier, boolean purge) { if (!isValidIdentifier(identifier)) { throw new NoSuchTableException( "Invalid identifier: %s", identifier); } Path tablePath = new Path(defaultWarehouseLocation(identifier)); TableOperations ops = newTableOps(identifier); TableMetadata lastMetadata = ops.current(); try { if (lastMetadata == null) { LOG.debug( "Not an iceberg table: {}", identifier); return false; } else { if (purge) { // Since the data files and the metadata files may store in different locations,
// so it has to call dropTableData to force delete the data file.
CatalogUtil. dropTableData( ops.io(), lastMetadata); } return fs.delete(tablePath, true /* recursive */); } } catch (IOException e) { throw new RuntimeIOException(e, "Failed to delete file: %s", tablePath); } }
Thanks, Manu
|
