924060929 commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r4068246533
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -552,59 +644,372 @@ private void initPrunedPartitions() throws UserException
{
throw new UserException(ExceptionUtils.getRootCauseMessage(e), e);
}
partitionInit = true;
+ ensureHmsRuntimeGeneration();
}
@Override
public void startSplit(int numBackends) {
+ ensureHmsRuntimeGeneration();
if (prunedPartitions.isEmpty()) {
splitAssignment.finishSchedule();
+ releaseFsViewOnce();
return;
}
- AtomicInteger numFinishedPartitions = new AtomicInteger(0);
+ acquireFsView();
Review Comment:
Fixed in `db16d80a7b7`.
`FsViewGeneration` now separates exact-lease acquisition (`acquireFsView`)
from the blocking remote sync (`synchronizeFsView`). `HudiScanNode` acquires
the lease first, then publishes the batch/listing owner (statement resource
plus `SplitAssignment` closeable) and registers the sync thread on that owner
*before* starting the I/O. Cancellation interrupts the sync thread, and the
owner releases the lease only from its actual-terminal path; `getSplits` also
releases through the owner on sync/submission/auth failures. Added blocked-sync
cancellation tests for both `BatchFsViewOwner` and `ListingFsViewOwner`.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -337,39 +332,40 @@ public void afterDropDb(String dbName) {
@Override
public boolean createTableImpl(CreateTableInfo createTableInfo) throws
UserException {
try {
- return executionAuthenticator.execute(() ->
performCreateTable(createTableInfo));
+ return executeCatalogOperation(() ->
performCreateTable(createTableInfo));
} catch (Exception e) {
throw new DdlException(
"Failed to create table: " + createTableInfo.getTableName() +
", error message is:" + e.getMessage(),
e);
}
}
- public boolean performCreateTable(CreateTableInfo createTableInfo) throws
UserException {
+ private boolean performCreateTable(CreateTableInfo createTableInfo) throws
UserException {
String dbName = createTableInfo.getDbName();
- ExternalDatabase<?> db = dorisCatalog.getDbNullable(dbName);
+ ExternalDatabase<?> db = getDatabaseWithinCatalogGeneration(dbName);
if (db == null) {
throw new UserException("Failed to get database: '" + dbName + "'
in catalog: " + dorisCatalog.getName());
}
String tableName = createTableInfo.getTableName();
// 1. first, check if table exist in remote
- if (tableExist(db.getRemoteName(), tableName)) {
+ if (tableExistsInternal(db.getRemoteName(), tableName)) {
if (createTableInfo.isIfNotExists()) {
LOG.info("create table[{}] which already exists", tableName);
return true;
} else {
ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR, tableName);
}
}
- // 2. second, check fi table exist in local.
+ // 2. second, check if the table conflicts under Doris case-folding
rules.
// This is because case sensibility issue, eg:
// 1. lower_case_table_name = 1
// 2. create table tbl1;
// 3. create table TBL1; TBL1 does not exist in remote because the
remote system is case-sensitive.
// but because lower_case_table_name = 1, the table can not be
created in Doris because it is conflict with
// tbl1
- ExternalTable dorisTable = db.getTableNullable(tableName);
- if (dorisTable != null) {
+ // Keep this lookup on the retained Iceberg catalog generation.
ExternalDatabase is resettable,
+ // so calling getTableNullable() here could enumerate a newer catalog
generation.
+ if (hasCaseFoldTableCollision(db.getRemoteName(), tableName)) {
Review Comment:
Fixed in `db16d80a7b7`.
`hasCaseFoldTableCollision` now checks the retained-generation view names in
addition to the table names, matching the combined local namespace the old
`listTableNamesFromRemote()` lookup exposed. Added
`testCreateTableCaseFoldCollisionIncludesViews`, which rejects `CREATE TABLE
TBL1` when the retained generation contains a view named `tbl1`.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -1800,18 +1922,40 @@ private Table useFrozenTableGeneration(Table
currentTable) {
"Unknown Iceberg system table type: %s",
systemTable.getSysTableType());
// Snapshot-selectable metadata tables must derive
their scans and schemas
// from the same frozen base generation as the
relation's snapshot fence.
+ frozenGenerationSource = cacheValue;
return
MetadataTableUtils.createMetadataTableInstance(frozenBaseTable, tableType);
}
+ // This scan falls back to the current generation, so its
asynchronous planner
Review Comment:
Fixed in `db16d80a7b7`.
`useFrozenTableGeneration` now classifies non-snapshot-selectable system
tables through `IcebergSysExternalTable.bindsToStatementGeneration()`: `ALL_*`
metadata tables stay on the frozen G1 table and retain it for async planning,
while truly static tables (`SNAPSHOTS`/`HISTORY`/`REFS`/...) fall back to the
current G2 table and clear the pinned runtime, mapping policy and planning
lease. Added `testAllMetadataTableRetainsFrozenGenerationForAsyncPlanning` and
`testStaticMetadataTableDropsPinnedGenerationResources`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
