Hi All,
While I was working on metadata change I have encountered the problem, which
Abdullah bumped into a year or so ago: we have several secondary indexes, which
are created during metadata bootstrap, and these indexes are not valid at all!
Example: here is on of the internal datatypes - DatasetRecordType:
open {
DataverseName: STRING,
DatasetName: STRING,
DatatypeName: STRING,
DatasetType: STRING,
InternalDetails: UNION(NULL, open {
FileStructure: STRING,
PartitioningStrategy: STRING,
PartitioningKey: [ [ STRING ] ],
PrimaryKey: [ [ STRING ] ],
GroupName: STRING,
Autogenerated: BOOLEAN,
CompactionPolicy: STRING,
CompactionPolicyProperties: [ open {
Name: STRING,
Value: STRING
}
]
}
),
ExternalDetails: UNION(NULL, open {
DatasourceAdapter: STRING,
Properties: [ open {
Name: STRING,
Value: STRING
}
],
GroupName: STRING,
LastRefreshTime: DATETIME,
TransactionState: INT32,
CompactionPolicy: STRING,
CompactionPolicyProperties: [ open {
Name: STRING,
Value: STRING
}
]
}
),
Hints: {{ open {
Name: STRING,
Value: STRING
}
}},
Timestamp: STRING,
DatasetId: INT32,
PendingOp: INT32
}
And here the secondary index GROUPNAME_ON_DATASET_INDEX is created on fields
GroupName,DataverseName,DatasetName, which is obviously wrong, since there is
no top-level field named GroupName (apparently at some point got substituted by
nested fields).
As Abdullah pointed out last time it is not clear if these indexes are used at
all by any test queries, but they at least are considered by the
Introduce{Select\Join}AccessMethod rule, which breaks my type validation logic.
For now I will remove incorrect indexes whatsoever, but I believe we should
plan to implement Mike’s solution for that problem: make bootstrap logic go
though compiler pipeline.
Best regards,
Ildar
