liuchang0520 commented on a change in pull request #6778:
URL: https://github.com/apache/incubator-pinot/pull/6778#discussion_r637075036
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
##########
@@ -1214,4 +1273,112 @@ private int
getMaxNumPartitionsPerInstance(InstancePartitions instancePartitions
return (numPartitions + numInstancesPerReplicaGroup - 1) /
numInstancesPerReplicaGroup;
}
}
+
+ // Pre fetch the LLC segment without deep store copy.
+ public void prefetchLLCSegmentWithoutDeepStoreCopy(String tableNameWithType)
{
+ TableType tableType =
TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == TableType.REALTIME) {
+ TableConfig tableConfig =
_helixResourceManager.getTableConfig(tableNameWithType);
+ if (tableConfig == null) {
+ LOGGER.warn("Failed to find table config for table: {}",
tableNameWithType);
+ return;
+ }
+
+ PartitionLevelStreamConfig streamConfig = new
PartitionLevelStreamConfig(tableConfig.getTableName(),
+ IngestionConfigUtils.getStreamConfigMap(tableConfig));
+ if (!streamConfig.hasLowLevelConsumerType()) {
+ return;
+ }
+
+ List<LLCRealtimeSegmentZKMetadata> segmentZKMetadataList =
ZKMetadataProvider.getLLCRealtimeSegmentZKMetadataListForTable(_propertyStore,
tableNameWithType);
Review comment:
Hi @mcvsubbu , this change is following your suggestion. Please correct
me if there is any missing steps.
During commit phase, [enqueue the segment without download url.
](https://github.com/apache/incubator-pinot/pull/6778/files/647f1de17ff0bc60bef20468604e65b44dbc057c#diff-ffc91e381e1c93dc73fbeb18ef872ec812e25f15760dbf35896ac81793ce1795R643)
`uploadToSegmentStoreIfMissing` reads in-memory segments list to fix, only
removes the segment from the in-memory list after successful upload.
Only `prefetchLLCSegmentWithoutDeepStoreCopy ` has access to ZK, [which is
triggered when setup the periodic
jobs](https://github.com/apache/incubator-pinot/pull/6778/files/647f1de17ff0bc60bef20468604e65b44dbc057c#diff-0a9ffdab99ed314e7a8e58bf86c125f020b7d6f86493d1f6d719beab62b7f106R71).
Leadership is also checked in this step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
