himanshug commented on a change in pull request #8038: Making optimal usage of multiple segment cache locations URL: https://github.com/apache/incubator-druid/pull/8038#discussion_r308485496
########## File path: server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java ########## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.segment.loading; + +import com.fasterxml.jackson.annotation.JsonSubTypes; +import com.fasterxml.jackson.annotation.JsonTypeInfo; +import com.google.common.collect.ImmutableList; +import org.apache.druid.timeline.DataSegment; + +/** + * Interface which represents a strategy to select a {@link StorageLocation} from available segment cache locations + * for segment distribution. + */ +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "tier", defaultImpl = LeastBytesUsedStorageLocationSelectorStrategy.class) +@JsonSubTypes(value = { + @JsonSubTypes.Type(name = "leastBytesUsed", value = LeastBytesUsedStorageLocationSelectorStrategy.class), + @JsonSubTypes.Type(name = "roundRobin", value = RoundRobinStorageLocationSelectorStrategy.class) +}) +public interface StorageLocationSelectorStrategy +{ + /** + * Find the best {@link StorageLocation} to load the given {@link DataSegment} into according to the location selector strategy. + * + * @return The storage location to load the given segment into or null if no location has the capacity to store the given segment. + */ + StorageLocation select(DataSegment dataSegment, String storageDirStr); Review comment: > Implemented both the strategies and made the strategy configurable. However there is one implementation glitch due to which SegmentLoaderLocalCacheManagerTest.testRetrySuccessAtSecondLocation() is failing. Here's the scenario. For example, assume strategy configured is least bytes used strategy and there are two locations - loc1 and loc2 each on different disks disk1 and disk2 respectively. loc1 has the least bytes used. The strategy picks loc1 and before SegmentLoaderLocalCacheManager loads a segment if disk1 fails or is not writable the segment loading fails. The strategy has no way (with my impl) to find if loc1 is bad, this results in the strategy picking loc1 every time failing all segment load attempts. What is a clean way to handle this ? good catch, I think, for that reason the interface here should be something like.. ``` Iterator<StorageLocation> getLocations(..) ``` so that caller can go through all of them like it does currently and caller should be responsible for calling the `reserve(..)` method not the impls of this. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
