kfaraz commented on code in PR #16328: URL: https://github.com/apache/druid/pull/16328#discussion_r1581735643
########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL + } + + private final Mode mode; + private final List<String> lookupsToLoad; + + public static final LookupLoadingSpec ALL = new LookupLoadingSpec(Mode.ALL, null); + public static final LookupLoadingSpec NONE = new LookupLoadingSpec(Mode.NONE, Collections.emptyList()); + + private LookupLoadingSpec(Mode mode, List<String> lookupsToLoad) + { + this.mode = mode; + this.lookupsToLoad = lookupsToLoad; + } + + public static LookupLoadingSpec partial(List<String> lookupsToLoad) Review Comment: Nit: Since we are changing the enum value `PARTIAL`, maybe this method should also be renamed to something like `loadOnly`. `LookupLoadingSpec.loadOnly(lookupsToLoad)` reads better than `LookupLoadingSpec.partial(lookupsToLoad)`. ########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL + } + + private final Mode mode; + private final List<String> lookupsToLoad; + + public static final LookupLoadingSpec ALL = new LookupLoadingSpec(Mode.ALL, null); + public static final LookupLoadingSpec NONE = new LookupLoadingSpec(Mode.NONE, Collections.emptyList()); Review Comment: Nit: Why does `NONE` require an empty list? For similarity with `ALL`, we should just pass null. ########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL + } + + private final Mode mode; + private final List<String> lookupsToLoad; + + public static final LookupLoadingSpec ALL = new LookupLoadingSpec(Mode.ALL, null); + public static final LookupLoadingSpec NONE = new LookupLoadingSpec(Mode.NONE, Collections.emptyList()); + + private LookupLoadingSpec(Mode mode, List<String> lookupsToLoad) + { + this.mode = mode; + this.lookupsToLoad = lookupsToLoad; + } + + public static LookupLoadingSpec partial(List<String> lookupsToLoad) Review Comment: ```suggestion /** * Creates a LookupLoadingSpec which loads only the lookups present in the given list. */ public static LookupLoadingSpec loadOnly(List<String> lookupsToLoad) ``` ########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL + } + + private final Mode mode; + private final List<String> lookupsToLoad; + + public static final LookupLoadingSpec ALL = new LookupLoadingSpec(Mode.ALL, null); + public static final LookupLoadingSpec NONE = new LookupLoadingSpec(Mode.NONE, Collections.emptyList()); + + private LookupLoadingSpec(Mode mode, List<String> lookupsToLoad) + { + this.mode = mode; + this.lookupsToLoad = lookupsToLoad; + } + + public static LookupLoadingSpec partial(List<String> lookupsToLoad) + { + if (lookupsToLoad == null) { + throw new ISE("Expected non-null list of lookups to load."); Review Comment: Throw an `InvalidInput.exception()` instead. ########## server/src/main/java/org/apache/druid/query/lookup/LookupReferencesManager.java: ########## @@ -373,10 +376,25 @@ private void takeSnapshot(Map<String, LookupExtractorFactoryContainer> lookupMap } } - private void loadAllLookupsAndInitStateRef() + /** + * Load a set of lookups based on the injected value in {@link DataSourceTaskIdHolder#getLookupLoadingSpec()}. + */ + private void loadLookupsAndInitStateRef() { - List<LookupBean> lookupBeanList = getLookupsList(); - if (lookupBeanList != null) { + LookupLoadingSpec lookupLoadingSpec = lookupListeningAnnouncerConfig.getLookupLoadingSpec(); + List<LookupBean> lookupBeanList; + if (lookupLoadingSpec.getMode() == LookupLoadingSpec.Mode.NONE) { + lookupBeanList = Collections.emptyList(); + } else { + lookupBeanList = getLookupsList(); + if (lookupLoadingSpec.getMode() == LookupLoadingSpec.Mode.PARTIAL && lookupBeanList != null) { + lookupBeanList = lookupBeanList.stream() + .filter(lookupBean -> lookupLoadingSpec.getLookupsToLoad().contains(lookupBean.getName())) + .collect(Collectors.toList()); + } + } + Review Comment: +1 ########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL Review Comment: Instead of `PARTIAL`, we can use `ONLY_REQUIRED`. "Partial" sounds a little ambiguous. ########## server/src/main/java/org/apache/druid/server/lookup/cache/LookupLoadingSpec.java: ########## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.lookup.cache; + +import org.apache.druid.java.util.common.ISE; + +import java.util.Collections; +import java.util.List; + +/** + * This class defines the spec for loading of lookups for a given task. It contains 2 fields: + * <ol> + * <li>{@link LookupLoadingSpec#mode}: This mode defines whether lookups need to be + * loaded for the given task, or not. It can take 3 values: </li> + * <ul> + * <li> ALL: Load all the lookups.</li> + * <li> NONE: Load no lookups. </li> + * <li> PARTIAL: Load only the lookups defined in lookupsToLoad </li> + * </ul> + * <li>{@link LookupLoadingSpec#lookupsToLoad}: Defines the lookups to load when the lookupLoadingMode is set to PARTIAL.</li> + * </ol> + */ +public class LookupLoadingSpec +{ + public enum Mode + { + ALL, NONE, PARTIAL + } + + private final Mode mode; + private final List<String> lookupsToLoad; + + public static final LookupLoadingSpec ALL = new LookupLoadingSpec(Mode.ALL, null); + public static final LookupLoadingSpec NONE = new LookupLoadingSpec(Mode.NONE, Collections.emptyList()); + + private LookupLoadingSpec(Mode mode, List<String> lookupsToLoad) + { + this.mode = mode; + this.lookupsToLoad = lookupsToLoad; + } + + public static LookupLoadingSpec partial(List<String> lookupsToLoad) + { + if (lookupsToLoad == null) { + throw new ISE("Expected non-null list of lookups to load."); + } + return new LookupLoadingSpec(Mode.PARTIAL, lookupsToLoad); + } + + public Mode getMode() Review Comment: I wonder if we even need to expose the `Mode` enum outside of this class. An alternate approach could be to expose methods `boolean isLoadAll()` , `boolean isLoadNone()`, `boolean isLoadRequired()`. Just a thought, I am okay with the current approach as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
