xiangfu0 commented on code in PR #20213: URL: https://github.com/apache/pinot/pull/20213#discussion_r4532732257
########## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/DistributedTaskLockManagerInterleavingTest.java: ########## @@ -0,0 +1,367 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.controller.helix.core.minion; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Supplier; +import javax.annotation.Nullable; +import org.apache.helix.AccessOption; +import org.apache.helix.store.zk.ZkHelixPropertyStore; +import org.apache.helix.zookeeper.datamodel.ZNRecord; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; +import static org.testng.Assert.assertTrue; +import static org.testng.Assert.fail; + + +/// Exhaustively checks [DistributedTaskLockManager] against every interleaving of its ZooKeeper operations. +/// +/// Each actor runs the real acquire / hold / release sequence on its own thread against a model of one ZooKeeper +/// ephemeral node. A scheduler lets exactly one ZooKeeper operation run at a time and explores, depth first, every +/// choice of which actor runs next, every outcome of each create (applied, applied but the response lost so the +/// client's retry reports NODE_EXISTS, applied but the call throws, not applied and the call throws) and at most one +/// ZK session expiry at any point. Every run is checked for: +/// - mutual exclusion: two actors never hold the lock while the first holder's session is alive; +/// - no orphan: once every actor has finished, the lock node does not exist, so the table is not blocked until a +/// session ends; +/// - no theft: an actor never takes a lock node written by another attempt. +/// +/// The check is bounded (two actors, one acquire each, one expiry) but complete within that bound. +public class DistributedTaskLockManagerInterleavingTest { + private static final String TABLE = "testTable_OFFLINE"; + + @DataProvider + public Object[][] actorConfigurations() { + return new Object[][]{ + // Two scheduling threads of one controller, which share its ZK session + {new String[]{"controller1", "controller1"}}, + // Two controllers with their own sessions + {new String[]{"controller1", "controller2"}} + }; + } + + @Test(dataProvider = "actorConfigurations") Review Comment: **MAJOR (CI cost, non-blocking):** This test took 211 s in [Unit Test Set 2](https://github.com/apache/pinot/actions/runs/42905426945/job/130142056672) and 521 s locally. Its WARN logs (about 330k `DistributedTaskLockManager` lines) grow that job's log from roughly 70-80k lines on master to 568k lines (113 MB). Could you silence that logger for this class, e.g. `Configurator.setLevel(DistributedTaskLockManager.class, Level.OFF)` in `@BeforeClass` and restore it in `@AfterClass`? And could the full exhaustive sweep be opt-in (a system property or TestNG group), with a bounded subset running by default? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
