Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15235
Change subject: IMPALA-9152: Make Impala ranger plugin singleton in FE Tests ...................................................................... IMPALA-9152: Make Impala ranger plugin singleton in FE Tests Fix the flakiness of Ranger FE tests in AuthorizationStmtTest which is caused by a row filter policy not being cleanly deleted. There is a bug in Ranger that policies being deleted in Ranger Admin server but still exist in Ranger plugins when there are concurrent create policy and get policy requests (RANGER-2727). It's more possible to hit the bug if we have more ranger plugins running, since each plugin instance will poll policies in each 30s regularly. Impalad and Catalogd servers only initialize one ImpalaRangerPlugin instance. However, AuthorizationStmtTest has embedded Frontend and CatalogServiceCatalog objects. It will initialize two ranger plugin instances totally. What's worse, the JUnit testing framework make a new object for each test method run. Currently there are 29 test methods in AuthorizationStmtTest, which means 29 AuthorizationStmtTest objects will be created. So we finally have 58 ranger plugin instances running, which make RANGER-2727 easy to happens. The failure can be reproduced by adding the following new test and run it with all existing tests: @Test public void testRangerPolicyRepeatedly() throws Exception { if (authzProvider_ == AuthorizationProvider.SENTRY) return; for (int i = 0; i < 100; ++i) { testRowFilterEnabled(); testColumnMaskEnabled(); } } We only explicitly create policies for column masking and row filtering (other tests are using grant/revoke requests). This test increases the number of create policy requests, so increases the possibility with concurrent get policies requests polling from other ranger plugin instances created by previous tests. The fix is to make ImpalaRangerPlugin a singleton class so we will have only one ranger plugin instance, which dramatically reduces the possibility of hitting RANGER-2727. The thorough fix is bumping CDP version after RANGER-2727 is resolved. Codes added in the previous patch (c1244c2f04e629cc07b0830a597c70317be92768) are removed. Tests: - Ran AuthorizationStmtTest with the above new test. Change-Id: I91f2bad1a9ce897b45cfc42f97b192fe2f8c7e06 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java 5 files changed, 38 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/15235/1 -- To view, visit http://gerrit.cloudera.org:8080/15235 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I91f2bad1a9ce897b45cfc42f97b192fe2f8c7e06 Gerrit-Change-Number: 15235 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>